DS NiFi Integration Flow Examples

Examples Process Group in DS NiFi

The Data Spine (DS) NiFi contains a Process Group (PG) called ‘Examples PG’. It is present inside the ‘p1/8091’ PG and all the DS Users have read/operate access to it. The Examples PG consists of various integration flows (iFlows) that realise different use cases. These iFlows are summerized here.

DS NiFi Examples PG Figure 1: DS NiFi Examples PG

1. Routing

  • The Examples PG itself is an example of how routing can be implemented in NiFi. The Examples PG makes use of routing to accommodate multiple child PGs/iFlows and route an incoming HTTP request to the correct PG/iFlow inside it.
  • The Examples PG uses the ‘UpdateAttribute’ and ‘RouteOnAttribute’ processors of NiFi to have multiple request-response iFlows.
  • Configuration of UpdateAttribute processor:

Configuration of UpdateAttribute processor Figure 2: Configuration of UpdateAttribute processor

  • Configuration of RouteOnAttribute processor:

Configuration of RouteOnAttribute processor Figure 2: Configuration of RouteOnAttribute processor

2. Creation of a reverse proxy endpoint (ex1)

  • The iFlow in this PG creates a reverse proxy endpoint for an external service.
  • It listens for HTTP GET requests and upon receiving a request, calls SMECluster’s Marketplace service, gets the response and then returns the response to the caller without making any changes to it.
  • For invoking this (ex1) endpoint and other endpoints exposed by the iFlows in DS NiFi, follow these steps.

iFlow to create a reverse proxy endpoint (ex1) Figure 3: iFlow to create a reverse proxy endpoint (ex1)

3. Simple Data transformation using Jolt Shiftr (ex2)

  • The iFlow in this PG uses the JoltTransformJSON processor in NiFi to perform data model transformation.

iFlow to data transformation using Jolt Shiftr (ex2) Figure 4: iFlow to data transformation using Jolt Shiftr (ex2)

4. Data transformation using XSLT (ex3)

  • The iFlow in this PG uses the TransformXml processor in NiFi to perform data model transformation.

iFlow to data transformation using XSLT (ex3) Figure 5: iFlow to data transformation using XSLT (ex3)

5. Data transformation using Jolt Chainr (ex6)

  • The iFlow in this PG uses the JoltTransformJSON processor in NiFi to perform data model transformation.
  • This iFlow (1) calls NIMBLE platform’s Marketplace Service directly with the EFPF credentials passed in the request, (2) gets and transforms the response’s payload to the one that conforms to the (Mockup) Integrated Marketplace’s data model, and (3) returns the resulting payload that conforms to the (Mockup) Integrated Marketplace’s data model
  • The Jolt spec chains together shift, modify-default-beta and remove operations to perform the data transformation.

iFlow to data transformation using Jolt Chainr (ex6) Figure 4: iFlow to data transformation using Jolt Chainr (ex6)

6. MQTT to HTTP/REST using NiFi iFlow (ex7)

  • The iFlow1 in this PG subscribes to messages from a Message Broker and iFlow2 makes the latest message available over an HTTP/REST endpoint.

MQTT to HTTP/REST using NiFi iFlow (ex7) Figure 5: MQTT to HTTP/REST using NiFi iFlow (ex7)

7. Carry forward query parameters (ex8)

  • The iFlow in this PG carries forward query parameters while calling an external service.

iFlow to carry forward query parameters (ex8) Figure 6: iFlow to carry forward query parameters (ex8)

8. Data Transformation using ExecuteScript/Javascript (ex9)

  • The iFlow in this PG uses Javascript code in ExecuteScript processor to perform data model transformation.

Data Transformation using ExecuteScript/Javascript (ex9) Figure 7: Data Transformation using ExecuteScript/Javascript (ex9)

9. Lookup, Retrieval and Data Enrichment using a Controller Service (a01)

  • The iFlows in this PG use a lookup controller service for lookup and retrieval / data enrichment.
  • iFlow1 uses LookupAttribute processor for performing lookup and retrieval
  • iFlow2 uses LookupRecord processor for data enrichment

iFlows using a controller service for lookup, retrieval and data enrichment (a01) Figure 8: iFlows using a controller service for lookup, retrieval and data enrichment (a01)

10. A simple HTTP Server (a02)

  • The iFlow in this PG listens for HTTP GET requests and returns a static JSON as the response payload.

iFlow to create a simple HTTP Server (a02) Figure 9: iFlow to create a simple HTTP Server (a02)

11. MQTTS Pub/Sub Example

  • iFlows that use MQTTS for Pub/Sub operations.

MQTTS Pub/Sub Example Figure 10: MQTTS Pub/Sub Example iFlows

12. ISO to EPOCH Timestamp Conversion

  • The iFlow in this PG performs data model mapping between two JSON Schemas and converts timestamp from ISO 8601 date format to EPOCH format.

ISO to EPOCH Timestamp Conversion Example Figure 11: ISO to EPOCH Timestamp Conversion Example

Data Spine NiFi Documentation

References

[1] NiFi Documentation: https://nifi.apache.org/docs.html
[2] NiFi Source Code Repository: https://github.com/apache/nifi

Next