DS NiFi Integration Flow Examples

Examples Process Group in DS NiFi

The Data Spine (DS) NiFi contains a Process Group (PG) called ‘Examples PG’. It is present inside the ‘p1/8091’ PG and all the DS Users have read/operate access to it. The Examples PG consists of various integration flows (iFlows) that realise different use cases. These iFlows are summerized here.

Figure 1: DS NiFi Examples PG

1. Routing

• The Examples PG itself is an example of how routing can be implemented in NiFi. The Examples PG makes use of routing to accommodate multiple child PGs/iFlows and route an incoming HTTP request to the correct PG/iFlow inside it.
• The Examples PG uses the ‘UpdateAttribute’ and ‘RouteOnAttribute’ processors of NiFi to have multiple request-response iFlows.
• Configuration of UpdateAttribute processor:

Figure 2: Configuration of UpdateAttribute processor

• Configuration of RouteOnAttribute processor:

Figure 2: Configuration of RouteOnAttribute processor

2. Creation of a reverse proxy endpoint (ex1)

• The iFlow in this PG creates a reverse proxy endpoint for an external service.
• It listens for HTTP GET requests and upon receiving a request, calls SMECluster’s Marketplace service, gets the response and then returns the response to the caller without making any changes to it.
• For invoking this (ex1) endpoint and other endpoints exposed by the iFlows in DS NiFi, follow these steps.

Figure 3: iFlow to create a reverse proxy endpoint (ex1)

3. Simple Data transformation using Jolt Shiftr (ex2)

• The iFlow in this PG uses the JoltTransformJSON processor in NiFi to perform data model transformation.

Figure 4: iFlow to data transformation using Jolt Shiftr (ex2)

• For more details on Jolt, see the DS NiFi Data Transformation Guide
• Template to load and run in a local instance of NiFi Jolt Example 1 Template

4. Data transformation using XSLT (ex3)

• The iFlow in this PG uses the TransformXml processor in NiFi to perform data model transformation.

Figure 5: iFlow to data transformation using XSLT (ex3)

• For more details on XSLT, see the DS NiFi Data Transformation Guide
• Template to load and run in a local instance of NiFi XSLT Example 1 Template

5. Data transformation using Jolt Chainr (ex6)

• The iFlow in this PG uses the JoltTransformJSON processor in NiFi to perform data model transformation.
• This iFlow (1) calls NIMBLE platform’s Marketplace Service directly with the EFPF credentials passed in the request, (2) gets and transforms the response’s payload to the one that conforms to the (Mockup) Integrated Marketplace’s data model, and (3) returns the resulting payload that conforms to the (Mockup) Integrated Marketplace’s data model
• The Jolt spec chains together shift, modify-default-beta and remove operations to perform the data transformation.

Figure 4: iFlow to data transformation using Jolt Chainr (ex6)

• For more details on Jolt, see the DS NiFi Data Transformation Guide
• To get more details on the setup of this example in a local instance of NiFi, see this documentation.
• Template to load and run in a local instance of NiFi Jolt Chainr Example Template

6. MQTT to HTTP/REST using NiFi iFlow (ex7)

• The iFlow1 in this PG subscribes to messages from a Message Broker and iFlow2 makes the latest message available over an HTTP/REST endpoint.

Figure 5: MQTT to HTTP/REST using NiFi iFlow (ex7)

• Template to load and run in a local instance of NiFi MQTT to HTTP/REST Example Template

7. Carry forward query parameters (ex8)

• The iFlow in this PG carries forward query parameters while calling an external service.

Figure 6: iFlow to carry forward query parameters (ex8)

8. Data Transformation using ExecuteScript/Javascript (ex9)

• The iFlow in this PG uses Javascript code in ExecuteScript processor to perform data model transformation.

Figure 7: Data Transformation using ExecuteScript/Javascript (ex9)

• For more details on the ExecuteScript processor, see the DS NiFi Data Transformation Guide
• Template to load and run in a local instance of NiFi ExecuteScript Example Template

9. Lookup, Retrieval and Data Enrichment using a Controller Service (a01)

• The iFlows in this PG use a lookup controller service for lookup and retrieval / data enrichment.
• iFlow1 uses LookupAttribute processor for performing lookup and retrieval
• iFlow2 uses LookupRecord processor for data enrichment

Figure 8: iFlows using a controller service for lookup, retrieval and data enrichment (a01)

10. A simple HTTP Server (a02)

• The iFlow in this PG listens for HTTP GET requests and returns a static JSON as the response payload.

Figure 9: iFlow to create a simple HTTP Server (a02)

11. MQTTS Pub/Sub Example

• iFlows that use MQTTS for Pub/Sub operations.

Figure 10: MQTTS Pub/Sub Example iFlows

12. ISO to EPOCH Timestamp Conversion

• The iFlow in this PG performs data model mapping between two JSON Schemas and converts timestamp from ISO 8601 date format to EPOCH format.

Figure 11: ISO to EPOCH Timestamp Conversion Example

• Template to load and run in a local instance of NiFi ISO_to_EPOCH_Template

Data Spine NiFi Documentation

• Data Spine NiFi Overview
• Apache NiFi Quickstart Guide
• Data Spine NiFi User Guide
• NiFi Data Transformation Examples
• Data Spine NiFi Admin Guide

References

[1] NiFi Documentation: https://nifi.apache.org/docs.html
[2] NiFi Source Code Repository: https://github.com/apache/nifi