Data Spine NiFi

Overview

Apache NiFi was selected to be the Integration Flow Engine of the Data Spine. Apache NiFi is a Dataflow Management Platform based on the concepts of Flow-based programming. It automates the flow of information between systems through directed graphs called dataflows. The dataflows support communication, data routing, data transformation and system mediation logic with the help of ‘processors’ as their vertices.

Apache NiFi GUI Elements Figure 1: Apache NiFi GUI Elements

The processors are responsible for handling data ingress, egress, routing, mediation and transformation. The edges that connect these processors with each other are called ‘Connections’. Apache NiFi offers a Web-based, highly configurable, drag-and-drop style GUI for creating such dataflows. Figure 1 highlights the elements of NiFi’s GUI and also shows a sample dataflow. NiFi’s GUI offers a functionality to search for a particular processor and view its short description to include it in a dataflow as shown in Figure 2. NiFi contains as many as 284 different processors as of version 1.11.3. In the context of Data Spine, the Integration Flows translate to dataflows in NiFi. Henceforth, dataflows would be referred to as Integration Flows in this documentation.

Apache NiFi Processors Figure 2: Apache NiFi Processors

Data Spine NiFi Documentation

References

  1. Apache NiFi Documentation: https://nifi.apache.org/docs.html