DS NiFi User Guide

Target Audience

  • Data Spine NiFi provides a low-code development environment and a drag-and-drop style GUI for an easy and intuitive creation integration flows or iFlows (dataflows/workflows).
  • This guide is intended towards the users of the EFPF ecosystem who want to create integration flows for realising composite applications, performing data model transformation, data enrichment, and/or protocol translation, etc.

Connection Details

The Production environment will be available soon

Environment Component URL Description
Production NiFi GUI https://efpf.smecluster.com/nifi GUI for creating iFlows. Email EFPF Support for getting access.
Production Proxy APIs exposed by iFlows (iFlow API) Example: http://nifi:8091/p00/ex2 Details below. Not directly accessible over the Internet.
Production Secure proxy APIs of iFlows Example: https://efpf.smecluster.com/apis/example-service-id/example-api-id/p00/ex2 Details below. Accessible to EFPF users.

Prerequisites

  • Register at the EFPF Portal to get an EFPF user account (Details: User Guide 101).
  • Basic knowledge and hands-on experience with using Apache NiFi.
  • Not familiar with Apache NiFi? No problem, take a look at the NiFi Overview and the Apache NiFi Quickstart Guide pages.
  • To use Pub/Sub through DS NiFi, you must obtain a DS RabbitMQ user account using the Pub/Sub Security Service GUI (Details: User Guide 101).

Notes

  • To create Integration Flows in Data Spine NiFi, a collaboration space called ‘Process Group (PG)’ would be given.
  • PGs can be given per organization (preferably) or per project.
  • Multiple users can be given access to a single PG and each PG can contain multiple Integration Flows or even other child PGs.
  • After getting an EFPF user account, you need to send an email to the EFPF Support Team for obtaining basic access privileges in NiFi mentioning the email id linked with your EFPF user account. If you try to login to NiFi without that, you’ll get an ‘Insufficient Permissions’ error.
  • The DS NiFi in EFPF Production environment already has PGs created for Open Call experiments, and the Open Call participants have been given access to their respective PGs.

Steps for Getting a New PG

  • If you want a new PG for your company, send an email to the EFPF Support Team mentioning the name of your company and your email id (which is linked with your EFPF user account). You would be given admin privileges for this PG.
  • You will receive an acknowledgement email once the PG has been created with
    1. the path to your PG in the form <parent PG> » <path to your PG> » <your PG>,
    2. a three letter identifier for the PG, and,
    3. the internal base URL for your PG.
  • Let’s say, the name of your company is ‘CompanyX’, and the PG given to your company is called ‘(p02) CompanyX’, it is inside a parent PG called ‘Projects’ which is inside another PG called ‘p1: EFPF Projects (nifi:8091)’ and its identifier is ‘p02’. Then the acknowledgement email will contain the following information:
    1. Path to your PG: NiFi Flow » p1: EFPF Projects (nifi:8091) » Projects » (p02) CompanyX
    2. Identifier for your PG: p02 (Hint: The PGs name are usually prefixed with their identifiers in round brackets)
    3. The internal base URL for your PG: http://nifi:8091/p02 (Hint: The URL path is composed using the PG identifiers)
  • Login to Data Spine NiFi and navigate to CompanyX PG (see Figure 1)

Navigation through Parent PGs Figure 1: Navigation through Parent PGs

  • Alternatively, you can also search for your PG in the Search box and navigate to it in one-shot as shown in Figure 2.

PG Search Figure 2: PG Search

  • PGs other than CompanyX PG won’t be visible and accessible to you (except for the PGs that contain examples) as shown in Figure 3.
  • The PGs that contain example iFlows are visible and accessible to all the users.

Visibility and Access Control Figure 3: Visibility and Access Control

  • Inside the CompanyX PG, you will find an input port ‘p02’. If you want to create Integration Flows that involve HTTP-based request-response type of communication, you can connect this port, p02 to your Integration Flow to receive requests. Connecting this port won’t be required in the case of Integration Flows that involve Pub/Sub (MQTT, AMQP) type of communication.

iFlow inside CompanyX PG Figure 4: iFlow inside CompanyX PG

  • If you want to have multiple HTTP-based request-response iFlows inside the CompanyX PG, you can use ‘UpdateAttribute’ and ‘RouteOnAttribute’ processors of NiFi to create the necessary routing structure.
  • Example: routing structure in ‘Examples (exp)’ PG shown in Figure 5:

Examples PG Contains Multiple Child PGs and Integration Flows Figure 5: Examples PG Contains Multiple Child PGs and Integration Flows

Steps for Configuring Access to your PG

  • You are given admin privileges for your own PG (in this example - CompanyX PG).
  • If you want to give more users access to CompanyX PG, right click on CompanyX PG and select ‘Manage access policies’.
  • Then add users against each access policy to give them that respective access permission as shown in Figure 6.

Manage Access Policies for CompanyX PG Figure 6: Manage Access Policies for CompanyX PG

  • The policies and the corresponding privileges are explained in the table below:
Policy Privilege
view the component Allows users to view component configuration details
modify the component Allows users to modify component configuration details
operate the component Allows users to operate components by changing component run status (start/stop/enable/disable), remote port transmission status, or terminating processor threads
view provenance Allows users to view provenance events generated by this component
view the data Allows users to view metadata and content for this component in flowfile queues in outbound connections and through provenance events
modify the data Allows users to empty flowfile queues in outbound connections and submit replays through provenance events
view the policies Allows users to view the list of users who can view/modify a component
modify the policies Allows users to modify the list of users who can view/modify a component

Steps for Invoking the HTTP Endpoints Exposed by the iFlows in NiFi

Example 1: iFlow in CompanyX PG (enabling only AuthN, and not AuthZ)

  • The internal URL for an endpoint exposed by an Integration Flow in NiFi can be composed as below:
    http://nifi:<port, e.g., '8091'>/<the three-letter identifier for your PG, e.g., 'p02'>/<path with three-letter identifiers if you have a routing structure inside your PG>/<the rest of the URL path specific to the Integration Flows inside your PG, if any>
  • The CompanyX PG has just one iFlow (see Figure 4 above) and therefore no routing structure. Hence, it’s internal URL would be http://nifi:8091/p02
  • Such internal endpoints exposed by the iFlows in DS NiFi cannot be invoked directly from the Internet. The corresponding secure proxy endpoints/routes in the API Security Gateway (ASG) for these endpoints need to be invoked instead, where authentication (and authorization, if desired) is enforced by the EFS.
  • The ASG checks the Service Registry for new service registrations/updates to existing services periodically (currently, every 1 hour) and creates routes to proxy <service>.<api>.url for each API in the service object. (Service Registry details: SR User Guide)
  • Therefore, you need to register the API exposed by in the iFlow in CompanyX PG as a service in the Service Registry ( Service Registry API Doc).
  • cURL for service registration:
curl --location --request PUT 'https://efpf.smecluster.com/apis/sr/companyx-service-1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer replace_this_text_with_token' \
--data-raw '{
    "id": "companyx-service-1",
    "type": "efpf.example",
    "title": "CompanyX iFlow Service Example",
    "description": "An example of iFlow service API from the CompanyX PG",
    "apis": [{
            "id": "companyx-service-1-api-1",
            "title": "Example API 1 of CompanyX iFlow Service",
            "description": "Example API 1 of CompanyX iFlow Service",
            "protocol": "HTTP",
            "url": "http://nifi:8091/p02",
            "spec": {
                "mediaType": "application/vnd.oai.openapi+json;version=3.0",
                "url": "https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
                "schema": {}
            }
        }
    ],
    "meta": {},
    "doc": "https://docs.efpf.linksmart.eu/swagger-ui/?url=https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
    "ttl": 864000000
}'
  • The service registration object in the cURL above (--data-raw) does not contain the enableAuthZ flag. Therefore, only authN will apply to its secure proxy route, i.e., all the users with an EFPF user account will be able to access its secure proxy endpoint.

  • After this service registration, ASG will automatically create a proxy route for http://nifi:8091/p02 which can be composed as below:
    https://<ASG root URL>/apis/<service id>/<api id>/<the URL path from <service>.<api>.url, i.e., without the "http://nifi:<port>" part>

  • Therefore, the secure proxy endpoint for http://nifi:8091/p02 would be: https://efpf.smecluster.com/apis/companyx-service-1/companyx-service-1-api-1/p02

  • How to invoke this secure proxy endpoint:

  • Step 1: Get an access token from the EFS Keycloak:

curl --location --request POST 'https://efpf.smecluster.com/auth/realms/efpf/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=user-terminal' \
--data-urlencode 'username=your-efpf-user@companyx.org' \
--data-urlencode 'password=replace_this_with_password'
  • Step 2: Use the access token to invoke the secure proxy endpoint:
curl --location --request GET 'https://efpf.smecluster.com/apis/companyx-service-1/companyx-service-1-api-1/p02' \
--header 'Host: efpf.smecluster.com' \
--header 'Authorization: Bearer replace_this_text_with_token'
  • OR how to invoke using Postman with Authorization Type ‘Bearer Token’:

img.png

  • OR how to invoke using Postman with Authorization Type ‘OAuth2.0’:

  • This method lets you generate token and use it to call an endpoint in the same request page, in 3 easy steps:

  • Step 1: Set the Oauth2.0 Configuration Options and click on ‘Get New Access Token’ button postman-step-1.png

  • Step 2: After getting the token, click on ‘Use Token’ button postman-step-2.png

  • Step 3: After the token gets added to the Access Token field, click on ‘Send’ to invoke the endpoint postman-step-3.png

Example 2: iFlow in Example_p01 PG (enabling AuthN as well as AuthZ)

iFlow in Example_p01 PG with AuthZ enabled Figure 7: iFlow in Example_p01 PG with AuthZ enabled

  • This example shows how authorization (AuthZ), in addition to authentication (AuthN) can be enabled for the endpoint exposed by the iFlow in Example_p01 PG.
  • Internal URL: http://nifi:8091/p01
  • cURL for service registration:
curl --location --request PUT 'https://efpf.smecluster.com/apis/sr/example-service-p1-p01' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer replace_this_text_with_token' \
--data-raw '{
    "id": "example-service-p1-p01",
    "type": "efpf.example",
    "title": "example-service-p1-p01",
    "description": "An example of a service with authorization enabled.",
    "apis": [{
            "id": "example-service-p1-p01-api-1",
            "title": "API 1 of example-service-p1-p01 service",
            "description": "To access this API, example-service-p1-p01-admin role is needed.",
            "protocol": "HTTPS",
            "url": "http://nifi:8091/p01",
            "spec": {
                "mediaType": "application/vnd.oai.openapi+json;version=3.0",
                "url": "https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
                "schema": {}
            },
            "meta": {
                "dataspine": {
                    "enableAuthZ": true
                }
            }
        }
    ],
    "meta": {},
    "doc": "https://docs.efpf.linksmart.eu/swagger-ui/?url=https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
    "ttl": 864000000
}'
  • As the service registration object contains the enableAuthZ flag set to true, authZ will be enabled for its secure proxy endpoint.

  • The secure proxy route created by asg-importer currently enables support for two authorization-/access-levels: (1) either a user has full, admin-level access (CRUD) to the API, or (2) no access at all.

  • In EFS Keycloak, a new resource (example-service-p1-p01), a new scope (example-service-p1-p01-api-1_admin) and the corresponding policies and permissions need to be created.

  • A new realm role needs to be defined (example-service-p1-p01-api-1_admin), and assigned to the users who should have permission to call this secure proxy endpoint.

  • This config in EFS Keycloak is currently a manual process, and therefore, you would need to contact the EFPF Support Team for adding this configuration.

  • The secure proxy endpoint for http://nifi:8091/p01 would be: https://efpf.smecluster.com/apis/example-service-p1-p01/example-service-p1-p01-api-1/p01

  • How to invoke this secure proxy endpoint:

  • Prerequisite: Ensure that the user has the required role for calling the API endpoint

  • Step 1: Get an access token from the EFS Keycloak (as above)

  • Step 2: Use the access token to invoke the secure proxy endpoint:

curl --location --request GET 'https://efpf.smecluster.com/apis/example-service-p1-p01/example-service-p1-p01-api-1/p01' \
--header 'Host: efpf.smecluster.com' \
--header 'Authorization: Bearer replace_this_text_with_token'
  • OR invoke using Postman img_1.png

Example PGs

  • Every PG directly inside the root ‘NiFi Flow’ PG contains the first PG (p00 - inside Projects PG) as example PG
  • These example PGs are visible and accessible to all
  • An iFlow inside “Examples PG” (NiFi Flow » p1: EFPF Projects (nifi:8091) » Projects » (p00) Example_p00 (Examples PG-AccessToALL) » (ex2) Jolt Simple Shiftr Data Transformation):

iFlow (ex2) inside Examples PG Figure 8: iFlow (ex2) inside Examples PG

  • How to invoke:
curl --location --request GET 'https://efpf.smecluster.com/apis/example-service-id/example-api-id/p00/ex2' \
--header 'Host: efpf.smecluster.com' \
--header 'Authorization: Bearer replace_this_with_token'

Steps for Configuring Access to your Templates

  • NiFi Templates can be used to reuse, share and port (download/upload as XML files) complete integration flows or certain parts of an integration flow. Please refer to Apache NiFi’s User Guide for creating and managing templates.
  • By default, the templates created by a user inherit the access policies of the PG they are created from. But, these can be overridden.
  • To configure access policies for your template (i.e., to override the default inherited access policies), click on the ‘Global Menu’ on NiFi’s GUI and select ‘Templates’.
  • Locate your template and click on ‘Access Policies’ (🔑).
  • On the ‘Access Policies’ page for your template, for each access policy from the dropdown list, add or remove users or user groups as required. These policies are the same as the ones for PGs, which are enlisted in the table above in ‘Steps for Configuring Access to your PG’ section.

Data Spine NiFi Documentation

Previous
Next