DS NiFi User Guide

Connection details

Environment Component URL Description
Development NiFi GUI https://dataspine.efpf.linksmart.eu/nifi GUI for creating integration flows. Accessible to EFPF users.
Development Proxy APIs exposed by integration flows (iFlow API) Example: https://dataspine.efpf.linksmart.eu/p1/exp/ex1 Details below. Not directly accessible to EFPF users.
Development APIs secured through API Security Gateway Example: https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex1/ Details below. Accessible to EFPF users.

Prerequisites

  • Register at the EFPF Portal to get an EFS account.

Notes

  • To create Integration Flows in Data Spine NiFi, a collaboration space called ‘Process Group (PG)’ would be given.
  • PGs can be given per organization or per project.
  • Multiple users can be given access to a single PG and each PG can contain multiple Integration Flows or even other child PGs.
  • After getting an EFS account, you need to send an email to the Data Spine Technical Support Team for obtaining basic access privileges in NiFi mentioning the email id linked with your EFS account. If you try to login to NiFi without that, you’ll get an ‘Insufficient Permissions’ error.

Steps for Getting a New PG

  • If you want a new PG, send an email to the Data Spine Technical Support Team mentioning the name of the PG to be created and your email id (this is linked with the EFS account). You would be given admin privileges for this PG.
  • You will receive an acknowledgement email once the PG has been created with (1) the path to your PG in the form <parent PG>/<your PG> and, (2) a three letter identifier for the PG. Let’s say, the name of your PG is ‘myPG’, it is inside a parent PG called ‘Organisations’ which is inside another PG called ‘p1/8091’ and its identifier is ‘exo’. Then the acknowledgement email will contain the following information:
    1. Path to your PG: <p1/8091>/<Organisations>/<myPG>
    2. Identifier for your PG: exo
  • Login to Data Spine NiFi and navigate to myPG (see Figure 1)

Navigation through Parent PGs Figure 1: Navigation through Parent PGs

  • PGs other than myPG won’t be visible and accessible to you as shown in Figure 2.

Visibility and Access Control Figure 2: Visibility and Access Control

  • Inside myPG, you will find an input port (say, ‘in1’). If you want to create Integration Flows that involve request-response type of communication, you can connect this port in1 to your Integration Flow to receive requests. Connecting this port won’t be required in the case of Integration Flows that involve publish-subscribe type of communication.
  • Inside myPG, you can use ‘UpdateAttribute’ and ‘RouteOnAttribute’ processors of NiFi to have multiple request-response Integration Flows as done in ‘Examples (exp)’ PG shown in Figure 3.

Examples PG Contains Multiple Child PGs and Integration Flows Figure 3: Examples PG Contains Multiple Child PGs and Integration Flows

Steps for Configuring Access to your PG

  • If you want to give more users access to myPG, right click on myPG and select ‘Manage access policies’.
  • Then add users against each access policy to give them that respective access as shown in Figure 4.

Manage Access Policies for myPG Figure 4: Manage Access Policies for myPG

  • The policies and the corresponding privileges are explained in the table below:
Policy Privilege
view the component Allows users to view component configuration details
modify the component Allows users to modify component configuration details
operate the component Allows users to operate components by changing component run status (start/stop/enable/disable), remote port transmission status, or terminating processor threads
view provenance Allows users to view provenance events generated by this component
view the data Allows users to view metadata and content for this component in flowfile queues in outbound connections and through provenance events
modify the data Allows users to empty flowfile queues in outbound connections and submit replays through provenance events
view the policies Allows users to view the list of users who can view/modify a component
modify the policies Allows users to modify the list of users who can view/modify a component
receive data via site-to-site Allows a port to receive data from NiFi instances
send data via site-to-site Allows a port to send data from NiFi instances

Steps for Invoking the Endpoints Exposed by Integration Flows in NiFi

  • The URL for an endpoint exposed by an Integration Flow in NiFi can be composed as below:
    https://<nifi root URL>/<the two letter identifier of the parent PG directly inside the 'NiFi Flow' PG>/<identifier for your PG>/<the rest of the URL path specific to the Integration Flows inside your PG>
  • The Examples PG is visible to all. Below are the endpoints exposed by Integration Flows inside this PG:
  1. ex1: https://dataspine.efpf.linksmart.eu/p1/exp/ex1/smecluster/GetAllProducts
  2. ex2: https://dataspine.efpf.linksmart.eu/p1/exp/ex2
  • These endpoints exposed by Integration Flows in NiFi cannot be invoked directly. The corresponding proxy endpoints in the API Security Gateway (ASG) for these endpoints need to be invoked instead, where authentication and authorization is enforced by EFS.
  • The ASG checks the Service Registry for new service registrations/updates to existing services periodically (currently, every 1 hour) and creates routes to proxy api.url for each API in the service object.
  • Therefore, we need to register the API exposed by Examples PG as a service in the Service Registry ( Service Registry API Doc):
curl --location --request PUT 'https://efpf-security-portal.salzburgresearch.at/apis/sr/example-service-id' \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer replace_this_text_with_token' \
--data-raw '{
	"id": "example-service-id",
	"type": "EFPF.example",
	"title": "Example Service",
	"description": "Example Service to demonstrate automated creation of proxy endpoints in APISIX based on this service description in SR",
	"meta": {},
	"apis": [
		{
			"id": "example-api-id",
			"title": "Example API",
			"description": "Example API contains original endpoints https://dataspine.efpf.linksmart.eu/p1/exp/ex1/smecluster/GetAllProducts and https://dataspine.efpf.linksmart.eu/p1/exp/ex2. Proxy endpoints for these in APISIX would be https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex1/smecluster/GetAllProducts and https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex2 respectively.",
			"protocol": "HTTPS",
			"url": "https://dataspine.efpf.linksmart.eu/p1/exp",
			"spec": {
				"mediaType": "application/vnd.oai.openapi+json;version=3.0",
				"url": "https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
				"schema": {}
			},
			"meta": {}
		}
	],
	"doc": "https://linksmart.eu/swagger-ui/dist/?url=https://docs.efpf.linksmart.eu/projects/data-spine-nifi/apidoc/examples-swagger.json",
	"ttl": 8640000
}'
  • ASG will create a proxy route for https://dataspine.efpf.linksmart.eu/p1/exp which can be composed as below:
    https://<ASG root URL>/apis/<service id>/<api id>/<the URL path from api.url minus the NiFi root URL>
  • Therefore, the proxy endpoints for ex1 and ex2 endpoints are:
  1. https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex1/smecluster/GetAllProducts
  2. https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex2
  • Calling proxy endpoint (1) above:
curl --location --request GET 'https://efpf-security-portal.salzburgresearch.at/apis/example-service-id/example-api-id/p1/exp/ex1/smecluster/GetAllProducts' \
--header 'Authorization: bearer replace_this_text_with_token' \
--header 'Host: efpf-security-portal.salzburgresearch.at' \
--header 'Accept: application/json'

p.s.:

  • How to get a user token from EFS
curl --location --request POST 'https://efpf-security-portal.salzburgresearch.at/auth/realms/master/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=apisix' \
--data-urlencode 'client_secret=replace_this_with_client_secret' \
--data-urlencode 'username=your-efpf-user@gmail.com' \
--data-urlencode 'password=pwd'

Steps for Configuring Access to your Templates

  • NiFi Templates can be used to reuse, share and port (download/upload as XML files) complete integration flows or certain parts of an integration flow. Please refer to Apache NiFi’s User Guide for creating and managing templates.
  • By default, the templates created by a user inherit the access policies of the PG they are created from. But these can be overridden.
  • To configure access policies for your template (i.e. to override the default inherited access policies), click on the ‘Global Menu’ on NiFi’s GUI and select ‘Templates’.
  • Locate your template and click on ‘Access Policies’ (🔑).
  • On the ‘Access Policies’ page for your template, for each access policy from the dropdown list, add or remove users or user groups as required. These policies are the same as the ones for PGs, which are enlisted in the table above in ‘Steps for Configuring Access to your PG’ section.
Previous