DS NiFi Admin Guide

Prerequisites

  • Create a new client named ‘nifi-client’ in EFS Keycloak with the following properties:
  • Under ‘Settings’ tab:
	Client ID: nifi-client
	Enabled: ON
	Client Protocol: openid-connect
	Access Type: confidential
	Standard Flow Enabled: ON
	Direct Access Grants Enabled: ON
	Valid Redirect URIs: https://dataspine.efpf.linksmart.eu:443/* // assuming that 'dataspine.linksmart.eu' is the domain name for NiFi
  • Under ‘Credentials’ tab:
	Client Authenticator: Client Id and Secret
	// and note down the Secret

Scope

  • Deployment: The deployment of Data Spine (DS) NiFi in the form of a Docker container (with OIDC support).
  • Configuration: The configuration for design-time user authentication for DS NiFi.
  • Multi-Tenancy Setup: The initial setup of a Multi-Tenant Authorization environment which (1) enables collaboration between users while designing integration flows in NiFi’s GUI and, (2) supports separation of ‘spaces’ called ‘Process Groups (PGs)’ where the integration flows are created. This separation is achieved by employing fine-grained access policies.
  • Maintenance: Lifecycle management of users, user-groups, policies and PGs, etc.

Deployment and Configuration

  • Assume that:

    • the root directory for NiFi is /mnt/data/services/nifi
    • dockerized nginx will act as a (reverse proxy) gateway for NiFi and the hostname for Docker gateway configured inside nginx is ‘docker-host’ (This can be configured by passing the argument --add-host=docker-host:172.17.0.1 to docker run command for nginx)
  • Create sub-directories for data and config inside the root directory:
    cd /mnt/data/services/nifi
    mkdir data
    mkdir conf
    mkdir customconf

  • Add the attached start.sh file customized for DS NiFi to /mnt/data/services/nifi/customconf directory that enables ‘OIDC’ user authentication option when NiFi is deployed as a Docker container

  • To obtain copies of conf files and scripts from NiFi, run a temporary nifi container ‘nifiTemp’:
    docker run -d --name nifiTemp apache/nifi:latest

  • In order to setup a secure instance of NiFi, use the tls-toolkit command line utility embedded inside the nifiTemp container to automatically generate the required keystores, truststore, and relevant configuration files:

	docker exec nifiTemp /opt/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -n 'docker-host(1)' --nifiPropertiesFile '/opt/nifi/nifi-current/conf/nifi.properties' --outputDirectory '/opt/nifi/nifi-current/conf' --isOverwrite 'true'
  • This generates the following files (inside the container nifiTemp):
    in /opt/nifi/nifi-current/conf directory:
    nifi-cert.pem
    nifi-key.key
    and, in /opt/nifi/nifi-current/conf/docker-host :
    keystore.jks
    nifi.properties
    truststore.jks
  • You can view the generated files with
    docker exec nifiTemp ls /opt/nifi/nifi-current/conf/
  • You can view the generated certificate with
    docker exec nifiTemp openssl x509 -in /opt/nifi/nifi-current/conf/nifi-cert.pem -noout -text
  • Copy files in /opt/nifi/nifi-current/conf/docker-host to /opt/nifi/nifi-current/conf:
    docker exec nifiTemp cp -rv /opt/nifi/nifi-current/conf/docker-host/. /opt/nifi/nifi-current/conf
  • Now, copy conf and scripts directories from nifiTemp to host:
    docker cp nifiTemp:/opt/nifi/nifi-current/conf /mnt/data/services/nifi/conf
    docker cp nifiTemp:/opt/nifi/scripts /mnt/data/services/nifi/conf
  • Create directories on the host to mount NiFi directories to:
    mkdir /mnt/data/services/nifi/data/logs
    mkdir /mnt/data/services/nifi/data/state
    mkdir /mnt/data/services/nifi/data/database_repository
    mkdir /mnt/data/services/nifi/data/flowfile_repository
    mkdir /mnt/data/services/nifi/data/content_repository
    mkdir /mnt/data/services/nifi/data/provenance_repository
  • Remove the nifiTemp container now:
    docker stop nifiTemp
    docker rm nifiTemp
  • Replace /mnt/data/services/nifi/data/scripts/start.sh with the customized start.sh that has the ‘oidc’ auth option enabled:
    cp /mnt/data/services/nifi/customconf/start.sh /mnt/data/services/nifi/conf/scripts
  • Create a file ‘run.sh’ and add the following command to it:
	docker run -d --name nifi \
	--restart=unless-stopped \
	-p 8091:8091 \
	-p 8092:8092 \
	-p 9443:9443 \
	-e NIFI_WEB_HTTPS_PORT='9443' \
	-e NIFI_WEB_PROXY_HOST='dataspine.linksmart.eu:9443' \
	-e NIFI_WEB_PROXY_CONTEXT_PATH='/' \
	-v /mnt/data/services/nifi/conf/scripts:/opt/nifi/scripts \
	-v /mnt/data/services/nifi/conf/conf:/opt/nifi/nifi-current/conf \
	-v /mnt/data/services/nifi/data/logs:/opt/nifi/nifi-current/logs \
	-v /mnt/data/services/nifi/data/state:/opt/nifi/nifi-current/state \
	-v /mnt/data/services/nifi/data/database_repository:/opt/nifi/nifi-current/database_repository \
	-v /mnt/data/services/nifi/data/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository \
	-v /mnt/data/services/nifi/data/content_repository:/opt/nifi/nifi-current/content_repository \
	-v /mnt/data/services/nifi/data/provenance_repository:/opt/nifi/nifi-current/provenance_repository \
	-e KEYSTORE_PATH=/opt/nifi/nifi-current/conf/keystore.jks \
	-e KEYSTORE_TYPE=JKS \
	-e KEYSTORE_PASSWORD=<to_be_updated> \
	-e TRUSTSTORE_PATH=/opt/nifi/nifi-current/conf/truststore.jks \
	-e TRUSTSTORE_PASSWORD=<to_be_updated> \
	-e TRUSTSTORE_TYPE=JKS \
	-e AUTH=oidc \
	-e NIFI_SECURITY_USER_OIDC_DISCOVERY_URL=https://efpf-security-portal.salzburgresearch.at/auth/realms/master/.well-known/openid-configuration \
	-e NIFI_SECURITY_USER_OIDC_CONNECT_TIMEOUT='100 secs' \
	-e NIFI_SECURITY_USER_OIDC_READ_TIMEOUT='5 secs' \
	-e NIFI_SECURITY_USER_OIDC_CLIENT_ID=nifi-client \
	-e NIFI_SECURITY_USER_OIDC_CLIENT_SECRET=<to_be_updated> \
	-e NIFI_SECURITY_USER_OIDC_PREFERRED_JWSALGORITHM=RS256 \
	-e INITIAL_ADMIN_IDENTITY='admin@efpf.org' \
	-e NIFI_JVM_HEAP_INIT=2g \
	-e NIFI_JVM_HEAP_MAX=4g \
	apache/nifi:latest
  • In the command above, update the NIFI_SECURITY_USER_OIDC_CLIENT_SECRET from the ‘nifi-client’ we created earlier in EFS Keycloak (see ‘Prerequisites’ section)
  • In addition, update the keystore and truststore passwords in the docker run command above with the corresponding ones in /mnt/data/services/nifi/conf/conf/nifi.properties file. The passwords can be obtained directly by executing the following commands:
    more /mnt/data/services/nifi/conf/conf/nifi.properties | grep nifi.security.keystorePasswd=
    more /mnt/data/services/nifi/conf/conf/nifi.properties | grep nifi.security.truststorePasswd=
  • In the command above, set the INITIAL_ADMIN_IDENTITY property to admin’s email id (that is linked with his/her EFS account)
  • The docker run command inside run.sh publishes the ports 8091 and 8092 to be used by the Integration Flows in NiFi. More such ports can be published as required by adding the new port mapping to this command and redeploying the ‘nifi’ container:
    docker stop nifi
    docker rm nifi
    sh run.sh
  • Import Java/JVM trusted certs into NiFi’s truststore to make NiFi trust those root certs:
    docker exec nifi keytool -importkeystore -srckeystore /usr/local/openjdk-8/lib/security/cacerts -destkeystore /opt/nifi/nifi-current/conf/truststore.jks -deststorepass replace_with_nifi_truststore_password

Importing Java certs to NiFi’s truststore enables HTTPS/MQTTS calls from processors such as InvokeHTTP, ConsumeMQTT (that is if the target’s ssl certs are trusted by the JVM).

  • Update nginx’s configuration to proxy requests to NiFi:
	# NiFi
	proxy_ssl_trusted_certificate /mnt/data/services/nifi/conf/conf/nifi-cert.pem;
    location / {
        proxy_pass https://docker-host:9443;
        proxy_set_header Host $host;
        proxy_set_header X-ProxyScheme https;
        proxy_set_header X-ProxyHost $host;
        proxy_set_header X-ProxyPort 443;
        proxy_set_header X-ProxyContextPath /;
    }

    # NiFi Integration Flows
    location /p1/ {
        proxy_pass http://docker-host:8091/;
    }
    location /p2/ {
        proxy_pass http://docker-host:8092/;
    }

Troubleshooting

  • From the vm where NiFi is deployed:
    curl -i --insecure https://localhost:9443
    openssl s_client -showcerts -connect localhost:9443
    openssl x509 -in ./conf/conf/nifi-cert.pem -noout -text

Multi-Tenancy Setup

Background, reasoning and guidelines:

  • NiFi’s Web-based GUI is intended to be used by multiple users for creating Integration Flows. The access and visibility of such Integration Flows needs to be restricted to their creators only and, restricted and hidden from the other users. Furthermore, user collaboration over an Integration Flow or a Process Group containing several Integration Flows needs to be facilitated. This requires not only configuring who has access to the Process Groups but also the level of their access. NiFi provides this functionality through its ‘Multi-Tenant Authorization’ policy governance framework. The Multi-Tenant Authorization enables multiple groups of users to collaboratively view, control and manipulate different parts of the Integration Flows, with different levels of authorization. Thus, when a logged in user attempts to view or update a particular resource through NiFi’s GUI, NiFi, based on the configured privileges for the user allows or denies that particular action. To define such privileges for individual users or user groups, the access policies need to be defined.

  • For DS NiFi, two different user roles would be defined: DS-Admin and DS-User. These would be defined in the form of NiFi user groups named ‘DS-Admins’ and ‘DS-Users’ respectively.

  • The access policies for the users belonging to the DS-Admins group are configured by the ‘First User’ (configured before with the ‘INITIAL_ADMIN_IDENTITY’ environment variable) enabling them to configure access policies for DS-Users.

  • The access policies for these user roles/groups need to be defined at three different levels:

    1. Global Access Policies: The Global Access Policies define privileges for uses that are applicable system-wide. The following table shows the Global Access Policies defined for NiFi in the EFPF Project.

    NiFi Global Access Policies Table 1: Global Access Policies

    1. Component-level Access Policies for the Root Process Group (‘NiFi Flow’): These policies define privileges for uses that are applicable to the Root Process Group of NiFi and the components (Process Groups, Integration Flows, processors, etc.) present in it. The following table shows the Component-level Access Policies for the Root Process Group ‘NiFi Flow’ in the EFPF Project.

    Component-level Access Policies for the Root Process Group 'NiFi Flow' Table 2: Component-level Access Policies for the Root Process Group 'NiFi Flow'

    1. Component-level Access Policies for a particular component (Process Group) PG_X: These policies define privileges for uses that are applicable to that particular component (Process Group, Template, etc.). When a user, user1 requests for the creation of a new PG, DS-Admin creates such PG, ‘PG_X’ and grants user1 ‘admin’ level privileges for PG_X by configiring policies as shown in Table 3 below. user1, the new component-owner/admin for PG_X can then grant permissions for PG_X to others (e.g. either directly or by creating user groups such as ‘PG_X-Admins’, ‘PG_X-Users’, etc.).

    Component-level Access Policies for a particular component (Process Group) PG_X Table 3: Component-level Access Policies for a particular component (Process Group) PG_X

  • In addition, when creating integration flows in NiFi that involve request-response type of communication, a new port needs to be published, added to docker run command and nifi container needs to be redeployed. To avoid that, we publish a predefined set of ports, create a PG per port in NiFi’s root PG and create routing integration flows inside these PGs. The user-specific PGs are then created inside these port-specific PGs and each user-specific PG is given a three letter identifier. The routing integration flows inside these port-specific PGs route requests to the corresponding user-specific PGs based on these identifiers in the URL path.

  • The following figure illustrates an example of NiFi’s Multi-Tenant Authorization. The users ‘admin’ and ‘user1’ have roles DS-Admin and DS-User respectively and user1 is the Component-Owner for the Process Group ‘ProjectA’. Therefore, only ProjectA component is visible and accessible to user1 and not other Process Groups and processors and not even some of the GUI elements.

NiFi Multi-Tenant Authorization Figure 1: NiFi Multi-Tenant Authorization

Multi-Tenancy Setup Steps

Creation of Integration Flows for Routing

A NiFi template that contains the initial admin-specific routing integrations flows' structure is attached at the end of this section

  • As the First User, login to NiFi and create port-specific PGs in NiFi’s root PG ‘NiFi Flow’ as shown in the figure below:

NiFi Port-Specific Integration Flows Figure 2: NiFi Port-Specific Integration Flows

  • Navigate inside ‘p1/8091’ PG and create a routing integration flow using ‘UpdateAttribute’ and ‘RouteOnAttribute’ processors to route to child PGs based on their three letter identifiers as shown in the figure below:

Routing Integration Flow in p1/8091 PG Figure 3: Routing Integration Flow in p1/8091 PG

  • Configure the properties for individual processors as given below:

HandleHttpRequest Processor Configuration Figure 4: HandleHttpRequest Processor Configuration

UpdateAttribute Processor Configuration Figure 5: UpdateAttribute Processor Configuration

RouteOnAttribute Processor Configuration Figure 6: RouteOnAttribute Processor Configuration

HandleHttpResponse Processor Configuration Figure 7: HandleHttpResponse Processor Configuration

LogMessage Processor Configuration Figure 8: LogMessage Processor Configuration

  • In this way, the root PG ‘NiFi Flow’ and the port-specific PGs inside it are admin-specific and are used for routing and management purposes. The PGs inside these port-specific PGs can be assigned directly to users (e.g. the ‘Examples’ PG is user-specific PG) or another level of PGs can be used to manage the connections better visually (e.g. the ‘Organisations’ PG is admin-specific and it’s child PGs are user-specific)

Another Level of Admin-Specific PGs: 'Organisations' PG Figure 9: Another Level of Admin-Specific PGs: 'Organisations' PG

Creation of User-Groups and Configuration of Policies

  1. Global Access Policies:
  • Click on the ‘Global Menu’ on NiFi’s GUI and select ‘Users’

Global Menu - Users Figure 10: Global Menu - Users

  • Click on the add user/group icon on the right hand side, select the ‘Group’ radio button and add two new user groups ‘DS-Admins’ and ‘DS-Users’ for the admins and users of Data Spine respectively. Add the email id for the First User (here: admin@efpf.org) to the DS-Admins group while creating it.

Creation of User-Groups Figure 11: Creation of User-Groups

  • Click on the ‘Global Menu’ on NiFi’s GUI and select ‘Policies’. Configure the global access policies for DS-Admins and DS-Users groups as shown in Table 1 above
  1. Component-level Access Policies for the Root PG ‘NiFi Flow’:
    User Groups:
  • Right click anywhere in the NiFi canvas in ‘NiFi Flow’ PG and select ‘Manage access policies’
  • Configure the access policies for DS-Admins and DS-Users groups as shown in Table 2 above
    Controller Services:
  • For the ‘NiFi Flow’ root PG, controller services ‘StandardHttpContextMap’ and ‘StandardRestrictedSSLContextService’ also needs to be enabled
  • To do that, right click anywhere on the canvas in ‘NiFi Flow’ PG and select ‘Configure’
  • On the NiFi Flow Configuration page, in ‘CONTROLLER SERVICES’ tab, click on add (+)
  • Search for ‘StandardHttpContextMap’, select it and click on ‘ADD’.
  • To enable it, click on ‘Enable’ (⚡)
  • To configure access policies for it, click on ‘Access Policies’ (🔑)

Enabling StandardHttpContextMap Figure 12: Enabling StandardHttpContextMap

  • For the policy ‘view the component’, add ‘DS-Admins’ and ‘DS-Users’ to the list of users
  • This StandardHttpContextMap is now ready to be used across different integration flows
  • Repeat the same procedure for adding, enabling and configuring access policies for ‘StandardRestrictedSSLContextService’ controller service
  1. Component-level Access Policies for a particular component (Process Group) PG_X:
  • The Data Spine Admin only needs to configure access policies for the Component-Owner/admin of the new Process Group PG_X
  • The person who requested the creation of PG_X is considered as the Component-Owner/admin for PG_X
  • Right click on PG_X and select ‘Manage access policies’
  • Configure the access policies for ‘Component-Owner for PG_X’ as shown in Table 3 above

Creation of Users

  • The users in EFS Keycloak are not directly replicated in NiFi and thus need to be added explicitely.
  • An EFS user, who wants to access DS NiFi, needs to send an email to the EFPF Support Team mentioning his/her email id (which is linked with the EFS account)
  • On receipt of such an email, click on the ‘Global Menu’ on NiFi’s GUI and select ‘Users’.
  • Click on the add user/group icon on the right hand side, select the ‘User’ radio button, type in the email id of the user to add him/her.
  • While creating the user in this way, also select the groups among DS-Admins and DS-Users to add the user to. All the users who are not DS-Admins are DS_Users. The policies configured for these groups become applicable to the existing users in those groups as well as to the newly added users.

Creation of new Process Groups

  • An EFS user, who wants a new PG in DS NiFi, needs to send an email to the EFPF Support Team mentioning the name of the PG to be created and his/her email id (which is linked with the EFS account)
  • Let’s assume that the user, ‘user1’ has requested the creation of a new PG named ‘myPG’ and myPG fits inside a parent PG called ‘Organisations’ which is a part of the admin-specific routing integration flow structure.
  • Designate an unique three letter identifier to myPG. Let’s say, the identifier is ‘exo’.

The database of already assigned three letter identifiers needs to be maintained by the Data Spine Admin!

  • Navigate inside the ´Organisations PG and create a new PG called ‘myPG’.
  • Inside myPG, create a new port ‘exo’.
  • Inside Organisations PG, create a new port named ‘exo’, and join it to myPG’s input port also named ‘exo’.

Creation of a new Process Group 'myPG' Figure 13: Creation of a new Process Group 'myPG'

  • Inside Organisations PG’s parent PG ‘p1/8091’, update the ‘RouteOnAttribute’ processor to add a new ‘Routing Strategy’ for myPG based on its identifier ‘exo’

Configuration of a new Routing Strategy in RouteOnAttribute Processor Figure 14: Configuration of a new Routing Strategy in RouteOnAttribute Processor

  • Create a new connection from RouteOnAttribute Processor to Organisations PG linking relationship ‘exo’ to input ‘exo’

Linking of RouteOnAttribute Processor to Organisations PG for 'exo' Figure 15: Linking of RouteOnAttribute Processor to Organisations PG for 'exo'

  • Start the relevant connections and processors
  • To make user1 the admin for myPG, right click on myPG and select ‘Manage access policies’
  • Configure the access policies for myPG as shown in Table 3 above
  • In ‘Access Policies’ window, when ‘view the component’ policy is selected in the dropdown list by default, you’ll see a message ‘Showing effective policy inherited from Process Group NiFi Flow. Override this policy.’
  • Click on Override. Then, in the ‘Override Policy’ prompt, select ‘Copy’ radio button and click on ‘OVERRIDE’
  • Remove ‘DS-Users’ from the list of users and add user1’s email id there.
  • Repeat this procedure for the other policies
  • In case of ‘view the policies’ and ‘modify the policies’ in the drop down, click on ‘Add’ and then add ‘DS-Admins’ and user1’s email id to the list of users
  • Finally, right click on the input port ‘exo’ and select ‘Manage access policies’
  • Configure only the ‘view the component’ policy in the same way as above and all the other policies can be left to their default inherited state
  • Access policies are now configured and user1 is now the admin for myPG
  • Send out an acknowledgement email to user1 that contains the following information:
    1. Path to your PG: <p1/8091>/<Organisations>/<myPG>
    2. Identifier for your PG: exo

Testing

  • To make sure that the initial setup was done correctly, the following steps can be followed for invoking the endpoints exposed by an example integration flow in NiFi
  • The URL for an endpoint exposed by an Integration Flow in NiFi can be composed as below: https://<nifi root URL>/<the two letter identifier of the parent PG directly inside the 'NiFi Flow' PG>/<identifier for your PG>/<the rest of the URL path specific to the Integration Flows inside your PG>

Examples PG Figure 16: Examples PG

  • The URLs for endpoints exposed by the integration flows inside Examples PG: https://dataspine.efpf.linksmart.eu/p1/exp/ex1/smecluster/GetAllProducts
    https://dataspine.efpf.linksmart.eu/p1/exp/ex2
  • These endpoints exposed by Integration Flows in NiFi cannot be invoked directly from outside. The corresponding proxy endpoints in the API Security Gateway for these endpoints need to be invoked instead, where authentication and authorization is enforced by EFS.
  • To test the correctness of the intial setup, from the vm where NiFi is deployed, invoke:
    curl --request GET 'https://localhost:8091/exp/ex1/smecluster/GetAllProducts'
    curl --request GET 'https://localhost:8091/exp/ex2'

Maintenance

  • To publish a new port, see the ‘Deployment and Configuration’ section above
  • To enable EFS users in DS NiFi, see the ‘Creation of Users’ section above
  • To create a new PG, see the ‘Creation of new Process Groups’ section above

Data Spine NiFi Documentation

References

[1] NiFi System Administrator’s Guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
[2] NiFi TLS Toolkit: https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit
[3] Apache NiFi at Dockerhub: https://hub.docker.com/r/apache/nifi

Previous
Next