User Guide 101 - Composite Application Developer

Target audience

This guide is designed to provide support to Composite Application Developers, that is, the users who want to create applications that use the existing tools/services from the EFPF Ecosystem using the provided NiFi graphical tool.

An application can also be developed to provide interoperability between different data models. If a tool/service consumer expects data adhering to a different data model than what the tool/service provides, she/he can make use of Data Spine (DS) NiFi to create an integration flow (i.e., a workflow/dataflow) that uses the built-in data model transformation tools (e.g., Jolt, XSLT, ExecuteScript, etc. Details NiFi Data Transformation Examples, DS NiFi Integration Flow Examples). NiFi provides an intuitive drag-and-drop GUI to create integration flows easily.

An overall flowchart of the steps is illustrated in the following picture:

img

Create a Composite Application using Existing Services

  1. Set up your EFPF account: Setting up an account is the same for all types of users, so you can find the instructions here.

  2. Get an access token via the EFPF Security Portal (EFS) API: Following the documentation on getting an access token, you need to first get a token that you later will use to access the Service Registry. While invoking Service Registry’s API, you would need to pass the access token using ‘Authorization’ header as explained in the documentation.

  3. Retrieve the needed information from the Service Registry: Use the Service Registry REST API to search for the existing services you want to use and retrieve the needed data. The URL of the Service Registry can be found here. (Details: Service Registry Overview, Service Registry User Guide)

  4. Get permissions to access NiFi: Send an email to the EFPF Support Team asking for permissions to access NiFi. To create Integration Flows in DS NiFi, a collaboration space called ‘Process Group (PG)’ would be given to you. You would be given admin privileges for your PG enabling you to not only create and execute integration flows, but also give other users access to your PG in order to collaborate. (Detailed procedure: DS NiFi User Guide. To get familiar with Apache NiFi, take a look at the NiFi Overview and the Apache NiFi Quickstart Guide pages.)

  5. Log in to NiFi: After getting the necessary access permissions, log in to NiFi’s GUI. To develop your application that uses the existing services in the EFPF Ecosystem, you will need to create integration flows using the drag-and-drop GUI of the Integration Flow Engine (NiFi).

  6. Get familiar with NiFi tool: The Integration Flow Engine of the DS is realised using the Apache NiFi dataflow management tool. To get familiar with Apache NiFi, take a look at the NiFi Overview and the Apache NiFi Quickstart Guide pages. The official documentation of Apache NiFi will give you more detailed information on using it. (EFPF specific details: DS NiFi User Guide)

  7. Develop your composite application using the existing services: This is the core part of the work to be done. This is where you have to create one or more integration flows to implement the application logic in NiFi. NiFi provides built-in processors to accomplish various tasks with ease, e.g., for data transformation to achieve interoperability at Data Model level, it provides processors such as Jolt, TransformXml (XSLT), ExecuteScript, etc. To easily create integration flows, you can refer to the example integration flows available in the same instance of NiFi you are using. Inside the ‘p1’ Process Group in NiFi you will find another Process Group called ‘Examples’. This Process Group contains some simple, most commonly used integration flows. (Details: DS NiFi User Guide)

In case you need to make some information from your application accessible from outside of NiFi (i.e., you take on the role of a Service Provider), you need to follow the next steps, depending on how you plan on doing this. You have two options:

  • Using Pub/Sub: You can make information from your integration flow in NiFi accessible from the outside via the DS Message Bus (RabbitMQ) using MQTT/AMQP.

    1. Request a username/password for RabbitMQ: Send an email to the EFPF Support Team (Details: DS RabbitMQ User Guide)

    2. Publish your data: In your integration flow in NiFi publish your data to RabbitMQ with the username and password provided in the previous step.

    3. Update the Service Registry: It is recommended to register the new Pub/Sub API to the Service Registry. If the data being published is to be consumed by other companies, the registration is mandatory. Else, if the data being published is meant to be consumed by the same users who published it or by users in the same company, it’s not mandatory but it’s still recommended to register the API containing information such as the topic name, payload syntax, etc., to the SR as it would be useful in the long run. Follow the steps below for registering/updating a service

  • Using your HTTP API: You can configure your integration flow in NiFi to expose an HTTP API endpoint that can be accessed like any other service/API in the EFPF Ecosystem. (Details: DS NiFi User Guide)

    1. Register your Endpoint as a Service: Follow the steps below for registering/updating a service and the API Security Gateway will create a secure proxy endpoint.

Register/Update a Service

  1. Create a JSON Description of your service: The JSON object should have all the required metadata for the service you want to register or update. The data model of the service can be found here.

  2. Ask Admin Rights: As stated before, if you need to register/update a service, you need to send an email to the EFPF Support Team asking for admin (update/delete) rights for the Service Registry.

  3. Get an access token via the EFS API: Following the documentation on getting an access token, you need to first get a token that you later will use to access the Service Registry.

  4. Register/Update your Service: The JSON object created before is POSTed/PUT to the Service Registry API using the token you got before. (Details: Service Registry User Guide)

  5. Check your new service: Use the provided API in the Service Registry to retrieve the registered services and check that the one you just registered/updated is there.

EFPF Ecosystem Documentation

See also

  • Data Spine User Guide: This guide enlists the activities that the service providers need to do in order to provide their services through the Data Spine and the activities the service consumers need to do in order to consume the services provided through the Data Spine with the help of examples.