User guide 101

Target audience

This guide is designed to provide support to the following group of users:

  • Tool/Service consumers: You want to consume a tool/service that is already present in the EFPF Platform.
  • Composite Application Developers: You want to create applications that use tools/services from the EFPF Platform.

If you want to provide a new tool/service to the EFPF platform, unfortunately this guide is NOT for you. The guide for providers of tools/services is currently under development and will be linked here when it is ready.

Overview

As a new user, you might have some doubts about how to use or where to access certain components of the EFPF system. This guide tries to fill that gap and offers a step-by-step procedure to get you ready.

Whenever you feel lost and cannot find information about any component, come back to this page and you will probably find your answer.

An overall flowchart of the first steps for a new user is seen in the following picture:

img

Setting up your account

  1. Register to the EFPF Portal: This is the main website where you will find the dasbhoard to access all your components and services. On the top right section, you will find the Register button. Follow the regular procedure to create your account.

  2. Wait: this is an easy step. You need to wait for your account to be activated. You will receive an email when this happens.

  3. Access your dashboard: Once your account is activated, you will be able to access your dashboard. Go again to the same page as before and look for the Login button. Enter your credentials and access your dashboard. It is recommended to have a quick look on the information available there.

  4. Select the environment to work on: The environment you will be working on depends on the type of user you are: Service/tool consumer, Composite Application Developer user or Tool/Service/Platform Provider. The environment to choose from and a short recommendation for each type of user is given as follow:

    • Dev environment: For development, experimentation, trying out new versions and functionalities of components such as NiFi - so the Dev environment would be highly unstable and would be used mainly for major enhancements to the infrastructure/ecosystem enablers such as the Data Spine (DS), the Portal, etc.

    • Test environment: Replica of Prod environment when it comes to setup and configuration. This will be used mainly for upgrading the existing components in the ecosystem. E.g., if a new version of NiFi is available and we want to check if it is compatible with the other DS components, it’ll be deployed in Test environment, integration (and system) testing would be performed, and if all tests pass, the new version would be “promoted” to the Prod environment. - in the project, this environment would be used for implementing the pilot scenarios to ensure that the setup and configuration of infrastructure/ecosystem enablers is correct and the (eco)system is stable before we setup the Prod environment following the same deployment scripts, setup and configuration - so this environment will be much more stable as compared to the Dev environment.

    • Prod environment: This would be the most stable environment and would be used for Open Calls and thereafter.

A user who want to only create composite applications with the existing tools and services in the EFPF ecosystem, can directly use the Prod environment and won’t need to use the Test environment at all.

The users who are Tool/Service/Platform Providers would need to use both Test and Prod Environments. At first, they would need to integrate their Tool/Service/Platform with the Test environment EFPF ecosystem and define and execute integration test scenarios. And when the tests pass, their Tool/Service/Platform can be promoted to the Prod environment. Same procedure can be used for upgrades of these components.

Accessing the Service Registry

Once you have your account set up and you know the environment you will be working on, the next step will be to get familiar with the Service Registry. Depending on the type of user you are, you might want to register/update services in the Service Registry or you will just want to get information about existing services.

The Service Registry does not have a graphical interface, but it can be accessed using its REST API. The URL of the Service Registry is https://efpf-security-portal.salzburgresearch.at/apis/sr. The user guide of the Service Registry can be found here.

All users are automatically allowed to get information from the Service Registry, but if you need to register/update a service in the Service Registry, you will need to get admin rights for your account. You need to send an email to the Data Spine Technical Support Team asking for admin rights. In any case, you need to first get a token to access the Servie Registry.

A detailed guide to access the Service Registry can be found here.

The procedure is divided in two at this point, depending on what you need to do:

  • Register/Update a service: Follow these step to register or update a Service in the Service Registry.
  • Create a composite application using existing services: Follow these steps to create composite applications that use already existing services.

Register/Update a Service

  1. Create a JSON Description of your service: The JSON file should have all the definition of the service you want to register or update. The model of the service can be found here.

  2. Ask Admin Rights: As stated before, if you need to register/update a service, you need to send an email to the Data Spine Technical Support Team asking for admin rights.

  3. Get an access token via the Security Portal API: Following the documentation on accessing the Service Registry, you need to first get a token that you later will use to access the Service Registry.

  4. Register/Update your Service: The JSON file created before is POSTed to the Service Registry API using the token you got before.

  5. Check your new service: Use the provided API in the Service Registry to retrieve the registered services and check that the one you just registered/updated is there.

Create a Composite Application using Existing Services

  1. Get an access token via the Security Portal API: Following the documentation on accessing the Service Registry, you need to first get a token that you later will use to access the Service Registry.

  2. Retrieve the needed information from the Service Registry: Use the Service Registry API to search for the existing services you want to use and retrieve the needed data.

  3. Get permissions to access the NiFi Tool: Send an email to the Data Spine Technical Support Team asking permissions to access the NiFi tool.

  4. Access the NiFi tool: To develop your application that uses the already existing services, you will do it using the NiFi tool. You can access the NiFi tool here

  5. Get familiar with NiFi tool: NiFi is not a tool from EFPF, but an already existing tool from Apache. The official documentation will give you the information you need to start using it. The EFPF portal offers a user guide for NiFi which can also help you which can be found here.

  6. Develop your composite application using the existing services: This is the core part of the work to be done. This is where you have to implement the logic in NiFi following what you learnt in the step before. Inside the NiFi tool you can find some examples that can help you. In the General Process Group in NiFi you will find another Process Group called Examples. This Process Group contains some simple example applications.

In case you need to make some information from your application accessible from outside of the NiFi tool, you need to follow the next steps, depending on how you plan on doing this. You have two options:

  • Using Pub/Sub: You can make information from your NiFi application accessible from outside via the MQTT Broker.

    1. Request a username/password from the broker: Send an email to the Data Spine Technical Support Team

    2. Publish your data: In your NiFi application publish your data to the broker with the username and password provided in the previous step.

    3. Update the Service Registry: It’s recommended to register the new Pub/Sub API to the Service Registry. If the data being published is to be consumed by other companies, the registration is mandatory. Else, if the data being published is meant to be consumed by the same users who published it or by users in the same company, it’s not mandatory but it’s still recommended to register the API containing information such as the topic name, payload syntax, etc. to the SR as it would be useful in the long run.

  • Using your own endpoint: You can have your own endpoint in the EFPF system that can be accessed like a regular service. Users accessing the endpoint will get the data from your NiFi application.

    1. Register your Endpoint as a Service: Follow the steps for registering/updating a service and the API Security Gateway will create a proxy Enpoint.