Data Spine FAQs


Question 1: Do I need to register my services to the Service Registry (SR)? Why?

Answer:

Case 1: Asynchronous Pub/Sub communication example

Asynchronous Pub/Sub communication example

  • Registration of APIs with topics ‘p1/topic1’, and ‘s1-topic1’ is necessary

  • [MUST] p1/topic1: Needs to be discovered by the potential Service Consumers (SCs) and the SCs need to know the API spec of the service in order to create integration flows. Hence, this API needs to be registered to the SR.

  • [HIGHLY RECOMMENDED] s1/topic1: In this case, the Service Provider and the Service Consumer is the same user ‘subscriber1’ (i.e., subscriber1 created the integration flow that consumes ‘p1/topic1’, transforms data, and publishes the transformed data over topic ‘s1/topic1’. subscriber1’s service ‘sub1’ then subscribes to this topic ‘s1/topic1’). It’s still recommended to register this API to the SR as it would be useful in the long run. For example, if the integration flow needs to be updated in the future or if the project/task is handed over to another user. In such cases, if the Service/API isn’t registered, the details would then have to be “reverse engineered” from the code or from the data transformation mapping rules or by subscribing to the topic and examining the response payload for different inputs - which could be time consuming and error-prone. Moreover, if the integration flow transforms the data to conform to a standard data model (e.g., OGC-SensorThings), more service consumers that expect the data conforming to that data model could benefit from the service/API registration.

    Case 2: Synchronous request-response communication example

    Synchronous request-response communication example

  • Registration of ‘EP1’ and ‘EP1p-C’ is necessary

  • [MUST] EP1: Needs to be discovered by the potential Service Consumers (SCs) and the SCs need to know the API spec of the service in order to create integration flows. Hence, this API needs to be registered to the SR.

  • [MUST] EP1p-C: (1) The user consumer1 intends to consume the endpoint ‘EP1p-C’ from their service ‘CS1’, which is not accessible directly. A secure proxy endpoint ‘EP1p-Cp’ must be created in the API Security Gateway (ASG) for ‘EP1p-C’, which can then be consumed. In the case of synchronous request-response communication, whenever a secure proxy API needs to be created for a given API, the API needs to be registered to the SR. The ASG periodically scans the SR for new registrations and automatically creates secure proxy APIs for those. Therefore, ‘EP1p-C’ must be registered in the SR. (2) Regarding creating and registering the API spec (OpenAPI/Swagger spec) for ‘EP1p-C’ to the SR: In this case, the Service Provider and the Service Consumer is the same user ‘consumer1’ (i.e., the user ‘consumer1’ created the integration flow that listens for requests over ‘EP1p-C’, upon receiving a request, transforms the request payload if required, calls ‘EP1p’, transforms the response payload and returns the response. It is consumer1’s own service ‘CS1’ that intends to invoke ‘EP1p-C’). It’s still recommended to register the API spec for ‘EP1p-C’ to the SR as it would be useful in the long run. For example, if the integration flow needs to be updated in the future or if the project/task is handed over to another user. In such cases, if the Service/API isn’t registered, the details would then have to be “reverse engineered” from the code or from the data transformation mapping rules or by calling the endpoint and examining the response payload for different inputs - which could be time consuming and error-prone. Moreover, if the integration flow transforms the data to conform to a standard data model (e.g., OGC-SensorThings), more service consumers that expect the data conforming to that data model could benefit from the service/API registration.


Question 2: Do I need to use the API Security Gateway (ASG) together with the EFPF Security Portal (EFS) to secure the API endpoints of my service? OR How do I enable SSO for my tool’s GUI?

Answer:

  • This depends upon whether your service’s API is already secured or not.
  • If your service is a part of a 3rd party platform and its API is already secured with that platform’s Identity and Access Management (IAM) service, it would be possible to access the service’s API using EFPF user credentials once the platform’s IAM service is federated with the EFS. Therefore, you do not need to use the ASG and EFS explicitly to secure your service’s API.
  • If you are offering your service as a part of the EFPF platform and its API is not already secured, you must secure access to it using the EFS, and also ASG, if applicable.
  • If your service offers a GUI that is embedded in the EFPF Portal, the Portal’s SSO (authentication) will naturally apply to your service’s GUI as well. But you might need to define the authorization policies for your service separately.
  • If your service’s GUI isn’t embedded into the EFPF Portal, you’ll need to add functionality to your service to redirect to the EFS for login (similar to what the EFPF Portal does). This will also require creating a new client in the EFS Keycloak for your service and using OAuth2.0 Authorization Code Grant Flow to request an Access Token from the EFS Keycloak. You’ll also need to ensure that your service’s API is access protected.
  • If your service offers only an HTTP API and no GUI, the access to the HTTP API can be secured mainly in two different ways: (1) using a local instance of Apache APISIX (managed by you), that connects to the EFS for authentication and authorization decisions. You would need to create secure proxy endpoints/routes for your service’s API endpoints in that local instance of APISIX and use either the openid-connect or the authz-keycloak plugin of APISIX to delegate the authentication and (optionally) authorization decisions to the EFS Keycloak. You will also need to use the proxy-rewrite plugin of APISIX to configure the proxy routes. Or, (2) using a Policy Enforcement Point embedded into your service that communicates with the EFS for authentication and (optionally) authorization decisions
  • If you’ve configured your integration flow in NiFi to expose an HTTP endpoint, you must use ASG/APISIX (cloud-native instance managed by the EFPF Team) and EFS to secure it in order to make it accessible over the Internet. This can be achieved by registering this API endpoint to the Service Registry. (Details: DS NiFi User Guide)


Question 3: As a user of the Data Spine, I want to consume the existing services in the EFPF ecosystem to create composite applications by using NiFi integration flows (workflows/dataflows). Can I use the Data Spine NiFi’s GUI to create the integration flows myself or will the EFPF Support Team create the integration flows for me?

Answer:

  • You would need to send an email to the EFPF Support Team asking for a collaboration space called ‘Process Group (PG)’ on Data Spine NiFi as described in the DS NiFi User Guide.
  • After a new PG is given to you, you can create integration flows using the drag-and-drop style GUI of NiFi to realise your composite application. You can also collaborate with others by giving them access to your PG.


Question 4: Can I install the Data Spine locally?

Answer:

  • The Data Spine is designed as a cloud-native, central component in the EFPF ecosystem to enable collaboration and (authorized) sharing of data and services among its users.
  • Therefore, it is highly recommended to use this cloud-native version of the Data Spine managed by EFPF Administrators.


Data Spine Documentation

User Guide 101

Previous
Next