Data Spine User Guide

High-level Dataflow in the EFPF Ecosystem

High-level Dataflow through Data Spine Figure 1: High-level Dataflow through Data Spine

Service Integration through Data Spine

This section enlists the activities that the service providers need to do in order to provide their services through the Data Spine and the activities the service consumers need to do in order to consume the services provided through the Data Spine with the help of examples. These design-time aspects become the prerequisites to enabling communication through the Data Spine. This section is further divided into ‘Synchronous Communication’ and ‘Asynchronous Communication’ as the activities differ for both.

For details on how each activity can be performed, refer to the User Guides of the concerned components of Data Spine:

  1. Service Registry / SR (Linksmart Service Catalog)
  2. API Security Gateway / ASG (Apache APISIX)
  3. EFS (Keycloak and Policy Enforcement Service)
  4. Integration Flow Engine (Apache NiFi)
  5. Message Bus (RabbitMQ)

Synchronous Communication:

Synchronous Services' Integration through Data Spine Figure 2: Synchronous Services' Integration through Data Spine

Figure 2 shows how provider1’s service PS1 and consumer1’s service CS1 interact with the Data Spine components to provide and consume services respectively. The step-by-step instructions for service provision and consumption through the Data Spine are given in the subsequent sections.

Prerequisites:

  • The service provider ‘provider1’ and service consumer ‘consumer1’ are both EFPF users.
  • provider1 and consumer1 have the necessary access permissions required to register services to the Service Registry.

Design-time activities:

  1. Service Registration: provider1 registers his/her service ‘PS1’ to the Data Spine Service Registry with an appropriate service ‘type’ (e.g., ‘platform1.marketplace-service’). For simplicity, let us assume that PS1’s REST API’s endpoint is ‘EP1’. (Details: Follow ASG Guide to get an access token and SR User Guide to register the service)
  2. Creation of Secure Proxy API: The API Security Gateway (ASG), which checks for new service registrations/updates to services in the Service Registry periodically (currently, every 1 hour), creates a secure proxy API endpoint/route ‘EP1P’ for EP1 in ASG.
  3. Access Configuration: An EFPF administrator user ‘admin1’ defines/configures the access permissions for accessing EP1P in the EFS.
  4. Service Metadata Retrieval: consumer1 decides to consume PS1 and gets the technical metadata for PS1 including its API spec from the Service Registry.
  5. Access Configuration: consumer1 requests for and acquires the necessary access permissions to invoke EP1P.
  6. Access Configuration: consumer1 requests for and acquires the necessary access permissions to create integration flows in the Integration Flow Engine (NiFi) component of Data Spine. (Details: DS NiFi User Guide)
  7. Integration Flow Creation: consumer1 creates an integration flow using the GUI of the Integration Flow Engine to invoke EP1’s secure proxy endpoint EP1P, perform data transformation and finally create an ‘interoperability-proxy’ endpoint EP1P-C for EP1P in the integration flow.
  8. Service/API Registration: consumer1 registers this new EP1P-C API endpoint to the Service Registry.
  9. Creation of Secure Proxy API: ASG creates a secure proxy API endpoint EP1P-CP for EP1P-C.
  10. Access Configuration: consumer1 requests for and acquires the necessary access permissions to invoke EP1P-CP.
  11. Integration Complete: provider1’s service PS1 and consumer1’s service CS1 are now integrated through the Data Spine and can start communicating with each other.

Asynchronous Communication:

Asynchronous Services' Integration through Data Spine Figure 3: Asynchronous Services' Integration through Data Spine

Figure 3 shows how publisher1’s service pub1 and subscriber1’s service sub1 interact with the Data Spine components to provide and consume services respectively. The step-by-step instructions for service provision and consumption through the Data Spine are given in the subsequent sections.

Prerequisites:

  • The publisher ‘publisher1’ and subscriber ‘subscriber1’ are both EFPF users.
  • publisher1 and subscriber1 have the necessary access permissions required to register services to the Service Registry.

Design-time activities:

  1. Access Configuration and MB User Account: publisher1 requests for and acquires the necessary access permissions to publish to the Data Spine Message Bus (RabbitMQ) over topic ‘p1/topic1’. A new Message Bus (MB) user account would be created for publisher1, if it doesn’t exist already, requested access permissions would be granted and publisher1 would be given MB credentials to perform the requested operations. In case if publisher1 already has an MB user account, the requested access permissions would be configured for this account and publisher1 would be notified. (Details: DS RabbitMQ User Guide)
  2. Publisher Configuration: publisher1 configures his/her service ‘pub1’ to publish to the Data Spine Message Bus over the topic ‘p1/topic1’ using the received MB credentials.
  3. Service Registration: publisher1 registers pub1 that consists of this Pub/Sub API containing its publication information to the Service Registry. (Details: Follow ASG Guide to get an access token and SR User Guide to register the service)
  4. Service Metadata Retrieval: subscriber1 decides to subscribe to pub1’s topic ‘p1/topic1’ and gets the technical metadata for pub1 including its API spec from the Service Registry.
  5. Access Configuration (and MB User Account): subscriber1 requests for and acquires the necessary access permissions to subscribe to p1/topic and to publish back to the Message Bus over the topic ‘s1/topic1’ (similar to step 1)
  6. Access Configuration: subscriber1 requests for and acquires the necessary access permissions to create integration flows in the Integration Flow Engine (NiFi) component of Data Spine. (Details: DS NiFi User Guide)
  7. Integration Flow Creation: subscriber1 creates an integration flow using the GUI of the Integration Flow Engine to subscribe to p1/topic, perform data transformation and finally to publish the resulting data to the Message Bus over the topic ‘s1/topic1’ using his/her MB credentials.
  8. Service Registration: subscriber1 registers his/her service with the APIs containing its subscription and publication information to the Service Registry.
  9. Integration Complete: publisher1’s service and consumer1’s service are now integrated through the Data Spine.

Dataflow through Data Spine

This section describes the service-to-service communication that happens at run-time through the Data Spine in the form of workflows. In order to enable such a communication, the participant services need to be integrated through the Data Spine first by following the design-time activities enlisted in the section above. Therefore, in a way, the workflows described in this section are a continuation of the activities mentioned in the previous section. This section is further divided into ‘Synchronous Communication Workflow’ and ‘Asynchronous Communication Workflow’ as the workflows differ for both.

Synchronous Communication Workflow (Figure 4):

  1. consumer1’s service CS1 invokes EP1P-CP endpoint/route of ASG.
  2. ASG, acting as a reverse proxy, checks with EFS to ensure that CS1 has the necessary permissions to access EP1P-CP and to perform the requested operation.
  3. ASG, upon receiving a positive reply from EFS, invokes EP1P-C exposed by the integration flow.
  4. Integration Flow Engine does the necessary request (payload, query parameter, path parameter and/or header) transformation as defined by the integration flow and finally, invokes the endpoint EP1P - the secure proxy API endpoint of PS1’s API endpoint EP1.
  5. Upon receiving the response, the Integration Flow Engine performs payload data transformation as defined by the integration flow and finally, sends the resulting payload as response to ASG.
  6. ASG sends the received response to consumer1’s service CS1.

Example of Synchronous Communication Workflow Figure 4: Example of Synchronous Communication Workflow

Asynchronous Communication Workflow (Figure 5):

  1. publisher1’s service pub1 publishes data to the Data Spine Message Bus over topic p1/topic1 using their MB credentials.
  2. The integration flow created by subscriber1 subscribes to this topic p1/topic1 using their MB credentials, transforms the payload data from publisher1’s data model pDM to its own data model sDM and finally, publishes this transformed payload data to the Message Bus over topic s1/topic1.
  3. subscriber1’s service sub1 subscribes to the topic s1/topic offered through the Message Bus and starts receiving the data.

Example of Asynchronous Communication Workflow Figure 5: Example of Asynchronous Communication Workflow

Sync/Async Service Integration Example (Figure 6):

Example of Sync/Async Service Integration Figure 6: Example of Sync/Async Service Integration

Single Sign-On (SSO) Setup

Refer to the following tutorials to setup EFPF/EFS SSO in the Identity Providers of base/external platforms:

  1. EFS SSO Guide With Keycloak
  2. EFS SSO Guide With Non Keycloak IDPs
Previous