Data Spine User Guide

High-level Dataflow in the EFPF Ecosystem

High-level Dataflow through Data Spine Figure 1: High-level Dataflow through Data Spine

Service Integration and Dataflow through Data Spine

  • This section enlists the activities that the service providers need to do in order to provide their services through the Data Spine and the activities the service consumers need to do in order to consume the services provided through the Data Spine, with the help of examples. These design-time aspects become the prerequisites to enabling communication through the Data Spine.
  • This section further describes the service-to-service communication that happens at run-time through the Data Spine in the form of dataflows, after they are integrated through the Data Spine.
  • The activities and the dataflows depend upon the communication pattern used. This section presents the activities and the dataflows for both ‘Synchronous (request-response) Communication’ and ‘Asynchronous (Pub/Sub) Communication’.

For more details on how each activity can be performed, refer to the User Guides of the concerned components of Data Spine:

  1. Service Registry / SR (Linksmart Service Catalog)
  2. API Security Gateway / ASG (Apache APISIX)
  3. EFS (Keycloak and Policy Enforcement Service)
  4. Integration Flow Engine (Apache NiFi)
  5. Message Bus (RabbitMQ)

Synchronous (request-response) Communication

Synchronous Services' Integration through Data Spine Figure 2: Synchronous Services' Integration through Data Spine

Figure 2 shows how provider1’s service PS1 and consumer1’s service CS1 interact with the Data Spine components to provide and consume services respectively. The step-by-step instructions for service provision and consumption through the Data Spine are given in the subsequent sections.

Prerequisites:

  • The service provider ‘provider1’ and service consumer ‘consumer1’ are both EFPF users. (To get an EFPF user account, see the User Guide 101)
  • provider1 and consumer1 have the necessary access permissions required to register services to the Service Registry. (Details: SR User Guide)

Design-time activities:

  1. Service Registration: provider1 registers his/her service ‘PS1’ to the Data Spine Service Registry with an appropriate service ‘type’ (e.g., ‘platform1.marketplace-service’). For simplicity, let us assume that PS1’s REST API’s endpoint is ‘EP1’. (Details: Follow ASG Guide to get an access token and SR User Guide to register the service)
  2. Creation of Secure Proxy API: The API Security Gateway (ASG), which checks for new service registrations/updates to services in the Service Registry periodically (currently, every 1 hour), creates a secure proxy API endpoint/route ‘EP1P’ for EP1 in ASG.
  3. Access Configuration: An EFPF administrator user ‘admin1’ defines/configures the access permissions for accessing EP1P in the EFS.
  4. Service Metadata Retrieval: consumer1 decides to consume PS1 and gets the technical metadata for PS1 including its API spec from the Service Registry.
  5. Access Configuration: consumer1 requests for and acquires the necessary access permissions to invoke EP1P.
  6. Access Configuration: consumer1 requests for and acquires the necessary access permissions to create integration flows in the Integration Flow Engine (NiFi) component of Data Spine. (Details: DS NiFi User Guide)
  7. Integration Flow Creation: consumer1 creates an integration flow using the GUI of the Integration Flow Engine to invoke EP1’s secure proxy endpoint EP1P, perform data transformation and finally create an ‘interoperability-proxy’ endpoint EP1P-C for EP1P in the integration flow.
  8. Service/API Registration: consumer1 registers this new EP1P-C API endpoint to the Service Registry.
  9. Creation of Secure Proxy API: ASG creates a secure proxy API endpoint EP1P-CP for EP1P-C.
  10. Access Configuration: consumer1 requests for and acquires the necessary access permissions to invoke EP1P-CP.
  11. Integration Complete: provider1’s service PS1 and consumer1’s service CS1 are now integrated through the Data Spine and can start communicating with each other.

Dataflow

  1. consumer1’s service CS1 invokes EP1P-CP endpoint/route of ASG.
  2. ASG, acting as a reverse proxy, checks with EFS to ensure that CS1 has the necessary permissions to access EP1P-CP and to perform the requested operation.
  3. ASG, upon receiving a positive reply from EFS, invokes EP1P-C exposed by the integration flow.
  4. Integration Flow Engine does the necessary request (payload, query parameter, path parameter and/or header) transformation as defined by the integration flow and finally, invokes the endpoint EP1P - the secure proxy API endpoint of PS1’s API endpoint EP1.
  5. Upon receiving the response, the Integration Flow Engine performs payload data transformation as defined by the integration flow and finally, sends the resulting payload as response to ASG.
  6. ASG sends the received response to consumer1’s service CS1.

Example of Synchronous Communication Workflow Figure 3: Example of Synchronous Communication Workflow

Asynchronous (Pub/Sub) Communication

Asynchronous Services' Integration through Data Spine Figure 4: Asynchronous Services' Integration through Data Spine

Figure 4 shows how publisher1’s service pub1 and subscriber1’s service sub1 interact with the Data Spine components to provide and consume services respectively. The step-by-step instructions for service provision and consumption through the Data Spine are given in the subsequent sections.

Prerequisites:

  • The publisher ‘publisher1’ and subscriber ‘subscriber1’ are both EFPF users.
  • publisher1 and subscriber1 have the necessary access permissions required to register services to the Service Registry.

Design-time activities:

  1. Access Configuration and MB User Account: publisher1 requests for and acquires the necessary access permissions to publish to the Data Spine Message Bus (RabbitMQ) over topic ‘p1/topic1’. A new Message Bus (MB) user account would be created for publisher1, if it doesn’t exist already, requested access permissions would be granted and publisher1 would be given MB credentials to perform the requested operations. In case if publisher1 already has an MB user account, the requested access permissions would be configured for this account and publisher1 would be notified. (Details: DS RabbitMQ User Guide)
  2. Publisher Configuration: publisher1 configures his/her service ‘pub1’ to publish to the Data Spine Message Bus over the topic ‘p1/topic1’ using the received MB credentials.
  3. Service Registration: publisher1 registers pub1 that consists of this Pub/Sub API containing its publication information to the Service Registry. (Details: Follow ASG Guide to get an access token and SR User Guide to register the service)
  4. Service Metadata Retrieval: subscriber1 decides to subscribe to pub1’s topic ‘p1/topic1’ and gets the technical metadata for pub1 including its API spec from the Service Registry.
  5. Access Configuration (and MB User Account): subscriber1 requests for and acquires the necessary access permissions to subscribe to p1/topic and to publish back to the Message Bus over the topic ‘s1/topic1’ (similar to step 1)
  6. Access Configuration: subscriber1 requests for and acquires the necessary access permissions to create integration flows in the Integration Flow Engine (NiFi) component of Data Spine. (Details: DS NiFi User Guide)
  7. Integration Flow Creation: subscriber1 creates an integration flow using the GUI of the Integration Flow Engine to subscribe to p1/topic, perform data transformation and finally to publish the resulting data to the Message Bus over the topic ‘s1/topic1’ using his/her MB credentials.
  8. Service Registration: subscriber1 registers his/her service with the APIs containing its subscription and publication information to the Service Registry.
  9. Integration Complete: publisher1’s service and consumer1’s service are now integrated through the Data Spine.

Dataflow

  1. publisher1’s service pub1 publishes data to the Data Spine Message Bus over topic p1/topic1 using their MB credentials.
  2. The integration flow created by subscriber1 subscribes to this topic p1/topic1 using their MB credentials, transforms the payload data from publisher1’s data model pDM to its own data model sDM and finally, publishes this transformed payload data to the Message Bus over topic s1/topic1.
  3. subscriber1’s service sub1 subscribes to the topic s1/topic offered through the Message Bus and starts receiving the data.

Example of Asynchronous Communication Workflow Figure 5: Example of Asynchronous Communication Workflow

Abstract Use Case Examples

Realisation of an Integrated Marketplace application

Figure 6 illustrates an example of dataflow at run-time between synchronous services in the EFPF Ecosystem. It shows the realisation of the EFPF Integrated Marketplace that fetches a list of products and services from the marketplace services of the base platforms and displays them coherently onto a GUI. The Marketplace GUI initiates the call to its backend. The Marketplace Backend searches for services of type ‘marketplace’ in the Service Registry through a proxy endpoint in API Security Gateway and discovers the marketplace services of COMPOSITION and NIMBLE platforms. It then invokes these services through the Integration Flows created in the Integration Flow Engine (DS NiFi) of the Data Spine and gets a response. The data models of responses from these marketplace services are transformed to the EFPF Integrated Marketplace’s data model through the Integration Flows. Finally, the Marketplace backend aggregates the responses, hands over the data to the GUI component in order to display it to the users.

Integrated Marketplace application dataflow Figure 6: Integrated Marketplace application dataflow

Realisation of an Analytics/Alerting application

Figure 7 illustrates an example of dataflow at run-time between asynchronous services that follow the Pub/SUb communication pattern. This example presents a scenario where the Risk Tool receives data from the Factory Connectors through the Data Spine, analyses it, calculates risk, and the Event Reactor based on this risk generates an alert. In this example, the Factory Connectors/Gateways ‘IW Collect’ installed at ‘Factory 1’ and ‘HAL’ installed at ‘Factory 2’ publish data to the Data Spine Message Bus (DS RabbitMQ). Through the pre-configured Integration Flows, this data is transformed to align with the Risk Tool’s data model and is published again to the Message Bus over new topics. The Risk Tool subscribes to these new topics and gets the data. It then analyses the data and identifies the risk associated, if any. The data could be the status of production with a delivery date, for example, and delayed delivery date could be a risk that the Risk Tool identifies. Once the Risk Tool identifies a risk, it publishes the risk data to the Message Bus. Again, through another pre-configured Integration Flow, this data is transformed to align with the Event Reactor’s data model and is published again to the Message Bus over a new topic. The Event Reactor subscribes to this new topic, gets the risk data and generates an alert.

Analytics/Alerting application dataflow Figure 7: Analytics/Alerting application dataflow

Sync/Async Service Integration Example

Example of Sync/Async Service Integration Figure 8: Example of Sync/Async Service Integration

Single Sign-On (SSO) Setup for platforms

Refer to the following tutorials to setup EFPF/EFS SSO in the Identity Providers of base/external platforms:

  1. EFS SSO Guide With Keycloak
  2. EFS SSO Guide With Non Keycloak IDPs

Data Spine Documentation

Previous
Next