Data Spine Developer Guide

Data Spine Architecture and Components’ Interaction

Data Spine Architecture Figure 1: Architecture of the Data Spine

Figure 1 shows the conceptual components of the Data Spine. The relationships and interactions between these core components are also illustrated. The access to the GUI of the Integration Flow Engine and its elements is protected by the ‘Keycloak’ component of the EFS. The API Security Gateway acts as the Policy Enforcement Point for the Data Spine. API Security Gateway relies on the Policy Enforcement Service of the EFS to make the access control related decisions. The API Security Gateway is configured to check the Service Registry for new service registrations and service updates periodically to automatically create proxy endpoints or routes for protecting access to them in the API Security Gateway. The access to the REST API of the Service Registry is secured through proxy endpoints in API Security Gateway. The Service Registry publishes service status announcement related messages to the Message Bus. The design-time administration and management related endpoints of the Integration Flow Engine and the Message Bus are secured through the Identity and Access Management services internal to these respective components. The run-time access to the endpoints exposed by the Integration Flows in the Integration Flow Engine is protected through the corresponding proxy endpoints in the API Security Gateway, once they are registered to the Service Registry.

Moreover, Figure 1 also shows the run-time view of the communication between two services S1 and S2 happening through the Data Spine. As design-time prerequisites, service S1 is registered in the Service Registry, the Integration Flow to consume S1 and perform data transformation has already been created and activated and, finally, service S2 has acquired access rights for invoking S1 through the Data Spine. The operation at run-time:

  1. S2 makes a call to the proxy endpoint EP1-c in the API Security Gateway with an EFPF token.
  2. The API Security Gateway delegates the authentication and authorization responsibility to the EFS.
  3. The EFS verifies whether the token is valid and has necessary authorization to invoke the EP1-c endpoint and perform the specified operation and finally, replies to the API Security Gateway.
  4. Upon receiving a positive reply from the EFS, the API Security Gateway invokes the corresponding endpoint EP1-b exposed by the respective Integration Flow in the Integration Flow Engine.
  5. The Integration Flow transforms the request, if specified and invokes the original EP1-a endpoint of S1.
  6. Upon receiving response from S1, the Integration Flow transforms the response payload, if specified and returns the response to the API Security Gateway.
  7. The API Security Gateway returns the response to S2. In this way, the components of the Data Spine work together to enable integration of and communication between the services of different platforms.

Data Spine Components' Interaction Figure 2: Data Spine Components' Interaction

Figure 2 illustrates the technological platforms, tools and services selected to realise different components of the Data Spine and the interactions between them. The interactions are similar to the ones between their respective conceptual components illustrated in Figure 1. The figure also shows Service Provider providing service S1 through the Data Spine and Service Consumer’s service S2 consuming S1 through the Data Spine with the help of ‘Integration Flow 1’. Finally, Figure 2 summarises the technologies selected to realise the conceptual components of the Data Spine. In this way, the Data Spine provides the necessary integration infrastructure to bridge the interoperability gaps between heterogeneous services and enables communication in the EFPF ecosystem.

Source Code of Components

Conceptual Component Technology
Integration Flow Engine Apache NiFi
Service Registry LinkSmart Service Catalog
Message Bus RabbitMQ
EFS Keycloak & Policy Enforcement Service

Maintenance, Enhancements and Extension

Follow the documentation of individual components to maintain, enhance or extend them:

Tool/Service Developer Documentation
Apache NiFi NiFi Developer’s Guide
APISIX APISIX Contributor Guide
LinkSmart Service Catalog LinkSmart SC: Contributing
RabbitMQ RabbitMQ: Contributing
Keycloak Keycloak: Contributing

Data Spine Documentation