ROAM User Guide

Introduction

The ROAM (Risk, Opportunity, Analysis & Monitoring) Tool allows users to automatically use data streams (via MQTT) and transform them into an output; although the tool is capable of general transformations, we focus on risks and opportunities, and assume that the output will be a risk, opportunity, or something related like a notification. The tool is also capable of using REST API to receive data from external sources. Once the EFPF Secure Data Store is available, the tool will also provide direct support for this. When using the ROAM Tool, the user defines risk workflows. These workflows require a source; the MQTT topic on which to receive data. Then, the user has to choose the so-called risk recipes that transform this input into the desired output. The tool comes with built-in configurable recipes. These configurable recipes allow for the user to specify some parameters that tailor the recipes to the users specific situations. The tool supports workflows with multiple input or output topics. Whenever there are simultaneously multiple input and output topics, they have to have the same amounts, and will correspond to one another based on their order.

The ROAM Tool comes with a configuration user interface (UI) and a REST API. The configuration UI enables the user to create and edit recipes and workflows with a visual interface. The API has the same functionality, but requires the users to make an API call and supply a json object.

Configuration UI

This UI contains a panel with a list of all available recipes; the ones which are configurable, and the ones that have been configured. Furthermore, there’s also a panel with a list containing all defined workflows, which is empty by default. Each panel contains a list with searching functionality, sorting functionality, and filtering functionality. The recipes and workflows may be sorted by their name and id, while they can only be filtered by their tags. The user can define the tags of a workflow and recipe themselves. Moreover, the lists contain buttons that allow the user to create, edit, copy and remove recipes. For workflows, the same functionality exists, and on top of that the user may also start listening to the incoming data (or stop). The user can also view the last output contents. For workflows that support having a cache, the user can also reset their cache. This will be explained in further detail later on. The user can only see recipes and workflows they have access to. If they have shared access, and are not the owner or not an admin, the user cannot edit or delete. The user can however perform all other actions. Each recipe and workflow has a detail row that can be expanded. Along with all the details, this row also allows the user to test the recipes and workflows. This testing functionality is also available while creating/editing a recipe/workflow. This allows the user to make sure that a certain input invokes the expected output.

To show the configuration UI in practice, we will run you through the UI with screenshots, highlighting the different options and functionalities along the way:

Main frame

Upon starting the tool, ang loggin in with their EFPF login, the user will always be met with the following screen:

The main frame of the ROAM tool

Keep in mind that the app for this tool is a single page app and that no progress will be saved and it will always behave the same if the page is restarted.

On the top right, there are three buttons. The right most button logs the user out and redirects back to the login screen. The middle button refreshes the contents of the app, if that is for any reason necessary (such as creating a recipe through REST API, outside of the app; this is not reflected without a reload). This does not reset the collapsed state of the recipe and workflow panels, but reloads the contained recipes and workflows. The left most button redirects the user to this user guide. This main screen contains two expandable lists, one for recipes, and one for workflows.

Recipes

Let us first take a look at recipes. Upon expanding the recipe list, the user is met with something like this:

A typical recipe list

This list contains two kinds of recipes; the first kind we call base recipes, which either come with the ROAM tool. You can consider these as templates, which contain default parameter values and a generally elaborate description of what the recipe can do. The second kind is customized recipes, in which the user has already filled in the parameters, and tailored it to their own needs.

Base recipes cannot be edited or deleted, and they have no id, but they can be copy-and-edited. They can have tags, although they do not as of writing this guide. Editing, copy-and-editing, and deletion can be done by clicking on the icon buttons to the left.

Users can have access to recipes in three ways. They have created it themselves, another user has shared it with them, or they are a company admin and can see all their company’s recipes (and workflows). If a user has the “roam_admin” role for the roam keycloak client, the ROAM Tool considers them to be an admin of the users company. Users can always edit, copy-and-edit, and delete their own recipes, but they can only copy-and-edit a recipe that has been shared with them.

In the list, the user can filter by tag, or the user can search for (part of) a name or an id. The user is also able to sort the list by recipe name, id, and creation time (UTC). This can be done by clicking on the arrow next to “Name”, “Id”, or “Creation Time”. The arrow appears when hovering over the column label. Note: when searching, the user needs to press enter to finalize their search, and the filter functionality accepts multiple words, which also each need enter to be pressed.

Each recipe also has an expandable detail row, which lists the recipe description, parameters, and metadata. The metadata contains the recipe owner, the recipe owner’s company, who has shared access, and when it was created and last modified. A typical detail row looks as follows:

A typical recipe detail row

In this row, the user can also test the recipe. The button for that is placed below the “input” row in the image, but due to the formatting of the table, it is not visible in this screenshot.

When designing recipes, there are three options.

  1. You make one from scratch, by selecting a desired base recipe.
  2. You edit an existing customized recipe.
  3. You copy-and-edit an existing customized or base recipe. In the case of a base recipe, this is the same as creation, but the base recipe is already set.

The first can be accessed by the “+"-button in the top left of the list, the other two can be accessed by the aforementioned buttons.

Recipe Creation & Editing

The recipe creation window looks like this:

A typical recipe creation window

It consists of three parts. The first is the recipe creation method: the user may define it manually, or the user may upload a file with possibly multiple recipes. The second is completely visible here and the recipe definition is performed in this part. This one only shows up when defining the recipe manually. The third is the recipe testing part, which we will get to after introducing workflows.

When the recipe type has been chosen, a previously greyed out “Submit” button is enabled in the bottom right of the window, allowing the user to finalize their recipe. It may happen that the user is met with an error window when submitting a recipe or workflow. This then means that some field is not properly filled in, and an appropriate error message will point you in the right direction.

Furthermore, the two buttons in the top right (the save and trash bin icons) allow the user to exit the window (pressing outside the window, or pressing Esc will also do). However, only when pressing the save icon button, the progress is saved. This is only valid for the current configuration session, though. When restarting the tool, every kind of progress is lost, except for submitted recipes/workflows.

Of course, the user can choose a recipe name and write a recipe description. The user is also able to define tags for filtering purposes. Regarding the recipe metadata, the user can only set users to share the recipe with, but other metadata is not changeable.

Depending on the recipe type, the additional parameter fields differ. After choosing a recipe type, a button appears that spawns a window with recipe details. Moreover, all parameter fields have their own information icon, that, when hovered over, explain the use of that particular parameter. For example, when choosing the simple MeanRecipe, that averages values:

Recipe Creation for MeanRecipe A new button shows up that allows the user to see the recipe details. The user can then see what each parameter exactly does. Additionally, four new fields show up: “axis type”, “field”, “output” and “remove”.

Whenever a field is empty, and only shows its label (like for “field” here), its default is an empty string (”"). It is generally a good idea to change the defaults, but especially when they are empty.

In this case, the “axis type” field is set to None. Whenever it is set to “Number”, an additional “axis” field appears.

To illustrate all different kinds of supported fields, consider the situation where we select “AddDataRecipe”. For context, this recipe allows the user to insert their own data into the workflow in a json format, by means of an object form, that contains other types of forms inside:

Different field types in AddDataRecipe

Each field has appropriately been named to make it more obvious. This should show how to properly input all supported data types: arrays (or lists, if you will), strings, numbers (floats/integers), booleans, nested objects, and it also supports None types. Strings and numbers are straightforward. When inputting a boolean, the user has to choose True or False from a dropdown box. In an array, all values are separated by a “,”. All spacebars before and after the comma are removed when parsing it. In an object, the user can add key-value pairs, and choose the data type of the value by means of a button. Lastly, the user can also input a None (null) type, which obviously does not allow for additional input.

For the array form, the user may click on the icon to the right, to toggle between string and number arrays. Green means text/string, and red means number. For now, there is no way to create hybrid arrays, or arrays containing anything else than strings or numbers. If that is necessary, the user is encouraged to upload their recipe or workflow, either with a file through the frontend, or through the REST API.

When inputting text in number mode, the following happens:

Wrong data in an array

The form is underlined in red, and it shows “The input is invalid.” This can also occur when typing a string in the number form. Whenever this happens, the last valid input is saved. Let’s say the user typed “1, 2, 3” and changed it to “1, 2, 3a”, for sake of example, then it will be internally saved as an array reading [1, 2, 3].

When opting to upload the recipe by means of a file, the form looks as follows:

The form when uploading a file

The “Upload files” text functions as a button that allows the user to upload a file.

Workflows

When expanding the workflow window, the user is met with something similar to the following:

A typical workflow list

The workflow list functions much the same as the recipe list, with only a couple of minor differences. For starters, the user will encounter this list empty, upon first starting the tool, as we do not provide any default workflows. Of course, workflows are very much tailored to a specific user, so providing them does not make sense.

For demo purposes, we already made two demo workflows. As you can see, the creation, editing, copy-and-editing, deletion, sorting, filtering, and searching functionalities remain entirely the same. However, there are three new buttons; the user may “start” (and stop) their workflow, by clicking on the green play icon (resp. red stop icon that will take its place when it is already running). There is also a “Last workflow data” button. This is the last output this workflow has provided. It mostly serves for reference purposes, and the user may download it as well:

An empty last workflow data window

Of course, this will be populated once the workflow has been run, and any MQTT data was received at least once. This last output is downloadable with the button at the bottom.

Some workflows have caching capabilities. This allows the user to make use of the previous output of the recipes in a workflow. This can be useful if some historic data always needs to be present, or if one needs to consider multiple recent data points. Whenever the workflow has caching capabilities, there is a third button allowing the user to reset the cache of all recipes in the workflow. If the user, for some reason, wants to (re)initialize the workflow cache manually, that needs to be done via the REST API.

Additionally, workflows also have a slightly different detail row, containing a description, subscription details, and recipe list:

A typical workflow details

The subscription details can be collapsed to show information about how many times the workflow has received data, when it was started, when it was changed, any errors that popped up, and other subscription data. The recipes can also be expanded to show each recipe in the workflow. Each recipe can again be expanded to show its details, showing information similar to the recipe detail row mentioned before. Like in the recipe details, the metadata can be collapsed to show the owner of the workflow, the owner’s company, who has shared access, and when it was created and last modified, but also the last output time.

Typical subscription details Typical recipe and metadata details

Workflow Creation & Editing

Again, overall this window is similar to the workflow creation/edit window.

The general part of the workflow creation window The recipe part of the workflow creation window

The user may again input a name, description, tags, and user access. This time, however, the user will also be able to define MQTT functionality parameters, such as input, output, and error topics. This error topic contains {id} by default, which places the workflow id into the error topic. Input and output topics can be a list, in which case the corresponding topic type needs to be set to Multiple, instead of Single. In the case of multiple topics, the workflow will listen to each input topic, or publish to each output topic. If both are lists, they need to have the same length, and each input topic will correspond to an output topic. Lastly, the user can define a vhost on which to listen and subscribe.

The user should have access to the defined topics and vhost.

When dealing with workflows, there are two new boxes; the recipe order box, and the recipe configuration box. Additionally, the user may again upload a file, and test the workflow, which will be discussed in the next section.

In the recipe order box, the user may select the recipe under consideration (which is then highlighted, like the one in the image above), and add a recipe, that will default to “Choose a Recipe”, by clicking on the small “+"-button below the arrows. The user may highlight a new recipe in the list by clicking on its name, or by clicking on “Choose a Recipe” when none has been selected yet. When a new recipe is added, the focus will automatically be shifted to the new “Choose a Recipe” box.

The recipe configuration box allows user to remove the focused recipe, and to choose between base recipes and customized recipes, by toggling the radio buttons and selecting the desired recipe from the dropdown box.

The recipe configuration box with a selected recipe

When selecting a recipe, two buttons show up: an info button, showing the recipe details, and an edit button, allowing the user to edit recipes while creating a workflow. These recipe edits will not be saved permanently, although the user has the option to permanently save the recipe as well.

The recipe configuration box with toggled buttons

When clicking the edit button, the user may change the individual parameters, just like explained before, when creating or editing a recipe. This also spawns a save button. When clicking this save button, the user may also change the name, description, and tags.

When a recipe is left at “Choose a Recipe”, it will be removed from the workflow just before submission.

Testing

Let us finally move on to testing. We will first discuss workflow testing, as recipe testing is a special case of workflow testing.

Workflow Testing

The workflow testing window

When testing a workflow, the user can choose between three different ways of providing input data:

  1. Uploading a file. This is the recommended way, as the user probably already has a sample in json format somewhere anyway, but it is also much easier to work with here.
  2. Plain text. This will be parsed as json. The testing input should be an array with nested json objects, like [{“test”: “test”}]. It may contain multiple json objects.
  3. Empty. In the app, this is called “Data already present or not needed”. Usually this third option is not the case, but it can be, if the first recipe generates data, like “AddDataRecipe”, which we used as an example for recipe creation. In reality, this option uses [{}] as input.

Keep in mind that, when choosing for manual inputs, the field names should be surrounded by “”, just like in the above exmaple after option 2. ‘’s do not work! The same holds for strings.

The user also has the option to select whether the testing output should be compared to some value, to make sure that the workflow is working as expected. For example, if the only recipe being tested is the AddDataRecipe, adding the value ’test’ to field ’test’, with an empty input, the output should look like [{“test”: “test”}]. After the test has been submitted and completed, a new “Results” button shows up, next to the Submit button.

This button spawns the following window:

The workflow test results window

Here, the user can see the output by a series of expansion panels. The entire output array, or only individual json objects in the output array, can be downloaded. At the bottom the user can see whether the output is the same as expected. If it does not match, the user will get a message notifying which field(s) do(es) not satisfy.

Recipe Testing

Recipe testing works mostly the same; the user may choose an input and eventually an output for comparison. The user can also define a workflow to test the recipe in, if desired. Otherwise, it will be tested in a workflow containing only the tested recipe.

The recipe testing box

Upon toggling the workflow specification radio button, a “Specify Workflow” button appears. This button spawns the following window:

The workflow definition window for recipe testing

The to be tested recipe is greyed out, as it cannot be altered here, and it is denoted by its newly given name. For the remainder, this window is exactly the same as regular workflow creation.

API

The API specification can be found here.

Previous
Next