In order to use Onum, there are certain system requirements.

The installation process creates the Distributor and all Workers for each data source in the cluster.

System requirements

Onum supports the following browsers:

• Google Chrome

Once you have acquired an Onum account, there are a few steps required for Onum Installation:

Cloud

Onum’s Operations team will prepare the infrastructure on Onum’s SaaS based on the estimated volumetrics.

On-Premise

• The infrastructure requirements are shared with the Operations team of the client. Further steps cannot be conducted without the required infrastructure.

• Access to the infrastructure is granted to Onum’s team with the right permissions for conducting the installation.

• A validation script is run by Onum in order to assess if all requirements described in the Annex are met and all connectivities are opened.

• If the above point is successful, an installation slot is scheduled and agreed.

• Installation is conducted by Onum engineers using a Docker and a post-installation validation script is run.

You can now access its tenant, ingest data, invite users and use all of the Onum capabilities.

Hardware requirements

Hardware (per Virtual Machine):

• Distribution: Linux (Debian or Red Hat)

• Server Hardware: 16GB RAM and 8 CPU

• Disk Storage: 500GB

Access

In case of upcoming system maintenance, we kindly seek permission to access customer infrastructure. Our aim is to ensure seamless operations and address any potential issues promptly.

Action

A unit of work performing a given operation on an event.

API

Application Programming Interface. A set of defined methods of communication among various components.

Cluster

Various distributors and workers can be grouped and contained within a cluster. You can have as many clusters as required per Tenant.

Data sink

Where the data is routed after being processed by Onum.

Data source

Where the data is generated before ingesting it into Onum, e.g. application server logs, firewall logs, S3 bucket, Kafka Topic, etc.

Distributor

This service receives and processes the Listener data before sending it on to workers within a cluster.

Event

An event represents semi-structured data such as a log entry. Events can be parsed so that structured data can be generated and eventually processed by the engine. Events are composed of fields, which are referred to as Field. An action that produces a new field will be referred to as outputField.

Label

Used to sort events coming from Listeners into categories or sets that meet given filters to be used in a Pipeline.

Listener

A Listener retrieves events in a given IP address and a port, routing the data to the Pipelines so that it can be processed.

Lookup

A lookup refers to searching for and retrieving information from a specific source or dataset, typically based on a key or reference.

Multitenancy

Multitenancy is an architecture in which tenants share the same underlying infrastructure, including databases and application code, but their data and configurations are kept separate to ensure privacy and security.

Pipeline

A sequence of Actions connected through inputs/outputs to process a stream of data. Data comes from the Listener and eventually is routed to a Datasink.

Role

A role is assigned to a user in order to control the access they have to certain or all Onum features. This way, we can personalise the experience for each user.

Tag

Tags can be assigned to Listeners, Pipelines or Data sinks in order to classify them or make them easier to find. This is particularly useful if you have a wide database and want to avoid lengthy searching for the resources you wish to use.

Tenant

A Tenant is a domain that contains a set of data in your organization. You can use one or various tenants and grant access to as many as required.

Worker

This service runs the Pipelines, receiving data from its distributor and contained within a Cluster.

These articles contain information on functionalities across the entire platform.

Quick Links

Most popular

Overview

The exponential growth of data ingestion volumes can lead to reduced performance, slow response times, and increased costs. With this comes the need to implement optimization strategies & volume reduction control. We help you cut the noise of large data streams and reduce infrastructure by up to 80%.

Gain deep insights from any type of data, using any format, from any source.

All of this...

@ the Edge

By collecting and observing that data at the edge, as close as possible to where it’s being generated, gain real-time observations and take decisive action to prevent network downtime, payment system failures, malware infections, and more.

Unlike most tools that provide data observation and orchestration, Onum is not a data analytics space, which is already served well by security information and event management (SIEM) vendors and other analytics tools. Instead, Onum sits as close as possible to where the data is generated, and well in front of your analytics platforms, to collect and observe data across every aspect of your hybrid network.

Start with the basics

Welcome to Onum! This guide will help you start working with Onum, a powerful tool designed to enhance your data analysis and processing capabilities.

Accessing Onum

A Tenant is a domain that contains a set of data in your organization. You can use one or various Tenants and grant access to as many as required.

Navigating the Interface

You can access the rest of the areas in Onum using the left panel.

Create Your First Listener

Onum receives any data through Listeners.

These are logical entities created within a Distributor, acting as the gateway to the Onum system. Configuring a Listener involves defining an IP address, a listening port, and a transport layer protocol, along with additional settings depending on the type of Listener specialized in the data it will receive.

Create Your First Data Sink

Onum outputs data via Data sinks. Use them to define where and how to forward the results of your streamlined data.

Build Your First Pipeline

Use Pipelines to start transforming your data and build a data flow. Pipelines are made of the following components:

Use cases

Since Onum is able to process any data type, you may be wondering how to identify which is which. See the color legend below:

Overview

When opening Onum, the Home area is the default view. Here you can see an overview of all the activity in your .

Use this view to analyze the flow of data and the change from stage to stage of the process. Here you can locate the most important contributions to your workflow at a glance.

Metrics

The Home view shows various infographics that provide insights into your data flow. Some Listeners or Data Sinks may be excluded from these metrics if they are duplicates or reused.

Sankey Diagram

Each column of the Sankey diagram provides information and metrics on the key steps of your flow.

You can see how the data flows between:

Hover over a part of the diagram to see specific savings.

Show Metrics

You can narrow down your analysis even further by selecting a specific node and selecting Show metrics.

View Details

Click a node and select View details to open a panel with in-depth details of the selected piece.

From here, you can go on to edit the selected element.

Hide/Show Columns

You can choose which columns to view or hide using the eye icon next to its name.

Add New Elements

You can add a new Listener, Label, Pipeline or Data sink using the plus button next to its name.

You can also create all of the aforementioned elements using the Create new button at the top-right:

Overview

Essentially, Onum receives any data through Listeners. These are logical entities created within a Distributor, acting as the gateway to the Onum system. Due to this, configuring a Listener involves defining an IP address, a listening port, and a transport layer protocol, along with additional settings depending on the type of Listener specialized in the data it will receive.

Click the Listeners tab on the left menu for a general overview of the Listeners configured in your Tenant and the events generated.

• The graph at the top plots the volume ingested by your listeners. The line graph represents the events in, and the bar graph represents bytes in. Learn more about this graph in this article.

• At the bottom, you have a list of all the Listeners in your Tenant. You can switch between the Cards view, which shows each Listener in a card, and the Table view, which displays Listeners listed in a table. Learn more about the cards and table views in this article.

Narrow Down Your Data

There are various ways to narrow down what you see in this view:

Add Filters

Add filters to narrow down the Listeners you see in the list. Click the + Add filter button and select the required filter type(s). You can filter by:

• Name: Select a Condition (Contains, Equals, or Matches) and a Value to filter Listeners by their names.

• Type: Choose the Listener type(s) you want to see in the list.

• Version: Filter Listeners by their version.

• Created by: Selecting this option opens a User drop-down where you can filter by creator.

• Updated by: Selecting this option opens a User drop-down where you can filter by the last user to update a pipeline.

The filters applied will appear as tags at the top of the view.

Select a Time Range

If you wish to see data for a specific time period, this is the place to click. Go to this article to dive into the specifics of how the time range works.

Select Tags

You can choose to view only those Listeners that have been assigned the desired tags. You can create these tags in the Listener settings or from the cards view. Press the Enter key to confirm the tag, then Save.

To filter by tags, click the + Tags button, select the required tag(s) and click Save.

Create a Listener

Depending on your permissions, you can create a new Listener from this view. To do it, simply click the New listener button at the top right corner.

This will open the Listener configuration.

Configure a Listener

Configuring your Listener involves various steps. You can open the configuration pane by creating a new Listener or by clicking a Listener in the Listener tab or the Pipeline view and selecting Edit Listener in the pane that opens.

Alternatively, click the ellipses in the card or table view and select Edit.

Type

The first step is to define the Listener Type. Select the desired type in this window and select Configuration.

Configuration

The configuration is different for each Listener type. Check the different Listener types and how to configure them in this section.

Labels

Use Onum's labels to cut out the noise with filters and search criteria based on specific metadata. This way, you can categorize events sent on and processed in your Pipelines.

Learn more about labels in this article.

Overview

This article outlines the more complex calculations that go on behind the graphs you see.

In the Listeners, Pipelines, and Data sinks views, you will see detailed metrics on your events and bytes in/out, represented in a graph at the top of these areas.

The line graph represents the events in/out, and the bar graph represents bytes in/on. Hover over a point on the chart to show a tooltip containing the events and bytes in for the selected time, as well as a percentage of how much increase/decrease has occurred since the previous lapse of time since the one currently selected.

Events

The values on the left-hand side represent the events in/out for the selected period.

Bytes

The values on the right-hand side represent the bytes in/out for the selected period.

Overview

In the Listeners, Pipelines, and Data sinks areas, you can view all the resources in your Tenant as cards or in a table.

In both views, you can:

• Click the magnifying glass icon to look for specific elements in the list. You can search by name, status, or tag.

• Display all the elements individually in a list or grouped by Status or Type. These grouping options vary depending on the area you are.

Table View

In the Table view, you can click the cog icon to begin customizing the table settings. You can reorder the columns in the table, hide or display the required ones or pin them.

Changes will be automatically applied. Click the Reset button to recover the original configuration.

• Use the buttons at the top right part of the table to expand or collapse each row in the table. This will change the level of detail of each element.

• Click the ellipsis button on each row to edit the element, copy its ID, or remove it.

Cards View

In this view, each element is displayed as a card that shows details about it.

• Click the ellipsis button on each card to edit the element, copy its ID, or remove it.

• Click the Add tag button and add the required tags to an element. For each tag you enter in the box, hit the Enter key. Click Save to add the tags.

Overview

Onum supports integration with Google Pub/Sub. Select Google Sub from the list of Listener types and click Configuration to start.

Configuration

Now you need to specify how and where to collect the data, and establish a connection with your Google account.

Metadata

Enter the basic information for the new Listener.

• Name* - Enter a name for the new Listener.

• Description - Optionally, enter a description for the Listener.

• Tags - Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Now add the configuration to establish the connection.

• Project ID* - This is a unique string found in the Manage all projects area of the projects list.

• Subscription Name* - Find your subscription in the Google Cloud console, Pub/Sub Subscriptions page, Metrics tab.

• Credentials File* - The Google Pub/Sub connector uses OAuth 2.0 credentials for authentication and authorization. Create a secret containing these credentials or select one already created.

Bulk Mesages Configuration

• Enabled* - Decide whether or not to activate the bulk message option.

• Message Format - Choose the required message format.

• Delimiter Char Codes - Enter the characters you want to use as delimiters.

Click Create labels to move on to the next step and define the required Labels if needed.

Easy, flexible deployment in any environment while keeping them as close as possible to where the data is produced delivers unparalleled speed and efficiency, enabling you to cut the infrastructure you have dedicated to orchestration by up to 80%.

The Onum infrastructure consists of:

• Distributor: this is the service that hosts the Listener before forwarding it to Workers.

• Worker: this is the service that runs the Pipelines, receiving data from its Distributor and contained within a Cluster.

• Cluster: a container grouping Distributors and Workers. You can have as many clusters as required per Tenant.

Listeners are hosted within Distributors and are placed as close as possible to where data is generated. The Distributor pulls tasks from the data queue passing through the pipeline and distributes it to the next available worker in a Cluster. As soon as a Worker completes a task it becomes available again, and the Distributor in turn will assign it the next task from the queue.

The installation process creates the Distributor and all Workers for each data source in the cluster.

How it works

Deployment types

The Onum Platform supports any deployment type ― including on-premises, the Onum public cloud, or your own private cloud.

In a typical SaaS-based deployment, most processing activities are conducted in the Cloud.

Client-side components can be deployed on a Linux machine or on a Kubernetes cluster for easy, flexible deployment in any environment. Onum supports all major cloud environments, including Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure.

Delivery methods

Onum supports all major standards such as Netflow, Syslog, and Kafka to orchestrate data streams to any desired destination, including popular data analytics tools such as Splunk and Devo, as well as storage environments such as S3.

Overview

Throughout the entire Onum platform, you can set a period to either narrow down or extend the data shown. You can either select a predefined period or apply a custom time range.

The related graph and resources will be automatically updated to display data from the chosen period. To remove a selected period, simply click the bin icon that appears next to the period to go back to the default time range (1 hour ago).

Predefined and Custom time ranges

As well as predefined time intervals, you can also define a custom time range. To do it, simply select the required starting and ending dates in the calendar.

Comparisons

The interesting thing about Onum is that you can directly see how much volume you have saved compared to past ingestions, telling you what is going well and what requires further streamlining.

The comparison is direct/equivalent, meaning all data shown is analyzed compared to the previously selected equivalent time range.

For example, if the time range is 1 hour, the calculation of differences will be carried out using the previous one hour before the current selection =

• Range selected: 10:00-11:00

• Comparison: 09:00-10:00

Again, let´s say you now wish to view data over the last 7 days. The percentages will be calculated by measuring the volume retrospectively two weeks ago with the previous week.

Although there are only a limited number of types available for use, the integration possibilities are endless. Alternatively, you can contact us to request a Listener type.

Click a Listener to see how to configure it.

Overview

Onum supports integration with Cisco System NetFlow. Select Flow from the list of Listener types and click Configuration to start.

Configuration

Now you need to specify how and where to collect the data, and how to establish a connection with Cisco NetFlow.

Metadata

Enter the basic information for the new Listener.

• Name* - Enter a name for the new Listener.

• Description - Optionally, enter a description for the Listener.

• Tags - Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Now add the configuration to establish the connection.

Socket

• Transport protocol* - Currently, Onum supports the UDP protocol.

• Port* - Enter the required IP port number.

Flow

• Protocols to process* - Select the required protocol(s) from the list.

• Fields to include* - Select all the fields you wish to include in the output data.

Access control

• Access control type* - Choose between None, Whitelist, or Blacklist.

• IPs - Enter the IPs you wish to apply the access control to. Click Add element to add as many as required.

Click Create labels to move on to the next step and define the required Labels if needed.

Overview

Onum supports integration with Transmission Control Protocol. Select TCP from the list of types.

Configuration

Now you need to specify how and where to collect the data, and how to establish a connection with TCP.

Metadata

Enter the basic information for the new Listener.

• Name* - Enter a name for the new Listener.

• Description - Optionally, enter a description for the Listener.

• Tags - Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

• Port* - Enter the IP port number.

• Trailer Character* - Choose between LF, CR+LF, or NULL.

TLS configuration

• Certificate* - This is the predefined TLS certificate.

• Private Key* - The private key of the corresponding certificate.

• CA chain - The path containing the CA certificates.

• Client Authentication Method* - Choose between No, Request, Require, Verify, and Require & Verify.

• Minimum TLS version* - Select a version from the menu.

The default tab that opens when in the Pipeline area is the Listeners tab, which shows all Listeners in your Tenant, as well as their labels.

Use the search bar to find a specific Listener or Label.

Edit a Listener

You can edit a label from the list by clicking the ellipses next to its name and selecting Edit.

Create Listener

If the Listener you wish to use in the Pipeline does not already exist, you can create it directly from this view using the Create Listener button in the bottom right of the tab. This will open the Listener Configuration window.

Add a Listener to your Pipeline

Overview

The Pipeline canvas provides infinite possibilities to use your data.

1. General settings

This pane shows the general properties of your Pipeline. Click the ellipses next to its name to Copy ID.

Depending on your permissions, you can view or modify:

• Name: When you create a pipeline by default, the first recommendation is to change the default name. But you can modify the name at any time clicking on the pencil icon close to the Pipeline's name

• Tags: Click the tag icon to open the menu.

• Clusters: Here you can see how many clusters your Pipeline is running in, as well as update them.

• Versions: View and run multiple versions of the Pipeline.

• Stop/Start Pipeline: Stop and start the Pipeline in some or all of the clusters it is running in.

• Publish

When you modify your Pipeline, you will be creating a new version. When your modifications are complete, you can Publish this new version using this button in the top right.

2. The metrics bar

If the Pipeline is running, the Metrics bar provides a visual, graphical overview of the data being processed in your Pipeline.

• Events In: View the total events in per second for the selected period, compared to the previous range (in %).

• Bytes In: The total bytes in per second for the selected time range, compared to the previous (in %).

• Events Out: View the total events out per second for the selected period, compared to the previous range (in %).

• Bytes Out: The total bytes out per second for the selected time range, compared to the previous (in %).

• Latency: The time (in milliseconds) it takes for data to travel from one point to another, compared to the previous (in %).

Set a time range

You can set a time range to view the metrics for a specific period of time. This will be used to calculate the percentages, compared to the previous time of the same period selected.

Hide/Show metrics

Use the Hide metrics/Show metrics button to hide/show the metrics pane.

3. Add to the Pipeline

Simply drag and drop an element from the left-hand side onto the canvas to add it to your Pipeline.

4. Canvas

The canvas is where you will build your Pipeline. Drag and drop an element from the left pane to add it to your Pipeline.

Delete a node

If you have enough permissions to modify this Pipeline, click the node in the canvas and select the Remove icon.

5. Navigation options

Zoom in/out, Center, undo, and redo changes using the buttons on the right.

Use the window in the bottom-right to move around the Canvas.

Connect the separate nodes of the canvas to form a Pipeline from start to finish.

Simply click the port you wish to link from and drag to the port you wish to link to. When you let go, you will see a link form between the two.

To Unlink, click anywhere on the link and select unlink in the menu.

Ports

Notice the ports of each element in the canvas. Ports are used as connectors to other nodes of the Pipeline, linking either incoming or outgoing data.

Listener: As a Listener is used to send information on, there are no in ports, and one out port.

Action: Actions generally have one in port, injecting it with data. When information is output, it will be sent via the default port. If there are problems sending on the data, it will not be lost, bur rather output via the error port.

Datasink: A datasink is the end stop for our data, so there is only one in port that receives your processed data.

Click one to read more about how to configure them:

Overview

Use Onum's labels to cut out the noise with filters and search criteria based on specific metadata. This way, you can categorize the events that Listeners receive before being processed in your .

As different log formats are being ingested in real-time, the same Listener may ingest different technologies. Labels are useful for categorizing events based on specific criteria.

When creating or editing a Listener, use Labels to categorize and assign filters to your data.

For most Listeners, you will see two main event categories on this screen:

• All Data - Events that follow the structure defined by the specified protocol, for example, Syslog events with the standard fields, or most of them.

• Unparsed - These are events that do not follow the structure defined in the selected protocol.

You can define filters and rules for each of these main categories.

What Are Labels Used For?

Once you've defined your labels to filter specific events, you can use them in your Pipelines.

Instead of using the whole set of events that come into your Listeners, you can use your defined labels to use only specific sets of data filtered by specific rules.

Creating Your First Label

When you create a new Listener, you'll be prompted to the Labels screen after configuring your Listener data.

From here, you have various options:

Create a new label

To create a new subset of data, select the + sign that extends directly from the All data or Unparsed bars. Be aware that if you select the + sign extending from the header bar, you will create a subheader.

Create a sub-label

You can create a branch from your primary header by clicking the plus button that extends from the main header. There is no limit to the amount that you can add.

Notice that the subheader shows a filter icon with a number next to it to indicate the string of filters applied to it already.

Duplicate your label

To duplicate a label, simply select the duplicate button in its row.

Delete a label

To delete a label, simply select the delete button in its row.

Once you have completed your chain, click Save.

Unlabeled

Any data that has not been assigned a label will be automatically categorized as unlabeled. This allows you to see the data that is not being processed by any Pipeline, but has not been lost.

This label will appear in the list of Labels for use in your Pipeline so that you can process the data in its unfiltered form.

Your Listener is now ready to use and will appear in the list.

Overview

Onum receives data from Syslog, supporting TCP and UDP protocols. Select Syslog from the list of types.

Configuration

Now you need to specify how and where to collect the data, and how to establish a connection with Syslog.

Metadata

Enter the basic information for the new Listener.

• Name* - Enter a name for the new Listener.

• Description - Optionally, enter a description for the Listener.

• Tags - Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

• Port* - Enter the IP port number.

• Protocol* - Onum supports TCP and UDP protocols.

• Framing Method* - Choose the required framing method between: Auto-Detect, Non-Transparent (newline), Non-Transparent (zero), or Octet Counting (message length).

TLS configuration

• Certificate* - This is the predefined TLS certificate.

• Private key for this listener* - The private key of the corresponding certificate.

• CA chain - The path containing the CA certificates.

• Client authentication method* - Choose between No, Request, Require, Verify, and Require & Verify.

• Minimum TLS version* - Select the required version from the menu.

Overview

Use Pipelines to transform your data and build a data flow linking from and to .

Select the Pipelines tab at the left menu to visualize all your Pipelines in one place. Here's what you will find and the actions you can perform in this area:

Narrow Down Your Data

There are various ways to narrow down what you see in this view, both the Pipeline list and the informative graphs. To do it, use the options at the top of this view:

Add Filters

Add filters to narrow down the Pipelines you see in the list. Click the + Add filter button and select the required filter type(s). You can filter by:

• Name: Select a Condition (Contains, Equals, or Matches) and a Value to filter Pipelines by their names.

• Status: Choose the status(es) you want to filter by: Draft, Running, and/or Stopped. You'll only see Pipelines with the selected status(es).

• Created by: Filter for the creator of the Pipeline in the window that appears.

• Updated by: Filter for users to see the Pipeline they last updated.

The filters applied will appear as tags at the top of the view.

Select a Time Range

Select Tags

You can choose to view only those Pipelines that have been assigned the desired tags. You can create these tags in the Pipeline settings or from the cards view. Press the Enter key to confirm the tag, then Save.

To filter by tags, click the + Tags button and select the required tag(s).

Metrics

Just below the filters, you will see 3 metrics informing you about various components in your Pipelines.

Listeners

View the events per second (EPS) ingested by all Listeners in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous lapse.

Data Sink

View the events per second (EPS) sent by all Data Sinks in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous.

Data Volume

See the overall data volume processed by all Pipelines for the selected time range, and the difference in percentage with the previous.

Visualize Your Data In/Out

Select between In and Out to see the volume received or sent by your Pipelines for the selected time range. The line graph represents the Events and the bar graph represents Bytes.

Hover over a point on the chart to show a tooltip containing the Events and Bytes in/out for the selected time, as well as a percentage of how much increase/decrease has occurred since the previous lapse of time since the one currently selected.

You can also analyze a different time range directly on the graph. To do it, click a starting date in the map and drag the frame that appears until the required ending date. The time range above will be also updated.

Pipelines List

At the bottom, you have a list of all the Pipelines in your tenant.

Use the Group by drop-down menu at the right area to select a criterion to organize your Pipelines in different groups (Status or None). You can also use the search icon to look for specific Pipelines by name.

Use the bottoms at the left of this area to display the Pipelines as Cards or listed in a Table:

Cards View

In this view, Pipelines are displayed as cards that display useful information. Click a card to open the Pipeline detail view, or double-click it to access it.

This is the information you can check on each card:

• The percentage at the top left corner indicates the amount of data that goes out of the Pipeline compared to the total incoming events, so you can check how data is optimized at a glance. Hover over it to see the in/out data in bytes and the estimation over the next 24 hours.

• You can also see the status of the Pipeline (Running, Draft, or Stopped).

• Next to the status, you can check the Pipeline current version.

• Click the Add tag button to define tags for the Pipeline. To assign a new tag, simply type the name you wish to assign, make sure to press Enter, and then select the Save button. If the Pipeline has tags defined already, you'll see the number of tags next to the tag icon.

• Click the ellipses in the right-hand corner of the card to reveal the options to Edit, Copy ID, or Remove it.

Table view

In this view, Pipelines are displayed in a table, where each row represents a Pipeline. Click a row to open the Pipeline detail view, or double-click it to access it.

Click the cog icon at the top left corner to rearrange the column order, hide columns, or pin them. You can click Reset to recover the default configuration.

Pipeline detail view

The details pane is split into three tabs showing the Pipeline at different statuses:

• Running: select the drop-down to see which clusters the Pipeline is currently running in.

• Draft

• Stopped

In each one, you can see the various versions of this Pipeline at this stage. Open one to see a preview of the pipeline, creation data, data metrics, and the Listeners and Data sinks it contains.

Once you have located the Pipeline to work with, click+ Edit Pipeline to open it.

Duplicate a Pipeline

If you wish to use a Pipeline just like the one you are currently working on, click the ellipses in the Card or Table view and select Duplicate, or from the Configuration pane.

Create a Pipeline

Depending on your permissions, you can create a new Pipeline from this view. There are several ways to create a new Pipeline:

This will open the new Pipeline, ready to be built.

Keep reading to learn how to build a Pipeline from this view.

Build your Pipeline

Overview

Onum supports integration with HTTP. Select HTTP from the list of Listener types and click Configuration to start.

Configuration

Now you need to specify how and where to collect the data, and how to establish an HTTP connection.

Metadata

Enter the basic information for the new Listener.

• Name* - Enter a name for the new Listener.

• Description - Optionally, enter a description for the Listener.

• Tags - Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Socket

• Port* - Enter the IP port number.

TLS Configuration

• Certificate* - This is the predefined TLS certificate.

• Private key for this listener* - The private key of the corresponding certificate.

• CA chain - The path containing the CA certificates.

• Client authentication method* - Choose between No, Request, Require, Verify, and Require & Verify.

• Minimum TLS version* - Select the required version from the menu.

Endpoint

• HTTP Method* - Choose GET, POST, or PUT method.

• Request path* - Enter the RegEx used to request access.

Message extraction

• Strategy* - Choose what and how to extract.

• Extraction info - Any additional information on the strategy to use.

General behavior

• Propogate headers strategy - Choose between None or Allow.

• Header keys - Enter the required header keys in this field. Click Add element for each one.

• Exported headers format - Choose the required format for your headers.

• Maximum message length - Maximum characters.

• Response code - Specify the response code to show when successful.

• Response Content-Type - Choose the text or application type.

• Response Text - The text that will show in case of success.

The Actions tab shows all available actions to be assigned and used in your Pipeline.

Use the search bar at the top to find a specific action.

Hover over an action in the list to see a tooltip, as well as the option to View details.

To add an action to a Pipeline, drag it onto the canvas.

Action Versioning

We are constantly updating and improving Actions, therefore you may come across old or even discontinued actions.

If there is an updated version of the Action available, it will show update available in its Definition, above the node when added to a Pipeline, and Details pane.

If you have added an Action to a Pipeline that is now discontinued, it will show as deactivated in the Canvas.

Add an Action to your Pipeline

Actions list

Redis is a powerful in-memory data structure store that can be used as a database, cache, and message broker. It provides high performance, scalability, and versatility, making it a popular choice for real-time applications and data processing.

This action stores data in its cache and makes it available on command.

Find Redis in the Actions tab and drag it onto the canvas to use it.

To use this action, you will need to have access to the Redis service, either on-premise or cloud, installed.

Configuration

Redis Configuration

• Redis endpoint* - enter the endpoint used to establish connection to the redis server.

• Commands - the command to retrieve the data from the server. Choose between SET and GET.

• Redis Key* - if the model has a version, enter it here.

• Event in field - the message to input to redis.

• Output - enter a name for the output event.

Rate Limit

• Maximum requests - set a limit on the number of requests per second to launch on the server.

Click Save to complete.

Evaluate any AI model built with the Cog library and deployed as an endpoint anywhere: SageMaker, HuggingFace, Replicate…

The HttpCog Action seamlessly integrates with models deployed on any platform, including the client's own infrastructure. It enables not only the utilization of state-of-the-art machine learning models but also the execution of code specific to a client's unique use case.

Find Http Enrichment in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

• Endpoint* - enter the endpoint used to establish connection to the model.

• Token - if the model has a token, use it here.

• Version - if the model has a version, enter it here.

• Input* - the message to input to the model.

• Output* - enter a name for the output event.

Click Save to complete.

These Actions involve improving raw data using advanced procedures.

Click an Action to learn more about how to use it.

Overview

This action adds new fields to your events using a given operation. You can select one or more operations to execute, and their resulting values will be set in user-defined event fields.

Find Field Generator in the Actions tab (under the Advanced group) and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

You can perform the following operations to define new fields in your events:

Operations

Click Save to complete.

For each element in the input list, the For Each action generates an output that adds to the input event and the index it occupies in the list.

For example, an input list containing [a,b,c] will generate three outputs, with these fields added to the event:

1. elementValueOutField: a; elementIndexOutField: 0

2. elementValueOutField: b; elementIndexOutField: 1

3. elementValueOutField: c; elementIndexOutField: 2

Find For Each in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration

Configuration

• Input* - the field that contains the input list. Valid types areString, Integer, Float, Boolean,Timestamp

• Output Field* - the field where each iterated element will be stored. This will be the same type as the input list field.

• Index Field* - where to stored the index of the returned out field in the input list.

Click Save to complete.

Example

127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,192.168.0.1.

In the input field, select the list type field.

Assign the value and the index fields a name.

The action will create a separate event for each element of the string, each event containing two fields (value and index).

The Lookup action allows you to retrieve information from your uploaded lookup. To learn more about how to upload data, go to .

Find Lookup in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

Select the table you wish to retrieve data from. The tables that show here in the list will be those you previously uploaded in the Enrichment view.

The Key column you selected during the upload will automatically appear. Select the field to search for this key column.

In the Outputs, choose the field that will be injected from the drop-down and assign it a name. Add as many as required.

Click Save to complete.

This Action accumulates events before sending them on.

Find Accumulator in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

Field list

• Fields - choose the Listener event fields you would like to accumulate. You can select an infinite number of fields using the Add element button.

• Accumulate type* - choose how to accumulate the events, by period or number of events.

  • Accumulate period - if you select by period, define the number of seconds to accumulate for.

  • Number of events - if you select by number of events, define how many.

• Output* - enter a name for the output event.

Click Save to complete.

This action allows you to configure and execute HTTP requests with custom settings for methods, headers, authentication, TLS, and more.

Find Http Request in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

• HTTP Method* - The HTTP method to use for the request (e.g., GET, POST, PUT, DELETE, PATCH).

• Server URL* - The target URL for the HTTP request.

• Payload field - JSON field to include as the request body.

• Output field - Field to store the HTTP response.

• HTTP headers - Key-value pairs for HTTP headers.

• Timeout - Timeout for the request (in seconds, minimum 1).

• Disable redirects - Select true to disable HTTP redirects or false to ignore.

• Content type - Set the request content type (default is application/json).

Authentication Configuration: if you require authentication.

• Authentication type* - None, Basic, Bearer, or API Key.

• Authentication credentials - Credentials required based on the authentication type.

  • Basic Authentication - Username and password for Basic authentication.

  • Bearer token - Token for Bearer authentication.

  • API key: -API key configuration with API Key Name and API Key Value.

Bulk Configuration

• Events per batch - Number of events per batch.

• Store as - Store response with options (Delimited, Without Delimiter, JSON Array).

• Delimiter - Custom delimiter (default is newline).

Rate Limit

• Maximum requests* - set a limit on the number of requests per second to launch on the server.

TLS Configuration

• Certificate - This is the predefined TLS certificate.

• Private Key - The private key of the corresponding certificate.

• CA Chain - The path containing the CA certificates.

• Minimum TLS version - choose the TLS version to use.

Proxy Configuration: if your organization uses proxy servers, establish the connection here.

• Proxy Scheme

• Username

• Password

• URL

Retry Configuration

• Max attempts - Maximum retry attempts.

• Wait between - Wait time (in milliseconds) between attempts.

• Backoff interval - Backoff interval for retries.

Configuration example

Click Save to complete.

A comprehensive list of the operations available in the Field Tranformation Action.

This action enriches based on the evaluation of the LLaMa2 Chat model. This model offers a flexible, advanced prompt system capable of understanding and generating responses across a broad spectrum of use cases for text logs.

By integrating LLaMA 2, Onum not only enhances its data processing and analysis capabilities but also becomes more adaptable and capable of offering customized and advanced solutions for the specific challenges faced by users across different industries.

Find MlLlama in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration

Configuration

• Token - this will be the API token of the model. See here for where to find these values.

• Model - the name of the model to connect to. It’s possible to select between the three available Llama2 models: Llama2-7b-Chat, Llama2-13b-Chat and Llama2-70b-Chat.

• Prompt - this will be the input field to call the model.

• Temperature - this is the randomness of the responses. If the temperature is low, the data sampled will be more specific and condensed, whereas setting a high temperature will acquire more diverse but less precise answers.

• System Prompt - describe in detail the task you wish the AI assistant to carry out.

• Max Length - the maximum number of characters for the result.

• Output - specify a name for the output field.

Click Save to complete.

These Actions involve enriching Onum data with external or additional data.

Click an Action to learn more about how to use it.

Aggregation Actions involve summarizing or grouping data points based on certain criteria.

Click an Action to learn more about how to use it.

These actions involve processing or transforming data in some way. This can be to remove unwanted data, to reformat, to extract, etc.

Click an Action to learn more about how to use it.

Summarize data by performing aggregations using keys and temporal keys (min, hour or day).

Find Aggregator in the Actions tab and drag it onto the canvas to use it.

Configuration

Grouping configuration

The Fields to Group option lists the fields coming from the linked Listener or Action for you to choose from. Choose one or more fields to group by.

Having defined which fields to group by, choose or create a Grouping Time. You can write the amount and unit (seconds, minutes, hours, days), or select a common amount.

Click Save.

Aggregations

Now you can add aggregation(s) to your grouping using the following operations:

• average: calculates the average of the values of each grouping.

• count: calculates the total occurrences for each grouping.

• countnotnull: calculates the total occurrences for each grouping, excluding null values.

• first: finds the first value found for each grouping . The first value will be the first in the workers queue.

• firstnotnull: finds the first not null value found for each grouping . The first value will be the first in the workers queue.

• ifthenelse: the operation will only be executed if the given conditions are met.

• last: finds the last value found for each grouping . The last value will be the last in the workers queue.

• lastnotnull: finds the last not null value found for each grouping . The last value will be the last in the workers queue.

• max: finds the highest value found.

• min: finds the lowest value found.

• sum: calculates the total of the values for each grouping.

To add another aggregation, use the Add item option.

Conditions

You can also carry out an advance configuration by Grouping By Conditionals.

Use the Add Condition option to add conditions to your Aggregation.

Click Save when complete.

Example

In this example, we will use the Group By action to summarize a large amount of data, grouping by IP address every 5 minutes and aggregate the number of requests by type per IP address.

[
  {"IP_Address": "192.168.1.1", "Request_Type": "GET", "Timestamp": "2025-01-09T08:00:00Z"},
  {"IP_Address": "192.168.1.2", "Request_Type": "POST", "Timestamp": "2025-01-09T08:05:00Z"},
  {"IP_Address": "192.168.1.1", "Request_Type": "POST", "Timestamp": "2025-01-09T08:10:00Z"},
  {"IP_Address": "192.168.1.3", "Request_Type": "GET", "Timestamp": "2025-01-09T08:15:00Z"},
  {"IP_Address": "192.168.1.2", "Request_Type": "GET", "Timestamp": "2025-01-09T08:20:00Z"},
  {"IP_Address": "192.168.1.1", "Request_Type": "GET", "Timestamp": "2025-01-09T08:25:00Z"},
  {"IP_Address": "192.168.1.3", "Request_Type": "POST", "Timestamp": "2025-01-09T08:30:00Z"}
]

{
  "aggregated_requests": [
    {
      "IP_Address": "192.168.1.1",
      "GET_Count": 2,
      "POST_Count": 1,
      "Total_Requests": 3
    },
    {
      "IP_Address": "192.168.1.2",
      "GET_Count": 1,
      "POST_Count": 1,
      "Total_Requests": 2
    },
    {
      "IP_Address": "192.168.1.3",
      "GET_Count": 1,
      "POST_Count": 1,
      "Total_Requests": 2
    }
  ]
}

This action evaluates a list of conditions for an event. If an event meets a given condition, it will be sent through an output port specific to that condition. If the event does not meet any condition, it will be sent through the default output.

Set any number of conditions on your data for filtering and alerting.

Find Conditional in the Actions tab and drag it onto the canvas to use it.

To open the configuration, click the Action in the canvas and select Configuration.

Configuration

In the panel that appears, you can begin to add conditions, each with their own port.

The Field option lets you choose not only the field to filter for, but the specific Action to take it from.

Choose between the following conditions for the filter (or use the arrow keys on your keyboard to navigate up and down the list):

• Number

  • (<) Less than

  • (≤) Less than or equal to

  • (>) Greater than

  • (≥) Greater than or equal to

  • (=) Equal to

  • (!=) Not equal to

• String

  • Contains

  • Doesn't contain

  • Equal

  • Not equal

• Boolean

  • Equal

  • Not equal

  • Contains

  • Does not contain

  • Equal to

  • Not equal to

Enter the value to filter in (remember to press enter if you´re writing one) and you have your condition.

Now you can add AND/OR clauses to your condition, or add a new condition entirely using the Add Condition option.

Click Save when complete.

Example

Let's say you have data on error and threat detection methods in storage devices and you wish to detect threats and errors using the Cyclic Redundancy Check methods crc8, crc16 and crc24.