Building a Pipeline
Last updated
Last updated
The Pipeline canvas provides infinite possibilities to use your data.
This pane shows the general properties of your Pipeline. Click the ellipses next to its name to Copy ID.
Depending on your permissions, you can view or modify:
Name: When you create a pipeline by default, the first recommendation is to change the default name. But you can modify the name at any time clicking on the pencil icon close to the Pipeline's name
Tags: Click the tag icon to open the menu.
Clusters: Here you can see how many clusters your Pipeline is running in, as well as update them.
Versions: View and run multiple versions of the Pipeline.
Stop/Start Pipeline: Stop and start the Pipeline in some or all of the clusters it is running in.
Publish
When you modify your Pipeline, you will be creating a new version. When your modifications are complete, you can Publish this new version using this button in the top right.
Go to Managing versions to learn more.
You can carry out all these actions in bulk if you wish to modify more than one Pipeline at a time.
If the Pipeline is running, the Metrics bar provides a visual, graphical overview of the data being processed in your Pipeline.
Events In: View the total events in per second for the selected period, compared to the previous range (in %).
Bytes In: The total bytes in per second for the selected time range, compared to the previous (in %).
Events Out: View the total events out per second for the selected period, compared to the previous range (in %).
Bytes Out: The total bytes out per second for the selected time range, compared to the previous (in %).
Latency: The time (in milliseconds) it takes for data to travel from one point to another, compared to the previous (in %).
You can set a time range to view the metrics for a specific period of time. This will be used to calculate the percentages, compared to the previous time of the same period selected.
Go to Selecting a Time Range to learn more about the specifics of how this works.
Use the Hide metrics/Show metrics button to hide/show the metrics pane.
Simply drag and drop an element from the left-hand side onto the canvas to add it to your Pipeline.
For Listeners, you can drag the specific Label down to the required level. Once in the Pipeline, you can see which Listener the label belongs to by hovering over it, or in the Metrics area of the configuration pane.
The canvas is where you will build your Pipeline. Drag and drop an element from the left pane to add it to your Pipeline.
Click it in the canvas to open its Properties.
If you have enough permissions to modify this Pipeline, click the node in the canvas and select the Remove icon.
Create links between your nodes to create a flow of data between them. Learn more about links below.
Zoom in/out, Center, undo, and redo changes using the buttons on the right.
Use the window in the bottom-right to move around the Canvas.
Connect the separate nodes of the canvas to form a Pipeline from start to finish.
Simply click the port you wish to link from and drag to the port you wish to link to. When you let go, you will see a link form between the two.
To Unlink, click anywhere on the link and select unlink in the menu.
Notice the ports of each element in the canvas. Ports are used as connectors to other nodes of the Pipeline, linking either incoming or outgoing data.
Listener: As a Listener is used to send information on, there are no in ports, and one out port.
Action: Actions generally have one in port, injecting it with data. When information is output, it will be sent via the default port. If there are problems sending on the data, it will not be lost, bur rather output via the error port.
Datasink: A datasink is the end stop for our data, so there is only one in port that receives your processed data.
Click one to read more about how to configure them:
Learn about how to set up and use Listeners
Discover Actions to manage and customize your data
Add the final piece of the puzzle for simpler data
