Data Streams > Data Pipeline Configuration

Data Pipeline Configuration

Create a Data Pipeline

Click on the + button to create a Data Pipeline.

Fill in the required details such as Name, Description, Instances, Threads per instance, and the Advanced Settings. Click on the Save button to add the pipeline.

View Pipeline

Click on the pipeline name to view the particular pipeline details.

There are 4 tabs under the ‘View’ option. You can get the details by clicking on each of them. Following are the tabs:

  1. Flows
  2. Settings
  3. Metrics

Flows

Under Flows, the pipeline flow chart is visible, and you can perform the following action:

  • Start: Start the pipeline.
  • Stop: Stop the pipeline.
  • Restart: Restart the pipeline.

Settings

You can view the Pipeline information from here.

Metrics

You can see the list of metrics under this tab.

Edit Pipeline

To Edit the pipeline, click on the Edit button located under the Actions column.

On clicking, you will be redirected to the following page.

Following are the two tabs present under the edit section.

  1. Pipeline Editor
  2. Pipeline Info

Pipeline Editor

Under the Pipeline Editor, you can edit the pipeline layout.

The first and last columns are reserved for data streams and the intermediate block you can use to add the plugin.

Click on any block to configure any desired changes. For example on clicking inputBlock, you may add the plugin and edit the filter expression.

To add the plugin to an empty block, click on any block except those in the first and last columns, and enter the block name.

Click the Save button. Then, if you click on the block once more, you’ll be able to insert plugins and the filter expression.

You can make changes to the Pipeline layout and further configure the edits as:

  • Debug: This is for debugging the pipeline.
  • Publish: Convert the draft pipeline to published and deploy the pipeline.
  • Revert to Published: The draft will be overwritten by the published pipeline.
  • Cancel: Discard the unsaved changes.
  • Save: Save the current changes in draft mode.

Pipelines Info

Another tab is the “Pipeline Info” under the edit section. You can configure the advanced pipeline settings from here.

Delete Pipeline

Select the pipeline you want to delete and click on the Delete button. You can delete the pipeline either using the Delete button at the top or under the Action column.

Click on the Delete button to confirm the delete action. 

The selected pipeline will be deleted and removed from the list.

Debug Pipeline

This Debug option is utilized for the production or publishing of the pipeline. There are other two debug options available:

  • Draft pipeline debug
  • Draft block debug

So there are three debug options available in the latest UI.

  • On the listing page: This runs the published version of the pipeline.
  • On the block editor page: This runs the draft version (Pipeline that is currently under construction).
  • On the block page: This shows the output after processing a particular block of the draft version of the pipeline.

You can access the debug option under the Actions column.

This will redirect you to the following page. You can click on Debug to start the debugging process. From the Data Preview, you can Download Input/ Output Events.

Further Reading

  1. I/O Streams Configuration
  2. DataStore Connector Configuration

Resources

Browse through our resources to learn how you can accelerate digital transformation within your organisation.

Unveiling our all powerful IBMB Observability ExperienceCenter. Click Here