nano SIEM
Pipelines

Getting started

Supported in:Self-hosted

Build, preview, publish, and roll back a pipeline

Build and preview a pipeline to check how your processing steps change incoming events. Publish a reviewed release when the output matches your intent.

Before you start

You need log_sources:view to open the pipeline and log_sources:edit to save a draft. Event previews require search:execute and source access. Publishing and rollback require log_sources:deploy.

1. Open a pipeline

Go to Ingestion → Pipelines and open the log source you want to process. The list shows each source's pipeline, its active version, draft, and latest release. If you need a new source, create it through Log Sources first.

An existing parser appears as an Existing parser Advanced VRL step. Keep its position in mind: steps before it see raw input; steps after it see the fields that parser produced. Adding a JSON Parse step does not by itself map vendor fields to OCSF.

2. Load representative input

In Draft preview, select a saved input sample, use Load recent events, tap live input, or paste raw events. Pasted input accepts one event per non-empty line, a JSON object, or a JSON array of events, with at most 100 inputs.

Load recent events retrieves stored events for this source from ClickHouse, under your source access, and reconstructs preview input from their original raw text. Rows without that text cannot be replayed. Recent and pasted samples stay in browser memory and are never attached to a release. To observe new incoming events, use Live capture.

3. Add and order steps

Add a step from Processing steps, select it, and configure its fields. Move steps up or down, duplicate them, or disable a step to compare behavior. Leave Execution condition blank to run on every event; otherwise supply a single VRL boolean expression.

For a raw JSON message, put Parse before field operations. For example, parse message, then use Filter with Predicate .level == "debug" and Matching events Drop. Check Step reference for the supported operations.

4. Preview and validate

Draft preview runs in an isolated preview worker, which your deployment enables with NANO_PIPE_PREVIEW_ENABLED=true; without it the preview controls explain why they are unavailable. Run the draft preview and inspect IN, OUT, and DIFF. Select an input to see whether it was retained, filtered, sampled out, split, combined, pending, or errored. For multiple outputs, select each output. When traces are available, compare the selected step with final output.

Preview runs a frozen batch of draft processing, with UTC as the implicit timezone. It excludes platform normalization, buffering, existing production sampling, and sinks. Stateful steps use isolated batch simulation: dedupe starts empty, and combine flushes partial groups at the sample's end. Step traces replay prefixes independently, so time or random functions can differ between stages.

Fix errors, choose Save draft, then Validate. Saving is required before validation and release review. A successful preview is evidence about the supplied sample, not a guarantee that every production event has the same shape.

5. Review and publish

Choose Review release. Inspect validation results and the configuration against the active version, then choose Publish release. Publishing requires log_sources:deploy. Publish re-reads the source and refuses if the saved draft changed since review. The release is live once Vector has loaded the new configuration; check release history and the source's health.

If reviewing a capture sample, saving captured events as the version's test samples is an explicit, unchecked-by-default choice. Those samples outlive the capture purge. Without that choice, the source's existing saved samples carry forward.

If another editor changes the source after your save or review, resolve the conflict and review the latest draft before publishing. Editing a reviewed draft makes the review stale.

Roll back a release

Open release history, find a release with a recorded restorable version, and choose Rollback. Review the confirmation, including any warning about draft changes, before requesting the rollback. Entries without a restorable version have rollback disabled. Check release history afterward to confirm which version is active.

Troubleshoot

Need help with these checks? Contact support with the deployment, request ID and time with timezone.

A release review becomes stale

Save the latest draft and review it again. If another editor changed the source, resolve the conflict before publishing.

A published release is not live

Check release history for the active version and the deploy error, if any. If Vector rejects a configuration, it keeps serving the last good one.

What's next

On this page