Step reference
Fields, behavior, examples, and limits for every pipeline step
Use this reference to configure processing steps and check their input, output, and limits. Every step has a Step name, an enabled flag, and an optional Execution condition (condition). A blank condition runs on every event; otherwise it must be a single-line VRL boolean expression. A false condition bypasses the step. The fields below include configuration keys in parentheses so JSON settings are unambiguous. Examples describe step configuration, not a separate API for creating pipelines.
Field paths may begin with a dot. Platform identity is protected: do not modify source_type, internal _nano* fields such as _nano_pipe_metadata and _nano_pipe_scope, or platform-owned source metadata such as metadata.log_source. Advanced VRL is also subject to server-side deploy-time guards.
Raw input commonly carries a string in message. Parse it before operating on vendor fields, or place operations after an existing parser that creates those fields. Processing errors follow the existing parser-failure restoration path; an error or VRL abort is not an intentional drop.
Parse
Fields: Format (format, currently only json) and Input field (field, default message).
Parses the field as a JSON object and merges its fields into the event root. Parsed values overwrite root fields with the same names. Invalid JSON, a non-string input, or a non-object result is an error.
Example: Input field message containing {"level":"info","user":{"email":"alice@example.com"}} creates level and user.email while preserving other root fields.
Rename field
Fields: Field path (field) and Target field (target).
Moves the value with target = del(field): removes the original field and assigns its value to the target, overwriting an existing target. A missing source can assign null; preview that case.
Example: field: "client_ip", target: "src_endpoint.ip" moves client_ip to src_endpoint.ip.
Remove field
Field: Field path (field).
Deletes the field. A missing field is harmless.
Example: field: "debug_payload" removes that payload before later processing.
Set field
Fields: Field path (field) and JSON value (value).
Assigns a static JSON value, replacing any existing value. Values can be strings, numbers, booleans, null, arrays, or objects.
Example: field: "unmapped.environment", value: "production" adds a fixed environment label.
Field operations
Fields: Operation (operation: rename, remove, set, or batch), Field path (field), Target field (target) for rename, JSON value (value) for set, and Field operations (operations) for a batch.
The first three operations have the semantics above. Multiple operations executes 1–64 operation objects in order, each with operation and the fields its operation requires.
Example: Choose Multiple operations and supply:
[
{"operation":"rename","field":"client_ip","target":"src_endpoint.ip"},
{"operation":"remove","field":"debug_payload"},
{"operation":"set","field":"unmapped.environment","value":"production"}
]Redact fields
Fields: Fields (fields, 1–64 paths) and Replacement (replacement, default [REDACTED], at most 1,024 bytes for text).
Assigns the replacement to each listed field. Replacement can create a field that was absent; it is whole-field replacement, not a regex search inside a string.
Example: Fields user.email, authorization, Replacement [REDACTED] replaces both values.
Mask or hash
Fields: Fields (fields, 1–64 paths), Operation (operation: replace, hash, or partial), Replacement (replacement), Hash algorithm (algorithm: sha256 or md5), Visible leading characters (keepStart), and Visible trailing characters (keepEnd).
- Redact replaces the whole value, as above.
- Hash requires strings and produces SHA-256 by default, or MD5 when selected. This is an unsalted deterministic hash; it is not encryption.
- Partial mask requires strings, keeps 0–64 leading and 0–64 trailing characters, and replaces each middle character. If the string is too short for the requested visible ends, it masks the entire string. The partial-mask replacement defaults to
*when omitted; set it explicitly in the editor.
Example: Fields user.email, Operation Partial mask, leading 1, trailing 0, Replacement * turns alice@example.com into a****************.
Filter
Fields: Predicate (predicate) and Matching events (action: drop or keep).
Drop discards matching events. Keep only discards non-matching events. The step's execution condition is separate: events that bypass the step are retained regardless of the predicate.
Example: Predicate .event.action == "heartbeat", Matching events Drop removes heartbeats. Intentional discards emit no output and do not use the parser error path.
Sample
Fields: Keep 1 in every N events (ratio, 1–1,000,000, default 10), Consistent sampling key (groupBy, default message), and Always keep condition (exempt, optional VRL boolean expression).
Hashes the JSON-encoded key deterministically and keeps the bucket selected by the ratio. Equal keys receive the same decision; this is an approximate fraction of traffic, not every Nth event in arrival order. The exemption retains matching events. A ratio of 1 keeps everything.
Example: Ratio 10, key session_id, exemption .severity == "critical" samples sessions consistently while keeping critical events.
Split events
Fields: Array field (field), Output field (as), Retained context fields (keep, up to 64 paths), and Maximum outputs per input (max, 1–1,000, default 50).
Emits one event per array member and puts that member at the output field. With no retained-context paths, each child keeps the parent context. With paths selected, each child starts with only those fields plus the output member. Later steps run on every child. An empty array produces no output; a non-array or an array exceeding the configured bound is an error, never silent truncation. The processing plan also bounds expanded output to 1,000.
Example: Array field items, Output field item, retained context request_id, maximum 50 turns {"request_id":"r1","items":[{"id":1},{"id":2}]} into two events carrying request_id and one item each.
Dedupe
Fields: Comparison fields (fields, 1–16 paths) and Maximum cached events (cacheEvents, default 1,000, effective maximum 5,000).
Drops repeated SHA-256 signatures of the selected fields using a bounded LRU cache. Signatures include value type and field presence, so missing and explicit null are distinct. There is no TTL setting. Larger saved cache limits are capped during compilation.
Example: Comparison fields message and cache 1000 discard another event with the same message while its signature remains cached.
Combine or summarize
Fields: Group keys (keys, 1–16 paths), Count field (countField, default unmapped.event_count), Merge rules (merge, up to 16 field-to-strategy mappings), Maximum events per group (maxEvents, 1–1,000, default 100), and Flush timeout (seconds) (timeoutSec, effective 1–5, default 5).
Groups events by key values and emits a reduced event at the event limit or idle timeout. At most 64 distinct group keys are tracked; overflow flushes the oldest group. Larger saved timeout values are capped at five seconds. Missing group keys behave as null. Unselected fields come from the last event in the group, and the count is written to the configured field.
| Merge strategy | Behavior |
|---|---|
retain | Last non-null value |
sum | Sum numeric values |
min / max | Minimum / maximum numeric value |
concat | Join strings with spaces |
concat_newline | Join strings with newlines |
array | Collect values, including null or missing values |
Other strategies skip null/missing values. Numeric and string strategies reject incompatible non-null inputs. The count field cannot also have a merge rule.
Example: Group keys src_endpoint.ip, count unmapped.event_count, merge rules {"traffic.bytes_in":"sum"}, maximum 100, timeout 5 summarize bytes per source IP. The output keeps the last member's remaining context.
State is per Vector process, clears on restart, and is not a cross-instance aggregate. Preview starts empty and flushes partial groups at the end of its sample; it does not reproduce production timing.
Route
Fields: Matching behavior (mode: first or fanout), Ordered routes (routes, 1–16 objects), Branch label field (field, default unmapped.pipe_route), and Unmatched events (fallback: keep or drop). Each route object has a unique label, a VRL predicate, and optional boolean drop.
First match uses the first matching branch. Fan out to all matches creates one copy per retained matching branch. Kept copies are tagged with the branch label and continue through the same pipeline to storage. A dropped branch emits nothing. Kept unmatched events receive the label unmatched. Labels use 1–64 letters, digits, underscores, or hyphens.
Example: Mode First match, field unmapped.pipe_route, fallback Keep, ordered routes:
[
{"label":"heartbeat","predicate":".event.action == \"heartbeat\"","drop":true},
{"label":"critical","predicate":".severity == \"critical\"","drop":false}
]Route only tags or drops within a log source's pipeline. It does not send to external destinations or move events to another source.
Lookup
Fields: Lookup name (table), Lookup version (version), Lookup snapshot rows (rows, 1–1,000), Match fields (match), Output mappings (outputs), and Missing match (missing: leave or error). Match and output mappings each contain 1–16 entries from table column names to event field paths.
Uses an immutable, case-sensitive, exact-match CSV snapshot pinned to the source release. Event match values are converted to strings. Rows must contain exactly the mapped columns, use string values bounded to 4,096 bytes, and have unique match-key combinations. Column names use letters, digits, or underscores. The lookup name uses 1–64 letters, digits, underscores, or hyphens; version is non-empty and at most 128 bytes.
Leave unchanged preserves an event with no match. Report error uses the processing error path. Customers supply rows and mappings, never filesystem paths or another source's table ID.
Example: Name asset_owners, version 1, match {"host":"device.hostname"}, outputs {"owner":"unmapped.owner"}, rows [{"host":"web-01","owner":"operations"}], missing Leave unchanged adds unmapped.owner="operations" for web-01.
Advanced VRL
Field: VRL code (code). Stored error behavior (onError) is pass; this runtime preserves parser failures and does not support another value.
Runs custom Vector Remap Language subject to platform validation. Local variables persist within consecutive stateless processing steps but do not cross a native dedupe, combine, route, or lookup stage. Put values in event fields when later stages need them. Source extensions run afterward in a separate scope.
Example: After parsing a JSON message:
.unmapped.environment = "production"Use Filter or Route for intentional drops. An abort follows parser-failure restoration and must not be treated as a drop policy. Preview and validate custom code before publishing.