Enrichment Architecture

How nano's enrichment system works internally: data flow, performance characteristics, and extensibility.

System Overview

The enrichment system updates source data on a schedule and swaps it into production atomically, so lookups never block during a refresh. It supports geolocation, threat intelligence, anonymizer detection, and custom enrichment sources.

Data Flow

Enrichment Data Sync

The sync process ensures enrichment data stays current while maintaining system availability:

Log Enrichment Flow

How individual logs get enriched during ingestion:

Custom Enrichment Data Flow

Custom enrichments follow a distinct path through the Deno sandbox:

1. Code Execution: TypeScript runs in a Deno sandbox
2. Data Storage: records are stored in the ClickHouse custom_enrichment_results table
3. Dictionary Refresh: the custom_enrichment_dict and custom_ioc_enrichment_dict dictionaries reload every 1-5 minutes (LIFETIME(MIN 60 MAX 300))
4. Log Enrichment: new logs are enriched with matching data at insert time

Performance Architecture

Zero-Downtime Updates

The staging table approach ensures continuous availability:

The update steps:

1. Download: new data is fetched in the background
2. Stage: data is loaded into a staging table
3. Validate: data integrity is verified
4. Swap: the production table is updated atomically
5. Cleanup: old data is removed

This ensures that:

• Log ingestion never stops
• Lookups always return results
• Updates are atomic and consistent

IP Lookups at Insert Time

IP enrichment is not a query-time JOIN. The nanosiem.logs table has 14 enriched_src_* / enriched_dest_* columns declared MATERIALIZED, each calling dictGetOrDefault against the ip_enrichment_dict dictionary when a row is inserted. The country/ASN values are written into the row, so search-time queries read a plain column with no lookup cost.

The dictionary uses an IP_TRIE layout keyed on CIDR networks, which gives longest-prefix matching directly in ClickHouse:

-- enriched_src_country, materialized on every logs insert
if(src_ip != '',
   if(isIPv4String(src_ip),
      dictGetOrDefault('nanosiem.ip_enrichment_dict', 'country', toIPv4OrDefault(src_ip), ''),
      dictGetOrDefault('nanosiem.ip_enrichment_dict', 'country', toIPv6OrDefault(src_ip), '')),
   '')

The dictionary reloads from its source table every 5-10 minutes (LIFETIME(MIN 300 MAX 600)); a sync updates the source table, and the next reload picks it up.

Database Schema

All enrichment data — and every lookup that reads it — lives entirely in ClickHouse: the payload tables plus the dictionaries that resolve them (moved off PostgreSQL in NAN-1114 / NAN-1117 so a cross-database read can't stall ingestion). PostgreSQL stores no enrichment data. It holds only the small enrichment_sources config row, sitting alongside the rest of the platform's metadata (auth, rules, tenants).

Source Config (PostgreSQL)

enrichment_sources is config/metadata only — one row per source describing where to fetch it, whether it's enabled, and how the last sync went. It carries no enrichment payload; that all lives in the ClickHouse tables below. (record_count is a denormalized mirror for display — the authoritative count is computed against ClickHouse.)

CREATE TABLE enrichment_sources (
    id VARCHAR PRIMARY KEY,
    name VARCHAR NOT NULL,
    source_type VARCHAR NOT NULL,
    description TEXT,
    download_url TEXT,
    last_sync_at TIMESTAMPTZ,
    last_sync_status VARCHAR,
    record_count BIGINT DEFAULT 0,
    config JSONB DEFAULT '{}',
    enabled BOOLEAN DEFAULT true,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

Enrichment Payload (ClickHouse)

ip_enrichments is a ReplacingMergeTree keyed by (source_id, network); it backs the ip_enrichment_dict dictionary. network stays a CIDR String because IP_TRIE builds its trie from the CIDR text. deleted tombstones CIDRs that a newer feed dropped.

CREATE TABLE nanosiem.ip_enrichments (
    network        String,
    source_id      LowCardinality(String) DEFAULT 'ipinfo_lite',
    country        String DEFAULT '',
    country_code   String DEFAULT '',
    continent      String DEFAULT '',
    continent_code String DEFAULT '',
    asn            String DEFAULT '',
    as_name        String DEFAULT '',
    as_domain      String DEFAULT '',
    updated_at     DateTime64(3) DEFAULT now64(3),
    deleted        UInt8 DEFAULT 0
)
ENGINE = ReplacingMergeTree(updated_at)
ORDER BY (source_id, network);

Lookup Dictionary

ip_enrichment_dict loads from ip_enrichments (deduped via argMax on updated_at, dropping tombstoned rows) and resolves the materialized columns:

CREATE DICTIONARY nanosiem.ip_enrichment_dict (
    network String,
    country String DEFAULT '',
    country_code String DEFAULT '',
    continent String DEFAULT '',
    continent_code String DEFAULT '',
    asn String DEFAULT '',
    as_name String DEFAULT '',
    as_domain String DEFAULT ''
)
PRIMARY KEY network
LIFETIME(MIN 300 MAX 600)
LAYOUT(IP_TRIE());

Disabling a source blanks the dictionary by deleting its rows from ip_enrichments; the next reload returns empty defaults for those CIDRs.

Extensibility Framework

Source Plugin Architecture

New enrichment sources implement standard interfaces:

pub trait EnrichmentSource {
    fn source_type(&self) -> &str;
    fn download(&self, config: &SourceConfig) -> Result<Vec<u8>>;
    fn parse(&self, data: &[u8]) -> Result<Vec<EnrichmentRecord>>;
    fn schema(&self) -> TableSchema;
}

pub trait EnrichmentLookup {
    fn lookup_single(&self, key: &str) -> Result<Option<EnrichmentResult>>;
    fn lookup_bulk(&self, keys: &[&str]) -> Result<HashMap<String, EnrichmentResult>>;
}

Current Implementations

Monitoring & Observability

Metrics Collection

Key metrics tracked by the enrichment system:

Health Checks

Automated monitoring ensures system reliability:

1. Sync Health

   • Last successful sync timestamp
   • Sync failure detection and alerting
   • Data freshness monitoring

2. Lookup Performance

   • Query latency percentiles
   • Cache effectiveness
   • Database connection health

3. Data Quality

   • Record count validation
   • Data integrity checks
   • Coverage analysis

Security Considerations

Data Protection

• URL Security: enrichment URLs contain tokens and are stored securely
• Access Control: API endpoints require appropriate permissions
• Data Validation: All input data validated before storage
• Audit Logging: All configuration changes logged

Network Security

• HTTPS Only: external downloads use encrypted connections
• SSRF validation: every download URL and allowed-domain entry is re-validated immediately before each fetch, rejecting loopback, RFC1918, and cloud-metadata addresses. The resolved IP is pinned for the connection and redirects are not followed.
• Rate Limiting: agent providers enforce per-minute and per-day caps locally
• Firewall Rules: restrict outbound connections as needed

Custom Enrichment Sandbox

Custom code runs in a restricted Deno sandbox:

• Network Restrictions: only allowed domains can be reached (max 10 per enrichment)
• No File System Access: cannot read or write local files
• Memory Limits: 1 GB V8 heap by default
• Execution Timeout: 600 seconds by default, configurable up to 1800 (MAX_TIMEOUT_SECS)
• Code size: 1 MB maximum (MAX_CODE_SIZE)
• Concurrency: at most 10 sandboxes run in parallel (MAX_CONCURRENT_SANDBOXES)