nano SIEM
Getting Started

Monitor your Self-run install

Supported in:Self-hosted

Watch nano's own health with Prometheus and Grafana on a single node, a role-split cluster or Kubernetes, and send nano's health alerts into nano.

nano can watch its own health with Prometheus and Grafana. It is optional and off by default. Turn it on, and you get:

  • Prometheus, scraping every nano component and every host.
  • Grafana, with five ready dashboards.
  • Alert rules for the problems that matter: stalled ingest, a down ClickHouse replica, PostgreSQL or Valkey with no primary or two, disk pressure, and more.
  • Optionally, nano's own health alerts sent into nano as events, so you can search them and run detections on them.

This is metrics only. If the monitoring stack stops, you lose dashboards. Ingest and search keep working.

nano's built-in SIEM Health page keeps working with or without this. Installs running the enterprise edition can also send SIEM Health to any OTLP/HTTP receiver: see docs/administration/health-export.md in the enterprise release bundle. The open-source edition does not include that exporter.

A release bundle also includes this guide, at docs/administration/monitor-self-run.md. A git checkout from get.nano.rs has no docs/administration/, so use this page. Both have the dashboards and alert rules under config/.

Pick the section for your install:

Single node

Add monitoring to COMPOSE_PROFILES in .env, then rerun the installer:

cd ~/nano
current=$(sed -n 's/^COMPOSE_PROFILES=//p' .env | tail -n 1 | tr -d "\"'")
sed -i '/^COMPOSE_PROFILES=/d' .env
echo "COMPOSE_PROFILES=${current:+$current,}monitoring" >> .env
NANO_SKIP_GIT_UPDATE=1 ./install.sh

Rerunning the installer can upgrade nano. If you installed with get.nano.rs (a git checkout in ~/nano), a plain ./install.sh first updates the checkout to the latest release, and then installs that release. NANO_SKIP_GIT_UPDATE=1 keeps your current release. To upgrade at the same time, leave it out. An extracted release bundle never updates itself.

The installer:

  • generates NANO_SERVICE_TOKEN (the bearer token Prometheus uses for /metrics) and NANO_GRAFANA_ADMIN_PASSWORD in .env, if they are missing;
  • sets NANO_JOBS_BIND_ADDR=0.0.0.0, so Prometheus can reach nano-jobs on the install's private Compose network (/metrics still needs the token);
  • keeps about 1 GiB of memory aside for the monitoring stack before it sizes the nano services.

Open Grafana through an SSH tunnel. It listens on 127.0.0.1:3030:

ssh -L 3030:127.0.0.1:3030 you@nano-host
# then browse to http://localhost:3030 and sign in as admin
grep '^NANO_GRAFANA_ADMIN_PASSWORD=' ~/nano/.env

To reach Grafana from your network instead, set NANO_GRAFANA_BIND_ADDRESS to the host's private address in .env and rerun the installer. Prometheus has no login, so it always stays on 127.0.0.1:9090.

Several hosts

Give one host the monitoring role in inventory.yaml. It can be a small dedicated VM (2 vCPU, 4 GB RAM, 50 GB disk), or a host you already have, such as a PostgreSQL/Keeper host or an app host. One monitoring host is enough even on a fully highly available cluster. Only one host may have the role.

Dedicated:

hosts:
  # ... your existing hosts, unchanged ...
  - name: monitoring-1
    id: monitoring-a
    address: 10.20.0.40
    roles: [monitoring]

Or on an existing host:

  - name: pg-1
    id: pg-a
    address: 10.20.0.20
    roles: [postgres, keeper, monitoring]

Adding, moving or removing the monitoring role does not change the cluster's durable identity. No data moves.

Then:

  1. Copy the updated inventory.yaml and the shared .env to every host. On the first app host, rerun its role first: it adds NANO_SERVICE_TOKEN and NANO_GRAFANA_ADMIN_PASSWORD to .env if they are missing. Copy that .env again.

  2. Rerun install.sh with each host's roles, as for any role change. Every host now runs node-exporter and publishes its exporters on its private address. App hosts serve search's and jobs' metrics through their private API port.

  3. Install the monitoring host last:

    NANO_HOST_NAME=monitoring-1 ./install.sh --role monitoring --inventory /etc/nano/inventory.yaml
  4. Update the host firewall (next section, and Host firewall).

Grafana listens on port 3030 of the monitoring host, on its bind_address (every interface unless the inventory sets one). The firewall script below lets only your management network (MGMT_CIDR) reach it. Without that script, restrict 3030 yourself, or set the host's bind_address to its private address. Sign in as admin with NANO_GRAFANA_ADMIN_PASSWORD from .env. Prometheus listens on that host's 127.0.0.1:9090 only.

The nano-deploy planner can place the role for you: plan.py --monitoring colocate (on a PostgreSQL/Keeper host) or plan.py --monitoring dedicated (its own VM).

Host firewall

The monitoring host scrapes these ports on the other hosts. Open each one to the monitoring host's address only, and Grafana to your management network only. plan.py --firewall-script with a monitoring host in the plan writes exactly these rules into the nano-docker-firewall unit (DOCKER-USER) and the UFW rules it already generates. The ports marked "already open to cluster hosts" are not narrowed: every cluster host reaches them today.

PortOnWhatFirewall
9100every hostnode-exporter (host network)UFW
3000app hostsapi, search (/metrics/search) and jobs (/metrics/jobs) metrics, TLS, bearer tokenalready open to cluster hosts
8405app and ingest hostsclickhouse-proxy metricsDOCKER-USER
8406app hostsingest-proxy metricsDOCKER-USER
8407ingest hostsapp-api-proxy metrics (2+ app hosts)DOCKER-USER
8408app hostspostgres-router metrics (PostgreSQL HA)DOCKER-USER
8409app hostsvalkey-router metrics (Valkey HA)DOCKER-USER
9598ingest hostsVectorDOCKER-USER
9363ClickHouse hostsClickHouseUFW
9364Keeper hostsKeeperUFW
8008PostgreSQL hostsPatroni (HA), TLSalready open to cluster hosts
2379etcd hostsetcd (HA), TLS with the cluster client certificatealready open to cluster hosts
6379, 26379PostgreSQL hostsValkey and Sentinel (HA), TLS with the cluster client certificatealready open to cluster hosts
3030monitoring hostGrafana, for peopleDOCKER-USER, MGMT_CIDR only

Kubernetes

The chart can create Prometheus Operator objects. All three switches are off by default:

monitoring:
  serviceMonitor:
    enabled: true    # api, search, Vector, ClickHouse; a PodMonitor for jobs
  prometheusRule:
    enabled: true    # the alert rules
  dashboards:
    enabled: true    # one ConfigMap per dashboard for the Grafana sidecar

This works with a stock kube-prometheus-stack, with no edits, if you installed it under the release name kube-prometheus-stack. Its Prometheus picks up objects labeled release: kube-prometheus-stack in every namespace, and its Grafana sidecar loads ConfigMaps labeled grafana_dashboard: "1". Another release name? Set monitoring.serviceMonitor.labels.release and monitoring.prometheusRule.labels.release to it.

Turning on serviceMonitor also:

  • moves nano-jobs' metrics listener off loopback (it still needs the token);
  • turns on ClickHouse's Prometheus endpoint on port 9363, and adds a headless Service for it.

api, search and jobs need the bearer token in the chart Secret's service_token key. The chart generates it. If you bring your own Secret (secrets.existingSecret), add a random service_token to it.

Keeper, the bundled PostgreSQL and Valkey are not scraped on Kubernetes.

Your own Prometheus

Already running Prometheus and Grafana? Scrape nano directly. Whatever the install shape:

  • Use the job names below (nano-api, nano-search, nano-jobs, nano-vector, clickhouse, node, ...). The shipped dashboards and alert rules select on them.
  • Give every target a host label with the nano host's name, the same value for every job on that host. The dashboards' Host picker lists these values, and alerts name the host with them. On a role-split install, use the host's name from inventory.yaml.
  • api, search and jobs need the bearer token NANO_SERVICE_TOKEN. It is in .env in the install directory (~/nano/.env by default; on a role-split install, the shared .env, which is the same on every host). The installer generates it. Give it to Prometheus as a file that only the Prometheus user can read, with authorization.credentials_file. Never put it in prometheus.yml or on a command line (see below).
  • Import the dashboards from config/grafana/dashboards/ (Grafana 11 or later, any Prometheus data source), and load the rules from config/prometheus/rules/nanosiem_alerts.yml.

Give Prometheus the token

On the Prometheus host, copy the token straight from the nano host into a file that only root and Prometheus's group can read (root:<group>, mode 0440). The token never appears on screen or in your shell history.

A packaged Prometheus (it runs as the prometheus user and group):

ssh you@nano-host "sed -n 's/^NANO_SERVICE_TOKEN=//p' ~/nano/.env | tail -n 1 | tr -d '\"'" \
  | sudo install -m 0440 -o root -g prometheus /dev/stdin /etc/prometheus/nano-service-token

Prometheus in a container (the stock prom/prometheus image runs as nobody): make a dedicated group for the file, and add the container to it. Do not give the file to nobody, which other services share.

sudo groupadd --system nano-metrics            # once
sudo install -d -m 0755 /etc/prometheus
ssh you@nano-host "sed -n 's/^NANO_SERVICE_TOKEN=//p' ~/nano/.env | tail -n 1 | tr -d '\"'" \
  | sudo install -m 0440 -o root -g nano-metrics /dev/stdin /etc/prometheus/nano-service-token
docker run ... --group-add "$(getent group nano-metrics | cut -d: -f3)" \
  -v /etc/prometheus/nano-service-token:/etc/prometheus/nano-service-token:ro prom/prometheus ...

(In Compose, the same is group_add: ["<gid>"] on the Prometheus service.) If the token changes in .env, copy it again.

Your own Prometheus: single node

Needs nano v2.0.25 or later. A single-node install publishes no metrics ports by default: every service talks to the others on the install's private Docker network. To let your own Prometheus in, set NANO_METRICS_BIND_ADDRESS to the nano host's private address (the one your Prometheus can reach), then rerun the installer:

cd ~/nano
echo 'NANO_METRICS_BIND_ADDRESS=10.0.0.12' >> .env    # your host's private IPv4
NANO_SKIP_GIT_UPDATE=1 ./install.sh

The installer checks the address first, and changes nothing if it is wrong. It must be:

  • one of this host's own IPv4 addresses (ip -4 -o addr show);
  • private: in 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or 127.0.0.0/8. 127.0.0.1 works for a Prometheus on the same host (outside Docker). A public address needs NANO_METRICS_ALLOW_PUBLIC=1 as well: set it only when a network firewall already limits these ports to your Prometheus. 0.0.0.0 is always refused.

This needs Docker Compose 2.24.4 or later (docker compose version).

The installer then:

  • starts metrics-edge, a small proxy that publishes these ports on that address only. Each port passes only GET and HEAD requests for /metrics to its service, so the rest of the API, search and ClickHouse stay private;
  • runs node-exporter, which otherwise runs only with the monitoring profile;
  • sets NANO_JOBS_BIND_ADDR=0.0.0.0, so the proxy can reach nano-jobs on the private Docker network (/metrics still needs the token);
  • records the setting in .env. Later reruns, nano commands and nano upgrade keep it, and nano upgrade updates the proxy with the rest of nano.
JobTargetAuth
nano-api<address>:3000bearer token
nano-search<address>:3002bearer token
nano-jobs<address>:3003bearer token
nano-vector<address>:9598none
clickhouse<address>:9363none
node<address>:9100none

All are plain HTTP on /metrics. Already run a node_exporter on the host? Set NANO_METRICS_NODE_PORT in .env to another port, or scrape yours instead.

To stop publishing, delete NANO_METRICS_BIND_ADDRESS from .env and rerun the installer. It removes the proxy and node-exporter, and moves nano-jobs back to loopback (unless the monitoring profile is on).

On a role-split install, NANO_METRICS_BIND_ADDRESS does nothing (the installer warns): see several hosts.

Firewall. Docker publishes these ports in front of the host firewall, so UFW rules do not apply to them. Allow only your Prometheus to reach them, with your cloud firewall or security group. Or add DOCKER-USER rules on the nano host. Replace both addresses; if you set NANO_METRICS_NODE_PORT, use it instead of 9100. Persist the rules the way you persist your other iptables rules.

for port in 3000 3002 3003 9598 9363 9100; do
  sudo iptables -I DOCKER-USER -p tcp -m conntrack --ctorigdst 10.0.0.12 \
    --ctorigdstport "$port" --ctdir ORIGINAL ! -s 10.0.0.50 -j DROP
done

The rules match the host address and port a client connected to (--ctorigdst, --ctorigdstport), because Docker has already translated them to the container's address when DOCKER-USER runs. --ctdir ORIGINAL keeps them off the replies, which carry the same connection details.

A working prometheus.yml (nano at 10.0.0.12, named nano-1). The nano_role label says which part of nano a target is, as nano's own Prometheus labels it.

global:
  scrape_interval: 15s

rule_files:
  - /etc/prometheus/rules/nanosiem_alerts.yml

scrape_configs:
  - job_name: nano-api
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/nano-service-token
    static_configs:
      - targets: ["10.0.0.12:3000"]
        labels: {host: nano-1, nano_role: app}
  - job_name: nano-search
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/nano-service-token
    static_configs:
      - targets: ["10.0.0.12:3002"]
        labels: {host: nano-1, nano_role: app}
  - job_name: nano-jobs
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/nano-service-token
    static_configs:
      - targets: ["10.0.0.12:3003"]
        labels: {host: nano-1, nano_role: app}
  - job_name: nano-vector
    static_configs:
      - targets: ["10.0.0.12:9598"]
        labels: {host: nano-1, nano_role: ingest}
  - job_name: clickhouse
    static_configs:
      - targets: ["10.0.0.12:9363"]
        labels: {host: nano-1, nano_role: clickhouse, shard: "1", replica: "1"}
  - job_name: node
    static_configs:
      - targets: ["10.0.0.12:9100"]
        labels: {host: nano-1, nano_role: "app,clickhouse,ingest,postgres"}

Copy config/prometheus/rules/nanosiem_alerts.yml from the nano host to /etc/prometheus/rules/ on the Prometheus host. Then check that Status > Targets shows all six UP.

Single node has no Valkey metrics for your own Prometheus (the bundled monitoring profile scrapes Dragonfly itself). The Valkey panels stay empty.

Your own Prometheus: several hosts

On a role-split install, the exporter ports open only when the inventory has a monitoring host, and the firewall script opens them to that host only. So:

  1. Give the monitoring role to the host your Prometheus runs on, or to a host on its network (see Several hosts), and regenerate the firewall script. If you do not want nano's own Prometheus and Grafana, stop them on that host: docker compose stop prometheus grafana.
  2. Start from the scrape config the installer renders on the monitoring host: .nano-generated/monitoring/prometheus.yml in the install directory (JSON, which Prometheus also reads as YAML). It lists every target with its host label. Change its file paths to yours: the token file, and the cluster CA (/etc/nano/tls/ca.crt there: the file cluster.tls.ca names in inventory.yaml). etcd also needs the cluster client certificate and key.
JobTargetPathAuth
nano-apieach app host :3000/metricsbearer token, TLS with the cluster CA (server_name is cluster.tls.server_name)
nano-searcheach app host :3000/metrics/searchsame
nano-jobseach app host :3000/metrics/jobssame
nano-vectoreach ingest host :9598/metricsnone
clickhouseeach ClickHouse host :9363/metricsnone
keepereach Keeper host :9364/metricsnone
nodeevery host :9100/metricsnone
haproxyapp and ingest hosts :8405-:8409/metricsnone
patronieach PostgreSQL host :8008/metricsTLS with the cluster CA
etcdeach etcd host :2379/metricsTLS, cluster client certificate
valkey, valkey-sentinela redis_exporter /scrape?target=rediss://host:6379 (or :26379)TLS, cluster client certificate

Without TLS on the private network (allow_plaintext_private_network), api, search and jobs are plain HTTP on the same port and paths.

Dashboards

DashboardShows
nano OverviewIs everything up? Ingest rate, API traffic and latency, firing alerts, targets down.
nano API, Search and JobsRequest rate, errors and latency per endpoint; search admission and queueing; the jobs leader; detection schedule lag.
nano ClickHouse and IngestInserts, queries, merges, parts, replication queue and delay; Vector throughput, buffers, errors and drops.
nano HostsCPU, memory, disk and network per host.
nano HAPostgreSQL primary and replication lag, etcd leader, Valkey master and Sentinel quorum, ClickHouse replicas and Keeper leader, router backends, the jobs leader.

Every dashboard has a Host picker. Panels for parts you do not run (for example Patroni on a single PostgreSQL host) stay empty.

Alerts

The rules are in config/prometheus/rules/nanosiem_alerts.yml. Each has a severity label: critical means act now, warning means look today.

AlertFires when
NanoIngestStalledVector's sources receive events but its ClickHouse sinks send none for 10 minutes. Vector's internal sources and nano's own demo_logs reload markers do not count, so an idle install stays quiet (v2.0.24 and later).
NanoIngestDropping, NanoIngestBufferFilling, NanoIngestErrorsVector loses events, its disk buffer passes 80%, or a component keeps failing.
NanoClickHouseReplicaDown, NanoClickHouseReadonlyReplicaA ClickHouse server is down, or has read-only tables (lost Keeper).
NanoClickHouseReplicationLag, NanoClickHouseReplicationQueue, NanoClickHouseTooManyPartsReplication or merges fall behind.
NanoKeeperDownA Keeper is down.
NanoPostgresNoPrimary, NanoPostgresTwoPrimaries, NanoPostgresReplicaLagPostgreSQL HA has no primary, two, or a lagging replica.
NanoEtcdNoLeader, NanoEtcdQuorumAtRisketcd lost its leader or its majority.
NanoValkeyNoMaster, NanoValkeyTwoMasters, NanoSentinelQuorumLostValkey HA has no master, two, or Sentinel lost its quorum.
NanoPostgresRouterNotOnePrimary, NanoRouterNoBackend, NanoRouterBackendDownAn app host's router sees no primary, two, or a dead peer.
NanoJobsNoLeader, NanoJobsTwoLeadersNo jobs instance, or two, hold the scheduler lease.
NanoApiDown, NanoSearchDown, NanoApiErrorRate, NanoApiLatencyThe API or search is down, failing or slow.
NanoDiskPressure, NanoDiskCritical, NanoHostMemoryPressureA disk passes 85% or 92% full, or memory runs low.
NanoTargetDownPrometheus cannot scrape something.

The bundled Alertmanager sends no notifications on its own. To be paged, use self-ingest and a nano detection, or point Prometheus at your own Alertmanager. The always-firing NanoHeartbeat alert (a separate rules file, nano_heartbeat.yml) exists for self-ingest; dashboards leave it out.

Certificate expiry is not covered: no exporter here reads certificates. Use a blackbox exporter if you need it.

Send nano's health into nano (self-ingest)

Off by default. When on, the bundled Alertmanager posts every alert notification to nano's HTTP ingest, as one JSON event: the alert names, labels (such as host and severity), summaries and status (firing or resolved). An alert is sent when it starts, every 4 hours while it fires, and when it resolves. The NanoHeartbeat alert is re-sent every 5 minutes, so a detection can also tell when the reports stop.

  1. In nano, create a log source for the events and issue an ingest token for it.

  2. Add the token to .env (the shared .env on a role-split install):

    NANO_SELF_INGEST_TOKEN=<the ingest token>
  3. Rerun the installer (role-split: the monitoring host's role). Alertmanager reads the token at start; without one it sends nothing.

Single node posts to the install's own Vector (http://vector:8080/ingest). Role-split posts to the first ingest host's private address. Set NANO_SELF_INGEST_URL (for example your ingest load balancer's /ingest URL) to change that.

Search for the events with the log source you created, for example "NanoHeartbeat" or "NanoIngestStalled". A useful detection: no NanoHeartbeat event for 15 minutes.

Resources and retention

The monitoring stack has fixed, small limits so it cannot starve nano:

SettingDefault
NANO_PROMETHEUS_MEM_LIMIT / NANO_PROMETHEUS_CPU_LIMIT512M / 0.5
NANO_GRAFANA_MEM_LIMIT / NANO_GRAFANA_CPU_LIMIT256M / 0.5
NANO_ALERTMANAGER_MEM_LIMIT / NANO_ALERTMANAGER_CPU_LIMIT64M / 0.1
NANO_PROMETHEUS_RETENTION15d
NANO_PROMETHEUS_RETENTION_SIZE4GB
NANO_MONITORING_RESERVE_MIB1024 (memory set aside on a shared host)

Set any of them in .env and rerun the installer.

Troubleshoot

A target is DOWN in Prometheus

Open Status > Targets in Prometheus. Check the host firewall opens that port to the monitoring host. For api, search or jobs, a 401 means the token in Prometheus does not match NANO_SERVICE_TOKEN: copy the shared .env again and rerun both hosts.

nano-jobs is DOWN on a single node

NANO_JOBS_BIND_ADDR must be 0.0.0.0 in .env. The installer sets it when the monitoring profile is on or NANO_METRICS_BIND_ADDRESS is set; rerun it.

Your own Prometheus gets "connection refused" from a single node

NANO_METRICS_BIND_ADDRESS is not set, or is not the address Prometheus connects to. Check with docker ps --filter name=nano-metrics-edge on the nano host. A timeout instead means a firewall in between.

Grafana does not start

NANO_GRAFANA_ADMIN_PASSWORD is missing from .env. Rerun the installer (on a role-split install, on the first app host), then copy .env again.

The old Grafana password still works

The password changed in .env, but Grafana keeps the password it was first started with. Change it in Grafana.

What's next

On this page