Monitor your Self-run install
Watch nano's own health with Prometheus and Grafana on a single node, a role-split cluster or Kubernetes, and send nano's health alerts into nano.
nano can watch its own health with Prometheus and Grafana. It is optional and off by default. Turn it on, and you get:
- Prometheus, scraping every nano component and every host.
- Grafana, with five ready dashboards.
- Alert rules for the problems that matter: stalled ingest, a down ClickHouse replica, PostgreSQL or Valkey with no primary or two, disk pressure, and more.
- Optionally, nano's own health alerts sent into nano as events, so you can search them and run detections on them.
This is metrics only. If the monitoring stack stops, you lose dashboards. Ingest and search keep working.
nano's built-in SIEM Health page keeps working with or without this. Installs
running the enterprise edition can also send SIEM Health to any OTLP/HTTP
receiver: see docs/administration/health-export.md in the enterprise
release bundle. The open-source edition does not include that exporter.
A release bundle also includes this guide, at
docs/administration/monitor-self-run.md. A git checkout from get.nano.rs
has no docs/administration/, so use this page. Both have the dashboards and
alert rules under config/.
Pick the section for your install:
- Single node (Docker Compose)
- Several hosts (role-split Compose)
- Kubernetes (Helm)
- Your own Prometheus
Single node
Add monitoring to COMPOSE_PROFILES in .env, then rerun the installer:
cd ~/nano
current=$(sed -n 's/^COMPOSE_PROFILES=//p' .env | tail -n 1 | tr -d "\"'")
sed -i '/^COMPOSE_PROFILES=/d' .env
echo "COMPOSE_PROFILES=${current:+$current,}monitoring" >> .env
NANO_SKIP_GIT_UPDATE=1 ./install.shRerunning the installer can upgrade nano. If you installed with
get.nano.rs (a git checkout in ~/nano), a plain ./install.sh first
updates the checkout to the latest release, and then installs that release.
NANO_SKIP_GIT_UPDATE=1 keeps your current release. To upgrade at the same
time, leave it out. An extracted release bundle never updates itself.
The installer:
- generates
NANO_SERVICE_TOKEN(the bearer token Prometheus uses for/metrics) andNANO_GRAFANA_ADMIN_PASSWORDin.env, if they are missing; - sets
NANO_JOBS_BIND_ADDR=0.0.0.0, so Prometheus can reach nano-jobs on the install's private Compose network (/metricsstill needs the token); - keeps about 1 GiB of memory aside for the monitoring stack before it sizes the nano services.
Open Grafana through an SSH tunnel. It listens on 127.0.0.1:3030:
ssh -L 3030:127.0.0.1:3030 you@nano-host
# then browse to http://localhost:3030 and sign in as admin
grep '^NANO_GRAFANA_ADMIN_PASSWORD=' ~/nano/.envTo reach Grafana from your network instead, set
NANO_GRAFANA_BIND_ADDRESS to the host's private address in .env and rerun
the installer. Prometheus has no login, so it always stays on 127.0.0.1:9090.
Several hosts
Give one host the monitoring role in inventory.yaml. It can be a small
dedicated VM (2 vCPU, 4 GB RAM, 50 GB disk), or a host you already have, such
as a PostgreSQL/Keeper host or an app host. One monitoring host is enough even
on a fully highly available cluster. Only one host may have the role.
Dedicated:
hosts:
# ... your existing hosts, unchanged ...
- name: monitoring-1
id: monitoring-a
address: 10.20.0.40
roles: [monitoring]Or on an existing host:
- name: pg-1
id: pg-a
address: 10.20.0.20
roles: [postgres, keeper, monitoring]Adding, moving or removing the monitoring role does not change the cluster's durable identity. No data moves.
Then:
-
Copy the updated
inventory.yamland the shared.envto every host. On the first app host, rerun its role first: it addsNANO_SERVICE_TOKENandNANO_GRAFANA_ADMIN_PASSWORDto.envif they are missing. Copy that.envagain. -
Rerun
install.shwith each host's roles, as for any role change. Every host now runs node-exporter and publishes its exporters on its private address. App hosts serve search's and jobs' metrics through their private API port. -
Install the monitoring host last:
NANO_HOST_NAME=monitoring-1 ./install.sh --role monitoring --inventory /etc/nano/inventory.yaml -
Update the host firewall (next section, and Host firewall).
Grafana listens on port 3030 of the monitoring host, on its bind_address
(every interface unless the inventory sets one). The firewall script below
lets only your management network (MGMT_CIDR) reach it. Without that
script, restrict 3030 yourself, or set the host's bind_address to its
private address. Sign in as admin with NANO_GRAFANA_ADMIN_PASSWORD from
.env. Prometheus listens on that host's 127.0.0.1:9090 only.
The nano-deploy planner can place the role for you: plan.py --monitoring colocate (on a
PostgreSQL/Keeper host) or plan.py --monitoring dedicated (its own VM).
Host firewall
The monitoring host scrapes these ports on the other hosts. Open each one to
the monitoring host's address only, and Grafana to your management network
only. plan.py --firewall-script with a monitoring host in the plan writes
exactly these rules into the nano-docker-firewall unit (DOCKER-USER) and
the UFW rules it already generates. The ports marked "already open to cluster
hosts" are not narrowed: every cluster host reaches them today.
| Port | On | What | Firewall |
|---|---|---|---|
| 9100 | every host | node-exporter (host network) | UFW |
| 3000 | app hosts | api, search (/metrics/search) and jobs (/metrics/jobs) metrics, TLS, bearer token | already open to cluster hosts |
| 8405 | app and ingest hosts | clickhouse-proxy metrics | DOCKER-USER |
| 8406 | app hosts | ingest-proxy metrics | DOCKER-USER |
| 8407 | ingest hosts | app-api-proxy metrics (2+ app hosts) | DOCKER-USER |
| 8408 | app hosts | postgres-router metrics (PostgreSQL HA) | DOCKER-USER |
| 8409 | app hosts | valkey-router metrics (Valkey HA) | DOCKER-USER |
| 9598 | ingest hosts | Vector | DOCKER-USER |
| 9363 | ClickHouse hosts | ClickHouse | UFW |
| 9364 | Keeper hosts | Keeper | UFW |
| 8008 | PostgreSQL hosts | Patroni (HA), TLS | already open to cluster hosts |
| 2379 | etcd hosts | etcd (HA), TLS with the cluster client certificate | already open to cluster hosts |
| 6379, 26379 | PostgreSQL hosts | Valkey and Sentinel (HA), TLS with the cluster client certificate | already open to cluster hosts |
| 3030 | monitoring host | Grafana, for people | DOCKER-USER, MGMT_CIDR only |
Kubernetes
The chart can create Prometheus Operator objects. All three switches are off by default:
monitoring:
serviceMonitor:
enabled: true # api, search, Vector, ClickHouse; a PodMonitor for jobs
prometheusRule:
enabled: true # the alert rules
dashboards:
enabled: true # one ConfigMap per dashboard for the Grafana sidecarThis works with a stock kube-prometheus-stack, with no edits, if you installed
it under the release name kube-prometheus-stack. Its Prometheus picks up
objects labeled release: kube-prometheus-stack in every namespace, and its
Grafana sidecar loads ConfigMaps labeled grafana_dashboard: "1". Another
release name? Set monitoring.serviceMonitor.labels.release and
monitoring.prometheusRule.labels.release to it.
Turning on serviceMonitor also:
- moves nano-jobs' metrics listener off loopback (it still needs the token);
- turns on ClickHouse's Prometheus endpoint on port 9363, and adds a headless Service for it.
api, search and jobs need the bearer token in the chart Secret's
service_token key. The chart generates it. If you bring your own Secret
(secrets.existingSecret), add a random service_token to it.
Keeper, the bundled PostgreSQL and Valkey are not scraped on Kubernetes.
Your own Prometheus
Already running Prometheus and Grafana? Scrape nano directly. Whatever the install shape:
- Use the job names below (
nano-api,nano-search,nano-jobs,nano-vector,clickhouse,node, ...). The shipped dashboards and alert rules select on them. - Give every target a
hostlabel with the nano host's name, the same value for every job on that host. The dashboards' Host picker lists these values, and alerts name the host with them. On a role-split install, use the host'snamefrominventory.yaml. - api, search and jobs need the bearer token
NANO_SERVICE_TOKEN. It is in.envin the install directory (~/nano/.envby default; on a role-split install, the shared.env, which is the same on every host). The installer generates it. Give it to Prometheus as a file that only the Prometheus user can read, withauthorization.credentials_file. Never put it inprometheus.ymlor on a command line (see below). - Import the dashboards from
config/grafana/dashboards/(Grafana 11 or later, any Prometheus data source), and load the rules fromconfig/prometheus/rules/nanosiem_alerts.yml.
Give Prometheus the token
On the Prometheus host, copy the token straight from the nano host into a
file that only root and Prometheus's group can read (root:<group>, mode
0440). The token never appears on screen or in your shell history.
A packaged Prometheus (it runs as the prometheus user and group):
ssh you@nano-host "sed -n 's/^NANO_SERVICE_TOKEN=//p' ~/nano/.env | tail -n 1 | tr -d '\"'" \
| sudo install -m 0440 -o root -g prometheus /dev/stdin /etc/prometheus/nano-service-tokenPrometheus in a container (the stock prom/prometheus image runs as
nobody): make a dedicated group for the file, and add the container to it.
Do not give the file to nobody, which other services share.
sudo groupadd --system nano-metrics # once
sudo install -d -m 0755 /etc/prometheus
ssh you@nano-host "sed -n 's/^NANO_SERVICE_TOKEN=//p' ~/nano/.env | tail -n 1 | tr -d '\"'" \
| sudo install -m 0440 -o root -g nano-metrics /dev/stdin /etc/prometheus/nano-service-token
docker run ... --group-add "$(getent group nano-metrics | cut -d: -f3)" \
-v /etc/prometheus/nano-service-token:/etc/prometheus/nano-service-token:ro prom/prometheus ...(In Compose, the same is group_add: ["<gid>"] on the Prometheus service.)
If the token changes in .env, copy it again.
Your own Prometheus: single node
Needs nano v2.0.25 or later. A single-node install publishes no metrics ports by default: every service
talks to the others on the install's private Docker network. To let your own
Prometheus in, set NANO_METRICS_BIND_ADDRESS to the nano host's private
address (the one your Prometheus can reach), then rerun the installer:
cd ~/nano
echo 'NANO_METRICS_BIND_ADDRESS=10.0.0.12' >> .env # your host's private IPv4
NANO_SKIP_GIT_UPDATE=1 ./install.shThe installer checks the address first, and changes nothing if it is wrong. It must be:
- one of this host's own IPv4 addresses (
ip -4 -o addr show); - private: in 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or
127.0.0.0/8.
127.0.0.1works for a Prometheus on the same host (outside Docker). A public address needsNANO_METRICS_ALLOW_PUBLIC=1as well: set it only when a network firewall already limits these ports to your Prometheus.0.0.0.0is always refused.
This needs Docker Compose 2.24.4 or later (docker compose version).
The installer then:
- starts
metrics-edge, a small proxy that publishes these ports on that address only. Each port passes onlyGETandHEADrequests for/metricsto its service, so the rest of the API, search and ClickHouse stay private; - runs node-exporter, which otherwise runs only with the
monitoringprofile; - sets
NANO_JOBS_BIND_ADDR=0.0.0.0, so the proxy can reach nano-jobs on the private Docker network (/metricsstill needs the token); - records the setting in
.env. Later reruns,nanocommands andnano upgradekeep it, andnano upgradeupdates the proxy with the rest of nano.
| Job | Target | Auth |
|---|---|---|
nano-api | <address>:3000 | bearer token |
nano-search | <address>:3002 | bearer token |
nano-jobs | <address>:3003 | bearer token |
nano-vector | <address>:9598 | none |
clickhouse | <address>:9363 | none |
node | <address>:9100 | none |
All are plain HTTP on /metrics. Already run a node_exporter on the host?
Set NANO_METRICS_NODE_PORT in .env to another port, or scrape yours
instead.
To stop publishing, delete NANO_METRICS_BIND_ADDRESS from .env and rerun
the installer. It removes the proxy and node-exporter, and moves nano-jobs
back to loopback (unless the monitoring profile is on).
On a role-split install, NANO_METRICS_BIND_ADDRESS does nothing (the
installer warns): see several hosts.
Firewall. Docker publishes these ports in front of the host firewall, so
UFW rules do not apply to them. Allow only your Prometheus to reach them,
with your cloud firewall or security group. Or add DOCKER-USER rules on the
nano host. Replace both addresses; if you set NANO_METRICS_NODE_PORT, use
it instead of 9100. Persist the rules the way you persist your other iptables
rules.
for port in 3000 3002 3003 9598 9363 9100; do
sudo iptables -I DOCKER-USER -p tcp -m conntrack --ctorigdst 10.0.0.12 \
--ctorigdstport "$port" --ctdir ORIGINAL ! -s 10.0.0.50 -j DROP
doneThe rules match the host address and port a client connected to
(--ctorigdst, --ctorigdstport), because Docker has already translated
them to the container's address when DOCKER-USER runs. --ctdir ORIGINAL
keeps them off the replies, which carry the same connection details.
A working prometheus.yml (nano at 10.0.0.12, named nano-1). The
nano_role label says which part of nano a target is, as nano's own
Prometheus labels it.
global:
scrape_interval: 15s
rule_files:
- /etc/prometheus/rules/nanosiem_alerts.yml
scrape_configs:
- job_name: nano-api
authorization:
type: Bearer
credentials_file: /etc/prometheus/nano-service-token
static_configs:
- targets: ["10.0.0.12:3000"]
labels: {host: nano-1, nano_role: app}
- job_name: nano-search
authorization:
type: Bearer
credentials_file: /etc/prometheus/nano-service-token
static_configs:
- targets: ["10.0.0.12:3002"]
labels: {host: nano-1, nano_role: app}
- job_name: nano-jobs
authorization:
type: Bearer
credentials_file: /etc/prometheus/nano-service-token
static_configs:
- targets: ["10.0.0.12:3003"]
labels: {host: nano-1, nano_role: app}
- job_name: nano-vector
static_configs:
- targets: ["10.0.0.12:9598"]
labels: {host: nano-1, nano_role: ingest}
- job_name: clickhouse
static_configs:
- targets: ["10.0.0.12:9363"]
labels: {host: nano-1, nano_role: clickhouse, shard: "1", replica: "1"}
- job_name: node
static_configs:
- targets: ["10.0.0.12:9100"]
labels: {host: nano-1, nano_role: "app,clickhouse,ingest,postgres"}Copy config/prometheus/rules/nanosiem_alerts.yml from the nano host to
/etc/prometheus/rules/ on the Prometheus host. Then check that
Status > Targets shows all six UP.
Single node has no Valkey metrics for your own Prometheus (the bundled
monitoring profile scrapes Dragonfly itself). The Valkey panels stay empty.
Your own Prometheus: several hosts
On a role-split install, the exporter ports open only when the inventory has a monitoring host, and the firewall script opens them to that host only. So:
- Give the
monitoringrole to the host your Prometheus runs on, or to a host on its network (see Several hosts), and regenerate the firewall script. If you do not want nano's own Prometheus and Grafana, stop them on that host:docker compose stop prometheus grafana. - Start from the scrape config the installer renders on the monitoring
host:
.nano-generated/monitoring/prometheus.ymlin the install directory (JSON, which Prometheus also reads as YAML). It lists every target with itshostlabel. Change its file paths to yours: the token file, and the cluster CA (/etc/nano/tls/ca.crtthere: the filecluster.tls.canames ininventory.yaml). etcd also needs the cluster client certificate and key.
| Job | Target | Path | Auth |
|---|---|---|---|
nano-api | each app host :3000 | /metrics | bearer token, TLS with the cluster CA (server_name is cluster.tls.server_name) |
nano-search | each app host :3000 | /metrics/search | same |
nano-jobs | each app host :3000 | /metrics/jobs | same |
nano-vector | each ingest host :9598 | /metrics | none |
clickhouse | each ClickHouse host :9363 | /metrics | none |
keeper | each Keeper host :9364 | /metrics | none |
node | every host :9100 | /metrics | none |
haproxy | app and ingest hosts :8405-:8409 | /metrics | none |
patroni | each PostgreSQL host :8008 | /metrics | TLS with the cluster CA |
etcd | each etcd host :2379 | /metrics | TLS, cluster client certificate |
valkey, valkey-sentinel | a redis_exporter /scrape?target=rediss://host:6379 (or :26379) | TLS, cluster client certificate |
Without TLS on the private network (allow_plaintext_private_network), api,
search and jobs are plain HTTP on the same port and paths.
Dashboards
| Dashboard | Shows |
|---|---|
| nano Overview | Is everything up? Ingest rate, API traffic and latency, firing alerts, targets down. |
| nano API, Search and Jobs | Request rate, errors and latency per endpoint; search admission and queueing; the jobs leader; detection schedule lag. |
| nano ClickHouse and Ingest | Inserts, queries, merges, parts, replication queue and delay; Vector throughput, buffers, errors and drops. |
| nano Hosts | CPU, memory, disk and network per host. |
| nano HA | PostgreSQL primary and replication lag, etcd leader, Valkey master and Sentinel quorum, ClickHouse replicas and Keeper leader, router backends, the jobs leader. |
Every dashboard has a Host picker. Panels for parts you do not run (for example Patroni on a single PostgreSQL host) stay empty.
Alerts
The rules are in config/prometheus/rules/nanosiem_alerts.yml. Each has a
severity label: critical means act now, warning means look today.
| Alert | Fires when |
|---|---|
| NanoIngestStalled | Vector's sources receive events but its ClickHouse sinks send none for 10 minutes. Vector's internal sources and nano's own demo_logs reload markers do not count, so an idle install stays quiet (v2.0.24 and later). |
| NanoIngestDropping, NanoIngestBufferFilling, NanoIngestErrors | Vector loses events, its disk buffer passes 80%, or a component keeps failing. |
| NanoClickHouseReplicaDown, NanoClickHouseReadonlyReplica | A ClickHouse server is down, or has read-only tables (lost Keeper). |
| NanoClickHouseReplicationLag, NanoClickHouseReplicationQueue, NanoClickHouseTooManyParts | Replication or merges fall behind. |
| NanoKeeperDown | A Keeper is down. |
| NanoPostgresNoPrimary, NanoPostgresTwoPrimaries, NanoPostgresReplicaLag | PostgreSQL HA has no primary, two, or a lagging replica. |
| NanoEtcdNoLeader, NanoEtcdQuorumAtRisk | etcd lost its leader or its majority. |
| NanoValkeyNoMaster, NanoValkeyTwoMasters, NanoSentinelQuorumLost | Valkey HA has no master, two, or Sentinel lost its quorum. |
| NanoPostgresRouterNotOnePrimary, NanoRouterNoBackend, NanoRouterBackendDown | An app host's router sees no primary, two, or a dead peer. |
| NanoJobsNoLeader, NanoJobsTwoLeaders | No jobs instance, or two, hold the scheduler lease. |
| NanoApiDown, NanoSearchDown, NanoApiErrorRate, NanoApiLatency | The API or search is down, failing or slow. |
| NanoDiskPressure, NanoDiskCritical, NanoHostMemoryPressure | A disk passes 85% or 92% full, or memory runs low. |
| NanoTargetDown | Prometheus cannot scrape something. |
The bundled Alertmanager sends no notifications on its own. To be paged, use
self-ingest and a nano detection, or point Prometheus at your own
Alertmanager. The always-firing NanoHeartbeat alert (a separate rules file,
nano_heartbeat.yml) exists for self-ingest; dashboards leave it out.
Certificate expiry is not covered: no exporter here reads certificates. Use a blackbox exporter if you need it.
Send nano's health into nano (self-ingest)
Off by default. When on, the bundled Alertmanager posts every alert
notification to nano's HTTP ingest, as one JSON event: the alert names,
labels (such as host and severity), summaries and status
(firing or resolved). An alert is sent when it starts, every 4 hours while
it fires, and when it resolves. The NanoHeartbeat alert is re-sent every 5
minutes, so a detection can also tell when the reports stop.
-
In nano, create a log source for the events and issue an ingest token for it.
-
Add the token to
.env(the shared.envon a role-split install):NANO_SELF_INGEST_TOKEN=<the ingest token> -
Rerun the installer (role-split: the monitoring host's role). Alertmanager reads the token at start; without one it sends nothing.
Single node posts to the install's own Vector (http://vector:8080/ingest).
Role-split posts to the first ingest host's private address. Set
NANO_SELF_INGEST_URL (for example your ingest load balancer's /ingest
URL) to change that.
Search for the events with the log source you created, for example
"NanoHeartbeat" or "NanoIngestStalled". A useful detection: no
NanoHeartbeat event for 15 minutes.
Resources and retention
The monitoring stack has fixed, small limits so it cannot starve nano:
| Setting | Default |
|---|---|
NANO_PROMETHEUS_MEM_LIMIT / NANO_PROMETHEUS_CPU_LIMIT | 512M / 0.5 |
NANO_GRAFANA_MEM_LIMIT / NANO_GRAFANA_CPU_LIMIT | 256M / 0.5 |
NANO_ALERTMANAGER_MEM_LIMIT / NANO_ALERTMANAGER_CPU_LIMIT | 64M / 0.1 |
NANO_PROMETHEUS_RETENTION | 15d |
NANO_PROMETHEUS_RETENTION_SIZE | 4GB |
NANO_MONITORING_RESERVE_MIB | 1024 (memory set aside on a shared host) |
Set any of them in .env and rerun the installer.
Troubleshoot
A target is DOWN in Prometheus
Open Status > Targets in Prometheus. Check the host
firewall opens that port to the monitoring host. For api, search or jobs, a
401 means the token in Prometheus does not match NANO_SERVICE_TOKEN: copy
the shared .env again and rerun both hosts.
nano-jobs is DOWN on a single node
NANO_JOBS_BIND_ADDR must be
0.0.0.0 in .env. The installer sets it when the monitoring profile
is on or NANO_METRICS_BIND_ADDRESS is set; rerun it.
Your own Prometheus gets "connection refused" from a single node
NANO_METRICS_BIND_ADDRESS is not set, or is not the address Prometheus
connects to. Check with docker ps --filter name=nano-metrics-edge on the
nano host. A timeout instead means a firewall in between.
Grafana does not start
NANO_GRAFANA_ADMIN_PASSWORD is missing from
.env. Rerun the installer (on a role-split install, on the first app host),
then copy .env again.
The old Grafana password still works
The password changed in .env, but Grafana keeps the password it was first started with. Change it in Grafana.
What's next
- High availability: the role-split layout this page adds monitoring to.
- Kubernetes (Helm): the chart values for the monitoring switches.
- Deploy with an AI agent: the nano-deploy skill can set up monitoring for you.
Install on Kubernetes (Helm)
Install nano Self-run on your own Kubernetes cluster with the nano Helm chart, then size it, make it highly available and upgrade it.
Self-run
Start here if you bought nano Self-run: what it includes, your license values, installs per license, the minimum version, install methods and getting help.