High availability
Run nano Self-run on three Linux servers so event storage, search and ingest survive the loss of one server.
Run nano Self-run on three servers so that losing one doesn't lose events or stop search. This page is the standard three-node layout with Docker Compose; for Kubernetes, see Highly available values in the Helm guide, with measured failover behavior on EKS, GKE and AKS. Larger layouts are supported too; see Scaling past three nodes. For one server, see Install with Docker Compose. For the plan itself, see Self-run.
What stays up
| If you lose | What happens |
|---|---|
nano-2 or nano-3 | Nothing visible. Ingest and search continue on the other server, with no data loss |
nano-1 | The UI, search and alerting stop until it's back. Stored events are safe. Senders get 503 and must retry |
| Any two servers | ClickHouse goes read-only until one server is back |
nano-1 runs the app and PostgreSQL, so protect it with hypervisor HA (Proxmox HA, VMware HA or your cloud's auto-recovery) and tested backups. To keep the UI, search and alerting up when an app server is lost, run two or more app servers. To make PostgreSQL fail over too, run three postgres servers.
Before you start
For about 12.5 GB/day (about 150 events per second), you need three identical servers and a load balancer:
| Server | Runs | vCPU | RAM | Data disk (SSD) |
|---|---|---|---|---|
nano-1 | UI, API, search, PostgreSQL | 8 | 32 GB | 250 GB |
nano-2 | Event storage (copy 1), ingest | 8 | 32 GB | 1.5 TB for a year, or 500 GB for 90 days |
nano-3 | Event storage (copy 2), ingest | 8 | 32 GB | Same as nano-2 |
| Load balancer | TLS for both hostnames | 2 | 2 GB | — |
Each server also needs about 100 GB for the OS and Docker. With Docker 29 and later, image layers go to the containerd image store under /var/lib/containerd, not /var/lib/docker: a data disk mounted at /var/lib/docker holds the volumes (ClickHouse, PostgreSQL, Keeper, Vector buffers), so keep room for the images (about 15 GB per server) on the disk that holds /var/lib/containerd, or mount the data disk for both. The CPU, operating system and software are the same as for a single node. For the minimum per server and for other layouts, see Server sizing.
You also need:
-
The bundle, a registry login and the three enrollment values from app.nano.rs, as for a single node. An engineer or agent who holds only the license values can download the bundle with them; see With the license values.
-
A private network between the three servers. Put them on different physical hosts: three VMs on one hypervisor aren't highly available.
-
Two DNS names that point at the load balancer: one for the UI (
nano.example.com) and one for senders (ingest.nano.example.com). -
A private TLS certificate for traffic between the servers. It must be valid for all three private addresses and one private name, and you need its CA certificate.
-
A load balancer that terminates public TLS and forwards plain HTTP:
nano.example.comtonano-1on port 80;ingest.nano.example.comtonano-2andnano-3on port 80;- health checks on
GET /healthz, andX-Forwarded-Forreplaced with the client's address.
Use a managed load balancer, or run your own: two small Linux hosts with HAProxy and keepalived, sharing one floating IP that both DNS names point at. The HAProxy example and the keepalived example are a complete pair. With no floating IP, see Load balancer options. The HAProxy package from your distribution works: 2.6 or later, which Debian 12 (2.6) and Ubuntu 24.04 (2.8) ship. Install both with
sudo apt install haproxy keepalived psmiscorsudo dnf install haproxy keepalived psmisc, then checkhaproxy -v. -
Host firewall rules on every server, before the first install. Docker-published ports bypass UFW. See Host firewall.
Server sizing
This table is the recommended size for each server in a production multi-server install, up to about 100 GB/day.
| Server | vCPU | RAM | Data disk (SSD) |
|---|---|---|---|
| ClickHouse, with ingest on the same server (the default up to about 100 GB/day) | 8 | 32 GB | daily volume × hot days ÷ 5 × 1.15 ÷ 0.7, plus 100 GB for Docker images, logs and system tables |
app | 4 | 16 GB | 50 GB |
postgres and keeper (also etcd, Valkey and Sentinel with three postgres servers) | 4 | 16 GB | 100 GB |
app, postgres and keeper on one server (nano-1 in the three-node layout) | 8 | 32 GB | 250 GB |
| Ingest on its own server (above about 100 GB/day) | 4 | 16 GB | 128 GB |
| HAProxy host | 2 | 2 GB | — |
- Hot days is how long events stay on local disk. Older events can move to object storage with storage tiering. At 12.5 GB/day, including the 100 GB, a year on local disk is about 1.5 TB per ClickHouse server, 90 days about 500 GB and 30 days about 250 GB.
- A VM sold as 16 GB reports about 15 GiB to Linux. That counts as 16 GB here.
- Above about 100 GB/day, use Sizing by daily volume. The nano-deploy planner raises these figures for your volume.
- The table gives recommended sizes. The nano-deploy planner and preflight enforce lower minimums: for example, a ClickHouse server with ingest on it can be 6 vCPU and 28 GiB at about 12.5 GB/day. A server at the minimum has less headroom for growth, and for carrying the load of a failed peer.
- 4 vCPU and 16 GB on every server is enough for evaluation and functional tests only. The planner's
--laboption plans such a cluster and labels the plan NOT FOR PRODUCTION.
Prepare the data disk
Mount each server's data disk at /var/lib/docker before you install Docker, so Docker's volumes land on it. Mount it by UUID with nofail, and make Docker depend on it. nofail lets the server boot when the disk doesn't mount (a slow device, a check that stops); without the dependency, Docker then starts on an empty /var/lib/docker on the root disk and reports itself running, with none of the server's containers or data. With it, Docker waits for the disk and doesn't start without it:
# On a new, empty disk only: check with lsblk -f first.
D=/dev/<data disk>
sudo mkfs.ext4 -L nano-data "$D"
U=$(sudo blkid -s UUID -o value "$D")
sudo mkdir -p /var/lib/docker
echo "UUID=$U /var/lib/docker ext4 defaults,noatime,nofail,x-systemd.device-timeout=90s,x-systemd.mount-timeout=300s 0 2" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload && sudo mount /var/lib/docker
# Docker and containerd wait for the disk and never start without it:
for unit in docker containerd; do
sudo mkdir -p /etc/systemd/system/$unit.service.d
printf '[Unit]\nRequiresMountsFor=/var/lib/docker\n' | sudo tee /etc/systemd/system/$unit.service.d/nano-data-disk.conf
done
sudo systemctl daemon-reloadThe dependency keeps Docker off the root disk, but on its own it can leave a server down. After a hard reset, the disk's /dev/disk/by-uuid link can be missing: the mount waits for a device that never appears, times out, and Docker correctly stays stopped until someone mounts the disk by hand. Add a small timer that retries during the first 30 minutes after boot. From v2.0.12, the installer installs it on a server whose Docker data is on a separate nofail disk: with passwordless sudo it does so itself, otherwise it prints the one command to run (sudo sh scripts/lib/data-disk-heal.sh install /var/lib/docker, from ~/nano). On v2.0.11 and earlier, install it yourself. While the disk is unmounted, it asks udev to re-scan block devices and starts the disk's systemd mount unit, so the filesystem check and the unit ordering still apply (it never calls mount directly). It starts Docker only once the disk is mounted, and does nothing while the disk's filesystem check (systemd-fsck@) has failed, because that needs you. It never formats anything:
sudo tee /usr/local/sbin/nano-data-disk-heal >/dev/null <<'EOF'
#!/bin/sh
# nano data-disk self-heal: mount the data disk and start Docker if boot missed it.
MOUNT=${2:-/var/lib/docker}
[ "${1:-}" = heal ] || { echo "usage: $0 heal [MOUNT_POINT]" >&2; exit 2; }
mountpoint -q "$MOUNT" && exit 0
awk -v m="$MOUNT" '$1 !~ /^#/ && $2 == m { f = 1 } END { exit !f }' /etc/fstab || exit 0
[ "$(cut -d. -f1 /proc/uptime)" -le 1800 ] || exit 0
unit=$(systemd-escape -p --suffix=mount "$MOUNT")
for dep in $(systemctl show -p Requires -p After --value "$unit"); do
case "$dep" in
systemd-fsck@*.service)
[ "$(systemctl show -p ActiveState --value "$dep")" = failed ] && { echo "$dep failed: check the filesystem by hand"; exit 0; } ;;
esac
done
udevadm trigger --subsystem-match=block --action=change
udevadm settle --timeout=60
systemctl reset-failed "$unit"
{ systemctl start "$unit" && mountpoint -q "$MOUNT"; } || exit 0
systemctl reset-failed containerd.service docker.socket docker.service
systemctl start containerd.service docker.socket docker.service
EOF
sudo chmod 0755 /usr/local/sbin/nano-data-disk-heal
printf '[Unit]\nDescription=nano: mount the data disk and start Docker if boot missed it\nAfter=local-fs.target\nConditionPathExists=/usr/local/sbin/nano-data-disk-heal\n\n[Service]\nType=oneshot\nExecStart=/usr/local/sbin/nano-data-disk-heal heal /var/lib/docker\n' \
| sudo tee /etc/systemd/system/nano-data-disk-heal.service
printf '[Unit]\nDescription=nano: retry the data disk mount after boot\n\n[Timer]\nOnBootSec=120s\nOnUnitInactiveSec=30s\nAccuracySec=5s\n\n[Install]\nWantedBy=timers.target\n' \
| sudo tee /etc/systemd/system/nano-data-disk-heal.timer
sudo systemctl daemon-reload && sudo systemctl enable --now nano-data-disk-heal.timerThis is the same logic and the same units as scripts/lib/data-disk-heal.sh in the v2.0.12 bundle and the nano-deploy skill's scripts/data-disk-heal.sh, which also install and remove them (sudo sh <script> install or uninstall). x-systemd.device-timeout=90s bounds the boot's wait for the device, the timer's first run comes after it (120 s), and x-systemd.mount-timeout=300s gives a large disk time to mount after an unclean stop. A LABEL= entry behaves the same as UUID=; a /dev/vdX path is stable only where the hypervisor keeps the device order, so prefer the UUID.
Before disk maintenance within 30 minutes of a boot (unmounting /var/lib/docker on purpose, resizing, a filesystem check), stop the timer, or it mounts the disk again and starts Docker: sudo systemctl stop nano-data-disk-heal.timer. It starts again at the next boot. To remove it:
sudo systemctl disable --now nano-data-disk-heal.timer
sudo rm /etc/systemd/system/nano-data-disk-heal.timer /etc/systemd/system/nano-data-disk-heal.service /usr/local/sbin/nano-data-disk-heal
sudo systemctl daemon-reloadIf a server comes back with Docker stopped (findmnt /var/lib/docker prints nothing), mount the disk through its systemd unit (which checks the filesystem first), check that it holds the server's data, then start Docker:
sudo udevadm trigger --subsystem-match=block --action=change && sudo udevadm settle --timeout=60
sudo systemctl reset-failed var-lib-docker.mount; sudo systemctl start var-lib-docker.mount && findmnt /var/lib/docker
sudo ls /var/lib/docker/volumes | head # this server's volumes, not an empty directory
sudo systemctl reset-failed docker.service containerd.service; sudo systemctl start dockerIf the mount unit fails because its filesystem check failed (systemctl --failed lists systemd-fsck@…), run sudo fsck /dev/disk/by-uuid/<uuid> on the unmounted disk, then start the mount unit again.
On a server that already has a data disk, add the two drop-ins and the timer. From v2.0.10 the installer refuses to run while the data disk listed in /etc/fstab for Docker's directory isn't mounted. It warns about a missing drop-in only for a nofail or noauto entry: a plain partition, such as a separate /var, is already required at boot. If the disk is mounted some other way, for example by a systemd .mount unit, run the installer with NANO_DATA_MOUNT=<mount point> so it checks that mount, or with NANO_DATA_MOUNT=none to skip the check.
Install
Run every command as an ordinary user in the docker group, not with sudo, unless the command shows sudo.
Unpack the bundle on every server
On nano-1, nano-2 and nano-3, verify and unpack the bundle, log in to the registry and select the customer images, exactly as in single-node step 2. Verify the signature on every server, or verify once and copy the checked files over a trusted path:
VERSION=<version> # the release shown on your deployment's page
tar -xzf "nano-compose-enterprise-$VERSION.tar.gz" -C "$HOME" && mv "$HOME/nano-$VERSION" "$HOME/nano"
export NANO_IMAGES_LOCK=images.enterprise.customer.lockWrite the inventory and stage the certificate
Save this as inventory.yaml. Change the addresses, hostnames and deployment_id, which must be your deployment ID from app.nano.rs:
version: 1
cluster:
name: production
deployment_id: deploy_01jxxxxxxxxxxxxxxxxxxxxxxx # from app.nano.rs; can't change later
tls:
cert: /var/lib/nano/tls/staged/server.crt
key: /var/lib/nano/tls/staged/server.key
ca: /var/lib/nano/tls/staged/ca.crt
server_name: nano-private.example.com
trusted_proxy_cidrs: [10.20.0.5/32] # your load balancer's private address
app_hostname: nano.example.com
ingest_hostname: ingest.nano.example.com
hosts:
- name: nano-1
id: nano-a
address: 10.20.0.11
roles: [app, postgres, keeper]
keeper: {"id": 1}
- name: nano-2
id: nano-b
address: 10.20.0.12
roles: [clickhouse, keeper, ingest]
clickhouse: {"shard": 1, "replica": 1}
keeper: {"id": 2}
- name: nano-3
id: nano-c
address: 10.20.0.13
roles: [clickhouse, keeper, ingest]
clickhouse: {"shard": 1, "replica": 2}
keeper: {"id": 3}On every server, install it to /etc/nano, then stage the private certificate. The source certificate files must be owned by root, with the key at mode 0600:
sudo install -d -m 0755 /etc/nano
sudo install -m 0600 -o "$USER" -g "$(id -gn)" inventory.yaml /etc/nano/inventory.yaml
cd "$HOME/nano"
sudo ./scripts/nano-tls-stage --cert /root/nano-pki/server.crt --key /root/nano-pki/server.key \
--ca /root/nano-pki/ca.crt --output /var/lib/nano/tls/staged --installer-user "$USER"Then log out and back in, so your new nano-tls group membership applies.
Prepare on nano-1 and add your license
On nano-1:
cd "$HOME/nano"
export NANO_BASE_URL=https://nano.example.com
export NANO_ADMIN_EMAIL=secops-admin@example.com NANO_ADMIN_NAME='SecOps Admin'
printf 'Admin password: '; stty -echo; read -r NANO_ADMIN_PASSWORD; stty echo; echo; export NANO_ADMIN_PASSWORD
NANO_HOST_NAME=nano-1 NANO_PREPARE_ONLY=1 ./install.sh --role app,postgres,keeper --inventory /etc/nano/inventory.yaml
cat >> .env <<'EOF'
LICENSE_URL=https://api.nano.rs/api/license/status
LICENSE_TOKEN=<your enrollment token>
NANO_DEPLOYMENT_ID=<your deployment ID>
EOFThe installer asks the same questions as on a single node, including the corporate proxy.
For a large install, also add your planned volume to the same .env before you copy it. Every server reads it from the shared .env and sizes its own role from it. The planned-volume tiers change nothing unless you set it:
cat >> .env <<'EOF'
NANO_PLANNED_GB_PER_DAY=1000
NANO_AVG_EVENT_BYTES=1000
EOFNANO_PLANNED_GB_PER_DAY is your planned daily volume in whole GB. NANO_AVG_EVENT_BYTES is your average event size: about 300 for firewall and flow, 1000 for mixed feeds, 2000 for EDR. Every server must use the same values. The installer doesn't compare servers with each other, so copy the same .env to every server and check that sha256sum ~/nano/.env matches everywhere. See Plan for your daily volume for what each tier changes.
Copy .env to the other servers
Copy ~/nano/.env from nano-1 to ~/nano/.env on nano-2 and nano-3, over SSH or your approved secret-transfer path. Keep it owned by your user, mode 0600. Don't send it through chat or tickets.
.env holds the shared secrets and settings, so every server needs the same copy. One rule covers every case: after any installer run that changed .env, copy that server's .env to every other server, the first app server included, before you run the installer anywhere else. Two runs change it:
- the first app server's: preparing, adding a server, and other topology changes;
- with three
postgresservers,./install.sh --record-pg-ha-identityon the firstpostgresserver in the inventory, after all three are installed (see PostgreSQL servers). It records the cluster's identity (NANO_PG_HA_SYSTEM_ID) and prints "The shared .env changed on this host (NANO_PG_HA_SYSTEM_ID)". Copy that server's.enveverywhere. Copying the first app server's older.envover it would drop the identity. The otherpostgresservers may have recorded the same line in the same place already; the copy leaves theirs unchanged.
When unsure, compare sha256sum ~/nano/.env on every server and copy from the one with the newest change. From v2.0.10 the installer, and from v2.0.15 scripts/nano upgrade, set every key on the line where it already is, so copies stay identical. Only a key the file doesn't have yet is appended. (In v2.0.13 the upgrade moved NANO_VERSION to the end of the file.) A key you assigned twice is collapsed to the value in effect, at its first line, and both tools name the collapsed keys, never values: Collapsed duplicate .env line(s) for <keys> (kept the value in effect). Any run on any server that changes the shared .env ends with "The shared .env changed" and the names of the keys it changed (never their values). With a role split, a sizing key such as NANO_NGINX_MEM_LIMIT in the shared .env overrides every server's own sizing in .nano-role.env, so leave per-server sizing to the installer. Copy that server's .env to every other server before the next run anywhere. Earlier releases could append a duplicate line (for example a second NANO_LICENSE_MODE) or move lines without saying so. Never copy .nano-role.env or .nano-generated/: they are per server and rebuilt on every run.
The deployment ID is an identifier, not a secret; it appears in installer output and in the inventory. The enrollment token (LICENSE_TOKEN) is the secret.
Start the servers in order
Run each command on the server named in its comment, top to bottom:
cd "$HOME/nano"
# 1. Data first
NANO_HOST_NAME=nano-1 ./install.sh --role postgres,keeper --inventory /etc/nano/inventory.yaml # on nano-1
NANO_HOST_NAME=nano-2 ./install.sh --role clickhouse,keeper --inventory /etc/nano/inventory.yaml # on nano-2
NANO_HOST_NAME=nano-3 ./install.sh --role clickhouse,keeper --inventory /etc/nano/inventory.yaml # on nano-3
# 2. The app, in the shell from step 3
NANO_HOST_NAME=nano-1 ./install.sh --role app,postgres,keeper --inventory /etc/nano/inventory.yaml # on nano-1
# 3. Ingest last
NANO_HOST_NAME=nano-2 ./install.sh --role clickhouse,keeper,ingest --inventory /etc/nano/inventory.yaml # on nano-2
NANO_HOST_NAME=nano-3 ./install.sh --role clickhouse,keeper,ingest --inventory /etc/nano/inventory.yaml # on nano-3The app run creates the database schema on both storage servers and your admin account. From now on, always rerun a server with its full role list, as in the last three lines.
When every server is installed, log out of the registry on each one: docker logout us-central1-docker.pkg.dev. Docker stores the pull credential unencrypted in ~/.docker/config.json. Log in again with a fresh credential before every rerun or upgrade, or use a Docker credential helper.
Verify it works
curl -fsS https://nano.example.com/api/health
curl -fsS https://ingest.nano.example.com/healthz
curl -o /dev/null -w '%{http_code}\n' https://nano.example.com/ingest # must print 404Sign in at https://nano.example.com. SIEM Health shows both storage replicas and the Keeper quorum as healthy. Issue an ingest token under Ingestion → Log Sources → Credentials, send a test event to https://ingest.nano.example.com/ingest and search for it.
Next steps
- Set up your first feed, sending to
ingest.nano.example.com - Set up backups
- Plan retention and storage tiering
Prefer to have a coding agent run the install? The nano-deploy agent skill drives this layout and checks each step.
Make senders retry
If nano-1 is down, every ingest node reports not ready about 15 seconds later, so the load balancer stops ingest until nano-1 is back. Configure senders to buffer to disk and retry on 429 and 5xx. Keep nano-1's recovery fast with hypervisor HA and monitoring on https://nano.example.com/api/health.
Test failover
Do this before you go live, and again after any change to the load balancer or the network. Use a hard power-off from your hypervisor or cloud console, not a clean shutdown, because that is what a real failure looks like. Without console access, sudo systemctl stop docker.socket docker on the server is a usable substitute, but softer: the containers stop cleanly and the network stays up, so it doesn't test a lost network path.
Some limits look like failures in a test. Plan your probes around them:
- Sign-in is rate limited. nano allows 100 sign-in requests per client address in 5 minutes, successful ones included, and answers
429with aRetry-Afterheader (up to 300 seconds) above that. Failed sign-ins also slow down per account: after 5 failures for one account, from any address, each further attempt waits up to 30 seconds or is refused, until the failures stop for a minute. A probe that signs in every second hits the address limit, and one that uses a wrong password hits the delay. Sign in once and reuse the session, or use an API key. - A new ingest token takes a few seconds to reach every ingest server, so the first requests with it can get
401. Wait about 10 seconds after creating it. From v2.0.15 the token's create response says so (active_within_secs), and a401from an ingest server carries anX-Nano-Retry-Hintheader: retry before you treat the token as wrong. - HEC requests are rate limited per token and per sender. See Request limits.
- After a hard reset, check that each server's data disk is mounted (
findmnt /var/lib/docker) before you read anything else. A data disk that fails to mount at boot leaves Docker running empty on the root disk. See Prepare the data disk.
-
Send a steady stream of numbered test events from a machine outside the cluster, and leave it running:
i=0; while sleep 1; do i=$((i+1)) curl -sS -o /dev/null -w "$i %{http_code}\n" -X POST "https://ingest.nano.example.com/ingest" \ -H "Authorization: Bearer $NANO_INGEST_TOKEN" -H "Content-Type: application/json" \ -d "{\"message\": \"failover-test seq=$i\"}" done | tee failover-test.log -
Power off
nano-2. Within about 10 seconds, the load balancer marks it down. Check thathttps://nano.example.comstill loads, a search forfailover-testreturns new events, and the stream keeps printing200. SIEM Health shows one storage replica down. -
Power
nano-2back on. Wait until SIEM Health shows both replicas and the Keeper quorum healthy again. -
Repeat steps 2 and 3 for
nano-3. -
Power off the load balancer that holds the floating IP, if you run your own pair. The other one takes the address within about 3 seconds. The stream may print a few failed requests while connections reset, then
200again. Power it back on. It stays the backup and doesn't take the address back. -
Power off
nano-1. The UI and search stop, and about 15 seconds later ingest answers503. Your real senders must buffer and retry; see Make senders retry. Power it back on. The UI returns in seconds and ingest in about 20 seconds. -
Stop the stream and count. Search
failover-test | stats countover the test window. The count must equal the number of requests that got200infailover-test.log:grep -c ' 200$' failover-test.log
With full HA
With two or more app servers, three postgres servers and a redundant load balancer, nothing should stop for long. Keep the stream from step 1 running, power off one server at a time, wait until SIEM Health is healthy again after you power it back on, then move to the next:
| Power off | What must keep working |
|---|---|
| An app server | The UI, search and ingest continue. The load balancer drops it after its /readyz checks fail, within about 5 seconds. On v2.0.12 and earlier, check it after it returns: see An app server is DOWN for the ingest servers after a reboot |
| The PostgreSQL primary | Writes, sign-ins and searches fail for about 20–25 seconds (20–24 seconds measured when the primary's server is powered off or reset), then resume with nothing lost. Sign-ins and saves in that window fail with 500 or 503 and succeed on retry. A planned switchover, as during an upgrade, pauses them for only a few seconds. Pages already open stay up and ingest over HEC and HTTP continues. patronictl list shows a new leader |
| A ClickHouse server | As for nano-2 above |
| An ingest server | The other ingest servers carry the stream. The load balancer drops it within about 10 seconds |
| One HAProxy host | Depends on the load balancer option: about 3 seconds with a floating IP, about 1–2 minutes with DNS failover, and an outage with a single load balancer |
Then count as in step 7. The count must equal the number of 200 responses.
A 200 means the ingest server has written the event to its disk buffer. From v2.0.15 it has also synced it to disk first, so it survives a power loss or hard reset of that server. In v2.0.13 and earlier the buffer synced every 500 ms, so an event answered 200 less than half a second before a hard reset of its ingest server could be lost (a stop or restart of the Vector container never lost one). With HEC indexer acknowledgements, the same holds for an ackId that reports true.
Delivery to storage is at least once: if a ClickHouse server goes away while a batch is being written, the batch can be written again, so a few events from that second may be stored twice. A count slightly above the number of 200 responses is that, not a fault; failover-test | stats dc(message) counts each numbered event once. A count below it is a fault. See What to expect during a failure for the measured case.
If a count doesn't match or a server doesn't rejoin, contact support before you go live, with the deployment ID, the times and failover-test.log.
Back up
Run the backup from nano-1. From v2.0.5, it backs up PostgreSQL (alerts, cases, fired detections, rules and the rest of nano's configuration) and the configuration and secrets in .env as one signed, encrypted set, without stopping ingest or search:
cd "$HOME/nano"
scripts/nano backup-set --inventory /etc/nano/inventory.yaml \
--backup-key-file /secure/nano-backup/operator.pem \
--backup-output /mnt/backup/nano-$(date -u +%Y%m%dT%H%M%SZ)From v2.0.6 the backup also includes the lookup tables (the rows behind | lookup), as on a single node. nano-1 reads them from ClickHouse over the network with a read-only backup user, and a restore loads them back. On v2.0.5, add --include-clickhouse-data to keep them.
Other ClickHouse data isn't included: the two replicas and object storage protect the events, and IOCs and enrichment re-sync from their sources. To also back up one copy of the events, add --include-clickhouse-data. Then the backup key and the backup folder's parent must exist at the same paths on nano-1 and nano-2; use a shared mount for the output.
Where the backup is written. Each part of the backup is written on the server that owns it: the PostgreSQL dump on whichever postgres server leads at that moment, and, with --include-clickhouse-data, the ClickHouse parts on the ClickHouse servers. With separate postgres servers, create the --backup-output folder's parent, owned by the user that installed nano, on the first app server and on every postgres server, because the leader can move (with --include-clickhouse-data, on every ClickHouse server too):
sudo install -d -o "$USER" -g "$(id -gn)" -m 0700 /mnt/backupFrom v2.0.15 the upgrade's dry run has a backup-output row for each of those servers and fails, with this command, on any server where the folder can't be created. In v2.0.13 the dry run passed and the upgrade then stopped at its backup step with mkdir: cannot create directory ... Permission denied; create the folder there and run the same command with --resume.
What pauses during a backup. Ingest, search, dashboards and sign-in keep working. To capture PostgreSQL and the lookup tables as one consistent set, the backup briefly stops the background writers on every app server: detections (scheduled and real-time), alerting and connectors. While they are stopped, requests that change something through the API, such as acknowledging an alert or editing a case, get 503 with "temporarily paused for a consistent backup"; retry them. The backup prints the pause as writers_down_ms; a 7-server test measured about 86 seconds. It grows with the number of app servers and the size of the lookup tables, not with event volume. The database dump and the encryption run after the writers restart. Nothing is lost: scheduled rules pick up from their last completed run and scan the paused minutes, and real-time detections turn the signals into alerts from where they stopped, so alerts arrive late, not never. A rule with a custom lookback shorter than its schedule interval plus the pause scans only its lookback. Schedule backups for a quiet period.
Create the key as for a single node.
The backup includes the Vector native sender anchors in ~/nano/.nano-pki only when the first postgres server is also the first app server, as nano-1 is here; see Back up the anchors.
Vector native (port 6000)
From v2.0.30. Upstream Vector aggregators and forwarders can send to nano with Vector's native protocol (v2) on port 6000 of every ingest server. The protocol has no token, so a client certificate is the credential: every connection must present one issued for this deployment, or the TLS handshake is refused before any event is read. There is no plaintext mode and no switch to turn it off. If a TLS file is missing or unreadable, the port refuses every sender. A plaintext lab cluster (allow_plaintext_private_network) has no listener: the port refuses every sender.
On v2.0.29 and earlier, port 6000 was plaintext with no credential check, and on a role-split cluster no ingest server published it.
Events arrive as the vector_native source, whichever sender sent them. For one log source per sender, use HEC or HTTP ingest with an ingest token instead.
What lives where
| What | Where | Secret |
|---|---|---|
| One trust anchor per sender (a small CA that signs only that sender's certificate), with its key | ~/nano/.nano-pki/vector-native on nano-1 (the first app server), mode 0700 | Yes. Never in .env, never copied to another server |
| The anchors' public certificates | NANO_VECTOR_NATIVE_CLIENT_ANCHORS in the shared .env | No |
| The trust bundle the listener checks senders against | ~/nano/config/vector-native/client-ca.crt on each ingest server, built from .env | No |
| The listener certificate | The private cluster certificate you staged (cluster.tls.cert) | Its key is the staged one |
| What senders verify nano with | Your private CA (cluster.tls.ca), with tls.server_name set to cluster.tls.server_name | No |
No CA key ever reaches an ingest server. Revoking a sender removes only its anchor, so other senders keep working with the files they have.
Issue a sender certificate
-
On
nano-1, issue one certificate per sender:cd "$HOME/nano" scripts/nano vector-client-cert edge-dc1 # [--out DIR] [--days N] [--address HOST[:PORT]] scripts/nano vector-client-cert edge-dc2 --csr edge-dc2.csr # the key stays on the senderWithout
--csr, it writesedge-dc1.key(mode0600, never printed),edge-dc1.crtandca.crtto./nano-vector-edge-dc1/. With--csr, the sender makes its own key and request, and only the certificate comes back. Certificates last 825 days unless you pass--days(up to 3650). The command records the new anchor innano-1's.envand prints the sink config. -
Copy
nano-1's.envto every other server and update every ingest server, as in Update every ingest server. Until you do, ingest servers refuse the new sender. -
Copy the files to the sender and add the printed sink to its Vector config, then restart the sender's Vector:
[sinks.nano] type = "vector" inputs = ["<your source or transform>"] address = "ingest.nano.example.com:6000" version = "2" tls.enabled = true tls.server_name = "nano-private.example.com" tls.crt_file = "/etc/vector/nano/edge-dc1.crt" tls.key_file = "/etc/vector/nano/edge-dc1.key" tls.ca_file = "/etc/vector/nano/ca.crt"ca.crtis a copy of your private CA certificate, andtls.server_nameiscluster.tls.server_name, so the address can be any name or address that reaches port6000on an ingest server.
Update every ingest server
Run this from nano-1 right after every issue or revoke. It copies .env to every other server, then rebuilds each ingest server's trust bundle and restarts its Vector. It is what the command does on an app server that also runs ingest. It uses the SSH access that scripts/nano upgrade needs; change the addresses to yours:
OTHERS="10.20.0.12 10.20.0.13" # every server except nano-1
INGEST="10.20.0.12 10.20.0.13" # every ingest server
for h in $OTHERS; do scp -p "$HOME/nano/.env" "$h:nano/.env"; done
for h in $INGEST; do
ssh "$h" 'cd "$HOME/nano" &&
bash scripts/lib/vector-native-pki.sh ensure-ingest --tls-dir config/vector-native --env-file .env &&
docker compose --project-directory "$HOME/nano/deploy/compose" --env-file "$HOME/nano/.env" \
--env-file "$HOME/nano/.nano-role.env" -f "$HOME/nano/deploy/compose/docker-compose.enterprise.yml" restart vector'
doneEach ingest server prints anchors=<number of active senders> changed=1 when its bundle changed (changed=0 if it was already up to date, for example on a second run). Check that sha256sum ~/nano/.env matches on every server. Re-running each ingest server's install command with its full role list does the same, more slowly.
Restarting Vector pauses all ingest on that server for a few seconds, and the load balancer takes it out until /healthz answers 200 again. Senders that retry lose nothing.
Revoke a sender
cd "$HOME/nano"
scripts/nano vector-client-cert --revoke edge-dc1It removes the sender's anchor from nano-1's records and from .env, and deletes the anchor key. No other sender needs new files.
Renew a sender
scripts/nano vector-client-cert --renew edge-dc1 # [--days N] [--csr FILE] [--out DIR]It re-signs the sender under the same anchor, so .env doesn't change and no ingest server needs an update or a restart. Copy the new edge-dc1.crt to the sender and restart the sender's Vector. With --csr, the sender keeps its key.
Expiry
scripts/nano vector-client-cert --listonnano-1shows each sender asactive,expiring(under 30 days),expiredorrevoked. On a role-split cluster the installer andscripts/nano upgradedon't warn about expiring sender certificates, so check--liston a schedule. Renew before expiry; nothing renews a sender on its own.- The listener certificate is your private cluster certificate. nano never renews it. When you replace it, stage it again on every server as in Write the inventory and stage the certificate, rerun each server's install command, and restart Vector on each ingest server: Vector reads the certificate only when it starts. If the new certificate comes from a new CA, every sender needs the new CA certificate as its
ca.crt.
Back up the anchors
scripts/nano backup-set writes its backup on the first postgres server in the inventory, and includes that server's ~/nano/.nano-pki. In the layout on this page that is nano-1, so the anchors are covered, and a restore puts them back. With PostgreSQL on servers of its own, the anchors on the first app server aren't in the backup. Then back them up yourself after every issue, renew or revoke, and protect the copy like the backup key, since it holds the anchor keys:
tar -C "$HOME/nano" -czf "/secure/nano-backup/nano-pki-$(date -u +%Y%m%dT%H%M%SZ).tgz" .nano-pkiTo restore it, unpack it into ~/nano on nano-1 and run chmod -R go-rwx ~/nano/.nano-pki.
- If you lose it, senders keep working, because their anchors are still in
.env. But you can't renew or revoke them. The next sender you issue publishes only the anchorsnano-1still has, so once the ingest servers are updated, every older sender is refused and needs a new certificate. - If you restore an older
.env, its anchor list is from the backup and can trust a sender you revoked since. Run--revokeagain for each of those senders (it republishes the current list), then update every ingest server.
Sender address
The address printed in the sink config is, in order:
NANO_VECTOR_NATIVE_PUBLIC_HOST, from the environment ornano-1's.env;cluster.ingest_hostname;- the
BASE_URLhost. Alocalhostor loopback host only prints a warning.
--address HOST[:PORT] sets it for one sender. cluster.ingest_hostname usually points at your HTTPS load balancer, which doesn't carry port 6000. So either add a TCP pass-through on port 6000 to the ingest servers (layer 4, no TLS termination: the client certificate must reach Vector), or set NANO_VECTOR_NATIVE_PUBLIC_HOST to a name senders can reach, or point each sender at one ingest server.
Bind address and firewall
Each ingest server publishes port 6000 on its inventory bind_address (0.0.0.0, every interface, unless you set one). If you set bind_address to a private address, senders outside that network can't connect; set NANO_VECTOR_NATIVE_BIND_ADDRESS=0.0.0.0 in the shared .env and rerun the ingest servers. It applies to every ingest server, so don't set it to one server's address.
Docker publishes port 6000, so UFW doesn't filter it, and the Docker rules below don't list it. Without a rule, anyone who can route to an ingest server reaches the TLS handshake, and is refused without a certificate. To allow only your senders:
-
In your cloud security group or network firewall, allow TCP
6000to the ingest servers only from your Vector senders (or from your TCP load balancer). -
In
/usr/local/sbin/nano-docker-firewallon each ingest server, make these changes, then runsudo systemctl restart nano-docker-firewall.service:PORTS="80 2379 2380 3000 5432 6000 6379 8008 8687 26379" # add 6000 SENDERS4="10.30.0.0/24" # new: Vector senders, port 6000 only SENDERS6="" # in chain(), after the load-balancer line: for s in $3; do echo "-A NANO-DOCKER -s $s -p tcp -m conntrack --ctorigdstport 6000 -j RETURN"; done # in apply(), pass the senders on: chain "$2" "$3" "$4" | "$1-restore" -w --noflush # at the end (the ip6tables line stays inside its existing if): apply iptables "$PEERS4" "$LB4" "$SENDERS4" apply ip6tables "$PEERS6" "$LB6" "$SENDERS6"A TCP load balancer in front of port
6000goes inSENDERS4too:LB4allows only port80.
Trade-offs and limits
- Propagation window. On a role-split cluster, an issue or a revoke on
nano-1takes effect on an ingest server only after you copy.envto it and update it. Until then that server refuses a new sender and accepts a revoked one. Block a compromised sender at the firewall first. - Vector restarts on every trust change. Vector has no TLS reload for this source, so each update restarts Vector on that ingest server (a few seconds; all ingest on that server pauses). Renewing a sender restarts nothing.
- The listener certificate isn't renewed for you. On a role-split cluster it's your cluster certificate, which you renew and stage. On a single server, nano re-issues it within 30 days of expiry, but only when
install.shorscripts/nano upgraderuns. - Sender keys are made on
nano-1and written to the output folder, unless you use--csrso the key never leaves the sender. Delete the output folder once the files are on the sender. - No automatic expiry warnings on a role-split cluster. Check
--list. .nano-pkiis inbackup-setonly when the firstpostgresserver is the first app server. Otherwise back it up yourself.- One source for every sender. Events land as
vector_native; the certificate doesn't choose a log source. - Single server: upgrading to v2.0.30 cuts off plaintext senders. On a single server,
6000used to accept plaintext on loopback (or wherever you bound it). After the upgrade it refuses every sender without a certificate.install.shandscripts/nano upgradeboth say so, andinstall.shalso counts thevector_nativeevents of the last 24 hours when it can. On a role-split cluster no sender breaks, because ingest servers didn't publish6000before.
For one server, see Vector native in the Vector aggregator guide. For Kubernetes, see Vector-native client certificates.
Upgrade
nano pulls the new images from the registry on every server. You upgrade with the new release's installer bundle, nano-compose-enterprise-<version>.tar.gz, with its .sha256 and .sha256.sig; see Get the bundle. There is no separate image bundle for registry installs.
From v2.0.10 or later
Every server runs v2.0.10 or a later release.
Before the first upgrade with scripts/nano upgrade, set up three things that a fresh install doesn't:
- SSH from the first app server to every other server, as the user that installed nano, without a prompt:
ssh -o BatchMode=yes <server address> truemust succeed from the first app server for each one. The upgrade andscripts/nano support-bundleboth run from there. - Drain and rejoin hooks on every app and ingest server, at the same paths, for example
/usr/local/sbin/nano-lb-drainand/usr/local/sbin/nano-lb-rejoin, and thenano-lb-haproxywrapper on every load-balancer host. Test them by hand. See Load-balancer drain and rejoin hooks. - The HAProxy runtime socket at
level adminon every load-balancer host, so the hooks can drain a server. See the HAProxy example.
-
Put the bundle and its two signature files at the same path on every server, for example
/srv/nano/releases/, and verify them once as in single-node step 2. -
On every server, unpack the bundle over
~/nano. It holds no.env, no.nano-role.envand no data, so each server keeps its settings, and nothing restarts yet:tar -xzf /srv/nano/releases/nano-compose-enterprise-<version>.tar.gz --strip-components=1 -C "$HOME/nano" -
Log in to the registry on every server with a fresh pull credential. It lasts one hour, so do this right before the upgrade.
-
Permit SSH from
nano-1(with several app servers, the first app server) to the other servers, then dry-run from there:cd "$HOME/nano" scripts/nano upgrade --inventory /etc/nano/inventory.yaml \ --bundle /srv/nano/releases/nano-compose-enterprise-<version>.tar.gz \ --remote-bundle /srv/nano/releases/nano-compose-enterprise-<version>.tar.gz \ --backup-key-file /secure/nano-backup/operator.pem \ --backup-output /mnt/backup/upgrade-<version> \ --lb-drain-command /usr/local/sbin/nano-lb-drain \ --lb-rejoin-command /usr/local/sbin/nano-lb-rejoin \ --dry-runBefore the dry run, create the
--backup-outputfolder's parent on everypostgresserver (and every ClickHouse server with--include-clickhouse-data), as in Where the backup is written. -
When every check passes, run the same command without
--dry-run. If you logged in to the registry by hand, log out on every server. From v2.0.30, with an online license, each server logs in with the license values in its own.env, in a temporary Docker configuration that leaves~/.dockeruntouched.
scripts/nano upgrade verifies the bundle's signature and checks that every server has the new bundle unpacked. Before it takes the backup or restarts anything, it checks that the registry accepts each server's login and pulls each server's own images, by signed digest, so an expired login or an unreachable registry fails while nothing has changed. The dry run does this too: it fills each server's image cache and changes nothing that runs. Then it takes a backup and upgrades one server at a time, checking health between them: Keeper and ClickHouse, then PostgreSQL (standbys first, then a switchover, then the old primary), then the migrations, the app servers and the ingest servers. It takes each app and ingest server out of the load balancer before restarting it. If it stops, fix the cause (often an expired registry login) and run the same command with --resume.
From v2.0.13, scripts/nano upgrade also does this:
- It renders each server's configuration again from the new bundle, with the installer's renderer and your inventory: the nginx edges, HAProxy, ClickHouse and Keeper, PostgreSQL HA, Valkey and monitoring. The dry run shows a
config-renderrow for each server and prints the difference for each server's files. It never prints key contents. The upgrade applies the new configuration on each server before it upgrades that server, and recreates every container whose configuration changed, not only those with a new image. The previous configuration stays in~/nano/.nano-generated.previous, andscripts/nano rollbackputs it back. If the inventory's deployment or server identity changed, the upgrade refuses: that is a topology change, which needs the installer's--accept-topology-migration. - The dry run tests the drain and rejoin hooks on every server the upgrade drains, as in Load-balancer drain and rejoin hooks.
- It waits for a new SIEM Health report before it compares findings with the report from before the upgrade. The first report runs about 5 minutes after the API starts. It waits up to 600 seconds (
NANO_UPGRADE_SIEM_HEALTH_WAIT_SECONDS). If no new report arrives, it prints a warning and records that it didn't run this check. - It prints each step as it happens, also over a plain
ssh <server> 'scripts/nano upgrade ...': a==> <time> <step>: startedline and adone in <N>sline. Every step is also recorded in~/nano/upgrade-journal/. In v2.0.13 the preflight and the dry run print nothing until their result table, for 2.5 to 4.5 minutes on a 7-server cluster. From v2.0.15 they also print a line as each phase starts and ends, for example==> <time> preflight checks on nano-3: startedanddone in 41s (28/28 PASS). - From v2.0.15 each drain and rejoin hook call is journaled as an
lb_hookevent in~/nano/upgrade-journal/on the first app server, with the server,drainorrejoin, the exit code and the duration (never the hook's output).
From v2.0.9 or earlier
Your servers run a release before v2.0.10, whose scripts/nano upgrade can't use the installer bundle. Upgrade by running each server's install command again from the new bundle (v2.0.10 or later), one server at a time, in this order. Later upgrades follow From v2.0.10 or later.
-
Back up first.
-
On every server, unpack the new bundle over
~/nanoand log in to the registry. -
On each server in turn: take it out of the load balancer if it's an app or ingest server and wait for the drain, run its exact install command with its full role list, wait until it's healthy, and put it back in the load balancer. Where a server carries several roles (for example Keeper on the
postgresservers and ingest on the ClickHouse servers, as the nano-deploy planner lays them out), the one run upgrades all of them; never pass a shorter role list. -
After the first server's run, copy its
.envto every other server before the next run. The run records the newNANO_VERSIONand prints "The shared .env changed on this host (NANO_VERSION)". See Copy .env to the other servers, and check thatsha256sum ~/nano/.envmatches everywhere. -
Order: the ClickHouse servers (with Keeper or ingest when they carry it), then the
postgresservers (standbys first, the primary's server last), then the app servers one at a time, then any other ingest servers. -
Before each
postgresserver, move the cache's primary off it. A run on apostgresserver also restarts its Valkey cache member and Sentinel. If that member is the cache primary, have Sentinel move it first, then wait untilSENTINEL get-master-addr-by-name nanonames another server:cd ~/nano && docker compose --env-file .env --env-file .nano-role.env \ -f deploy/compose/docker-compose.enterprise.yml exec -T valkey-sentinel \ valkey-cli --tls --insecure --cert /etc/nano/tls/server.crt --key /etc/nano/tls/server.key \ -p 26379 SENTINEL failover nanoFrom v2.0.11 the installer switches the PostgreSQL primary over before restarting its server. With the v2.0.10 bundle, switch over by hand first, see PostgreSQL servers.
The nano-deploy skill's upgrade-plan.py reads every server and prints these steps in order for your cluster, marking the servers that hold the PostgreSQL and cache primaries.
A 7-server test upgraded this way in under 10 minutes without losing an event.
Load-balancer drain and rejoin hooks
With more than one ingest server, the upgrade takes each ingest server out of your load balancer before restarting it, and puts it back after. With two or more app servers it does the same for each app server. You provide the two commands. The upgrade runs them on the server being upgraded, with that server's inventory name (hosts[].name) as the only argument, and stops if one fails. They must exist at the same path on every app and ingest server. On an app server without hooks, the drain relies on your load balancer health-checking /readyz.
The bundle ships two example files for the HAProxy example, in deploy/compose/examples/load-balancer/. Each file's header has the install steps.
-
nano-lb-hookruns on the app and ingest servers. Install the one file under both names. Your load balancer usually knows the servers by other names (app-1,ingest-1and so on, as the nano-deploy planner names them), so edit itsSERVERSmap (inventory name, then<backend>/<server>) and itsLBSlist (every HAProxy host):sudo install -m 0755 nano-lb-hook /usr/local/sbin/nano-lb-drain sudo install -m 0755 nano-lb-hook /usr/local/sbin/nano-lb-rejoinIt runs
ssh nano-lb@<load balancer> /usr/local/sbin/nano-lb-haproxy set <backend>/<server> drain(orready) on each load balancer, and fails unless the reply confirms the new state. -
nano-lb-haproxyruns on every load-balancer host. It accepts onlyset <backend>/<server> drain|readyandstate <backend>/<server>. It applies the state through HAProxy's admin socket, reads it back withshow servers state, and fails unless HAProxy confirms it. Install it with anano-lbuser and a sudoers line for that onesocatcommand:sudo useradd --system --create-home --shell /bin/sh nano-lb sudo install -m 0755 nano-lb-haproxy /usr/local/sbin/nano-lb-haproxy echo 'nano-lb ALL=(root) NOPASSWD: /usr/bin/socat stdio /run/haproxy/admin.sock' \ | sudo tee /etc/sudoers.d/nano-lb && sudo chmod 0440 /etc/sudoers.d/nano-lb sudo visudo -cf /etc/sudoers.d/nano-lbThen add the SSH key of the upgrade user on each app and ingest server to
~nano-lb/.ssh/authorized_keys, restricted to the wrapper:command="/usr/local/sbin/nano-lb-haproxy",restrict ssh-ed25519 AAAA... nano@app-1
On v2.0.12 and earlier: the two files
The v2.0.12 bundle has no nano-lb-haproxy, and its nano-lb-hook sends the command that a forced command discards. On v2.0.12, install these two files instead. Later bundles ship them in deploy/compose/examples/load-balancer/.
nano-lb-hook, on every app and ingest server:
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Example load-balancer drain/rejoin hook for `scripts/nano upgrade` on a
# multi-server cluster (NAN-5269), for the HAProxy example beside this file.
#
# Install ONE copy under both names on every app and ingest server:
# sudo install -m 0755 nano-lb-hook /usr/local/sbin/nano-lb-drain
# sudo install -m 0755 nano-lb-hook /usr/local/sbin/nano-lb-rejoin
# and pass them to the upgrade:
# --lb-drain-command /usr/local/sbin/nano-lb-drain
# --lb-rejoin-command /usr/local/sbin/nano-lb-rejoin
#
# The upgrade runs the hook ON the server it is upgrading, with that server's
# INVENTORY name (hosts[].name in inventory.yaml) as the only argument, and
# stops when the hook exits non-zero. The load balancer knows the server by
# its own name (app-1, ingest-1, ... in haproxy.cfg and in the nano-deploy
# planner's output), so the hook maps one to the other: edit SERVERS.
#
# Each load balancer needs the runtime socket at `level admin` (haproxy.cfg:
# stats socket /run/haproxy/admin.sock mode 600 level admin) and the
# load-balancer side of this hook, `nano-lb-haproxy` (beside this file; its
# header has the install steps). The upgrade user on every app and ingest
# server needs an SSH key that the load-balancer hosts accept for LB_USER,
# restricted to that wrapper with a forced command in authorized_keys:
# command="/usr/local/sbin/nano-lb-haproxy",restrict ssh-ed25519 AAAA...
# Do NOT force `sudo socat` itself: a forced command replaces the command this
# hook sends and gets no stdin, so socat would read nothing, exit 0, and the
# drain would silently do nothing. The wrapper reads the state back from
# HAProxy, so a drain that did not take effect fails the hook (and the
# upgrade, and its dry run) instead.
# With a managed cloud load balancer, replace lb_set with a call to its API
# (deregister / re-register the target) and check the result the same way.
set -eu
# <inventory name> <haproxy backend>/<haproxy server>, one per line.
SERVERS='
nano-1 nano_app_hosts/app-1
nano-2 nano_ingest_hosts/ingest-1
nano-3 nano_ingest_hosts/ingest-2
'
# Every HAProxy in front of the cluster (both of a keepalived pair).
LBS=${NANO_LB_HOSTS:-"lb-1.example.internal lb-2.example.internal"}
LB_USER=${NANO_LB_USER:-nano-lb}
# The wrapper on the load balancers (see nano-lb-haproxy).
LB_COMMAND=${NANO_LB_COMMAND:-/usr/local/sbin/nano-lb-haproxy}
case "$(basename "$0")" in
nano-lb-drain) state=drain ;;
nano-lb-rejoin) state=ready ;;
*) echo "install this file as nano-lb-drain and nano-lb-rejoin" >&2; exit 2 ;;
esac
[ $# -eq 1 ] || { echo "usage: $(basename "$0") <inventory host name>" >&2; exit 2; }
target=$(printf '%s\n' "$SERVERS" | awk -v h="$1" '$1 == h { print $2; exit }')
[ -n "$target" ] || { echo "$1 is not in SERVERS (edit $0)" >&2; exit 2; }
lb_set() {
# The reply is "<backend>/<server> <state>" only when HAProxy confirms it.
reply=$(ssh -o BatchMode=yes -o ConnectTimeout=5 "$LB_USER@$1" \
"$LB_COMMAND set $target $state") || return 1
[ "$reply" = "$target $state" ] || {
echo "$1 did not confirm $target $state (reply: ${reply:-none}); is $LB_COMMAND installed as the forced command?" >&2
return 1
}
}
for lb in $LBS; do
lb_set "$lb" || { echo "could not set $target to $state on $lb" >&2; exit 1; }
done
echo "$target: $state on $LBS"nano-lb-haproxy, on every load-balancer host:
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Load-balancer side of the example drain/rejoin hook (NAN-5279), for the
# HAProxy example beside this file. Install it on EVERY load-balancer host:
#
# sudo useradd --system --create-home --shell /bin/sh nano-lb
# sudo install -m 0755 nano-lb-haproxy /usr/local/sbin/nano-lb-haproxy
# echo 'nano-lb ALL=(root) NOPASSWD: /usr/bin/socat stdio /run/haproxy/admin.sock' \
# | sudo tee /etc/sudoers.d/nano-lb && sudo chmod 0440 /etc/sudoers.d/nano-lb
# sudo visudo -cf /etc/sudoers.d/nano-lb
#
# and restrict each app/ingest server's key to this one program in
# ~nano-lb/.ssh/authorized_keys:
#
# command="/usr/local/sbin/nano-lb-haproxy",restrict ssh-ed25519 AAAA... nano@app-1
#
# A forced command REPLACES the command the client sent, and sshd gives it the
# client's command in SSH_ORIGINAL_COMMAND, with no stdin. So the forced
# command must be this wrapper, never `sudo socat` itself: socat would read an
# empty stdin and exit 0, and the drain would silently do nothing.
#
# Accepted commands (from SSH_ORIGINAL_COMMAND, or arguments when run directly):
# nano-lb-haproxy set <backend>/<server> drain|ready
# nano-lb-haproxy state <backend>/<server>
# `set` applies the state, reads it back from HAProxy and prints
# "<backend>/<server> <state>"; it exits non-zero unless HAProxy confirms it.
set -eu
SOCKET=${NANO_LB_SOCKET:-/run/haproxy/admin.sock}
SOCAT=${NANO_LB_SOCAT:-/usr/bin/socat}
SUDO=${NANO_LB_SUDO:-sudo -n}
die() { echo "nano-lb-haproxy: $*" >&2; exit 2; }
if [ -n "${SSH_ORIGINAL_COMMAND:-}" ]; then
# Word-split the client's command without globbing; every accepted word
# is validated below, so nothing from it reaches a shell.
set -f
# shellcheck disable=SC2086
set -- $SSH_ORIGINAL_COMMAND
set +f
fi
case "${1:-}" in
nano-lb-haproxy|*/nano-lb-haproxy) shift ;;
esac
action=${1:-}
target=${2:-}
printf '%s\n' "$target" | grep -Eq '^[A-Za-z0-9_.:-]+/[A-Za-z0-9_.:-]+$' \
|| die "usage: nano-lb-haproxy set <backend>/<server> drain|ready | state <backend>/<server>"
backend=${target%/*}
server=${target#*/}
haproxy() {
# One runtime-API command on stdin; the reply on stdout.
printf '%s\n' "$1" | $SUDO "$SOCAT" stdio "$SOCKET"
}
current_state() {
# `show servers state <backend>`: field 4 is srv_name, field 7 is
# srv_admin_state (bit 0x08 = drained by the runtime API).
admin=$(haproxy "show servers state $backend" \
| awk -v s="$server" '$1 !~ /^#/ && NF >= 7 && $4 == s { print $7; exit }')
case "$admin" in
''|*[!0-9]*) echo unknown ;;
*) if [ $(( admin & 8 )) -ne 0 ]; then echo drain; else echo ready; fi ;;
esac
}
case "$action" in
set)
[ $# -eq 3 ] || die "usage: nano-lb-haproxy set <backend>/<server> drain|ready"
state=$3
case "$state" in drain|ready) ;; *) die "state must be drain or ready" ;; esac
reply=$(haproxy "set server $target state $state")
reply=$(printf '%s' "$reply" | tr -d '\r' | sed '/^[[:space:]]*$/d')
[ -z "$reply" ] || die "HAProxy refused 'set server $target state $state': $reply"
observed=$(current_state)
[ "$observed" = "$state" ] \
|| die "HAProxy reports $target as $observed after setting $state"
echo "$target $state"
;;
state)
[ $# -eq 2 ] || die "usage: nano-lb-haproxy state <backend>/<server>"
echo "$target $(current_state)"
;;
*)
die "usage: nano-lb-haproxy set <backend>/<server> drain|ready | state <backend>/<server>"
;;
esacThe HAProxy runtime socket needs level admin to change a server's state (level operator can only read it).
Test the hooks before the first upgrade. On an app server, with its inventory name:
/usr/local/sbin/nano-lb-drain nano-1
ssh nano-lb@lb-1.example.internal /usr/local/sbin/nano-lb-haproxy state nano_app/nano-1 # prints "nano_app/nano-1 drain"
/usr/local/sbin/nano-lb-rejoin nano-1
ssh nano-lb@lb-1.example.internal /usr/local/sbin/nano-lb-haproxy state nano_app/nano-1 # prints "nano_app/nano-1 ready"Each hook prints the target and its state on every load balancer, and exits non-zero if a load balancer doesn't confirm it. From v2.0.13, the dry run also checks the hooks on every server the upgrade drains, and runs a real drain and rejoin through them, one server at a time. If a round trip fails, it drains no further server and names the server that may still be drained. Set NANO_UPGRADE_LB_HOOK_TEST=0 to skip that round trip.
With a managed cloud load balancer, replace the hook's lb_set function with a call to its API (deregister and re-register the target), and check the result the same way.
Scaling past three nodes
Three nodes is a starting point, not a limit. The same installer and inventory format build larger layouts: more app servers, more ingest servers, more ClickHouse servers and dedicated Keeper servers. With one postgres server, PostgreSQL is the part that cannot fail over; run three postgres servers to remove that single point of failure.
App servers
The app (UI, API, search, detections and alerting) keeps no data of its own, so you can run it on two or more servers behind the load balancer. Losing one app server then stops nothing: the load balancer stops sending it requests, detections and alerting carry on from another app server, and ingest keeps flowing.
- Give each one the
approle. They can be dedicated servers or share a server with other roles, but not withpostgresif you want the app to survive that server. - Two or more app servers need the private certificate (the default). The plaintext lab mode is refused.
- The servers share one cache on the
postgresserver. Allow TCP 6379 to thepostgresserver from the app servers only. With threepostgresservers the cache fails over with PostgreSQL. - In the load balancer's app backend, list every app server on port 80 and health-check
GET /readyz. It answers200only while that server is ready, and503while an upgrade is draining it. No session affinity is needed. Retry only failed connections on another server: after a timeout or an error response the request may already have been applied, and sending it again would apply it twice. See the HAProxy example. - Create
.envon the first app server in the inventory and copy it to every other server, the other app servers included. The other app servers refuse to create their own. - Install the data servers first, then the app servers one at a time, first app server first, then the ingest servers.
- An upgrade updates the database once, then upgrades the app servers one at a time. Each one is taken out of the load balancer first, so nobody sees an outage. Set
NANO_UPGRADE_APP_DRAIN_SECS(default 15) above how long your load balancer takes to mark a server down.
Add an app server
You can add an app server to a running deployment. No data moves.
- Add a
hostsentry with a newname,idandaddress, androles: [app], and make sure the private certificate covers its address. Install the updated inventory on every server. - Rerun the installer on each server, in this order: the
postgresserver, the existing app server, the new app server, then the ingest servers. The first rerun refuses and prints the current identity; rerun each with--accept-topology-migration <identity>, as for adding an ingest server. - Add the new server to the load balancer's app backend once
GET /readyzreturns200.
When you go from one app server to two, the cache moves to the postgres server. Sign-outs from the last 15 minutes are carried over; searches running at that moment are lost and must be run again.
PostgreSQL servers
PostgreSQL holds alerts, cases, rules, users and nano's configuration. With one postgres server it is a single point of failure. Give three servers the postgres role and it fails over on its own:
- One server is the primary and two are standbys that stream every change. If the primary is lost, a standby takes over and writes, sign-ins and searches fail while it is promoted: about 20–25 seconds when the primary's server dies (20–24 seconds measured on a 7-server cluster; Patroni first waits for the lost primary's lease to expire). A planned switchover, as during an upgrade, takes a few seconds. Requests in that window fail, sign-ins with
500or503, and succeed on retry; nothing else is needed. - Part of that window is the leader timeout (
ttl, 20 seconds by default, which is also the lowest value Patroni accepts). A shorter timeout fails over sooner but can also fail over a healthy primary during a brief network stall. On a slow or long-distance private network, raise it undercluster, inpostgres_ha:, withttl: 30,loop_wait: 5andretry_timeout: 10(ttlmust be at leastloop_waitplus twiceretry_timeout). A longerttlmakes the pause longer. - Every app server sends database traffic to whichever server is the primary at that moment, so nothing in nano is reconfigured during a failover.
- By default every write waits until one standby has it, so a failover loses no saved change. This costs one round trip on the private network per write. Under
cluster, setpostgres_ha:withsynchronous_mode: strictto stop writes rather than ever accept one that only the primary holds, orsynchronous_mode: offto trade that guarantee for latency. - With two or more app servers, the shared cache moves with it: each
postgresserver holds a copy and a replacement takes over in about 10 seconds. Sign-outs survive the switch. - The servers can share hardware with other roles, but not with each other, and not with the only app server.
Two postgres servers work too if the deployment has three Keeper servers, which then act as the tie-breaker. Three postgres servers are recommended.
The private certificate must cover every postgres server's address and be usable as a client certificate (extendedKeyUsage = serverAuth, clientAuth). Allow these ports between the postgres servers, and from the app servers to them: 5432, 8008, 2379, 2380, 6379 and 26379.
Install: install the three postgres servers first, in any order. The first one in the inventory creates the database and the other two copy it. The database is created only once a second postgres server joins (the coordination service, etcd, needs two of its three members), so the first server's install can't record the cluster's identity yet and says so. After the third, record it on the first postgres server:
cd ~/nano && ./install.sh --record-pg-ha-identityIt reads the identity from that server's database and writes NANO_PG_HA_SYSTEM_ID to its .env; it changes nothing else and restarts nothing. It prints "The shared .env changed on this host (NANO_PG_HA_SYSTEM_ID)". Then copy the first postgres server's .env to every other server, as in Copy .env to the other servers, and install the other servers as usual. The other two postgres servers may have recorded the same line already; the copy leaves their .env unchanged.
From v2.0.11, rerunning the installer on the server that holds the primary first switches the primary over to the synchronous standby (a pause of a few seconds instead of a failover), and stops without restarting anything if that doesn't work within 90 seconds. If no standby is streaming and caught up, the installer warns and restarts the primary anyway: PostgreSQL then fails over to another server if one is eligible (about 20–25 seconds of failed writes), otherwise writes pause until the primary is back. Bring the standbys back first.
Add two postgres servers to a running deployment. No data moves; the existing server keeps its database and becomes the first primary.
- Back up first.
- Add the two new
hostsentries withroles: [postgres]after the existingpostgresserver, and make sure the private certificate covers their addresses. Install the updated inventory on every server. - Rerun the installer on the first app server. It adds two new secrets to
.env. Copy that.envto every server. - Rerun the installer on the existing
postgresserver, then straight away on the two new ones. Writes pause for a minute or two while they start. Each rerun first refuses and prints the current identity; rerun it with--accept-topology-migration <identity>. - Rerun the installer on every app server, then on the ingest servers.
Backups and restores run on whichever postgres server is currently the primary; on another server they stop and name it. Upgrades update the standbys first, then move the primary role to an updated standby, then update the old primary, so the database stays available throughout. An upgrade to a new PostgreSQL major version is refused; contact support to plan one.
Replace a lost PostgreSQL server
A rebuilt postgres server with an empty disk copies the database from the current primary when you rerun the installer on it. Nothing else is needed while the other two servers are up.
The coordination service on the postgres servers (etcd) keeps working on two of its three members, but a member whose disk was rebuilt can't rejoin on its own:
- On a healthy
postgresserver, list the members withetcdctl member list, remove the dead one withetcdctl member remove <id>, and add it back withetcdctl member add <host id> --peer-urls=https://<address>:2380. Runetcdctlinside theetcdcontainer with the private certificate; the multi-node guide has the full commands. - On the rebuilt server, run the installer once, set
initial-cluster-state: existingin.nano-generated/ha/etcd.yml, and startetcd. - Check that
etcdctl endpoint health --clusterreports three healthy members.
Change a PostgreSQL server's address
The cache's coordinators (Sentinel) remember server addresses on their own disk. After changing a postgres server's address in the inventory, rerun the installer on every postgres and app server. Then, on each postgres server in turn, stop valkey-sentinel, delete its sentinel-data volume and start it again so it picks up the new addresses.
If the first PostgreSQL server starts empty
nano never creates a new, empty database over an existing one. If the first postgres server comes back with an empty disk at a moment when the coordination service has also lost its state (for example, after rebuilding several servers at once), it refuses to start. It logs refusing to initialise a NEW PostgreSQL cluster. Your data is still on the other postgres servers:
- On the
postgresserver whose copy is most recent (normally the former synchronous standby), start its database once in adopt mode:NANO_PATRONI_ADOPT_EXISTING=true docker compose … up -d --force-recreate --no-deps postgres-ha. It becomes the primary with its own data. - Start the other
postgresservers as usual. The empty one copies from the new primary. - Rerun the installer on the adopted server to leave adopt mode.
If no postgres server has data, restore from backup. Contact support before you use the NANO_PATRONI_ALLOW_REINIT switch, which deliberately starts an empty database.
Ingest servers
Add ingest servers to handle more events per second and to spread sender load. Any number is supported.
Deploying a log source reloads one ingest server at a time from v2.0.15. Each ingest server waits its turn, withdraws its readiness so the load balancer takes it out (/healthz answers 503), reloads, rejoins, then lets the next one go. The waits are 6 seconds each, for a load-balancer check of inter 2s fall 2 rise 2; with a slower check, raise VECTOR_CONFIG_INSTALL_DRAIN_SECS and VECTOR_CONFIG_INSTALL_REJOIN_SECS in .env. A server drains only while another ingest server is ready, so a single ingest server, or one whose partner is down, reloads in place. In v2.0.13 every ingest server reloaded at once, and ingest stalled for 10 to 15 seconds with 503s on every deploy. Give each one the ingest role and put it behind the load balancer's ingest.nano.example.com backend, as with nano-2 and nano-3. At higher volumes, move ingest off the storage servers onto servers of its own, so parsing and storage don't compete for CPU.
ClickHouse shards and replicas
- Replicas add availability. Each replica of a shard holds a full copy of that shard's events. Two replicas, as on this page, survive the loss of one server. More replicas don't add write capacity.
- Shards add write capacity and storage. Events are split across shards, and each shard stores and merges only its part. Searches run on every shard in parallel.
In the inventory, each clickhouse server has a shard and a replica number. Shard numbers start at 1 with no gaps, and every shard has the same replicas. For example, two shards with two replicas each is four ClickHouse servers: {shard: 1, replica: 1}, {shard: 1, replica: 2}, {shard: 2, replica: 1} and {shard: 2, replica: 2}.
Choose the shard and replica count before the first install. The installer can't yet add a shard or a replica to a running deployment: a new ClickHouse server starts without nano's schema, and existing events aren't moved onto a new shard. Support for this is planned. Until then, contact support before you change ClickHouse servers on a live deployment.
Keeper servers
Any layout with replicas or more than one shard needs exactly three Keepers, with IDs 1, 2 and 3. The installer refuses any other number, so five Keepers aren't supported yet. Keepers can share servers with other roles, as on this page, or run on three small servers of their own. Dedicated Keepers keep ClickHouse's coordination steady when the storage servers are busy. Decide before the first install: changing Keeper servers later is a topology change, like ClickHouse.
Add an ingest server
You can add an ingest server to a running deployment. No data moves.
-
Add a
hostsentry with a newname,idandaddress, androles: [ingest]. Install the updated inventory to/etc/nano/inventory.yamlon every server, including the new one. -
Make sure the private certificate is valid for the new address. If it isn't, issue one from the same CA that covers every address and stage it on every server first.
-
On
nano-1, rerun with its full role list. The installer refuses, because any new server changes the recorded topology, and prints the current identity. Rerun with that value:NANO_HOST_NAME=nano-1 ./install.sh --role app,postgres,keeper --inventory /etc/nano/inventory.yaml \ --accept-topology-migration <identity printed by the refusal> -
Copy the updated
~/nano/.envfromnano-1to every other server, as in Copy .env to the other servers. It now records the new topology. -
On the new server, unpack the bundle and log in to the registry as in step 1, stage the certificate, and run
./install.sh --role ingest --inventory /etc/nano/inventory.yamlwithNANO_HOST_NAMEset to its name. -
Open the ports the other ingest servers use. Add the server to the load balancer once
GET /healthzreturns200.
A rerun never narrows a server's roles. The installer adds every role the server already runs to the roles you pass, and recreates all of them. To remove a role on purpose, change the inventory and rerun with NANO_ROLE_NARROW=1.
Moving from a single node to a multi-node layout isn't an in-place change either: back up, install the new layout, and restore. Contact support to plan it.
Sizing by daily volume
Starting points for a fresh install above the minimums in Server sizing. All rows have one app server and one postgres server; add app and postgres servers for failover as above.
| Daily volume | ClickHouse | Ingest | Keeper |
|---|---|---|---|
| Up to about 100 GB | 1 shard × 2 replicas, 8 vCPU and 32 GB each | On the two storage servers | 3, shared with other roles (this page) |
| About 100–500 GB | 1 shard × 2 replicas, 16 vCPU and 64 GB each | 2 or more servers of their own, 8 vCPU and 32 GB each | 3, shared or dedicated |
| About 1 TB | 1 shard × 2 replicas, 16 vCPU and 64 GB each, NVMe | 2–3 servers of their own, 8–16 vCPU each | 3 dedicated |
| Several TB | 2 or more shards × 2 replicas, about 5 vCPU per TB/day across the shards | Plan from events per second (below) | 3 dedicated |
Ingest CPU depends on event size. Parsing costs about the same per event, so a TB of small events needs more Vector CPU than a TB of large ones. For raw logs parsed by nano, plan about 41 vCPU per TB/day at 300 bytes per event (firewall, flow, DNS), 13 at 1,000 bytes (mixed, identity, cloud audit) and 7 at 2,000 bytes (EDR). Pre-shaped OCSF from Tenzir or Cribl needs about 0.5 vCPU per TB/day. With NANO_PLANNED_GB_PER_DAY and NANO_AVG_EVENT_BYTES set, the installer says how many ingest servers of the current size you need. See Plan Vector from events per second.
Prevalence lags at volume. With a planned volume of about 1 TB/day or more, the rare and new lists refresh every 30 minutes, and every 60 minutes at 10 TB/day and up. They can lag ingest by up to that long. Detections still check a new artifact against the real-time counts.
Each ClickHouse server needs about daily volume × retention days ÷ 10 ÷ number of shards of disk, plus room for merges. The ratios come from Size the host and are extrapolated above a few hundred GB/day until load-tested. Watch ClickHouse and ingest CPU once real data flows, and contact support to review a plan above 1 TB/day.
Load balancer options
Users and senders reach nano through the load balancer, so a full-HA install needs a load balancer that survives the loss of one of its own hosts. In order of preference:
| Option | Survives losing one load-balancer host | Use it when |
|---|---|---|
| Managed cloud load balancer | Yes, run by the provider | You run on a public cloud. Recommended there. Most clouds don't pass VRRP, so a floating IP isn't an option |
| Two HAProxy hosts with keepalived and a floating IP | Yes, in about 3 seconds | Both hosts are on the same network segment and VRRP (IP protocol 112) is allowed. See the keepalived example |
| Two HAProxy hosts and DNS failover | Yes, in about 1–2 minutes | You have no floating IP and no managed load balancer. See below |
| One load balancer | No | Labs, or when you accept an outage at the edge |
Two HAProxy hosts and DNS failover. Give both HAProxy hosts the same haproxy.cfg, and publish both addresses as A records for both hostnames at a DNS provider with health-checked failover, with a TTL of 60 seconds or less. The health check removes a dead host from DNS. Put both HAProxy addresses in trusted_proxy_cidrs. This works for HEC, OTLP/HTTP and HTTP senders that retry and look up DNS again: common collectors such as Vector, the OpenTelemetry Collector and Fluent Bit, and curl, try the next address when a connection fails. Failover takes the TTL plus the health-check interval, roughly 1–2 minutes, not seconds; nano hasn't measured it. Senders that cache DNS indefinitely, such as some appliances and Java applications with an unlimited DNS cache, won't fail over: point those at a managed load balancer or a floating IP. Browsers may show errors until DNS updates. The nano-deploy planner calls this option --lb haproxy-dns.
One load balancer is a single point of failure. While it is down, nobody can sign in and senders can't deliver. The cluster behind it keeps running, and senders that buffer to disk catch up when it returns.
No floating IP and no DNS provider? Then no option above removes the edge as a single point of failure. Before you settle for that, ask for one of them: most clouds and hosting providers sell a floating (reserved) IP that you can move between two HAProxy hosts, and a DNS zone with health-checked records costs little. If neither is possible, you can still run two HAProxy hosts and give senders both addresses: log shippers that accept a list of endpoints, and servers with both addresses in /etc/hosts or their resolver, try the second one when the first fails. Browsers don't, so users switch to the other address by hand while one load-balancer host is down. Otherwise, run one load balancer and accept the outage at the edge while it is down.
Where HAProxy runs. Give each HAProxy host its own small VM (2 vCPU, 2 GB). If there is no spare VM, the nano-deploy planner puts HAProxy on a postgres server, the most lightly loaded role (app, ingest and ClickHouse servers carry the request path), and says so in the plan. Losing that server then also loses a PostgreSQL member, which stops writes and sign-ins for about 20–25 seconds if it was the primary. Use the HAProxy package from your distribution, 2.6 or later.
Host firewall
The installer never changes the host firewall. Set these rules on every server before the first install.
| Ports | Service | Filter with |
|---|---|---|
| 80 (app and ingest), 3000, 8687 | App and ingest frontends, private API, Vector tap relay | DOCKER-USER |
| 6000 (ingest, from v2.0.30) | Vector native, mutual TLS. Open it only to your Vector senders; see Bind address and firewall | Cloud firewall, or DOCKER-USER |
| 5432, 8008 | PostgreSQL, Patroni | DOCKER-USER |
| 2379, 2380 | etcd | DOCKER-USER |
| 6379, 26379 | Valkey, Sentinel | DOCKER-USER |
| 8123, 9000, 9009, 9363 | ClickHouse (host networking) | UFW or the host's INPUT rules |
| 9181, 9234 | Keeper (host networking) | UFW or the host's INPUT rules |
Drop outside connections to Docker-published ports
On every server, run this with your own private addresses. It drops a packet only when all of these hold:
- it opens a new connection, so replies are never touched;
- it arrives from outside this server, not from its own containers (
docker0and the Composebr-*bridges); - its source is not one of your nano servers, and not the load balancer reaching port 80;
- its destination is one of the published ports.
The server's own containers can still talk to each other and connect out to other servers on the same port numbers, which PostgreSQL replication and etcd need. A systemd unit re-applies the rules at boot, before Docker starts, and after every Docker start. Run the script again after any address change:
sudo tee /usr/local/sbin/nano-docker-firewall >/dev/null <<'EOF'
#!/bin/sh
# nano: filter NEW inbound connections to Docker-published ports.
set -eu
PORTS="80 2379 2380 3000 5432 6379 8008 8687 26379"
PEERS4="10.20.0.11 10.20.0.12 10.20.0.13" # every nano server's private address
PEERS6=""
LB4="10.20.0.5" # load balancer source addresses or CIDR ranges: port 80 only
LB6=""
chain() {
echo "*filter"
echo ":NANO-DOCKER - [0:0]"
echo "-A NANO-DOCKER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN"
for i in lo docker0 br-+; do echo "-A NANO-DOCKER -i $i -j RETURN"; done
for s in $1; do echo "-A NANO-DOCKER -s $s -j RETURN"; done
for s in $2; do echo "-A NANO-DOCKER -s $s -p tcp -m conntrack --ctorigdstport 80 -j RETURN"; done
for p in $PORTS; do echo "-A NANO-DOCKER -p tcp -m conntrack --ctstate NEW --ctorigdstport $p -j DROP"; done
echo "COMMIT"
}
apply() {
$1 -w -N DOCKER-USER 2>/dev/null || true
chain "$2" "$3" | "$1-restore" -w --noflush
$1 -w -C DOCKER-USER -j NANO-DOCKER 2>/dev/null || $1 -w -I DOCKER-USER 1 -j NANO-DOCKER
}
apply iptables "$PEERS4" "$LB4"
if command -v ip6tables >/dev/null 2>&1 && ip6tables -w -S >/dev/null 2>&1; then
apply ip6tables "$PEERS6" "$LB6"
fi
EOF
sudo chmod 0755 /usr/local/sbin/nano-docker-firewall
sudo tee /etc/systemd/system/nano-docker-firewall.service >/dev/null <<'EOF'
[Unit]
Description=nano: filter inbound connections to Docker-published ports (DOCKER-USER)
Before=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/nano-docker-firewall
[Install]
WantedBy=multi-user.target
EOF
sudo mkdir -p /etc/systemd/system/docker.service.d
printf '[Service]\nExecStartPost=-/usr/local/sbin/nano-docker-firewall\n' |
sudo tee /etc/systemd/system/docker.service.d/nano-docker-firewall.conf >/dev/null
sudo systemctl daemon-reload
sudo systemctl enable nano-docker-firewall.service
sudo systemctl restart nano-docker-firewall.service
sudo iptables -S NANO-DOCKER--ctorigdstportmatches the published port as the client sent it, because Docker has already rewritten the destination when the packet reachesDOCKER-USER.- The rules use their own
NANO-DOCKERchain, replaced in one step, so running the script again never leaves the ports open. - If Docker isn't installed yet, the unit creates
DOCKER-USER, and Docker uses it when it starts. Docker must use its defaultiptablesfirewall backend: withfirewall-backendset tonftables,DOCKER-USERisn't used.
Allow only cluster sources to ClickHouse and Keeper
ClickHouse and Keeper use host networking, so UFW or the host's INPUT rules cover them. With UFW, allow SSH from your management network first, then deny by default and allow each private source:
sudo ufw allow from 192.0.2.0/24 to any port 22 proto tcp # 1. your management network, first
sudo ufw default deny incoming # 2. then deny by default
sudo ufw allow from 10.20.0.11 to any port 8123,9000 proto tcp # 3. each app, ingest and ClickHouse server
sudo ufw allow from 10.20.0.12 to any port 9009 proto tcp # each ClickHouse server
sudo ufw allow from 10.20.0.11 to any port 9363 proto tcp # each app server
sudo ufw allow from 10.20.0.12 to any port 9181 proto tcp # each app, ClickHouse and Keeper server
sudo ufw allow from 10.20.0.12 to any port 9234 proto tcp # each Keeper server
for ifc in docker0 br-+; do # 4. this server's own containers
sudo ufw allow in on "$ifc" to any port 8123,9000,9009,9363,9181,9234 proto tcp
sudo ufw allow in on "$ifc" to any port 80,2379,2380,3000,5432,6379,8008,8687,26379 proto tcp
done
sudo ufw --force enable # 5. last- Step 4 matters. A server's own containers reach its ClickHouse and Keeper, and its own published ports through its own address, from a Docker bridge address rather than from the server's address. Without it, the app server's ClickHouse proxy silently loses a ClickHouse replica on the same server.
- Also allow SSH from the first app server to the others, for
scripts/nano upgrade. - On HAProxy hosts (HAProxy runs as a host service), allow
443/tcponly from the networks your users and senders connect from, for examplesudo ufw allow from 10.20.0.0/16 to any port 443 proto tcp. If an earlier rule opened 443 to every source, remove it first withsudo ufw delete allow 443/tcp(it removes the IPv4 and IPv6 forms), then check thatsudo ufw statusshows no443/tcp ALLOW AnywhereorAnywhere (v6)row. On a keepalived pair, also allow VRRP from the other HAProxy host:sudo ufw allow from <other host> proto vrrp. Without it, both hosts claim the floating address. - On RHEL with firewalld, install the
DOCKER-USERpart as shown and express these rules as firewalld rich rules.
The nano-deploy planner prints both steps for your own addresses, as one script you run unchanged on every server and every HAProxy host: plan.py … --firewall-script. It refuses to run until you set MGMT_CIDR to your management network, so the SSH rule is never left out. It refuses hostnames, and it opens 443 and VRRP only on the HAProxy hosts. With HAProxy hosts it needs --edge-source-cidr <network> (repeat it for each user, VPN and sender network) and opens 443 only to those. It opens 443 to the whole internet only if you pass --edge-source-cidr 0.0.0.0/0. When you regenerate the script this way after an earlier version, it removes the old open-to-every-source 443 rule on the HAProxy hosts before adding the scoped ones; confirm with sudo ufw status.
Test from outside the cluster
From a machine that isn't a nano server, every one of these must fail to connect, on each server's public address:
for port in 3000 5432 6379 8008 2379 2380 26379 8687 8123 9000 9181 9234; do nc -zv -w 3 <server public address> $port; doneFrom an allowed server, the same ports must connect. The nano-deploy skill's preflight checks this for you: run preflight.sh --exposure-probe <server addresses> from a machine outside the cluster. It tries every sensitive port on every address and warns about any that answer.
ClickHouse network allow-list
ClickHouse users accept connections only from loopback and the inventory servers' address and client_address, never from any address. On a server that also runs the app, ingest, PostgreSQL or monitoring containers, the local Docker address pools are admitted too, because those containers reach ClickHouse from a container address. So each server must reach the ClickHouse servers from its inventory address.
Two optional cluster keys in the inventory add to this. Each takes IPs, CIDRs or DNS names; an any-address entry is refused:
cluster:
# Extra sources for every ClickHouse user, for unusual NAT between servers.
clickhouse_allowed_networks: [10.20.9.0/24]
# External Tenzir or Cribl writers. They get the ingest user only.
clickhouse_ingest_allowed_networks: [10.30.0.15/32]On a single-server install, the same external-writer setting is NANO_CLICKHOUSE_INGEST_ALLOWED_NETWORKS in .env; see ClickHouse network allow-list. The allow-list doesn't replace the host firewall: keep both.
On a multi-shard cluster, the Tenzir pipeline writes to the distributed table, so its events spread across the shards.
Advanced
HAProxy example
global
log stdout format raw local0
# Runtime API: `show stat`, and `set server ... state drain|ready` for the
# upgrade's drain and rejoin hooks, which needs level admin.
stats socket /run/haproxy/admin.sock mode 660 level admin
defaults
mode http
log global
timeout connect 5s
timeout client 60s
timeout server 310s # long searches
timeout check 5s
frontend nano_https
bind :443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
http-request del-header X-Forwarded-For
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https
# HEC channel affinity: header first, then the ?channel= parameter.
http-request set-var(txn.hec_channel,ifnotempty) urlp(channel) if { path_beg /services/collector }
http-request set-var(txn.hec_channel,ifnotempty) req.hdr(X-Splunk-Request-Channel) if { path_beg /services/collector }
use_backend nano_ingest if { hdr(host) -i ingest.nano.example.com }
default_backend nano_app
backend nano_app
option httpchk GET /healthz # GET /readyz with two or more app servers
http-check expect status 200
server nano-1 10.20.0.11:80 check inter 2s fall 2 rise 2
# With two or more app servers: list each one, health-check /readyz, and
# retry a failed connection on another server.
# balance roundrobin
# retries 3
# option redispatch
# retry-on conn-failure # connection failures only: never resend a request the app may have applied
# server app-2 10.20.0.14:80 check inter 2s fall 2 rise 2
backend nano_ingest
balance hash var(txn.hec_channel)
hash-type consistent
# A connection that fails or times out goes to the other ingest server. This
# covers HTTP and OTLP as well as HEC. Nothing else is retried.
retries 3
option redispatch 1
retry-on conn-failure
option httpchk GET /healthz
http-check expect status 200
server nano-2 10.20.0.12:80 check inter 2s fall 2 rise 2
server nano-3 10.20.0.13:80 check inter 2s fall 2 rise 2/etc/haproxy/certs/ holds one PEM (chain plus key) per hostname. Don't enable the PROXY protocol (send-proxy). The app hostname answers 404 for ingest paths, and the ingest hostname serves only /ingest, /enrichment, /services/collector and /otlp/v1/*.
Check the file with sudo haproxy -c -f /etc/haproxy/haproxy.cfg, then sudo systemctl enable --now haproxy.
keepalived example (floating IP for two HAProxy hosts)
Use this when you run your own two load balancers, lb-1 and lb-2, on the same network segment as the cluster. Copy the same haproxy.cfg to both, then save this as /etc/keepalived/keepalived.conf on both, changing the lines marked for lb-2:
global_defs {
router_id nano_lb_1 # nano_lb_2 on lb-2
enable_script_security
script_user root
}
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 1
fall 2
rise 2
}
vrrp_instance nano_vip {
state BACKUP # both BACKUP with nopreempt: no move back
nopreempt
interface eth0 # the cluster-facing interface
virtual_router_id 51 # unique on this network segment
priority 150 # 100 on lb-2
advert_int 1
unicast_src_ip 10.20.0.6 # this host (10.20.0.7 on lb-2)
unicast_peer {
10.20.0.7 # the other host (10.20.0.6 on lb-2)
}
authentication {
auth_type PASS
auth_pass changeme # replace: your own 8 characters, same on both
}
virtual_ipaddress {
10.20.0.5/24 # the floating IP both DNS names point at
}
track_script {
chk_haproxy
}
}killallcomes from thepsmiscpackage. Install it if it's missing.- Allow VRRP (IP protocol 112) between
lb-1andlb-2in the firewall. - If
haproxy.cfgbinds the floating address itself rather than:443, setnet.ipv4.ip_nonlocal_bind=1on both hosts. - VRRP's password is sent in clear text. Keep
lb-1andlb-2on a segment where only they can send VRRP. - Check the file with
sudo keepalived -t -f /etc/keepalived/keepalived.conf, thensudo systemctl enable --now keepalived.ip -4 addr show eth0onlb-1shows the floating IP. - Most public clouds don't pass VRRP or let a VM claim a floating IP this way. There, use the cloud's managed load balancer with the same health checks.
Ports between the servers
Only the load balancer should be reachable by users and senders, plus port 6000 on the ingest servers if you use Vector native. Permit the other ports on the private network only, and enforce it as in Host firewall: UFW alone doesn't cover the Docker-published ports.
| Traffic | Port | From → to |
|---|---|---|
| App frontend | 80 | Load balancer → nano-1 |
| Ingest frontend | 80 | Load balancer → nano-2, nano-3 |
| Private API | 3000 | nano-2, nano-3 → nano-1 |
| Vector tap relay | 8687 | nano-1 → nano-2, nano-3 |
| Vector native, mutual TLS (from v2.0.30) | 6000 | Your Vector senders → nano-2, nano-3. See Vector native |
| PostgreSQL | 5432 | Every server → nano-1 |
PostgreSQL failover (three postgres servers only) | 5432, 8008, 2379, 2380, 6379, 26379 | Between the postgres servers, and app servers → postgres servers |
| ClickHouse HTTP, native, interserver | 8123, 9000, 9009 | Every server → nano-2, nano-3 |
| Keeper client, Raft | 9181, 9234 | Between all three servers |
| SSH (upgrades) | 22 | nano-1 → nano-2, nano-3 |
Outbound access is the same as for a single node. Every server pulls images at install and upgrade. nano-1 makes the license check-in and the content updates.
How the layout works
The bundle splits nano into five roles: app (UI, API, search, jobs), postgres, clickhouse (event storage), keeper (ClickHouse's coordination service) and ingest (Vector and its edge). nano-2 and nano-3 each hold a full copy of the events. Three Keepers, one per server, keep ClickHouse writable while two of them are up.
Two copies add availability, not capacity: each copy stores and merges the full stream. The installer sizes each server's containers from its own CPU and RAM. On an 8 vCPU, 32 GB storage server, ClickHouse gets about 22 GB of memory and about 6 vCPU. For the minimum per server, see Server sizing.
Running app and ingest on the same server is supported on every release you can download (v2.0.4 and later). The standard layouts keep ingest off the app servers so sender load never competes with the UI.
The inventory's deployment_id, each server's name, id and roles, and the ClickHouse shard/replica and Keeper id name the data volumes. The installer refuses to change them after the first install. For other layouts, see Custom multi-node layouts.
Failure details
| Failure | Ingest | Search and UI | Recovery |
|---|---|---|---|
nano-2 or nano-3 down | Continues on the other node. The load balancer drops the failed node in about 9 s | Continue on the other copy | Start the server. It rejoins in about 15–18 s and fetches what it missed |
| ClickHouse stopped on one server | Continues | Continue | Start it; it catches up from its peer |
nano-1 down | Stops at the load balancer about 15 s later | Down | Start the server. The UI is back in about 6 s and ingest in 15–22 s |
| Two servers down | Events wait in disk buffers, then senders get 429 | Down or read-only | Start one server to restore the Keeper quorum |
Measured on a three-server test install at 141 events per second: stopping either storage server lost and duplicated no events.
Known limitations in 2.0.x:
- Each time you deploy a parser or change a source, every ingest node reports not ready for about 30 seconds while it loads the new configuration. Make these changes in a quiet period.
- If Deploy reports a failure on a multi-server install, contact support before you retry.
- A rerun with only some of a server's roles, or a failed image pull during a rerun, can leave ClickHouse not listening on the private network. Rerun that server with its full role list and a fresh registry login.
Troubleshoot
Need help? Contact support with your deployment ID, the inventory without secrets and the time, with timezone.
The installer refuses the inventory
The message names the field. Common causes: two roles on one server use the same port, a server name that doesn't match the machine (set NANO_HOST_NAME), or not exactly three Keepers.
The installer refuses the CPU
See single node. In this layout, every server runs ClickHouse or Keeper, so all three need a supported CPU.
The license shows as invalid
The inventory's deployment_id is what nano sends to the license check. It must equal the deployment ID in app.nano.rs. On a new install with the wrong value, remove it and reinstall with the correct one.
An ingest node stays out of the load balancer
/healthz returns 503 until the node has loaded its configuration from nano-1. Check that it reaches nano-1 on port 3000, then rerun that server with its full role list.
A storage copy is behind after a server returns
It catches up on its own. SIEM Health shows replica lag.
The API logs "Refusing to start: DualPool initialization failed" after a restart
For a few seconds after a server restarts, the API can't reach PostgreSQL or ClickHouse yet and restarts itself until they answer. It recovers on its own. If the message continues for more than a minute, check that the server reaches the postgres servers on 5432 and its ClickHouse proxy (cluster-health.py from the nano-deploy skill checks both).
An app server is DOWN for the ingest servers after a reboot
On v2.0.12 and earlier, after an app server reboots, its internal API edge (the app-internal-edge container) can keep the address the API container had before the reboot. The ingest servers check the app servers through that edge, so they mark that app server DOWN and send OTLP authentication and ingest checks to the other app servers only. Nothing in the UI shows it: /readyz answers 200 and the API is healthy. SIEM Health doesn't report it either, because it doesn't see containers.
After an app server reboots, run scripts/nano cluster-health from ~/nano on each ingest server. If the gw:app-api-proxy row fails for that app server, restart its edge on the app server:
docker restart "$(docker ps --format '{{.Names}}' --filter name=app-internal-edge)"From v2.0.13, the edges look up container addresses at request time, so this doesn't happen. scripts/nano cluster-health on an app server also has an edge:internal row that fails when the edge can't reach the API, and prints the restart command.
See the load balancer's view of each server
Add a runtime socket to the global section of haproxy.cfg (stats socket /run/haproxy/admin.sock mode 660 level admin, as in the HAProxy example; level operator is enough to read it, but the upgrade's drain hooks need level admin), reload HAProxy, then run echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18. Each server line shows UP or DOWN.
Self-run
Start here if you bought nano Self-run: what it includes, your license values, installs per license, the minimum version, install methods and getting help.
Custom multi-node layouts
Split nano Self-run across Linux servers in a layout other than the standard three-node HA design, with a reviewed inventory.