nano v2.0.29
nano v2.0.29: a security release. A package extract no longer widens ClickHouse's network allow-list or resets its sizing, busy single ingest hosts upgrade without pausing senders, and image bundles work on registry installs.
v2.0.29 includes everything in v2.0.28. It is a security release. Every Compose install that was upgraded to v2.0.28 by extracting the installer package should upgrade.
To upgrade, follow Upgrade a self-hosted deployment and the Upgrade notes below. v2.0.20 is a required stop: installs on a release older than v2.0.20 upgrade to v2.0.20 first.
Security
- Extracting the v2.0.28 package could widen ClickHouse's network allow-list and reset its sizing. Who is affected: enterprise Compose installs that were upgraded from v2.0.27 to v2.0.28 by extracting the installer package over the install directory. Up to v2.0.28, Compose mounted the rendered allow-list and sizing as single files over placeholders inside the shipped
clickhouse/config.dandclickhouse/users.ddirectories. Extracting a package replaces those placeholders, which detaches the mounts on Linux, and ClickHouse reloaded the placeholders. Every ClickHouse user then accepted connections from all private ranges (10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,fc00::/7) instead of the pinned nano network, and ClickHouse's sizing limits went back to the defaults. In v2.0.29 ClickHouse mounts only two generated directories under.nano-generated/, which no package contains, so an extract can't change what it reads. It fails closed: if those directories are missing or were not assembled, ClickHouse refuses to start instead of starting with a wider allow-list. What to do: upgrade to v2.0.29. The upgrade re-renders the directories and recreates ClickHouse, which restores the pinned allow-list and the sizing. The upgrade's lockout check recognizes this state and no longer asks forNANO_ACK_CH_NETWORK_LOCKOUT.
Fixes
- A busy single ingest host upgrades without pausing senders. In v2.0.28,
scripts/nano upgradestopped atrollout:ingest("Vector buffers did not drain within 300s") while senders pushed logs at full rate. Now the upgrade closes the host's ingest edge to new events while Vector drains and restarts, then reopens it as soon as Vector is healthy. See Busy single ingest host under Upgrade notes. - Image bundles work on registry installs.
scripts/nano upgrade --bundle nano-<version>-enterprise.tarused to fail atrollout:migration("No such image") on installs that pull from another registry repository, such as an install from app.nano.rs, after Keeper, ClickHouse and PostgreSQL had already been upgraded. The bundle's images are now retagged to the install'sNANO_IMAGE_PREFIXafter their digests verify. The dry run reports this asbundle-image-names, and refuses a bundle it can't match before any host changes. - Install prompts. In a terminal, answering anything other than y or n to a yes/no question (the outbound proxy questions, for example) could count as "no" and skip the proxy. Bad answers are now asked again with the reason, including a bad offline license path. The admin email must look like
local@domain, and a weak password is asked again instead of ending the install. The public base URL must behttp(s)://host[:port][/path]. Pressing Enter on an invalid recorded base URL no longer recordshttp://localhost. NANO_BASE_URLis validated. A value that is not anhttp(s)://URL now fails an unattended install at once. Trailing spaces and line endings are trimmed.- The Search sidebar never misses a field. It lists the fields from the events the search matches, so a field that only one host sends is listed. When the list or the field stats cover only part of the results, the section says so (
partial, or "Stats from the first N matching events"), with the reason. A failed request shows as a labeled section instead of a missing one. In-queryearliest=andlatest=now apply to the sidebar too, so with the time picker at 15 minutes andearliest=-7din the query, the sidebar covers 7 days like the results. - Disk stats work on several hosts. SIEM Health's disk table and the disk pressure check read disk details from every ClickHouse replica. On several hosts, that read failed with
Not enough privileges ... SELECT ON system.disks. The nano ClickHouse user now has the missing columns.
Changes
- An older upgrader warns on a newer bundle.
scripts/nano upgraderuns the steps of the release it belongs to. If it is older than the bundle, it now prints a warning that the new release's steps will be skipped, and says to extract the installer package first. It is a warning, not a refusal. - HEC
/1.0routes are rate limited./services/collector/event/1.0and/services/collector/raw/1.0now get the same admission and limits as/eventand/raw: an 8 MiB request cap (previously 100 MiB), 50 requests per second and 16 connections per client address. Many senders behind one NAT address can see429, and very large batches413. Point those senders at/eventor/raw, or split the batches. On Helm the ingestion pause now covers the/1.0routes too.
Upgrade notes
-
Required stop at v2.0.20. See above.
-
Extract first, then upgrade. The extracted release's upgrader is the one that runs. Do these steps in order:
- On every host, as the user that owns the install directory (not root): verify the v2.0.29 installer package,
nano-compose-<edition>-2.0.29.tar.gz, and extract it over the install directory. This applies to image-bundle upgrades too. - Several hosts only, on the app host: run the new installer as
NANO_PREPARE_ONLY=1 ./install.sh --inventory <inventory> --role <its roles>. It writes the new shared secrets in.envand starts nothing. - Several hosts only: copy that
.envto every host. - Dry run. From the app host, run the extracted
scripts/nano upgrade … --dry-run. On several hosts, pass--inventory …as usual. - Upgrade. When every check passes, run the same command without
--dry-run.
Between step 1 and step 5, don't run
docker compose upby hand. See the warning at the top of this page. - On every host, as the user that owns the install directory (not root): verify the v2.0.29 installer package,
-
HEC
/1.0limits. Check senders behind a shared NAT address and senders that post large batches to/event/1.0or/raw/1.0. See Changes above. -
Unattended installs with
NANO_BASE_URL. Make sure the value is a fullhttp(s)://URL before you re-runinstall.sh, or the install fails. -
Migrations run automatically. New PostgreSQL and ClickHouse migrations apply during the upgrade.
Busy single ingest host
Senders that push to nano can keep sending during the upgrade. Before the ingest host's Vector restarts, every ingest route (/ingest, /enrichment, OTLP, and HEC event and raw, including /1.0) answers 503 with Retry-After: 5, and /healthz answers 503 too. The window usually lasts a few seconds. Its upper bound is the 300-second drain limit plus Vector's restart, at most 180 seconds. Senders must keep retrying through it. Vector, OpenTelemetry exporters and HEC forwarders retry by default. Fluent Bit's default Retry_Limit is 1, which drops a chunk that fails twice: set Retry_Limit no_limits (or a high number) on every output that sends to nano.
- Through a load balancer, the LB sees
/healthzanswer503and takes the host out of the pool. Senders then get the LB's own answer, with noRetry-After. - The upgrade from v2.0.28 or earlier: the running edge predates the closed state, so for this one upgrade the edge is stopped for the window instead, and senders see a refused connection.
- HEC indexer acknowledgements: the restart forgets outstanding ack IDs, so a sender that uses them resends those batches and some events can arrive twice.
- Pull sources (Kafka, S3 through SQS, Pub/Sub, Event Hubs, syslog) keep receiving. For a Kafka, S3 or Pub/Sub source you configured yourself, enable
acknowledgementson it so it commits its position only once an event is in Vector's disk buffer. - If the upgrade is interrupted, the next
scripts/nano upgrade --resumeor--abandon-incompletereopens the edge first.scripts/nano cluster-healthon the host reports a closed ingest edge asclosed to new events (503), and fails with!!! CLOSEDonce it has been closed for more than 15 minutes. It prints the command to reopen it. With no upgrade running, reopen it by hand withdocker compose exec ingest-edge rm -f /run/nano-ingest/closed, ordocker compose up -d --no-deps ingest-edgeif it is stopped.
nano v2.0.30
nano v2.0.30: Vector native port 6000 always requires mutual TLS, with a certificate per sender. The installer checks the host before it asks anything, shows a review screen, and Self-run installs and upgrades log in to the registry themselves.
nano v2.0.28
nano v2.0.28: the interactive install.sh fix, one ClickHouse user per workload, ingest no longer refused at the hourly query quota, and upgrade and Helm fixes.