Install on Kubernetes (Helm)
Install nano Self-run on your own Kubernetes cluster with the nano Helm chart, then size it, make it highly available and upgrade it.
Installing on Kubernetes needs a Self-run license (paid). Open (free) installs with Docker Compose. You set your license values in step 3, and they unlock the Self-run features and high availability.
Install nano Self-run on a Kubernetes cluster that you operate, with the nano Helm chart. The chart runs the whole stack, including ClickHouse through the official ClickHouse operator. For one server without Kubernetes, see Install with Docker Compose.
Before you start
For about 12.5 GB/day (about 150 events per second), you need:
| Item | Requirement |
|---|---|
| Kubernetes | 1.28 or newer (1.30 or newer recommended), and Helm 3.14 or newer |
| Capacity | About 8 vCPU and 32 GB RAM free. For HA, six worker nodes with 8 vCPU and 32 GB each, two in each of three zones; see Highly available values. Check that your cloud quota allows 48 vCPUs or more of that instance type first |
| Storage | A default StorageClass, with 1.5 TiB for a year of events or 500 GiB for 90 days. GKE and AKS ship one; EKS has none out of the box, see Amazon EKS. Or name one with global.storageClass |
| CPU | Any x86-64 server CPU from about 2015 or later, or ARM64 |
| Ingress | A Gateway API implementation, such as Envoy Gateway, GKE Gateway, the AWS Load Balancer Controller v2.13 or newer, or Azure Application Gateway for Containers |
| Network policy | A network plugin that enforces Kubernetes NetworkPolicy. Most managed clusters don't by default; see Network policy enforcement |
| TLS | A cloud-managed certificate (ACM on EKS, a Certificate Manager map on GKE), or cert-manager |
| DNS | Two names in a domain you control: one for the UI (nano.example.com) and one for senders (ingest.nano.example.com) |
| PostgreSQL | Bundled for small installs. For production, a managed PostgreSQL 16 or newer with three extensions allowed; see Managed PostgreSQL requirements |
From your Self-run deployment you also need the following. Each comes from the deployment's page in app.nano.rs or, if you only have the three license values (deployment ID, license URL, license token), from the license API as noted:
-
The version to install, from the release notes or the deployment's page. Set it once; every command below uses it. This reads the current release; to install another one, set it by hand (
export NANO_VERSION=<version>):export NANO_VERSION=$(curl -fsS https://api.nano.rs/api/releases/latest \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["version"])') echo "$NANO_VERSION" -
The three license values, from 2 · License on the deployment's page. The token is shown only once. 3 · Install → Kubernetes shows them as Helm values. One Helm release is one install, however many api and jobs pods it runs; see How many installs a license covers.
-
A registry credential, from 4 · Registry access → Generate pull credential, or with only the license values:
( umask 077 printf 'Authorization: Bearer %s\n' "$(cat license-token.txt)" > license-header.txt curl -fsS -H @license-header.txt \ "https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<deployment ID>" \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"], end="")' > registry-password.txt rm -f license-header.txt )curl reads the token from a header file and the password goes straight to
registry-password.txt, so neither appears on a command line, in the process list or in your shell history. Thepasswordfield is the pull password (usernameoauth2accesstoken). It pulls the chart and the images, and lasts one hour. From chart 2.0.10, the chart keeps it fresh in the cluster; see Registry access.
You don't need the Compose bundle or a hand-built image lock: the chart ships the release's image lock and the image values made from it.
Network policy enforcement
Vector's OTLP receiver (4317 gRPC, 4318 HTTP) has no credential check of its own, and before chart 2.0.30 neither had its Vector-native listener (6000). How nano keeps other pods away from them depends on the chart version:
- From chart 2.0.11: both listen on loopback only, inside the Vector pod. The only way in is an
otlp-guardsidecar (onnano-vector-internal, port 4318) that forwards a request only when it carries the ingest edge's hop token (vector_otlp_hop_tokenin the chart secret). The edge sends that token after it has checked the sender's ingest token. Without it, the guard answers401, whatever the network plugin. OTLP/gRPC and Vector-native aren't on any Service. The chart's NetworkPolicy,nano-vector-ingress, stays as defense in depth. - Chart 2.0.10 and earlier: the listeners are reachable on the pod network, and the NetworkPolicy is the only thing that keeps other pods out.
Kubernetes stores a NetworkPolicy whatever the network plugin, but only a plugin that enforces it applies it. On a cluster without enforcement, nothing warns you. Turn it on: required on 2.0.10 and earlier, recommended from 2.0.11.
Vector-native (6000). From chart 2.0.30 it requires mutual TLS: every sender needs a client certificate; see Vector-native client certificates. With the default vector.native.enabled: false, it listens only on loopback inside the Vector pod and no Service exposes it. From 2.0.11 to 2.0.29 it had no credential check, and vector.native.enabled: true exposed it on nano-vector-internal for any pod the NetworkPolicy admitted. Before 2.0.11, the port was always reachable in-cluster.
Vector metrics (9598). Vector's Prometheus endpoint has no credential. It serves read-only counters (component names, throughput, errors), never events. On chart 2.0.11 the chart's policy admits it from every pod in the cluster; from 2.0.12 only from pods in nano's namespace (nano's own pods read it). To scrape it from a monitoring namespace, add that namespace with vector.metrics.networkPolicyFrom, for example [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}}].
Turn on enforcement when you create the cluster:
| Cluster | How |
|---|---|
| GKE | Dataplane V2 (--enable-dataplane-v2), which you can only choose at creation. On an existing cluster without it, turn on the network policy add-on instead (--update-addons=NetworkPolicy=ENABLED, then --enable-network-policy); this recreates the nodes. See GKE cluster |
| EKS | The VPC CNI network policy agent: VPC CNI v1.14 or newer with enableNetworkPolicy: "true" in the vpc-cni add-on's configuration. Set it when you create the cluster; on an existing cluster, follow Cluster exactly, because a plain update-addon can break pod networking |
| AKS | --network-dataplane cilium (Azure CNI powered by Cilium), or --network-policy azure or calico. See AKS cluster |
| Other clusters | A plugin that enforces NetworkPolicy, such as Calico or Cilium |
After the install, check from a throwaway pod in another namespace that it can't reach Vector's OTLP port:
kubectl run np-test -n default --rm -i --image=curlimages/curl:8.12.1 --restart=Never --command -- \
sh -c 'curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://nano-vector-internal.nano.svc.cluster.local:4318/v1/logs; true'Change nano in the host name if your release or namespace differs. With enforcement, the request times out and the command prints 000 (the pod still ends normally). Without enforcement, chart 2.0.11 and later answer 401 (the guard refuses a request without the hop token), which is safe but means the policy isn't enforced; on 2.0.10 and earlier, any HTTP status means anyone in the cluster can write OTLP.
Vector-native client certificates
From chart 2.0.30, the Vector-native listener (6000) always requires mutual TLS. There is no plaintext mode. vector.native.enabled: true publishes it on nano-vector-internal, and vector.service.exposeNative: true also on nano-vector. Both are ClusterIP Services, so senders run in the cluster. For senders outside it, put your own TCP load balancer in front (no TLS termination) and add its name or address to vector.native.tls.extraDnsNames or extraIPs.
The chart keeps two separate trust paths. With release nano:
nano-vector-native-ca: the listener CA. Generated once, kept on uninstall, mounted into no pod. It signsnano-vector-native-tls, the listener certificate (the two Vector Services' in-cluster names,localhost,127.0.0.1and your extra names), which is re-signed when the CA or the names change. Senders verify nano with this CA's certificate.nano-vector-native-clients(keyca.crt): the sender anchor bundle, one anchor per sender. It starts with a locked anchor, so no sender is accepted until you add one. The chart never overwrites it and keeps it on uninstall.
Helm reads both back with lookup. A renderer without cluster access (helm template, Argo CD) would make a new listener CA on every render, so GitOps installs set vector.native.tls.existingSecret: a Secret with tls.crt and tls.key (the listener) and ca.crt (the anchor bundle). With secrets.existingSecret and no TLS Secret, the listener is locked and vector.native.enabled is refused.
Issue a sender certificate on an operator workstation with the nano release bundle or a nano checkout:
umask 077; mkdir -p nano-vn
kubectl -n nano get secret nano-vector-native-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > nano-vn/listener-ca.crt
scripts/nano vector-client-cert edge-dc1 --pki-dir nano-vn --server-ca nano-vn/listener-ca.crt \
--address nano-vector-internal.nano.svc:6000
# publish the anchor bundle, then restart Vector so it reads it
kubectl -n nano create secret generic nano-vector-native-clients \
--from-file=ca.crt=nano-vn/client-ca.crt --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nano rollout restart deployment/nano-vectorChange nano in the names if your release or namespace differs. The command prints the Vector sink config (see Vector native). --csr FILE keeps the sender's key on the sender. --renew edge-dc1 --pki-dir nano-vn re-signs a sender under the same anchor, with no Secret change and no restart.
To revoke, run scripts/nano vector-client-cert --revoke edge-dc1 --pki-dir nano-vn, then apply the Secret and restart Vector as above. Only that sender's anchor leaves the bundle. The revocation takes effect only when the restart has replaced every Vector pod: until then, pods that haven't restarted still accept the sender. Check with kubectl -n nano rollout status deployment/nano-vector.
- Keep
nano-vn. It holds the anchors and their keys, and nothing backs it up for you. Without it you can't renew or revoke, and the next issue publishes only the anchors you still have. - Expiry.
scripts/nano vector-client-cert --list --pki-dir nano-vnmarks senders that expire within 30 days. The chart's listener certificate lasts 10 years. - Upgrading from 2.0.29 or earlier with
vector.native.enabled: true: senders without a client certificate are refused from the upgrade on. The 2.0.30 chart creates the listener CA during the upgrade, so issue each sender's certificate right after it (as above) and update the sender's sink config; plan for a short gap in that sender's data until you do.
Egress NetworkPolicy (optional)
From chart 2.0.27, networkPolicy.egress.enabled: true adds default-deny egress policies for nano's own pods. Every nano pod can then reach DNS and pods in nano's namespace. api, search, jobs and the connector runner can also reach the Kubernetes API (networkPolicy.egress.kubeApi.cidrs, required), external PostgreSQL (networkPolicy.egress.postgresql.cidrs), your egress proxy (networkPolicy.egress.proxy) and any networkPolicy.egress.extraEgress rules. Everything else is blocked. It needs a plugin that enforces egress policy, from the table above. On EKS that means Linux EC2 nodes; the VPC CNI doesn't enforce policy on Fargate or Windows nodes.
DNS. networkPolicy.egress.dns.mode sets where nano's pods may send DNS:
any(the default from 2.0.28): UDP and TCP 53 to any destination, and no other port. It works with every cluster DNS path without extra values. The tradeoff: a compromised nano pod could query an outside resolver.peers: only the peers innetworkPolicy.egress.dns.to. The default list has kube-dns/CoreDNS pods, NodeLocal DNSCache pods,169.254.20.10/32(NodeLocal DNSCache) and169.254.10.10/32and169.254.10.11/32(AKS LocalDNS). Check it against your cluster first. A wrong list cuts DNS for every nano pod.
| Cluster | DNS path | With peers |
|---|---|---|
| GKE Dataplane V2 with Cloud DNS, including Autopilot (its default) | NodeLocal DNSCache is on by default (Autopilot always; Standard from 1.34.1-gke.3720000). With Cloud DNS, pods query 169.254.20.10 on the node's host network | Not supported: the Cilium dataplane can't match a host-network address with an ipBlock. Use any |
| GKE Dataplane V2 without Cloud DNS | NodeLocal DNSCache pods run on the pod network | The default list |
| GKE without Dataplane V2 (network policy add-on) | NodeLocal DNSCache and kube-dns | The default list, plus the kube-dns Service IP as a /32: kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' |
| EKS | CoreDNS pods (k8s-app: kube-dns). NodeLocal DNSCache is optional and self-installed, on the host network at 169.254.20.10 | The default list. With NodeLocal DNSCache, prefer any |
| AKS | CoreDNS pods. With LocalDNS (AKS Automatic, and AKS Standard 1.37 and later in Preferred mode), pods query 169.254.10.10 and 169.254.10.11 on the node | With Cilium, an ipBlock can't select node addresses, so LocalDNS is unreachable. Use any |
| Calico or Cilium elsewhere | kube-dns pods, or a NodeLocal DNSCache on the host network | Calico: the default list. Cilium with a host-network cache: use any |
Kubernetes API addresses. Get them with kubectl get endpoints kubernetes -o jsonpath='{.subsets[*].addresses[*].ip}' and give them as /32s, with these exceptions:
- GKE private clusters: the control plane range.
- EKS: with the private endpoint on, the addresses are the control plane's network interfaces in the cluster subnets. EKS adds new ones during a cluster upgrade, so list the cluster subnet CIDRs. A public-only endpoint has public addresses that change; turn on the private endpoint first.
- AKS with API Server VNet Integration (the AKS Automatic default): the delegated API server subnet CIDR. Turning integration on changes the address.
- Self-managed clusters with Cilium, where the API server runs on the control plane nodes: Cilium doesn't match node addresses with an
ipBlockunless it runs withpolicy-cidr-match-mode=nodes.
The kubernetes Service ClusterIP (chart 2.0.30 and later). Pods reach the API at kubernetes.default.svc, a ClusterIP such as 172.20.0.1. The EKS VPC CNI network policy agent matches egress ipBlocks against that ClusterIP, not the addresses behind it. With only the addresses above, api, search, jobs and the connector runner stay in Init forever. networkPolicy.egress.kubeApi.serviceIP adds the ClusterIP to the Kubernetes API rule as a /32. It does nothing on GKE and AKS.
auto(the default):helm installandhelm upgradelook upService default/kubernetes.helm templateand GitOps renderers such as Argo CD can't look it up, so set the address there. If the installing user can't read Services indefault, the render fails: set the address, ornone.- An address:
kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'(dual-stack:"<IPv4>,<IPv6>"). none: onlykubeApi.cidrs.
When the rule has no ClusterIP, helm install and helm upgrade print a note with the command to add it. On chart 2.0.29 and earlier, add the ClusterIP as a /32 to kubeApi.cidrs instead.
EKS: no ipBlock.except in extraEgress or vectorExtraEgress. The VPC CNI agent applies except ranges as denies that override other policies' allows. A 0.0.0.0/0 rule with except covering the private ranges cuts DNS for every nano pod. List the public destinations as plain cidrs instead. From chart 2.0.30, helm install and helm upgrade warn when they see except in either list.
Pods that use workload credentials also need the metadata endpoint, through extraEgress:
- EKS Pod Identity:
169.254.170.23/32on TCP 80. - GKE Workload Identity:
169.254.169.252/32on TCP 988, plus169.254.169.254/32on TCP 80 with Dataplane V2.
Turn it on safely. The policies select pods by release, not version, so they apply at once to the running pods as well as new ones.
- Turn it on in its own
helm upgrade, with the same chart version and only thenetworkPolicy.egresschange. - Read the render first:
helm template ... --is-upgrade --show-only templates/networkpolicy-egress.yaml. - Run the upgrade with
--wait --timeout 10m --rollback-on-failure(Helm 4) or--atomic(Helm 3). If pods don't become ready, Helm restores the previous revision, which removes the policies. - Check DNS from a pod:
kubectl -n nano exec deploy/nano-api -- getent hosts kubernetes.default.svc.
To back out by hand, run helm upgrade with --set networkPolicy.egress.enabled=false, or helm rollback.
ClickHouse network allow-list
From chart 2.0.20, clickhouse.allowedNetworks sets the networks that every nano ClickHouse user accepts connections from. Set it. List your pod CIDRs, and your node CIDRs if a LoadBalancer sits in front of ClickHouse. Entries are IPs, CIDRs or DNS names. An any-address entry such as 0.0.0.0/0 or ::/0 is refused.
This starting value covers the pod networks of the common network plugins and of GKE, EKS and AKS:
clickhouse:
allowedNetworks: ["127.0.0.1", "::1", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10", "fc00::/7"]If your pods use other ranges, such as 198.18.0.0/15 or global IPv6, list those too.
While it's unset (the default, and on --reuse-values upgrades), the previous behavior stays: users accept connections from any address, limited only by the ClusterIP Services and the chart's NetworkPolicy. Nothing is locked out on install or upgrade. helm install and helm upgrade print a warning, and SIEM Health reports "ClickHouse users accept connections from any address".
To let a Tenzir or Cribl writer outside the cluster write events directly, add its networks to clickhouse.ingestAllowedNetworks. They are admitted as the ingest user only. With a LoadBalancer in front of ClickHouse, set externalTrafficPolicy: Local, or ClickHouse sees node addresses.
Amazon EKS
Skip this section on other clusters. On EKS, values-aws.yaml exposes nano through two ALBs from the AWS Load Balancer Controller's Gateway API support, with TLS from ACM. Prepare the cluster as follows before you install.
Cluster
A small HA cluster for about 12.5 GB/day: three Availability Zones, a NAT gateway per zone, OIDC for IAM roles for service accounts (IRSA), and two m6i.2xlarge nodes per zone on private subnets, one managed node group per zone. Change the name, region and zones:
# cluster.yaml: eksctl create cluster -f cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: nano
region: <region>
version: "1.37"
tags: {nano-deployment: nano} # on the CloudFormation stacks, VPC, subnets, NAT gateways and IAM roles
availabilityZones: ["<region>a", "<region>b", "<region>c"]
vpc: {cidr: 10.42.0.0/16, nat: {gateway: HighlyAvailable}}
iam: {withOIDC: true}
addons:
- name: vpc-cni
configurationValues: '{"enableNetworkPolicy": "true"}' # enforce NetworkPolicy
- name: coredns
- name: kube-proxy
- name: eks-pod-identity-agent
- name: metrics-server
- name: aws-ebs-csi-driver
wellKnownPolicies: {ebsCSIController: true}
managedNodeGroups:
- name: ng-a
availabilityZones: ["<region>a"]
instanceType: m6i.2xlarge
minSize: 2
desiredCapacity: 2
maxSize: 2 # six nodes in all; there is no cluster autoscaler
tags: {nano-deployment: nano} # on the instances and their volumes
volumeSize: 100
volumeType: gp3
privateNetworking: true
# ng-b and ng-c: the same, with availabilityZones ["<region>b"] and ["<region>c"]The vpc-cni configuration turns on the VPC CNI's network policy agent, which needs VPC CNI v1.14 or newer; see Network policy enforcement. Set it here, at creation.
On an existing cluster, keep the add-on's IAM role when you change its configuration. eksctl creates the vpc-cni add-on with a service account role; an update-addon without --service-account-role-arn drops it, and the new aws-node pods then crash-loop on ec2:DescribeNetworkInterfaces while the add-on sits in UPDATING. A node replaced in that state gets no pod networking. Read the role first, pass it back, and wait for the rollout:
ROLE=$(aws eks describe-addon --cluster-name nano --addon-name vpc-cni \
--query 'addon.serviceAccountRoleArn' --output text)
echo "$ROLE" # "None" means the add-on uses the node role; then leave out --service-account-role-arn
aws eks update-addon --cluster-name nano --addon-name vpc-cni \
--service-account-role-arn "$ROLE" --resolve-conflicts PRESERVE \
--configuration-values '{"enableNetworkPolicy": "true"}'
aws eks wait addon-active --cluster-name nano --addon-name vpc-cni
kubectl -n kube-system rollout status daemonset/aws-node --timeout=10m
kubectl -n kube-system get pods -l k8s-app=aws-node # every pod Running, no restartsIf rollout status stalls with an aws-node pod in CrashLoopBackOff, the role was lost: put it back with kubectl -n kube-system annotate sa aws-node eks.amazonaws.com/role-arn=<role> --overwrite, delete the failing pod, and rerun the update-addon above with --service-account-role-arn.
EBS volumes are bound to one Availability Zone, so a ClickHouse replica or Keeper member can only restart on a node in its own zone. With two nodes per zone, a drained or failed node's pods move to the other node in that zone. With one node per zone (three nodes), nano survives the loss of a node with its data safe, but that zone's ClickHouse replica and Keeper member stay Pending until a node returns to the zone, and you have no redundancy meanwhile. One node group spanning the three zones with desiredCapacity: 6 also works, but it doesn't guarantee two nodes in each zone.
Storage: EBS CSI driver and a default StorageClass
The aws-ebs-csi-driver add-on above runs with an IAM role (wellKnownPolicies: {ebsCSIController: true}). On an existing cluster, add it with eksctl create addon and the same policy. EKS has no default StorageClass, so every nano volume stays Pending until you create one:
# gp3.yaml: kubectl apply -f gp3.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
type: gp3
tagSpecification_1: "nano-deployment=nano" # on every nano EBS volume
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: trueIf you don't want a cluster default, set global.storageClass: gp3 and clickhouse.storage.storageClass: gp3 instead.
AWS Load Balancer Controller with Gateway API
You need controller v2.13 or newer, running with an IAM role (IRSA or Pod Identity), with its Gateway API support switched on. Pick the controller version first; these steps use v3.6.2, the version of our latest test:
-
Create the controller's IAM policy from its release, and a service account with an IAM role that has it (IRSA, through the cluster's OIDC provider from
iam: {withOIDC: true}):LBC_VERSION=v3.6.2 ACCOUNT=$(aws sts get-caller-identity --query Account --output text) curl -fsSLo lbc-iam-policy.json \ "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/$LBC_VERSION/docs/install/iam_policy.json" aws iam create-policy --policy-name nano-AWSLoadBalancerControllerIAMPolicy \ --policy-document file://lbc-iam-policy.json --tags Key=nano-deployment,Value=nano eksctl create iamserviceaccount --cluster nano --region <region> \ --namespace kube-system --name aws-load-balancer-controller \ --attach-policy-arn "arn:aws:iam::$ACCOUNT:policy/nano-AWSLoadBalancerControllerIAMPolicy" \ --approveUse the
iam_policy.jsonof the controller version you install: newer versions need permissions that older policies lack. When you delete the cluster, delete this policy after it (aws iam delete-policy). -
Install the Gateway API standard CRDs, for example v1.6.0:
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml -
Install the controller's own Gateway CRDs from the same controller release:
kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/$LBC_VERSION/config/crd/gateway/gateway-crds.yaml" -
Install or upgrade the controller with the
ALBGatewayAPIfeature gate, using the service account from step 1. The chart's version differs from the controller's:helm search repo eks/aws-load-balancer-controller --versionslists each chart version with its controller version (APP VERSION); use the chart version on the row for yours:helm repo add eks https://aws.github.io/eks-charts && helm repo update eks helm search repo eks/aws-load-balancer-controller --versions | head helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \ --version <chart version whose APP VERSION is $LBC_VERSION> \ --set clusterName=nano --set defaultTags.nano-deployment=nano \ --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller \ --set controllerConfig.featureGates.ALBGatewayAPI=true -
Create the GatewayClass that
values-aws.yamluses:apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: {name: aws-alb} spec: {controllerName: gateway.k8s.aws/alb}
From chart 2.0.10, the chart renders a TargetGroupConfiguration for each Gateway backend, with ip targets and a /healthz health check, so the controller's default settings work. You don't need defaultTargetType on the controller. The related values:
| Value | Default | Purpose |
|---|---|---|
gateway.aws.targetType | ip | instance also needs edge.service.type and ingest.service.type set to NodePort |
gateway.aws.crdApiVersion | "" (auto) | From chart 2.0.11, the chart uses gateway.k8s.aws/v1 when the cluster serves it (controller v3.x) and v1beta1 otherwise (v2.13 to v2.16). Before 2.0.11 the default was v1beta1, which controller v3.x reports as deprecated on every install and upgrade; set gateway.k8s.aws/v1 there |
gateway.appBackendTimeoutSec | 310 | The app ALB's idle timeout, so long searches finish |
Restrict exposure
By default, both ALBs are internet-facing, and the controller opens ports 80 and 443 to 0.0.0.0/0. From chart 2.0.11, you can limit each ALB to the networks that need it, or make it internal:
| Value | Default | Purpose |
|---|---|---|
gateway.aws.sourceRanges.app, gateway.aws.sourceRanges.ingest | [] (0.0.0.0/0) | CIDRs allowed to reach that ALB. The chart sets them as sourceRanges on the ALB's LoadBalancerConfiguration, which limits the inbound rules of its security group |
gateway.aws.scheme.app, gateway.aws.scheme.ingest | "" (tls.aws.scheme, internet-facing) | internal makes that ALB reachable only inside the VPC and networks connected to it |
For example, the UI only from your office and VPN, and an internal ingest ALB for senders in the VPC or over a VPN:
gateway:
aws:
sourceRanges:
app: [203.0.113.0/24, 10.8.0.0/16]
scheme:
ingest: internalAn internal ALB has private addresses, so only senders that can route into the VPC reach it. Point the ingest name at it with a CNAME record as usual.
Certificate and DNS
values-aws.yaml uses tls.mode: cloud-managed, so you need an ACM certificate, in the cluster's region, that covers both the app host and the ingest host, or one certificate for each. Request it for names in a domain you control and validate it before you install. After the install, point both names at the ALBs with CNAME records (see Point DNS at the Gateways).
PostgreSQL on RDS
For production, use RDS for PostgreSQL 16 or newer, Multi-AZ, in the cluster's private subnets, open on port 5432 only to the nodes. With eksctl managed node groups, the nodes use the EKS cluster security group (eks-cluster-sg-nano-…), not the stack's ClusterSharedNodeSecurityGroup; allowing the wrong one leaves the migration waiting for PostgreSQL:
VPC=$(aws eks describe-cluster --name nano --query cluster.resourcesVpcConfig.vpcId --output text)
NODE_SG=$(aws eks describe-cluster --name nano --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
SUBNETS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=tag:kubernetes.io/role/internal-elb,Values=1 \
--query 'Subnets[].SubnetId' --output text) # eksctl's private subnets
DB_SG=$(aws ec2 create-security-group --group-name nano-pg --description "nano PostgreSQL" --vpc-id $VPC \
--tag-specifications 'ResourceType=security-group,Tags=[{Key=nano-deployment,Value=nano}]' --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id $DB_SG --protocol tcp --port 5432 --source-group $NODE_SG
aws rds create-db-subnet-group --db-subnet-group-name nano-pg --db-subnet-group-description "nano" \
--subnet-ids $SUBNETS --tags Key=nano-deployment,Value=nano
aws rds create-db-instance --db-instance-identifier nano-pg \
--engine postgres --engine-version 17 --db-instance-class db.m7g.large \
--allocated-storage 50 --storage-type gp3 --storage-encrypted --multi-az \
--db-subnet-group-name nano-pg --vpc-security-group-ids $DB_SG --no-publicly-accessible \
--master-username nanoadmin --manage-master-user-password \
--backup-retention-period 7 --tags Key=nano-deployment,Value=nano
aws rds wait db-instance-available --db-instance-identifier nano-pg
aws rds describe-db-instances --db-instance-identifier nano-pg --query 'DBInstances[0].Endpoint.Address' --output text--manage-master-user-password keeps the admin password in Secrets Manager, so it never appears on a command line. See Managed PostgreSQL requirements for the database, user, extensions and password secret, then add the postgresql block from Highly available values to my-values.yaml, with the endpoint as postgresql.external.host.
Google Kubernetes Engine (GKE)
Skip this section on other clusters. On GKE, values-gke.yaml exposes nano through two GKE Gateways of the class gke-l7-global-external-managed, one global external Application Load Balancer for the app and one for ingest, with Google-managed certificates from a Certificate Manager map. Prepare the project and the cluster as follows before you install.
Use a GKE Standard cluster. nano hasn't been tested on Autopilot, and we don't support it yet. The chart expects control over the nodes: from the standard preset up, an ingest node pool labeled and tainted nano.rs/pool=ingest, local SSDs for ClickHouse on large and xlarge, and a known number of nodes in each zone for the zonal ClickHouse and Keeper volumes.
Project, APIs and network
Set the project and region once. Every gcloud command in this section passes --project "$PROJECT":
export PROJECT=<project id> REGION=<region>
gcloud projects describe "$PROJECT" --format='value(projectId,name)' # check it's the right projectTurn on the APIs that the commands below use: GKE (container), Compute Engine (compute: the VPC, quotas and static addresses), Cloud SQL Admin (sqladmin), Service Networking (servicenetworking: Cloud SQL's private IP) and Certificate Manager (certificatemanager). A new project often has some of them off; sqladmin in particular:
gcloud services enable container.googleapis.com compute.googleapis.com sqladmin.googleapis.com \
servicenetworking.googleapis.com certificatemanager.googleapis.com --project "$PROJECT"Create a VPC with one subnet in the region, or use your own (then change nano-vpc and nano-subnet below). GKE adds the Pod and Service ranges to the subnet itself:
gcloud compute networks create nano-vpc --project "$PROJECT" --subnet-mode=custom
gcloud compute networks subnets create nano-subnet --project "$PROJECT" --network=nano-vpc \
--region="$REGION" --range=10.40.0.0/21 --enable-private-ip-google-accessLabels. Unlike tags on EKS and AKS, GCP labels reach only some of what nano creates. Label what you can with nano-deployment=nano, and find the rest by name:
| Resource | How to label it |
|---|---|
| GKE cluster, its node VMs and their boot disks | --labels nano-deployment=nano on clusters create |
| nano's volumes (ClickHouse, Keeper, Vector) | A StorageClass like standard-rwo with the parameter labels: nano-deployment=nano, set as global.storageClass and clickhouse.storage.storageClass |
| Certificate Manager DNS authorizations, certificates and map | --labels nano-deployment=nano on each create |
| Cloud SQL instance | The Cloud SQL Admin API (settings.userLabels), as for the user in Managed PostgreSQL requirements; gcloud sql instances create has no label option |
| Reserved static IP addresses | The Compute Engine API's setLabels; gcloud compute addresses create has no label option |
| Not labeled: the VPC, subnet and peering; the load balancers the Gateways create (forwarding rules, proxies, URL maps, backend services, health checks, NEGs); GKE's firewall rules | GCP or GKE creates these without labels. Their names start with gke- or gkegw, or contain the cluster or namespace name |
So a label-based asset search doesn't find every nano resource on GKE; search by name as well.
Quota
A new project usually can't hold the HA layout. Check these quotas in the cluster's region before you create anything, and request increases first:
| Quota | What counts against it |
|---|---|
CPUS_ALL_REGIONS (project) and CPUS or N2_CPUS (region) | Every node's vCPUs: 48 for six n2-standard-8 nodes |
SSD_TOTAL_GB (region) | pd-balanced and pd-ssd disks, including each node's boot disk (100 GB pd-balanced by default) and every nano volume. Two 1,500 GiB ClickHouse volumes and six boot disks need about 3,900 GB; the default is often 500 GB |
describe has no --filter, so flatten the quotas and filter the output:
gcloud compute project-info describe --project "$PROJECT" --flatten=quotas \
--format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' | grep -E '^CPUS_ALL_REGIONS,'
gcloud compute regions describe "$REGION" --project "$PROJECT" --flatten=quotas \
--format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' | grep -E '^(CPUS|N2_CPUS|SSD_TOTAL_GB),'The documented HA layout needs about 48 vCPUs and 3,900 GB of SSD quota; a new project often has 32 vCPUs and 500 GB. If you can't raise the vCPU quota, six n2-highmem-4 nodes (24 vCPUs, 32 GB each) ran the highly available values at about 12.5 GB/day in our tests; see GKE cluster.
If you can't raise SSD_TOTAL_GB far enough, keep 90 days on the hot disk (clickhouse.storage.size: 500Gi) and turn on the cold tier on Cloud Storage. With two replicas that is still 1,000 GB of pd-balanced, more than the default 500 GB on its own. Two things help: create the nodes with --disk-type pd-standard, which moves their boot disks (600 GB for six nodes) off the SSD quota, and request the increase early. For an evaluation only, clickhouse.storage.size: 150Gi fits the default quota; don't go live on it. With values-standard.yaml (plannedGBPerDay: 100) that disk holds about 4 days, under the chart's 7-day minimum, so a first install refuses it: set sizing.plannedGBPerDay to the volume you'll send, or sizing.allowUndersizedDisk: true (see Upgrades).
GKE cluster
A small HA cluster for about 12.5 GB/day: a regional cluster over three zones with two nodes in each zone (six nodes), the GKE Gateway API controller, Dataplane V2 and Workload Identity, in the VPC from Project, APIs and network. Change the name and zones if you need to:
clusters create adds the cluster to your kubeconfig and makes it the current context. To keep ~/.kube/config and its current context as they are, point KUBECONFIG at a separate file first, in every terminal you use for nano:
export KUBECONFIG=$PWD/nano-kubeconfig
gcloud container clusters create nano --project "$PROJECT" --region "$REGION" \
--node-locations "$REGION-a,$REGION-b,$REGION-c" --num-nodes 2 \
--machine-type n2-standard-8 \
--network nano-vpc --subnetwork nano-subnet --enable-ip-alias \
--gateway-api=standard \
--enable-dataplane-v2 \
--workload-pool="$PROJECT.svc.id.goog" \
--release-channel regular --labels nano-deployment=nano| Flag | Why nano needs it |
|---|---|
--region with --node-locations and --num-nodes 2 | A regional control plane, and two nodes in each zone. --num-nodes counts per zone. Persistent disks are zonal, so a ClickHouse replica or Keeper member can only restart on a node in its own zone, as on EKS |
--gateway-api=standard | Installs the Gateway API CRDs and the GKE GatewayClasses. Without it there is no gke-l7-global-external-managed class, so the install fails or the Gateways never get an address. On an existing cluster, run gcloud container clusters update nano --project "$PROJECT" --region "$REGION" --gateway-api=standard. The class appears a few minutes after the cluster is ready (about 3 minutes in our test); wait for it before you install, as below |
--enable-dataplane-v2 | Enforces the chart's NetworkPolicy. GKE Standard doesn't enforce NetworkPolicy by default. See Network policy enforcement |
--workload-pool | Workload Identity, for the cold tier on Cloud Storage without keys |
--enable-ip-alias | A VPC-native cluster, which Cloud SQL's private IP needs. It is the default for new clusters |
Before you install, wait until the Gateway class exists (it can take a few minutes after the cluster is ready):
for i in $(seq 60); do
kubectl get gatewayclass gke-l7-global-external-managed >/dev/null 2>&1 && { echo "GatewayClass ready"; break; }
sleep 10
doneThis waits up to 10 minutes and works in any shell, macOS included (macOS has no timeout command).
n2-standard-8 (8 vCPU, 32 GB) matches the HA sizing in Before you start. In our test at about 12.5 GB/day, six n2-highmem-4 nodes (4 vCPU, 32 GB each, 24 vCPUs in total) also ran the highly available values without failures; watch CPU once real data flows.
Private nodes. With --enable-private-nodes, the nodes have no public addresses:
- Add Cloud NAT to the region's Cloud Router. The cluster must reach
api.nano.rsfor the license and the registry credential, the nano registry, and the public registries of the ClickHouse, Valkey and refresh images. - Make sure you can still reach the control plane. In our test, private nodes turned on authorized networks with an empty list, and
kubectltimed out. Either add your network with--enable-master-authorized-networks --master-authorized-networks <your CIDR>, or turn on the DNS-based endpoint with--enable-dns-accessand fetch credentials withgcloud container clusters get-credentials nano --project "$PROJECT" --region "$REGION" --dns-endpoint.
Storage. GKE's default StorageClass is standard-rwo: pd-balanced, zonal, WaitForFirstConsumer. It works for nano without changes; check it with kubectl get storageclass. The chart doesn't use regional persistent disks. For large and xlarge, add a node pool with local SSDs and a StorageClass for them, and set clickhouse.storage.storageClass.
Certificates and static IPs
values-gke.yaml uses tls.mode: cloud-managed: both Gateways take their certificates from a Certificate Manager map, set as tls.gke.certificateMap. Create it before you install, with a DNS authorization and a Google-managed certificate for each host name, then one map entry per host name:
-
Create the DNS authorizations, then add the CNAME record each one shows in your DNS zone:
gcloud certificate-manager dns-authorizations create nano-app --project "$PROJECT" \ --domain=nano.example.com --labels nano-deployment=nano gcloud certificate-manager dns-authorizations create nano-ingest --project "$PROJECT" \ --domain=ingest.nano.example.com --labels nano-deployment=nano gcloud certificate-manager dns-authorizations describe nano-app --project "$PROJECT" --format='value(dnsResourceRecord)' gcloud certificate-manager dns-authorizations describe nano-ingest --project "$PROJECT" --format='value(dnsResourceRecord)' -
Create the certificates, the map and its entries:
gcloud certificate-manager certificates create nano-app --project "$PROJECT" \ --domains=nano.example.com --dns-authorizations=nano-app --labels nano-deployment=nano gcloud certificate-manager certificates create nano-ingest --project "$PROJECT" \ --domains=ingest.nano.example.com --dns-authorizations=nano-ingest --labels nano-deployment=nano gcloud certificate-manager maps create nano --project "$PROJECT" --labels nano-deployment=nano gcloud certificate-manager maps entries create nano-app --project "$PROJECT" --map=nano \ --certificates=nano-app --hostname=nano.example.com gcloud certificate-manager maps entries create nano-ingest --project "$PROJECT" --map=nano \ --certificates=nano-ingest --hostname=ingest.nano.example.comEach certificate is ready when
gcloud certificate-manager certificates describe nano-app --project "$PROJECT" --format='value(managed.state)'printsACTIVE. With DNS authorization, that doesn't wait for the load balancers. -
Reserve a global static IP address for each Gateway, so the addresses, and the DNS records that point at them, survive an uninstall and reinstall. Create an A record for each host name with its address:
gcloud compute addresses create nano-app-ip nano-ingest-ip --project "$PROJECT" --global --ip-version=IPV4 gcloud compute addresses list --project "$PROJECT" --global --filter='name~^nano-'
Then add the map and the addresses to my-values.yaml:
tls:
gke:
certificateMap: nano
gateway:
addresses:
app: [{type: NamedAddress, value: nano-app-ip}]
ingest: [{type: NamedAddress, value: nano-ingest-ip}]Without gateway.addresses, GKE assigns ephemeral addresses, which change if you reinstall. The chart also renders a HealthCheckPolicy for each backend and a GCPBackendPolicy with a 310-second timeout for the app (gateway.appBackendTimeoutSec), so long searches finish. The chart has no setting yet to restrict source addresses on GKE: both load balancers accept traffic from the internet.
PostgreSQL on Cloud SQL
For production, use Cloud SQL for PostgreSQL with regional availability, reached only over a private IP in the cluster's VPC. See Managed PostgreSQL requirements for the version, the user and the password secret. On top of those:
-
Set up private services access on the VPC once: an allocated range and the peering with Service Networking. The first peering attempt in a project can fail with an authentication error while Google provisions the Service Networking agent; run it again.
gcloud compute addresses create google-managed-services-nano --project "$PROJECT" --global \ --purpose=VPC_PEERING --prefix-length=20 --network=nano-vpc gcloud services vpc-peerings connect --project "$PROJECT" --service=servicenetworking.googleapis.com \ --ranges=google-managed-services-nano --network=nano-vpc gcloud services vpc-peerings list --project "$PROJECT" --network=nano-vpc # lists servicenetworking-googleapis-comA
/20(4,096 addresses) leaves room for Cloud SQL in several regions and other Google-managed services. A/16also works but reserves 65,536 addresses of your private range, which may later overlap networks you peer or connect over VPN. -
Create the instance with a private IP only and encrypted connections.
--ssl-mode=ENCRYPTED_ONLYmatches nano'ssslmode: require; a mode that requires client certificates would refuse nano's connections.db-custom-2-8192(2 vCPU, 8 GB) allows enough connections for the HA values. Put the primary and the standby in two of the cluster's node zones with--zoneand--secondary-zone; without them, Cloud SQL picks the zones, and in our test it put the primary in a zone with no nodes:gcloud sql instances create nano-pg --project "$PROJECT" --database-version=POSTGRES_17 --edition=ENTERPRISE \ --region="$REGION" --zone="$REGION-b" --secondary-zone="$REGION-c" \ --tier=db-custom-2-8192 --availability-type=REGIONAL \ --network="projects/$PROJECT/global/networks/nano-vpc" --no-assign-ip \ --ssl-mode=ENCRYPTED_ONLY --storage-type=SSD --storage-size=50 --storage-auto-increase \ --backup-start-time=03:00 --enable-point-in-time-recovery -
Create the user and the database with
gcloud, as in Managed PostgreSQL requirements. A database created this way is owned bycloudsqlsuperuser, not bynanosiem. That works: the nano user is a member ofcloudsqlsuperuser. -
Set
postgresql.external.hostto the private IP:gcloud sql instances describe nano-pg --project "$PROJECT" --format='value(ipAddresses[0].ipAddress)'.
During a failover. In our tests, a Cloud SQL regional failover stalled API requests and sign-ins for about 10 to 25 seconds; on 2.0.18, some API requests and one sign-in took longer than a 15-second client timeout. OTLP writes got HTTP 503 for about 16 seconds on 2.0.18, as on AKS from 2.0.12 (on 2.0.11 they stalled instead). HEC writes slowed by up to about 3 seconds and never failed. Service resumed without a manual step, and no accepted event was lost. Clients must retry: senders on 503 and timeouts, API clients and sign-in monitors on timeouts. An OTLP request that times out at the client may or may not have been stored, so a sender that retries after a timeout can at worst store an event twice.
When you delete the VPC later, gcloud services vpc-peerings delete can refuse while Google still lists the connection as in use. Deleting the peering from the VPC side works: gcloud compute networks peerings delete servicenetworking-googleapis-com --project "$PROJECT" --network=nano-vpc.
Cold tier on Cloud Storage
For long retention, turn on the cold tier with provider: gcs and Workload Identity on the ClickHouse service account. The cluster's --workload-pool above is what makes that work without keys.
Azure AKS with Application Gateway for Containers
Skip this section on other clusters. On AKS, values-azure.yaml exposes nano through Application Gateway for Containers (AGC), with GatewayClass azure-alb-external and certificates from cert-manager. Prepare the cluster as follows before you install. The Azure steps are in Microsoft's Application Gateway for Containers documentation; this section lists what nano needs from them.
AKS cluster
A small HA cluster for about 12.5 GB/day: three availability zones, Azure CNI overlay, the OIDC issuer and workload identity on, and two 8 vCPU, 32 GB nodes per zone (six nodes, for example Standard_D8ds_v5 or Standard_D8ds_v6). New subscriptions often have too little vCPU quota for six such nodes; check the regional and VM-family quotas, and raise them to 48 vCPUs or more before you start.
Create the cluster in a virtual network of your own, with a subnet for the nodes and one for Application Gateway for Containers. Without --vnet-subnet-id, AKS creates its own virtual network in its managed MC_ resource group, and the AGC subnet would then have to go into a network that AKS manages, which Azure advises against. Change the names, region and tag:
export RG=<resource group> LOC=<region> TAGS="nano-deployment=nano"
az network vnet create -g $RG -n nano-aks-vnet -l $LOC --address-prefixes 10.50.0.0/16 --tags $TAGS
az network vnet subnet create -g $RG --vnet-name nano-aks-vnet -n aks --address-prefixes 10.50.0.0/20
az network vnet subnet create -g $RG --vnet-name nano-aks-vnet -n agc --address-prefixes 10.50.16.0/24 \
--delegations Microsoft.ServiceNetworking/trafficControllers
az aks create -g $RG -n nano -l $LOC --tier standard --tags $TAGS --nodepool-tags $TAGS \
--zones 1 2 3 --node-count 6 --node-vm-size Standard_D8ds_v6 \
--network-plugin azure --network-plugin-mode overlay --network-dataplane cilium \
--vnet-subnet-id "$(az network vnet subnet show -g $RG --vnet-name nano-aks-vnet -n aks --query id -o tsv)" \
--enable-oidc-issuer --enable-workload-identity --no-ssh-key--tier standard gives the control plane its uptime SLA; the free tier has none. The cluster's identity needs Network Contributor on a virtual network you bring; AKS grants it when your account may assign roles, otherwise give it before the create with a user-assigned identity (--assign-identity).
--network-dataplane cilium makes Cilium enforce NetworkPolicy; --network-policy azure or calico also works. Without one of them, AKS doesn't enforce it; see Network policy enforcement. On an existing cluster, az aks show -g <resource group> -n nano --query networkProfile.networkPolicy prints none or nothing when nothing enforces it.
AKS spreads one node pool's nodes over the zones on a best-effort basis; to guarantee two in each zone, create one node pool per zone. AKS ships a default StorageClass, default, on the Azure Disk CSI driver (like managed-csi). Azure Disk LRS volumes are zonal, so a ClickHouse replica or Keeper member can only restart in its own zone, as on EKS. Check that a default StorageClass exists with kubectl get storageclass.
Application Gateway for Containers
- Install the ALB controller with its Helm chart and workload identity, as in Microsoft's quickstart: a managed identity with a federated credential for the controller's service account in
azure-alb-system, and the role assignments the quickstart lists, including AppGw for Containers Configuration Manager. - Use the
agcsubnet from AKS cluster (delegated toMicrosoft.ServiceNetworking/trafficControllers, in the cluster's own virtual network), and give the controller's identity Network Contributor on it. - Choose how AGC is deployed, and tell the chart which one to use. The values go in
my-values.yaml:-
Managed by the controller. Create an
ApplicationLoadBalancerresource that names the delegated subnet, in a namespace of your choice. The controller creates the AGC and a frontend per Gateway. Point both Gateways at it:gateway: annotations: alb.networking.azure.io/alb-namespace: alb-infra # the ApplicationLoadBalancer's namespace alb.networking.azure.io/alb-name: nano-alb # and name -
Bring your own. Create the AGC, two frontends and the association with the delegated subnet yourself, for example with
az network alb. Each frontend's host name is known before you install. Point the Gateways at the AGC and give each its frontend:gateway: annotations: alb.networking.azure.io/alb-id: /subscriptions/<subscription>/resourceGroups/<resource group>/providers/Microsoft.ServiceNetworking/trafficControllers/<AGC name> addresses: app: [{type: alb.networking.azure.io/alb-frontend, value: <app frontend name>}] ingest: [{type: alb.networking.azure.io/alb-frontend, value: <ingest frontend name>}]
-
Each frontend has a host name under alb.azure.com. Point your two names at the frontends with CNAME records (see Point DNS at the Gateways). For an evaluation without your own domain, use the two frontend host names themselves; see Certificates.
Tags. --tags on your commands doesn't reach what Azure creates for you: the volumes' Azure Disks, in the MC_ resource group, come untagged. Pass --tags to az network alb create and to each az network alb frontend create too. After the install, tag nano's disks:
MC_RG=$(az aks show -g $RG -n nano --query nodeResourceGroup -o tsv)
for id in $(az disk list -g $MC_RG --query "[?tags.\"kubernetes.io-created-for-pvc-namespace\"=='nano'].id" -o tsv); do
az resource tag --is-incremental --ids "$id" --tags $TAGS -o none
doneTo tag disks as they are created instead, create a StorageClass like default with the Azure Disk parameter tags: nano-deployment=nano, and set it as global.storageClass and clickhouse.storage.storageClass.
Certificates
AGC terminates TLS from Kubernetes Secrets, so values-azure.yaml uses tls.mode: cert-manager. Install cert-manager and a ClusterIssuer that can issue certificates for both host names, and set tls.certManager.issuerRef.name. cloud-managed isn't available on Azure; the chart refuses it.
To use certificates you already have, set tls.mode: existingSecret with tls.existingSecret.app and tls.existingSecret.ingest. This is the one case where you set tls.mode yourself: you replace the preset's TLS on purpose, so the caution in Write your values doesn't apply.
Without a domain of your own, for an evaluation, use the two AGC frontends' host names (az network alb frontend show ... --query fqdn -o tsv) as ingress.host and ingest.hostname, with a private certificate authority that cert-manager runs. Don't use Let's Encrypt for *.alb.azure.com names: they share one rate limit with every other AGC user. Browsers and senders must trust the CA certificate:
# private-ca.yaml: kubectl apply -f private-ca.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: selfsigned-bootstrap}
spec: {selfSigned: {}}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: nano-private-ca, namespace: cert-manager}
spec:
isCA: true
commonName: nano-private-ca
secretName: nano-private-ca
privateKey: {algorithm: ECDSA, size: 256}
issuerRef: {name: selfsigned-bootstrap, kind: ClusterIssuer}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: nano-private-ca}
spec: {ca: {secretName: nano-private-ca}}Set tls.certManager.issuerRef.name: nano-private-ca, and give clients the CA certificate: kubectl -n cert-manager get secret nano-private-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > nano-ca.crt.
PostgreSQL on Azure
For production, use Azure Database for PostgreSQL flexible server. It needs the three extensions on its azure.extensions allow-list, and the nano user needs schema public, before you install; see Managed PostgreSQL requirements for both, the private networking and the password secret.
Create the server with private access, zone-redundant HA and a SKU the region offers. Choose the SKU from az postgres flexible-server list-skus -l <region> -o table. A listed SKU can still be refused with SkuNotAvailable (in one test, Standard_D2ds_v5 was refused and Standard_D2ads_v5 worked); try the other family of the same size. On Azure CLI 2.88 and newer, zone-redundant HA is --zonal-resiliency Enabled with --zone and --standby-zone; older versions use --high-availability ZoneRedundant:
az postgres flexible-server create -g $RG -n nano-pg -l $LOC --version 17 \
--tier GeneralPurpose --sku-name <SKU> --storage-size 128 \
--zonal-resiliency Enabled --zone 1 --standby-zone 2 \
--vnet <PostgreSQL VNet> --subnet <delegated subnet> --private-dns-zone <zone> \
--admin-user nanoadmin --tags $TAGSDon't pass --admin-password: on the command line it shows in the process list and your shell history. Without it, the CLI generates a password and prints it once in its output; save it to pg-admin-password.txt, readable only by you, for the bootstrap pod. Or create the server in the portal, or with Terraform reading the password from Key Vault. Zone-redundant HA isn't offered in every region or for every subscription; it may need a support request. Same-zone HA (--zonal-resiliency Enabled --allow-same-zone) survives a server failure but not the loss of the zone.
PostgreSQL in another region. Some subscriptions can't create a flexible server in some regions (Provisioning is restricted in this region). Ask Azure for access in the cluster's region if you can. To use another region, the server needs its own virtual network in that region, with a subnet delegated to Microsoft.DBforPostgreSQL/flexibleServers and an address range that doesn't overlap the cluster's. Then:
- Peer that network with the cluster's network in both directions (global VNet peering,
az network vnet peering create --allow-vnet-access, once from each side). - Create one private DNS zone, for example
nano-pg.private.postgres.database.azure.com, and link it to both networks (az network private-dns link vnet create, once for each). Pass it to the server create with--private-dns-zone.
Every query then crosses regions, which adds latency and is billed. In one test (eastus to eastus2), the first migration took about 6 minutes, well within --timeout 30m.
Known limitations on AKS
- The chart renders no backend timeout policy for AGC, so the app frontend uses AGC's default request timeout. Very long searches may be cut off; this hasn't been tested yet. On EKS and GKE,
gateway.appBackendTimeoutSec(310) applies. - The chart renders no health-check policy for AGC. AGC uses the pods' readiness probes, which works.
- The cold tier on Azure Blob storage needs account keys. Managed identity and workload identity aren't supported for it yet.
Managed PostgreSQL requirements
Production installs use a managed PostgreSQL (postgresql.mode: external). Before the migration Job (nano-migrate-r1) applies anything, nano checks the server version, that it can create tables in schema public (from 2.0.12) and that it can create every extension it needs. If something is missing, the Job fails with one message that names it and the fix for your service, and nothing is applied.
| Requirement | Detail |
|---|---|
| Version | PostgreSQL 16 or newer. nano is tested on 16, 17 and 18 |
| Extensions | pg_trgm, pgcrypto and uuid-ossp. nano creates them itself; they are trusted extensions, so the database owner can. No shared_preload_libraries change is needed |
| Database and user | A database nanosiem owned by the user nanosiem, or set postgresql.database and postgresql.user |
Schema public | The nano user must be able to create tables in schema public of its database. PostgreSQL 15 and newer no longer let every user do that, and on Azure flexible server public belongs to azure_pg_admin, so the first migration fails with permission denied for schema public. Give it the schema as below |
| Connections | At least 200 (about 20 for each nano pod with the highly available values) |
| TLS | Keep sslmode: require or stricter |
| Password | URL-safe, because it goes into the connection URL, and stored in a Kubernetes secret |
| Amazon RDS | Google Cloud SQL | Azure Database for PostgreSQL flexible server | |
|---|---|---|---|
| Minimum version | 16 | 16 | 16 |
Schema public | Usually nothing to do: the database owner has it. The bootstrap SQL below makes sure | Usually nothing to do: a database made with gcloud sql databases create and a user made through Cloud SQL have it. If the check reports it, run GRANT ALL ON SCHEMA public TO nanosiem; as postgres, connected to nanosiem | Required. ALTER SCHEMA public OWNER TO nanosiem; as the admin user, connected to nanosiem (in the bootstrap SQL below) |
| Allow the extensions | Nothing to do, unless your parameter group sets rds.allowed_extensions. Then add pg_trgm,pgcrypto,uuid-ossp to it | No database flags are needed. Create the nano user with gcloud sql users create: those users belong to cloudsqlsuperuser and can create the extensions. Roles created with CREATE ROLE in SQL can't | Required. Add pg_trgm,pgcrypto,uuid-ossp to the azure.extensions server parameter. It applies without a restart |
| Parameters | None. The default rds.force_ssl=1 works | None. On small tiers, raise the max_connections flag | azure.extensions. The default require_secure_transport=ON works. Avoid the Burstable B1ms and B1s tiers, which allow 50 connections and have no HA |
| Private networking | In the cluster's VPC, not publicly accessible, with a security group that allows 5432 from the nodes' security group | Private IP only, through private services access on the cluster's VPC (a VPC-native cluster), with no public IP | Private access (VNet integration) with a delegated subnet in, or peered with, the AKS VNet and a private DNS zone linked to it, or a private endpoint |
| High availability | Multi-AZ DB instance or Multi-AZ DB cluster | Regional availability (--availability-type=REGIONAL) | Zone-redundant HA. Not every region offers it, and the Burstable tier doesn't; there, use same-zone HA, which survives a server failure but not the loss of the zone |
On Azure, read the current value first, then append the three extensions to it. If the current value is empty, leave out <existing>,:
az postgres flexible-server parameter show -g <resource group> --server-name <server> \
--name azure.extensions --query value -o tsv
az postgres flexible-server parameter set -g <resource group> --server-name <server> \
--name azure.extensions --value <existing>,pg_trgm,pgcrypto,uuid-osspOn Cloud SQL, create the database with gcloud, and the user through the Cloud SQL Admin API. gcloud sql users create takes the password only as --password=..., which puts it in the process list and your shell history; it has no option to prompt. Write the password to pg-password.txt (readable only by you), then:
gcloud sql databases create nanosiem --instance=<instance> --project <project>
( umask 077
printf 'Authorization: Bearer %s\n' "$(gcloud auth print-access-token)" > gcp-auth-header.txt
python3 -c 'import json,sys; print(json.dumps({"name": "nanosiem", "password": open(sys.argv[1]).read().strip()}))' \
pg-password.txt > pg-user.json )
curl -fsS -X POST -H @gcp-auth-header.txt -H 'Content-Type: application/json' --data-binary @pg-user.json \
"https://sqladmin.googleapis.com/v1/projects/<project>/instances/<instance>/users"
curl -fsS -X PATCH -H @gcp-auth-header.txt -H 'Content-Type: application/json' \
--data '{"settings": {"userLabels": {"nano-deployment": "nano"}}}' \
"https://sqladmin.googleapis.com/v1/projects/<project>/instances/<instance>" # optional: label the instance
rm -f gcp-auth-header.txt pg-user.json
gcloud sql users list --instance=<instance> --project <project> # nanosiem, BUILT_INA user created through Cloud SQL (with gcloud or this API) is a member of cloudsqlsuperuser and can create the extensions.
Don't create the Cloud SQL user with CREATE ROLE in SQL: such roles can't create extensions, and the migration stops on pg_trgm.
On RDS and Azure, create the user and database as the admin user in psql. \password prompts for the password, so it doesn't end up in your shell history. On PostgreSQL 16 and newer, an admin who isn't a superuser, such as the admin user on RDS and Azure, must be a member of the new role before creating a database it owns:
CREATE ROLE nanosiem LOGIN;
\password nanosiem
GRANT nanosiem TO <admin user>;
CREATE DATABASE nanosiem OWNER nanosiem;
\connect nanosiem
ALTER SCHEMA public OWNER TO nanosiem;The last two lines give the nano user schema public in its database. They're required on Azure flexible server, where public belongs to azure_pg_admin, and harmless on RDS, where the database owner already has it. GRANT ALL ON SCHEMA public TO nanosiem; works too if you'd rather keep the owner.
Reaching a private RDS or Azure server. The database has no public address, so psql on your workstation can't reach it. Run it from inside the cluster, in a one-off pod. The passwords travel in a temporary secret, never on a command line. Put the admin password in pg-admin-password.txt (on RDS with --manage-master-user-password, read it from Secrets Manager: aws secretsmanager get-secret-value --secret-id <arn> --query SecretString --output text | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"], end="")' > pg-admin-password.txt) and the new nano password in pg-password.txt, then:
kubectl create namespace nano --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nano create secret generic pg-bootstrap \
--from-file=admin=./pg-admin-password.txt --from-file=nano=./pg-password.txt
kubectl -n nano run pg-bootstrap --rm -i --restart=Never --image=postgres:17-alpine --overrides='
{"spec": {"containers": [{"name": "pg-bootstrap", "image": "postgres:17-alpine", "stdin": true, "stdinOnce": true,
"command": ["sh", "-c", "PGPASSWORD=$(cat /s/admin) exec psql -v ON_ERROR_STOP=1 \"host=<endpoint> user=<admin user> dbname=postgres sslmode=require\" -f -"],
"volumeMounts": [{"name": "s", "mountPath": "/s", "readOnly": true}]}],
"volumes": [{"name": "s", "secret": {"secretName": "pg-bootstrap"}}]}}' <<'SQL'
\set pw `cat /s/nano`
CREATE ROLE nanosiem LOGIN;
ALTER ROLE nanosiem PASSWORD :'pw';
GRANT nanosiem TO CURRENT_USER;
CREATE DATABASE nanosiem OWNER nanosiem;
\connect nanosiem
ALTER SCHEMA public OWNER TO nanosiem;
SQL
kubectl -n nano delete secret pg-bootstrapIt prints psql's output (CREATE ROLE through ALTER SCHEMA) and exits 0 within a few seconds. Keep "stdinOnce": true: the overrides replace the container kubectl run generates, and without it the pod never sees the end of the SQL, so the command prints nothing, waits about a minute and exits 1, although the SQL ran. If it fails, psql names the statement, for example role "nanosiem" already exists on a second run. kubectl warns that the session is recorded in the container logs, including credentials. Here the passwords are read from the mounted secret, not typed, so they aren't in the logs.
On RDS, <endpoint> is the instance endpoint and <admin user> the master user name. On Azure, <endpoint> is <server>.postgres.database.azure.com and <admin user> the admin you created the server with. The pod resolves the private DNS zone through the cluster's virtual network, so the zone must be linked to it.
After you create the namespace, store the password in a secret from a file that holds only the password, with no trailing newline:
kubectl -n nano create secret generic nano-postgres --from-file=password=./pg-password.txtThe nano-deploy skill checks the version, schema public and the extensions from inside the cluster before helm install.
Install
Pull the chart
Save the registry credential's password to registry-password.txt (readable only by you, for example after umask 077), log in to the registry with it, then pull and unpack the chart:
helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken \
--password-stdin < ./registry-password.txt
helm pull oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano \
--version "$NANO_VERSION" --untarIf you have portal access, 1 · Get the installer on the deployment's page shows these commands with your credential filled in.
The install runs from the unpacked chart, so you can log out now. helm registry login may keep the credential in your operating system's credential store, such as the macOS keychain, outside your working directory:
helm registry logout us-central1-docker.pkg.devThe chart is now in ./nano, with its sizing and exposure presets beside values.yaml. It also holds the release's image lock, nano/images.enterprise.customer.lock, and nano/values-customer-images.yaml, which pins every nano image by digest. From chart 2.0.12 the ClickHouse operator image is pinned by digest too; 2.0.11 and earlier pull it by tag. You only need to regenerate that file if you change the lock:
bash nano/scripts/values-from-lock.sh nano/images.enterprise.customer.lock > nano/values-customer-images.yamlCreate the namespace and registry secret
kubectl create namespace nanoBuild the registry secret from registry-password.txt through a Docker config file, so the password never appears on a command line, where other users can see it in the process list and it stays in your shell history:
( umask 077; python3 - registry-password.txt us-central1-docker.pkg.dev oauth2accesstoken \
> dockerconfig.json <<'PY'
import base64, json, sys
cred, host, user = sys.argv[1:4]
password = open(cred, encoding="utf-8").read().rstrip("\n")
auth = base64.b64encode(f"{user}:{password}".encode()).decode()
print(json.dumps({"auths": {host: {"username": user, "password": password, "auth": auth}}}))
PY
)
kubectl -n nano create secret generic nano-registry \
--type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=dockerconfig.json
rm -f dockerconfig.jsonKeep the type kubernetes.io/dockerconfigjson: the refresh CronJob updates the secret in place.
From chart 2.0.10, a CronJob refreshes this secret before the credential expires. See Registry access.
Write your values
Save your enrollment token to license-token.txt, then create my-values.yaml with only your site's values:
nano:
baseUrl: https://nano.example.com
deploymentId: <your deployment ID>
license:
url: https://api.nano.rs/api/license/status
global:
imagePullSecrets: [nano-registry]
ingress:
host: nano.example.com # UI and API
ingest:
hostname: ingest.nano.example.com # sendersLeave tls.mode to the exposure preset. Add only the TLS keys that your preset asks for:
| Preset | Cluster | Add to my-values.yaml |
|---|---|---|
values-aws.yaml | EKS | tls.aws.appCertificateArn and tls.aws.ingestCertificateArn (the same ARN if one certificate covers both hosts) |
values-gke.yaml | GKE | tls.gke.certificateMap |
values-azure.yaml | AKS | tls.certManager.issuerRef.name, for example letsencrypt |
values-sovereign.yaml | Any other cluster (Envoy Gateway) | tls.certManager.issuerRef.name, your ClusterIssuer |
For example, on EKS:
tls:
aws:
appCertificateArn: arn:aws:acm:<region>:<account>:certificate/<id>
ingestCertificateArn: arn:aws:acm:<region>:<account>:certificate/<id>Also set clickhouse.allowedNetworks; see ClickHouse network allow-list. With external PostgreSQL, also add the postgresql block from Highly available values. For about 12.5 GB/day, save the 12.5 GB/day example as values-12gb.yaml.
Install
Layer the image values, one sizing preset, one exposure preset, then your own files. Later files win:
helm install nano ./nano -n nano --wait --timeout 30m \
-f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
-f values-12gb.yaml -f my-values.yaml \
--set-file secrets.licenseToken=./license-token.txtThis example is for EKS. Use values-gke.yaml or values-azure.yaml on those clouds, and values-sovereign.yaml (Envoy Gateway) anywhere else. With --wait, Helm reports the release as deployed only once its pods are ready; without it, Helm reports success while pods may still be crash-looping. Keep a note of your -f files and their order: every upgrade needs the same list.
The first install runs the database migration before the app pods start. Against a remote managed PostgreSQL, for example in another region, it can take 10 minutes or more. Follow it with kubectl -n nano logs -f job/nano-migrate-r1. From chart 2.0.11, the Deployments that wait on it allow 30 minutes (migrations.dependentProgressDeadlineSeconds, 1800), so install with --wait --timeout 30m: it matches that deadline, and --wait returns as soon as every pod is ready (about 8 minutes in our 2.0.18 tests). A longer --timeout doesn't help; raise both together if you must. On 2.0.10, see helm install fails with "Progress deadline exceeded".
Point DNS at the Gateways
kubectl -n nano get gatewayPoint nano.example.com at the app Gateway's address and ingest.nano.example.com at the ingest Gateway's address. On EKS and AKS, the addresses are host names, so use CNAME records. On GKE, they are IP addresses, so use A records; with reserved static IPs, you created them before the install.
Verify it works
kubectl -n nano get pods # every pod Running or Completed, every container ready
helm test nano -n nano # health of every service
curl -fsS https://ingest.nano.example.com/healthz # 200 from chart 2.0.10helm test checks that each service answers, not that every pod is ready, so check the pods as well: a pod with fewer ready containers than it has (for example Vector at 1/2) or a rising restart count needs a look with kubectl -n nano describe pod and kubectl -n nano logs. On chart 2.0.9, the ingest host answers /healthz with 404; that is expected.
Then open https://nano.example.com and create the first admin on the one-time setup page. A licensed install shows no license banner.
The setup page asks for the setup token, the setup_token key of nano-secrets (from 2.0.8). Write it to a file that only you can read, rather than printing it to your terminal, where it stays in scrollback and CI logs:
( umask 077; kubectl -n nano get secret nano-secrets -o jsonpath='{.data.setup_token}' | base64 -d > setup-token.txt )To create the admin without a browser, send the same request the page does, with the token as a bearer token and the password read from a file rather than typed on the command line. curl reads the header from a file, so the token stays out of the process list:
( umask 077; printf 'Authorization: Bearer %s\n' "$(cat setup-token.txt)" > setup-header.txt )
python3 -c 'import json,sys; print(json.dumps({"email": sys.argv[1], "name": sys.argv[2], "password": open(sys.argv[3]).read().strip()}))' \
admin@example.com "Admin" ./admin-password.txt | \
curl -fsS -X POST https://nano.example.com/api/setup/initialize \
-H @setup-header.txt -H 'Content-Type: application/json' --data-binary @-
rm -f setup-header.txtOnce the first admin exists, delete the token files: rm -f setup-token.txt setup-header.txt. The token isn't needed again.
The body is {"email", "name", "password"}; the token is needed only until the first admin exists.
Next steps
- Set up your first feed, sending to
ingest.nano.example.com - Make it highly available
- Plan retention and the cold tier
Prefer to have a coding agent run the install? The nano-deploy agent skill supports Helm.
Sizing presets
Pick one preset and layer your own values on top. The figures are a starting point; watch ClickHouse CPU and memory once real data flows.
| Preset | Daily volume | ClickHouse | Vector | PostgreSQL |
|---|---|---|---|---|
values-small.yaml | Up to ~10 GB | 2–4 vCPU, 4–6 GiB, 50 Gi | 1 × 0.5–2 vCPU | Bundled |
values-standard.yaml | ~50–100 GB | 4–8 vCPU, 32 GiB, 500 Gi | 2–4 × 2–4 vCPU | Bundled, external recommended |
values-large.yaml | ~1 TB | 8–16 vCPU, 64 GiB, 4 Ti NVMe | 2–4 × 6–8 vCPU (pre-shaped: 2–3 × 2–4 vCPU) | External |
values-xlarge.yaml | ~10 TB | 2 shards, 28–32 vCPU, 256 GiB, 16 Ti NVMe each | 4–7 × 28–31 vCPU (pre-shaped: 2–4 × 4–8 vCPU) | External |
The Vector pools are sized for an average event of 1,000 bytes. From standard up, the ingest edge and Vector run on an ingest node pool labeled and tainted nano.rs/pool=ingest, and the Vector autoscaler needs metrics-server. large and xlarge need a local-NVMe StorageClass for ClickHouse (clickhouse.storage.storageClass).
Planned volume and ingest path
Each preset records its nominal volume in sizing.plannedGBPerDay. Two more values tune the Vector estimate:
| Value | What it means | Default |
|---|---|---|
sizing.ingestPath | raw for logs nano parses. preshaped for OCSF that arrives already shaped from Tenzir or Cribl: the chart then uses the preset's smaller vector.preshaped pool | raw |
sizing.avgEventBytes | Your average event size, in bytes: about 300 for firewall and flow, 1000 for mixed feeds, 2000 for EDR | 1000 |
If sizing.plannedGBPerDay is more than the Vector pool can carry, helm install and helm upgrade print a warning. Firewall-heavy installs need a larger vector.hpa.maxReplicas.
Plan Vector from events per second. Parsing costs about the same per event, whatever its size:
events/s = GB/day × 10^9 ÷ 86,400 ÷ sizing.avgEventBytes
Vector vCPU = 2 (peak hour) × events/s ÷ events per coreRaw logs are planned at 1,900 events per second per core. 16 benchmarked shipped OCSF parsers measured 1,950–6,500. Per TB/day of raw logs, that is about 41 vCPU at 300 bytes per event, 13 at 1,000 and 7 at 2,000. Pre-shaped OCSF is planned at 45,000 events per second per core, about 0.5 vCPU per TB/day. Vector gets at least 1 GiB of memory per pod, and 8 GiB or more for raw parsing at 1 TB/day and up.
The large-deployment ClickHouse settings ship only in the large and xlarge presets, so small and standard keep their values:
| Setting | values-large.yaml (~1 TB/day) | values-xlarge.yaml (≥10 TB/day) |
|---|---|---|
| Insert flush | Adaptive, 1–5 s | Adaptive, 1–10 s |
async_insert_max_data_size | 256 MB | 256 MB |
parts_to_delay_insert / parts_to_throw_insert | 2,000 / 4,000 | 5,000 / 10,000 |
merges_mutations_memory_usage_to_ram_ratio | 0.5 | 0.5 |
| Vector sink concurrency | 6 | 12 |
Prevalence refresh (migrations.prevalenceRefreshMinutes) | 30 min | 60 min |
Prevalence refresh. The rare and new lists, and the host counts in the prevalence drawer, are refreshed every 10 minutes by default. Refreshing less often saves CPU at volume. On large and xlarge, the lists can lag ingest by up to 30 or 60 minutes. Detections and searches still check a new artifact against the real-time counts, so it is still caught as rare. To set the interval yourself, set migrations.prevalenceRefreshMinutes (10–1440). 0 means the default of 10 minutes; moving back to a smaller preset resets it.
Direct OCSF writers. A Tenzir or Cribl writer next to the release can load its ClickHouse connection from the <release>-direct-ocsf ConfigMap with envFrom. It carries NANO_CH_SHARDS and NANO_CH_TABLE. With more than one shard, the table is the distributed one, so events spread across shards. The ingest user's password stays in the release secret. To admit a writer outside the cluster, see ClickHouse network allow-list.
Example: about 12.5 GB/day
Start from values-small.yaml and give ClickHouse room for enrichment, queries and a year of data:
# values-12gb.yaml
clickhouse:
resources:
requests: {cpu: "2", memory: 12Gi}
limits: {cpu: "4", memory: 16Gi}
storage:
size: 1500Gi # 365 days; use 500Gi for 90 days
profiles: # about 50% of the 0.8 x 16 GiB server cap per query
default: {max_memory_usage: "6800000000", max_memory_usage_for_user: "9000000000", max_threads: "4"}
rawsql: {max_memory_usage: "6800000000", max_threads: "4"}See Retention and disk for the retention math.
Highly available values
To survive the loss of a node, add a second ClickHouse replica, three Keepers, two of each app service and an external, highly available PostgreSQL 16 or newer, such as Multi-AZ RDS, regional Cloud SQL or zone-redundant Azure flexible server (see Managed PostgreSQL requirements). Layer this after values-12gb.yaml:
# values-12gb-ha.yaml
clickhouse:
replicas: 2 # two full copies of the data: about 2x ClickHouse cost
antiAffinity: hard # one ClickHouse pod per node
zoneAware: true # replicas in different zones; Keeper over 3 zones
keeper:
replicas: 3
api: {replicas: 2}
search: {replicas: 2}
jobs: {replicas: 2}
web: {replicas: 2}
edge: {replicas: 2}
connectorRunner: {replicas: 2} # lease fencing: each feed runs on one runner at a time
vector:
replicas: 2
ingest:
edge:
replicas: 2
hecAcks: false # HEC acks need a single ingest edge
postgresql:
mode: external
external:
host: nano-pg.internal.example.com # on EKS, the RDS endpoint
sslmode: require
passwordSecret: {name: nano-postgres, key: password}Create the PostgreSQL password secret as in Managed PostgreSQL requirements, then install with both files:
kubectl -n nano create secret generic nano-postgres --from-file=password=./pg-password.txt
helm install nano ./nano -n nano --wait --timeout 30m \
-f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
-f values-12gb.yaml -f values-12gb-ha.yaml -f my-values.yaml \
--set-file secrets.licenseToken=./license-token.txtPlan at least two worker nodes in each of three zones: six nodes with 8 vCPU and 32 GB each. clickhouse.zoneAware: true puts the replicas of a shard in different zones and spreads Keeper over three zones (minDomains: 3), so set it only on a cluster with nodes in three zones. Three nodes, one per zone, also work: nano survives the loss of a node with its data safe, but that zone's ClickHouse replica and Keeper member stay Pending until the node returns. See Cluster on EKS, GKE cluster on GKE and AKS cluster on AKS.
Cloud block storage is zonal, so a ClickHouse replica or Keeper member can only restart in its own zone. On every cloud, give each zone that ClickHouse and Keeper use at least two nodes, and set clickhouse.zoneAware: true and antiAffinity: hard:
| Cloud | Volume | Node layout |
|---|---|---|
| EKS | EBS gp3, bound to one Availability Zone | One managed node group per zone, at least 2 nodes each, so the cluster autoscaler scales the right zone |
| GKE | pd-balanced, zonal. Regional persistent disks exist, but the chart doesn't use them | One node pool over three zones with --node-locations and --num-nodes 2 (2 per zone, 6 nodes), or one pool per zone. See GKE cluster |
| AKS | Azure Disk LRS, zonal | One node pool with --zones 1 2 3 and a count of 6 (AKS spreads nodes on a best-effort basis), or one pool per zone to guarantee 2 in each. See AKS cluster |
If any sender uses HEC acknowledgements, keep ingest.edge.replicas: 1 and hecAcks: true.
From chart 2.0.10, every component that runs more than one replica gets a PodDisruptionBudget with maxUnavailable: 1: api, search, jobs, web, edge, connector runner, ingest edge and Vector. A node drain then moves one pod at a time.
What to expect during a failure. Measured on one test install each, with these values and two nodes per zone:
| Event | Effect |
|---|---|
| A ClickHouse, Valkey or api pod deleted | No failed writes, sign-ins or API requests. After a ClickHouse pod delete on AKS, a few events written in the same second were stored twice (see below) |
| A node drained, more than an hour after install | Its pods were running elsewhere within about a minute, with images pulled using the refreshed registry credential. No requests failed |
| RDS Multi-AZ forced failover (EKS) | HEC writes never failed. API reads stalled for about 15 seconds. No sign-in failed |
| Cloud SQL regional failover (GKE) | HEC writes never failed (up to about 3 seconds slower). API requests and sign-ins stalled for about 10 to 25 seconds; on 2.0.18 some exceeded a 15-second client timeout. OTLP writes got 503 for about 16 seconds on 2.0.18 (they stalled on 2.0.11). No accepted event was lost. An OTLP request that timed out at the client may or may not have been stored, so a retry can at worst store it twice |
| Azure flexible server forced failover (AKS), same-zone and zone-redundant HA | Sign-ins and API writes failed for about 30 seconds, then resumed, and some API reads failed. No HEC write failed and no event was lost. OTLP writes failed for about 48 seconds while both api pods were unready: on 2.0.11 with HTTP 500 or a 15-second stall. From 2.0.12 they get 503 with Retry-After, which OpenTelemetry exporters retry, within 10 seconds |
In every test, every accepted event was searchable afterwards, and no failover needed a manual repair. Senders and clients should retry on 429, 502, 503, 504 and timeouts, as with any HA layout; buffer OTLP in a collector so a failover doesn't drop batches.
Delivery is at least once. An event that nano accepted is never lost, but in rare cases it is stored twice: when a ClickHouse replica goes away while a batch is being written, the batch's acknowledgement can be lost, and the batch is written again. In one AKS test, deleting a ClickHouse pod stored 10 of 150 HEC events and 10 of 150 OTLP events twice, all written in the second after the delete. Counts and detections over that minute can be slightly high. Each copy has the same metadata.uid, so dc(metadata.uid) gives the true count. Nothing reports the duplicates, and nothing needs repairing.
Sign-ins are rate limited. nano allows 100 sign-in requests per client address in 5 minutes, and 10 per account in 10 minutes, successful ones included. Above either, it answers 429 with a Retry-After header (up to 300 seconds). A failover probe that signs in as one user more often than once a minute hits the per-account limit. Sign in once and reuse the session, or use an API key.
After a ClickHouse pod restarts, the nano-deploy skill's health check on 2.0.11 can report distributed DDL entries pending > 10 min for good. A cluster-wide schema statement issued while the replica restarted stayed Inactive on that replica, although the replica ran every later statement and nothing diverged. ClickHouse removes such an entry after 7 days (distributed_ddl.task_max_lifetime); don't delete it from Keeper by hand. From 2.0.12 the check only warns about it.
Upgrades
Pull the new release's chart, with a fresh registry login, then upgrade with the same -f files as the install, in the same order, plus --wait --timeout 20m. The new version comes from helm pull --version. This example matches the HA install above:
export NANO_VERSION=<new version>
helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken \
--password-stdin < ./registry-password.txt
rm -rf nano && helm pull oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano \
--version "$NANO_VERSION" --untar
helm registry logout us-central1-docker.pkg.dev
helm upgrade nano ./nano -n nano --wait --timeout 20m \
-f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
-f values-12gb.yaml -f values-12gb-ha.yaml -f my-values.yamlThe upgrade doesn't need --set-file secrets.licenseToken: the chart keeps every key of its existing secret, the license token included, when you leave it out. Pass it only to replace the token.
Don't use --reuse-values. It renders the new chart with the previous release's values, including the previous chart's defaults, and ignores the new chart's values.yaml. Settings a release adds or changes in its defaults never arrive, and Helm still reports the new chart version. With charts 2.0.12 and earlier it also kept the old images: an upgrade to 2.0.12 with --reuse-values or --reset-then-reuse-values ran 2.0.11. From 2.0.13 the chart pins its own images, so both flags get the new release's images, but --reuse-values still drops new defaults. Pass the same -f files as above. If you no longer have them, --reset-then-reuse-values -f nano/values-customer-images.yaml is safe: it takes the new chart's defaults and keeps the values you set before. Check with --dry-run=server that every nano image is the new version before you upgrade.
To run an image other than the release's on purpose, set images.<name>.tag (and digest). An explicit tag always wins; leave image.tag empty.
Ingest during an upgrade. An ingest-edge or Vector rollout drops nothing only if the load balancer stops sending to an old pod before it exits, and has registered and health-checked the new pod before the old one stops. From chart 2.0.15 the ingest.rollout values handle this:
| Value | Default | Why |
|---|---|---|
minReadySeconds | 45 | A new pod serves this long before an old one stops |
edgePreStopSeconds | 35 | The ingest edge keeps serving this long after it is told to stop: on EKS, deregistration plus 3 × 10 s failed checks; on GKE, 2 × 5 s |
vectorPreStopSeconds | 40 | Longer than the 30 s cluster DNS cache plus nginx's 5 s re-read |
vectorMaxUnavailable | 0 | Vector rolls one extra pod at a time instead of stopping one |
GKE: the chart's HealthCheckPolicy checks every 5 s with a 3 s timeout, healthy after 1 success and unhealthy after 2 failures (gateway.gke.healthCheck). EKS: the TargetGroupConfiguration checks /healthz every 10 s with a 5 s timeout, healthy after 2 and unhealthy after 3. Also label the namespace elbv2.k8s.aws/pod-readiness-gate-inject=enabled. With your own load balancer, set edgePreStopSeconds above its check interval × unhealthy threshold, and minReadySeconds above its interval × healthy threshold plus its registration time. On chart 2.0.13 an HA upgrade under traffic answered about 19 retryable 502/503s over 40 seconds; senders that retry lost nothing.
Always pass --wait --timeout 20m: migrations wait for ClickHouse, Helm's default of 5 minutes covers the whole upgrade, and --wait reports failure while the new pods are still not ready instead of printing success. Migrations run before any Deployment changes; if they fail, the previous version keeps serving. Migrations are forward-only, so roll forward instead of using helm rollback across a schema change. Helm never upgrades CRDs: when a release moves to a newer ClickHouse operator, first run kubectl apply --server-side -f nano/crds/clickhouse-operator-crds.yaml.
Upgrading from 2.0.10 or earlier to 2.0.11: a short OTLP gap. Vector now accepts OTLP only through the otlp-guard, which requires the ingest edge's hop token. The ingest edge and Vector roll out together, so until every ingest-edge pod is new, an old edge pod can reach a new Vector pod and get 401 on /otlp. OpenTelemetry exporters don't retry a 401, so those batches are lost. HTTP /ingest and HEC aren't affected. The gap closes when kubectl -n nano rollout status deployment/nano-ingest-edge completes. If OTLP loss matters, pause your OTLP senders for the upgrade. If you bring your own secret (secrets.existingSecret), add vector_otlp_hop_token to it first (see the Secrets list under Advanced), and set vector.native.enabled: true if in-cluster Vector forwarders send to port 6000.
Upgrading to 2.0.27 or later on a smaller ClickHouse disk than the plan. From 2.0.27, with sizing.plannedGBPerDay set (every sizing preset sets it), the chart compares clickhouse.storage.size × shards with the planned volume and the retention.
- First install: under 7 days, it refuses to render.
helm upgradeof a running release: it goes ahead and prints a warning. For example,values-small.yaml(plannedGBPerDay: 13) with a 20Gi ClickHouse disk holds about 5 days, and still upgrades.- Renders that count as a first install:
helm templatewithout--is-upgrade,helm install --dry-run, and Argo CD (which renders withhelm template) refuse it.
Before you upgrade, compare the disk with GB/day × retention days ÷ 5 × 1.15 ÷ 0.70 (see Retention and disk). Then do one of these:
- Raise
clickhouse.storage.size. The StorageClass must allow expansion. GKEstandard-rwoand AKSmanaged-csido. On EKS, setallowVolumeExpansion: trueon yourgp3class; EBS allows one size change per volume every 6 hours. - Turn on the cold tier.
- Set
sizing.allowUndersizedDisk: truein the values your renders and GitOps tool use.
The check reads values, not the volume, so it behaves the same on every cloud.
Egress NetworkPolicy DNS changes in 2.0.28. See Egress NetworkPolicy.
Upgrade recovery
An upgrade runs two pre-upgrade hook Jobs before anything else changes:
nano-ch-config-heal-r<revision>(from chart 2.0.10). If ClickHouse is still unhealthy aftermigrations.clickhouseConfigHeal.graceSeconds(60) and is down on its configuration (no replica answers/ping, or a ClickHouse pod is inCrashLoopBackOffor exited with code 36), it merge-patches onlyspec.settingsof theClickHouseClusterwith the new release's values. A ClickHouse that crash-loops on its configuration gets the fix, and the migration can proceed. On a healthy cluster, or with one replica down for another reason (a node drain, aPendingpod), it does nothing.nano-migrate-r<revision>. It waits at mostmigrations.upgradeWaitSeconds(600) for every registered ClickHouse replica, then fails with a hint. A failed hook fails the upgrade, and the previous version keeps serving.
To recover, fix the value that broke ClickHouse and run the same helm upgrade --wait --timeout 20m again. If the first attempt timed out, run it again once the ClickHouse pods are Running.
Find the cause:
kubectl -n nano get pods -l app.kubernetes.io/name=clickhouse-server
kubectl -n nano logs <clickhouse pod> --previous | tail
kubectl -n nano logs job/nano-migrate-r<N>
kubectl -n nano logs job/nano-ch-config-heal-r<N>Charts before 2.0.10, or with migrations.clickhouseConfigHeal.enabled: false. Patch the offending key on the ClickHouseCluster yourself, wait for the ClickHouse pods to be Running, then run helm upgrade again. For example:
kubectl -n nano patch clickhousecluster nano --type merge \
-p '{"spec":{"settings":{"extraConfig":{"background_pool_size":"16"}}}}'First install. The migration Job retries up to migrations.backoffLimit (4) times. If it still fails, app pods stay in Init:Error on wait-for-migrations. Fix the cause, then run helm upgrade with the same values: migrations run as the hook, and the app pods no longer wait on the install Job.
If you fixed the cause while the Job was still retrying and a later attempt succeeded, the failed attempts' pods stay behind in Error until the next helm upgrade replaces the Job. They don't affect nano. Delete them once the install is healthy:
kubectl -n nano delete pod -l job-name=nano-migrate-r1 --field-selector=status.phase=FailedServer-side apply conflict. If a later helm upgrade reports a conflict with the field manager nano-ch-config-heal, run it again with --force-conflicts.
Known 2.0.9 issues fixed in 2.0.10
- ClickHouse exits with code 36 on
values-small.yaml. Itsbackground_pool_sizeof 8 is rejected by ClickHouse 26.10. On 2.0.9, setclickhouse.server.background_pool_size: "16". From 2.0.10, the chart refuses to render a pool setting that would stop ClickHouse. - With
clickhouse.replicas: 2, the migration waits for "database visible on 2/4 nodes". The ClickHouse operator's database sync makesdefaulta Replicated database, which lists a seconddefaultcluster. 2.0.10 keeps database sync on and counts only the configured cluster. On 2.0.9 there is no values workaround; upgrade to 2.0.10. As a last resort on 2.0.9 only, an advanced step: patch theClickHouseClusterwithspec.settings.enableDatabaseSync: false, runDROP DATABASE default SYNC; CREATE DATABASE default ENGINE=Atomicon each replica, then runhelm upgradeagain. Don't do this on 2.0.10; it isn't needed there and isn't recommended. - Vector never becomes ready (
Permission deniedreading the config generations). 2.0.10 runs Vector as the config sidecar's user, uid 100. - AWS: "TargetGroup port is empty". 2.0.10 renders a
TargetGroupConfigurationwithiptargets. On 2.0.9, install the AWS Load Balancer Controller with--set defaultTargetType=ip. - The connector runner crash-loops with
VRL sandbox self-test failed: VRL worker died: Operation not permittedwhere the container runtime gives pods a finite stack hard limit (seen on EKS AL2023 with containerd 2.3). The sandboxed VRL worker asked for a larger stack limit than the pod may set. No values workaround; 2.0.10 keeps the inherited limit, with the sandbox (seccomp included) still enforced. Pull feeds are unavailable until then.
Air-gapped clusters
- Load the signed image bundle into your internal registry.
- Set
global.imageRegistry: registry.internalandclickhouse-operator.global.imageRegistry: registry.internal. - On a connected host,
helm pullthe chart as in the install (without--untar), copynano-<version>.tgzin, and install it with the same values.
With global.imageRegistry set, the registry credential refresh stays off. See Air-gapped deployment for offline content, licensing and AI.
Advanced
Registry access
nano images come from us-central1-docker.pkg.dev/nano-rs/nanosiem-customer with a pull-only credential that lasts one hour. Pods that start later, for example after a node replacement, can't pull images with an expired credential.
From chart 2.0.10, the chart refreshes it. The CronJob nano-registry-cred-refresh runs every 20 minutes. It requests a new credential from GET https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<id> with your enrollment token (the license_token key of the chart secret, set from secrets.licenseToken), and updates the pull secret in place.
registryCredentialRefresh.enabled: auto(the default) turns it on whennano.license.url,nano.deploymentIdandglobal.imagePullSecretsare set andglobal.imageRegistryis empty.- It updates the first entry of
global.imagePullSecrets, herenano-registry, orregistryCredentialRefresh.secretName. The secret must already exist; create it as in the install. - Its own image,
curlimages/curl, comes from Docker Hub, not from the registry it refreshes. - With
secrets.existingSecret, your secret must holdlicense_token.
Check it, or refresh now:
kubectl -n nano get cronjob nano-registry-cred-refresh
kubectl -n nano create job --from=cronjob/nano-registry-cred-refresh refresh-now
kubectl -n nano logs job/refresh-nowOn older charts, or to avoid the dependency on the license API, mirror the images into your own registry once and set global.imageRegistry. To request a credential by hand, for example for helm registry login:
( umask 077; printf 'Authorization: Bearer %s\n' "$(cat license-token.txt)" > license-header.txt )
curl -fsS -H @license-header.txt \
"https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<your deployment ID>"
rm -f license-header.txtThe response has username (oauth2accesstoken), password and expiresAt. For the same call on a single host, see Reruns, the registry and TLS. ClickHouse, Keeper, Vector, PostgreSQL, Valkey and nginx come from public registries and are digest-pinned in values.yaml.
What the chart deploys
| Component | Kind | Notes |
|---|---|---|
| api, search, jobs, web | Deployments | jobs uses leader election, so several replicas are safe |
| edge | Deployment (nginx) | The app frontend (UI and API) |
| ingest edge | Deployment, autoscaler, PodDisruptionBudget | The ingest frontend, with its own hostname and Gateway |
| Vector | Deployment, autoscaler | A sidecar delivers parsers, sources and ingest tokens |
| connector runner | Deployment | Pull feeds. Lease fencing makes several replicas safe |
| Valkey | Deployment | Cache and coordination, one replica |
| PostgreSQL | StatefulSet, or external | Bundled for small installs; managed for production |
| ClickHouse and Keeper | ClickHouseCluster, KeeperCluster | Through the bundled ClickHouse operator |
| Schema migrations | Job | A pre-upgrade hook on upgrades, after the ClickHouse config heal hook |
| Registry credential refresh | CronJob | Online-licensed installs without a mirror |
Components with more than one replica also get a PodDisruptionBudget. The chart installs the ClickHouse operator with its admission webhooks off, so it doesn't need cert-manager. If the operator is already installed in the cluster, set clickhouse-operator.enabled=false and pass --skip-crds. Run one operator per cluster.
Values you will set
| Value | Purpose |
|---|---|
nano.baseUrl | Public URL. An http:// URL turns on dev mode (cookies without Secure) |
nano.deploymentId, nano.license.url, secrets.licenseToken | Online license. Leave nano.license.url empty for the signed offline license |
nano.schemaProfile | ocsf (default). Never change it on an existing install |
nano.retention.hotDays / warmDays | Retention policy |
nano.extraEnv | Extra environment for api, search and jobs |
secrets.existingSecret | Bring your own secret (External Secrets, Vault, sealed-secrets) |
global.storageClass | StorageClass for nano's volumes; empty uses the cluster default |
expose.mode, gateway.*, tls.* | Gateway API exposure and TLS (set by the exposure preset) |
postgresql.mode | bundled or external |
clickhouse.shards, resources, storage | Set by the preset. Plan the shard count up front |
clickhouse.replicas | HA switch, 1 by default |
clickhouse.zoneAware | Spread replicas and Keepers across zones |
clickhouse.coldStorage.* | Object-storage cold tier |
clickhouse.allowedNetworks, clickhouse.ingestAllowedNetworks | ClickHouse network allow-list |
sizing.plannedGBPerDay, sizing.ingestPath, sizing.avgEventBytes | Planned volume and ingest path |
migrations.prevalenceRefreshMinutes | Prevalence refresh interval. Set by the large and xlarge presets |
ingress.host | App hostname |
ingest.hostname, ingest.service.*, ingest.edge.* | Ingest frontend |
ingest.nodeSelector, tolerations, affinity | Ingest node pool |
registryCredentialRefresh.* | Registry credential refresh |
migrations.upgradeWaitSeconds, migrations.clickhouseConfigHeal.* | Upgrade recovery |
ClickHouse needs at least 6 GiB of memory with enrichment on. Per-query memory (clickhouse.profiles.default.max_memory_usage) should be about 50% of the server cap, which is 80% of the pod memory limit; change them together. Replicas add availability, not capacity, and cost about twice the ClickHouse resources. You can add replicas or shards later with helm upgrade, but never shrink a running Keeper ensemble.
On arm64 node pools that are tainted (GKE: kubernetes.io/arch=arm64:NoSchedule), add the toleration with global.tolerations and clickhouse-operator.manager.tolerations.
Secrets
Unless secrets.existingSecret is set, the chart generates nano-secrets (for the release name nano; with another release name, the prefix changes). It keeps it across upgrades and on helm uninstall. If unsure, run kubectl -n nano get secret | grep secrets. Back it up. Losing encryption_key makes every stored credential unrecoverable.
A secret you bring yourself (secrets.existingSecret) must hold the keys the chart would generate. Keep passwords URL-safe.
| Key | Required |
|---|---|
encryption_key | Yes. 64 hex characters |
jwt_secret | Yes |
postgres_password | With the bundled PostgreSQL. External PostgreSQL reads postgresql.external.passwordSecret instead |
clickhouse_default_password, clickhouse_password, clickhouse_admin_password, clickhouse_ingest_password, clickhouse_vector_password, clickhouse_rawsql_password, clickhouse_rawsql_noaudit_password | Yes |
vector_auth_token, otlp_edge_key, platform_ingest_token | Yes |
vector_config_delivery_token | Yes (falls back to vector_auth_token before v2.1) |
vector_tap_token | With vector.tap.enabled |
vector_otlp_hop_token | Yes, from 2.0.11. 32 or more characters from A-Z a-z 0-9 _ -. The ingest edge presents it to Vector's otlp-guard; without it, or in another format, the install or upgrade stops with instructions |
setup_token | To create the first admin on the setup page (from 2.0.8). Without it, setup works only from inside the api pod (loopback) |
license_token | With an online license. The registry credential refresh needs it too |
docs_search_token | Optional. pivt's grounding in the nano documentation |
health_export_token | Optional. Only with nano.healthExporter.endpoint |
The ingest frontend
Ingest never shares a frontend with the app, so a sender flood can't take the UI down. It has its own hostname, Gateway or LoadBalancer Service, edge pods and node pool. The ingest hostname serves:
| Protocol | Endpoint |
|---|---|
| HTTP | https://<ingest-host>/ingest |
| HEC | https://<ingest-host>/services/collector/event (and /services/collector/raw); see HEC |
| OTLP/HTTP | https://<ingest-host>/otlp/v1/{logs,traces,metrics}, binary protobuf only (Content-Type: application/x-protobuf, the OpenTelemetry SDK and Collector default). OTLP/JSON gets 415; see OTLP |
The app hostname answers 404 on these paths. Issue a token per source under Ingestion → Log Sources → Credentials.
nano's HEC endpoint has indexer acknowledgements on, so every request needs a channel: an X-Splunk-Request-Channel: <uuid> header or a ?channel=<uuid> parameter. Without one, it answers 400 with Data channel is missing. Send every event and every /services/collector/ack poll of a channel with the same channel. ingest.edge.hecAcks: false only lets the ingest edge scale out; it doesn't turn channels off. When Vector scales, a moved channel's pending acks are lost and senders re-send.
From chart 2.0.10, the ingest hostname also answers GET /healthz with 200.
Cold tier
The cold tier is optional and off by default (clickhouse.coldStorage.enabled: false). With it on, ClickHouse moves older parts to an object-storage bucket when the local disk passes about 70% full (move_factor 0.3). Moved parts stay searchable through a local cache (cacheSize, 20Gi by default). Only the table's retention (DELETE TTL) removes data. For 365-day retention on a cloud cluster, we recommend turning it on and sizing the hot disk for 30 to 90 days.
clickhouse:
coldStorage:
enabled: true
provider: s3 # s3 | gcs | azure | s3-compatible
bucket: acme-nano-cold
region: us-east-1 # the cluster's region
prefix: ""
useEnvironmentCredentials: true # or credentialsSecret: nano-cold
cacheSize: 200Gi
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/nano-clickhouse-coldThe ClickHouse service account is nano-clickhouse for the release name nano. To check it, run helm template with your values and look for kind: ServiceAccount.
| Provider | Credentials |
|---|---|
| S3 on EKS | useEnvironmentCredentials: true with IRSA or EKS Pod Identity on the ClickHouse service account (clickhouse.serviceAccount.annotations). Limit the IAM policy to s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket and s3:GetBucketLocation on that bucket and prefix |
| GCS on GKE | useEnvironmentCredentials: true with Workload Identity: annotate iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com, grant the Google service account roles/storage.objectAdmin on the bucket, and bind roles/iam.workloadIdentityUser for <namespace>/nano-clickhouse. Or HMAC keys in credentialsSecret |
| Azure Blob on AKS | Account keys only: account_name and account_key in credentialsSecret, plus azure.storageAccountUrl and azure.containerName. The container must exist. Managed identity and Workload Identity aren't supported for the cold tier yet; the chart refuses useEnvironmentCredentials with azure |
| Wasabi, RustFS, Civo Object Store | provider: s3-compatible plus endpoint, with keys in credentialsSecret (access_key_id, secret_access_key) |
Cost. Approximate list prices per GB-month (us-east-1, us-central1; check your region's prices):
| Cloud | Block disk | Object storage |
|---|---|---|
| AWS | EBS gp3, about $0.08 | S3 Standard, about $0.023 |
| Google Cloud | pd-balanced, about $0.10 | GCS Standard, about $0.020 |
| Azure | Managed disks, about $0.075 (Standard SSD) to $0.12 or more (Premium) | Blob Hot, about $0.018 to $0.02 |
For example, 12.5 GB/day compresses to about 2.5 GB/day on disk, or about 900 GB for 365 days per replica. On gp3 that is about $72 a month per replica, plus headroom. With the cold tier, it is about $21 a month in S3 plus a hot disk for 90 days (about 225 GB of gp3, about $18). With clickhouse.replicas: 2, each replica writes its own copy to the bucket, so double the object storage as well as the disks.
Requests and egress cost little for a SIEM's access pattern, but a bucket in another region adds egress charges: keep it in the cluster's region. Use a standard storage class. Infrequent-access and archive classes (S3 Standard-IA, Glacier) charge per GB read and have minimum storage durations, and ClickHouse reads this tier.
To enable it on an existing install, run this once per large table after the upgrade (add ON CLUSTER 'default' on clusters):
ALTER TABLE nanosiem.ocsf_logs MODIFY SETTING storage_policy = 'tiered'Backups and uninstall
- PostgreSQL: use your managed database's backups. In bundled mode, run
kubectl -n nano exec nano-postgres-0 -- pg_dump -U nanosiem nanosiem > nano-pg.sqlon a schedule. - ClickHouse:
BACKUP DATABASE nanosiem TO S3(...)per shard, or volume snapshots. With a cold tier, cold parts are already in your bucket. - The chart secret (
nano-secrets): back it up out of band, encrypted.
helm uninstall nano -n nano removes the release, including the ClickHouseCluster and KeeperCluster; you don't need to delete those yourself. It keeps the PVCs, the chart secret and any hook Jobs that are still there:
- From chart 2.0.11, the config heal (
nano-ch-config-heal-r<N>) and schema sync (nano-ch-schema-sync-r<N>) hook Jobs delete themselves when they succeed. The migration hook Jobs (nano-migrate-r<N>) stay so you can read their logs, and Kubernetes deletes them a day after they finish. Failed hook Jobs also stay for a day. - On chart 2.0.10, every hook Job stays after it finishes, and Kubernetes deletes it a day later.
To delete everything, including all stored events, after the uninstall:
kubectl -n nano delete jobs --all
kubectl -n nano delete pvc --all && kubectl -n nano delete secret nano-secretsOr delete the namespace, kubectl delete namespace nano, which removes all of it. With the default Delete reclaim policy, the cloud disks go with the PVCs. helm uninstall also keeps the vector-source-credentials secret (the ingest credentials Vector uses); the namespace delete removes it, or delete it with kubectl -n nano delete secret vector-source-credentials.
Tearing down the cluster too. Remove nano's load balancers while the controller that owns them is still running, then the controller, then the cluster. On EKS:
-
helm uninstall nano -n nanodeletes the Gateways. Wait untilkubectl -n nano get gatewaylists none and the ALBs are gone (aws elbv2 describe-load-balancersno longer lists them). -
kubectl delete gatewayclass aws-albwhile the AWS Load Balancer Controller still runs. Deleted after the controller, it hangs on thegateway.k8s.aws/gatewayclassfinalizer. -
Delete the namespace, then uninstall the controller (
helm uninstall aws-load-balancer-controller -n kube-system). -
Delete the RDS instance, its DB subnet group and its security group, in that order, waiting for each. They are in the cluster's VPC, and
eksctlcan't delete a VPC that still holds them:aws rds delete-db-instance --db-instance-identifier <instance> --skip-final-snapshot # or --final-db-snapshot-identifier <name> aws rds wait db-instance-deleted --db-instance-identifier <instance> aws rds delete-db-subnet-group --db-subnet-group-name <subnet group> aws ec2 delete-security-group --group-id <database security group id>If the instance has deletion protection, turn it off first with
aws rds modify-db-instance --db-instance-identifier <instance> --no-deletion-protection --apply-immediately. The instance takes 5 to 15 minutes to delete. Ifdelete-security-groupreportsDependencyViolation, the instance's network interface is still being released: wait a minute and run it again. -
eksctl delete cluster -f cluster.yaml --wait, then the controller's IAM policy.
If eksctl delete cluster fails with The vpc ... has dependencies and cannot be deleted, the cluster is already gone, but the VPC stack (eksctl-<cluster>-cluster in CloudFormation) is in DELETE_FAILED, and its NAT gateways keep billing. Delete whatever is left in the VPC, such as the RDS subnet group and security group, then run aws cloudformation delete-stack --stack-name eksctl-<cluster>-cluster and aws cloudformation wait stack-delete-complete --stack-name eksctl-<cluster>-cluster. Running eksctl delete cluster again only says that no cluster was found.
On GKE, uninstall nano and wait for the Gateways to go before you delete the cluster, so the load balancers and their network endpoint groups are removed with them.
Troubleshoot
Need help? Contact support with your deployment ID and the time, with timezone.
Pods stay in ImagePullBackOff at install
The credential in nano-registry was already expired or mistyped when you created the secret. Get a new one and recreate the secret, or mirror the images. See Registry access.
Pods go to ImagePullBackOff after an hour
The pull credential lasts one hour, so pods scheduled later can't pull. From chart 2.0.10, check that kubectl -n nano get cronjob nano-registry-cred-refresh exists and that its last Job succeeded; run one now with kubectl -n nano create job --from=cronjob/nano-registry-cred-refresh refresh-now. If the CronJob is missing, set nano.license.url, nano.deploymentId and global.imagePullSecrets, and leave global.imageRegistry empty. On older charts, refresh the secret by hand or mirror the images. See Registry access.
helm pull fails with 401 or 403
The registry credential expired, or the chart path is wrong. Use oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano; the customer credential can't read other paths. Generate a new credential, run helm registry login again, then pull.
ClickHouse or Keeper pod Pending after a drain
kubectl -n nano describe pod shows volume node affinity conflict. Its disk (EBS, persistent disk or Azure Disk) is in one zone, and no schedulable node is left in that zone. Uncordon the node or add one in that zone; plan two nodes per zone. See Cluster, GKE cluster or AKS cluster.
Volumes stay Pending
The cluster has no default StorageClass. Create one, for example the gp3 StorageClass on EKS, or set global.storageClass.
GKE Gateways get no address
kubectl get gatewayclass doesn't list gke-l7-global-external-managed: the cluster was created without the Gateway API. Run gcloud container clusters update nano --project "$PROJECT" --region "$REGION" --gateway-api=standard, wait for the class to appear, then run the same helm upgrade --install again. If the class exists, kubectl -n nano describe gateway names the problem, for example a certificate map that doesn't exist or a reserved address name that doesn't match gateway.addresses. See Google Kubernetes Engine (GKE).
The ALB shows "TargetGroup port is empty"
The AWS Load Balancer Controller created instance targets for ClusterIP Services. Upgrade to chart 2.0.10, which sets ip targets for every Gateway backend. On 2.0.9, reinstall the controller with --set defaultTargetType=ip.
ClickHouse exits with code 36
ClickHouse rejected a setting at startup; its log names it. On 2.0.9 with values-small.yaml, it is background_pool_size: set clickhouse.server.background_pool_size: "16". Then run helm upgrade --wait --timeout 20m again. See Upgrade recovery.
The migration waits for "database visible on N/M nodes"
On 2.0.9 with clickhouse.replicas: 2, the migrator counts nodes from a second default cluster that the ClickHouse operator's database sync adds, so it waits for nodes that don't exist. Upgrade to 2.0.10; see Known 2.0.9 issues for a last-resort step on 2.0.9. If N is lower than the number of ClickHouse pods you run, check those pods instead. See Upgrade recovery.
The license shows as invalid
Check that nano.deploymentId matches app.nano.rs exactly and that the cluster reaches https://api.nano.rs. If the token was lost or exposed, an owner or admin can select Regenerate license token under 2 · License on the deployment's page. The old token stops working at once. Save the new one to license-token.txt, run helm upgrade with the same values and --set-file secrets.licenseToken=./license-token.txt, then restart the api and jobs pods so they read it.
The chart refuses to render
The message names the value. Common causes: nano.license.url without nano.deploymentId, an ingest hostname equal to the app hostname, ingest.edge.replicas above 1 with hecAcks: true, or an empty ACM ARN with values-aws.yaml. A message that clickhouse.storage.size "holds only about N days" is the disk check: see Upgrades.
Pods wait on the migration Job
nano-migrate-r1 waits until every ClickHouse replica and PostgreSQL answer. Follow it with kubectl -n nano logs -f job/nano-migrate-r1. If it fails, see Upgrade recovery.
helm install fails with "Progress deadline exceeded"
On chart 2.0.10 and earlier, helm install --wait can fail after about 10 minutes with Error: resource Deployment/nano/nano-api not ready. status: Failed, message: Progress deadline exceeded, although the pods become ready a few minutes later. The first migration took longer than the Deployments' 10-minute progress deadline (progressDeadlineSeconds, 600), for example against a managed PostgreSQL in another region, or after retries on a missing extension. Nothing is lost and nothing needs cleaning up.
- Wait for the migration to finish:
kubectl -n nano logs -f job/nano-migrate-r1, thenkubectl -n nano get pods. - Once the Job has completed, run the install again as
helm upgrade --install nano ./nano -n nano --wait --timeout 30mwith the same-ffiles, in the same order, and the same--set-file. It completes the release. A secondhelm installrefuses the name, because the failed release still exists.
On 2.0.10, a longer --timeout doesn't help: the deadline is set on each Deployment, and Helm 4 (whose --wait reads the Deployment's progress condition) fails as soon as one passes it. From chart 2.0.11, those Deployments allow migrations.dependentProgressDeadlineSeconds (1800, 30 minutes). Install with --wait --timeout 30m, which matches it, or raise both together.
The migration fails with "transaction_timeout"
unrecognized configuration parameter "transaction_timeout" means nano 2.0.10 or earlier on PostgreSQL 16. Upgrade nano to v2.0.11 or later, or use PostgreSQL 17 or newer. See Managed PostgreSQL requirements.
The migration fails with "permission denied for schema public"
The nano user can't create tables in schema public. On Azure flexible server, public belongs to azure_pg_admin in every new database. As the admin user, connected to the nanosiem database (for a private server, from the one-off pod), run ALTER SCHEMA public OWNER TO nanosiem;, or GRANT ALL ON SCHEMA public TO nanosiem;. Then run the same helm upgrade --install again; if the Job is still retrying, its next attempt succeeds. The failed migration applied nothing. From 2.0.12, nano checks this before the migration and names the fix. On 2.0.11 the check passed and the migration failed.
The migration fails with "not allow-listed" or a missing extension
Azure flexible server refuses extensions that aren't in its azure.extensions server parameter. Append pg_trgm,pgcrypto,uuid-ossp to it as in Managed PostgreSQL requirements, then run the same helm upgrade --install again. The failed migration applied nothing, so the database needs no cleanup. On Cloud SQL, permission denied to create extension means the nano user was created with CREATE ROLE in SQL; create it with gcloud sql users create instead.
Install with Docker Compose
Install nano on a Linux server you control with Docker Compose, for Open (free) or Self-run (paid). Size the host, serve HTTPS, back up and upgrade.
Monitor your Self-run install
Watch nano's own health with Prometheus and Grafana on a single node, a role-split cluster or Kubernetes, and send nano's health alerts into nano.