nano SIEM
Getting Started

Install on Kubernetes (Helm)

Supported in:Self-hosted

Install nano Self-run on your own Kubernetes cluster with the nano Helm chart, then size it, make it highly available and upgrade it.

Installing on Kubernetes needs a Self-run license (paid). Open (free) installs with Docker Compose. You set your license values in step 3, and they unlock the Self-run features and high availability.

Install nano Self-run on a Kubernetes cluster that you operate, with the nano Helm chart. The chart runs the whole stack, including ClickHouse through the official ClickHouse operator. For one server without Kubernetes, see Install with Docker Compose.

Before you start

For about 12.5 GB/day (about 150 events per second), you need:

ItemRequirement
Kubernetes1.28 or newer (1.30 or newer recommended), and Helm 3.14 or newer
CapacityAbout 8 vCPU and 32 GB RAM free. For HA, six worker nodes with 8 vCPU and 32 GB each, two in each of three zones; see Highly available values. Check that your cloud quota allows 48 vCPUs or more of that instance type first
StorageA default StorageClass, with 1.5 TiB for a year of events or 500 GiB for 90 days. GKE and AKS ship one; EKS has none out of the box, see Amazon EKS. Or name one with global.storageClass
CPUAny x86-64 server CPU from about 2015 or later, or ARM64
IngressA Gateway API implementation, such as Envoy Gateway, GKE Gateway, the AWS Load Balancer Controller v2.13 or newer, or Azure Application Gateway for Containers
Network policyA network plugin that enforces Kubernetes NetworkPolicy. Most managed clusters don't by default; see Network policy enforcement
TLSA cloud-managed certificate (ACM on EKS, a Certificate Manager map on GKE), or cert-manager
DNSTwo names in a domain you control: one for the UI (nano.example.com) and one for senders (ingest.nano.example.com)
PostgreSQLBundled for small installs. For production, a managed PostgreSQL 16 or newer with three extensions allowed; see Managed PostgreSQL requirements

From your Self-run deployment you also need the following. Each comes from the deployment's page in app.nano.rs or, if you only have the three license values (deployment ID, license URL, license token), from the license API as noted:

  • The version to install, from the release notes or the deployment's page. Set it once; every command below uses it. This reads the current release; to install another one, set it by hand (export NANO_VERSION=<version>):

    export NANO_VERSION=$(curl -fsS https://api.nano.rs/api/releases/latest \
      | python3 -c 'import json,sys; print(json.load(sys.stdin)["version"])')
    echo "$NANO_VERSION"
  • The three license values, from 2 · License on the deployment's page. The token is shown only once. 3 · Install → Kubernetes shows them as Helm values. One Helm release is one install, however many api and jobs pods it runs; see How many installs a license covers.

  • A registry credential, from 4 · Registry access → Generate pull credential, or with only the license values:

    ( umask 077
      printf 'Authorization: Bearer %s\n' "$(cat license-token.txt)" > license-header.txt
      curl -fsS -H @license-header.txt \
        "https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<deployment ID>" \
        | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"], end="")' > registry-password.txt
      rm -f license-header.txt )

    curl reads the token from a header file and the password goes straight to registry-password.txt, so neither appears on a command line, in the process list or in your shell history. The password field is the pull password (username oauth2accesstoken). It pulls the chart and the images, and lasts one hour. From chart 2.0.10, the chart keeps it fresh in the cluster; see Registry access.

You don't need the Compose bundle or a hand-built image lock: the chart ships the release's image lock and the image values made from it.

Network policy enforcement

Vector's OTLP receiver (4317 gRPC, 4318 HTTP) has no credential check of its own, and before chart 2.0.30 neither had its Vector-native listener (6000). How nano keeps other pods away from them depends on the chart version:

  • From chart 2.0.11: both listen on loopback only, inside the Vector pod. The only way in is an otlp-guard sidecar (on nano-vector-internal, port 4318) that forwards a request only when it carries the ingest edge's hop token (vector_otlp_hop_token in the chart secret). The edge sends that token after it has checked the sender's ingest token. Without it, the guard answers 401, whatever the network plugin. OTLP/gRPC and Vector-native aren't on any Service. The chart's NetworkPolicy, nano-vector-ingress, stays as defense in depth.
  • Chart 2.0.10 and earlier: the listeners are reachable on the pod network, and the NetworkPolicy is the only thing that keeps other pods out.

Kubernetes stores a NetworkPolicy whatever the network plugin, but only a plugin that enforces it applies it. On a cluster without enforcement, nothing warns you. Turn it on: required on 2.0.10 and earlier, recommended from 2.0.11.

Vector-native (6000). From chart 2.0.30 it requires mutual TLS: every sender needs a client certificate; see Vector-native client certificates. With the default vector.native.enabled: false, it listens only on loopback inside the Vector pod and no Service exposes it. From 2.0.11 to 2.0.29 it had no credential check, and vector.native.enabled: true exposed it on nano-vector-internal for any pod the NetworkPolicy admitted. Before 2.0.11, the port was always reachable in-cluster.

Vector metrics (9598). Vector's Prometheus endpoint has no credential. It serves read-only counters (component names, throughput, errors), never events. On chart 2.0.11 the chart's policy admits it from every pod in the cluster; from 2.0.12 only from pods in nano's namespace (nano's own pods read it). To scrape it from a monitoring namespace, add that namespace with vector.metrics.networkPolicyFrom, for example [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}}].

Turn on enforcement when you create the cluster:

ClusterHow
GKEDataplane V2 (--enable-dataplane-v2), which you can only choose at creation. On an existing cluster without it, turn on the network policy add-on instead (--update-addons=NetworkPolicy=ENABLED, then --enable-network-policy); this recreates the nodes. See GKE cluster
EKSThe VPC CNI network policy agent: VPC CNI v1.14 or newer with enableNetworkPolicy: "true" in the vpc-cni add-on's configuration. Set it when you create the cluster; on an existing cluster, follow Cluster exactly, because a plain update-addon can break pod networking
AKS--network-dataplane cilium (Azure CNI powered by Cilium), or --network-policy azure or calico. See AKS cluster
Other clustersA plugin that enforces NetworkPolicy, such as Calico or Cilium

After the install, check from a throwaway pod in another namespace that it can't reach Vector's OTLP port:

kubectl run np-test -n default --rm -i --image=curlimages/curl:8.12.1 --restart=Never --command -- \
  sh -c 'curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://nano-vector-internal.nano.svc.cluster.local:4318/v1/logs; true'

Change nano in the host name if your release or namespace differs. With enforcement, the request times out and the command prints 000 (the pod still ends normally). Without enforcement, chart 2.0.11 and later answer 401 (the guard refuses a request without the hop token), which is safe but means the policy isn't enforced; on 2.0.10 and earlier, any HTTP status means anyone in the cluster can write OTLP.

Vector-native client certificates

From chart 2.0.30, the Vector-native listener (6000) always requires mutual TLS. There is no plaintext mode. vector.native.enabled: true publishes it on nano-vector-internal, and vector.service.exposeNative: true also on nano-vector. Both are ClusterIP Services, so senders run in the cluster. For senders outside it, put your own TCP load balancer in front (no TLS termination) and add its name or address to vector.native.tls.extraDnsNames or extraIPs.

The chart keeps two separate trust paths. With release nano:

  • nano-vector-native-ca: the listener CA. Generated once, kept on uninstall, mounted into no pod. It signs nano-vector-native-tls, the listener certificate (the two Vector Services' in-cluster names, localhost, 127.0.0.1 and your extra names), which is re-signed when the CA or the names change. Senders verify nano with this CA's certificate.
  • nano-vector-native-clients (key ca.crt): the sender anchor bundle, one anchor per sender. It starts with a locked anchor, so no sender is accepted until you add one. The chart never overwrites it and keeps it on uninstall.

Helm reads both back with lookup. A renderer without cluster access (helm template, Argo CD) would make a new listener CA on every render, so GitOps installs set vector.native.tls.existingSecret: a Secret with tls.crt and tls.key (the listener) and ca.crt (the anchor bundle). With secrets.existingSecret and no TLS Secret, the listener is locked and vector.native.enabled is refused.

Issue a sender certificate on an operator workstation with the nano release bundle or a nano checkout:

umask 077; mkdir -p nano-vn
kubectl -n nano get secret nano-vector-native-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > nano-vn/listener-ca.crt
scripts/nano vector-client-cert edge-dc1 --pki-dir nano-vn --server-ca nano-vn/listener-ca.crt \
  --address nano-vector-internal.nano.svc:6000
# publish the anchor bundle, then restart Vector so it reads it
kubectl -n nano create secret generic nano-vector-native-clients \
  --from-file=ca.crt=nano-vn/client-ca.crt --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nano rollout restart deployment/nano-vector

Change nano in the names if your release or namespace differs. The command prints the Vector sink config (see Vector native). --csr FILE keeps the sender's key on the sender. --renew edge-dc1 --pki-dir nano-vn re-signs a sender under the same anchor, with no Secret change and no restart.

To revoke, run scripts/nano vector-client-cert --revoke edge-dc1 --pki-dir nano-vn, then apply the Secret and restart Vector as above. Only that sender's anchor leaves the bundle. The revocation takes effect only when the restart has replaced every Vector pod: until then, pods that haven't restarted still accept the sender. Check with kubectl -n nano rollout status deployment/nano-vector.

  • Keep nano-vn. It holds the anchors and their keys, and nothing backs it up for you. Without it you can't renew or revoke, and the next issue publishes only the anchors you still have.
  • Expiry. scripts/nano vector-client-cert --list --pki-dir nano-vn marks senders that expire within 30 days. The chart's listener certificate lasts 10 years.
  • Upgrading from 2.0.29 or earlier with vector.native.enabled: true: senders without a client certificate are refused from the upgrade on. The 2.0.30 chart creates the listener CA during the upgrade, so issue each sender's certificate right after it (as above) and update the sender's sink config; plan for a short gap in that sender's data until you do.

Egress NetworkPolicy (optional)

From chart 2.0.27, networkPolicy.egress.enabled: true adds default-deny egress policies for nano's own pods. Every nano pod can then reach DNS and pods in nano's namespace. api, search, jobs and the connector runner can also reach the Kubernetes API (networkPolicy.egress.kubeApi.cidrs, required), external PostgreSQL (networkPolicy.egress.postgresql.cidrs), your egress proxy (networkPolicy.egress.proxy) and any networkPolicy.egress.extraEgress rules. Everything else is blocked. It needs a plugin that enforces egress policy, from the table above. On EKS that means Linux EC2 nodes; the VPC CNI doesn't enforce policy on Fargate or Windows nodes.

DNS. networkPolicy.egress.dns.mode sets where nano's pods may send DNS:

  • any (the default from 2.0.28): UDP and TCP 53 to any destination, and no other port. It works with every cluster DNS path without extra values. The tradeoff: a compromised nano pod could query an outside resolver.
  • peers: only the peers in networkPolicy.egress.dns.to. The default list has kube-dns/CoreDNS pods, NodeLocal DNSCache pods, 169.254.20.10/32 (NodeLocal DNSCache) and 169.254.10.10/32 and 169.254.10.11/32 (AKS LocalDNS). Check it against your cluster first. A wrong list cuts DNS for every nano pod.
ClusterDNS pathWith peers
GKE Dataplane V2 with Cloud DNS, including Autopilot (its default)NodeLocal DNSCache is on by default (Autopilot always; Standard from 1.34.1-gke.3720000). With Cloud DNS, pods query 169.254.20.10 on the node's host networkNot supported: the Cilium dataplane can't match a host-network address with an ipBlock. Use any
GKE Dataplane V2 without Cloud DNSNodeLocal DNSCache pods run on the pod networkThe default list
GKE without Dataplane V2 (network policy add-on)NodeLocal DNSCache and kube-dnsThe default list, plus the kube-dns Service IP as a /32: kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
EKSCoreDNS pods (k8s-app: kube-dns). NodeLocal DNSCache is optional and self-installed, on the host network at 169.254.20.10The default list. With NodeLocal DNSCache, prefer any
AKSCoreDNS pods. With LocalDNS (AKS Automatic, and AKS Standard 1.37 and later in Preferred mode), pods query 169.254.10.10 and 169.254.10.11 on the nodeWith Cilium, an ipBlock can't select node addresses, so LocalDNS is unreachable. Use any
Calico or Cilium elsewherekube-dns pods, or a NodeLocal DNSCache on the host networkCalico: the default list. Cilium with a host-network cache: use any

Kubernetes API addresses. Get them with kubectl get endpoints kubernetes -o jsonpath='{.subsets[*].addresses[*].ip}' and give them as /32s, with these exceptions:

  • GKE private clusters: the control plane range.
  • EKS: with the private endpoint on, the addresses are the control plane's network interfaces in the cluster subnets. EKS adds new ones during a cluster upgrade, so list the cluster subnet CIDRs. A public-only endpoint has public addresses that change; turn on the private endpoint first.
  • AKS with API Server VNet Integration (the AKS Automatic default): the delegated API server subnet CIDR. Turning integration on changes the address.
  • Self-managed clusters with Cilium, where the API server runs on the control plane nodes: Cilium doesn't match node addresses with an ipBlock unless it runs with policy-cidr-match-mode=nodes.

The kubernetes Service ClusterIP (chart 2.0.30 and later). Pods reach the API at kubernetes.default.svc, a ClusterIP such as 172.20.0.1. The EKS VPC CNI network policy agent matches egress ipBlocks against that ClusterIP, not the addresses behind it. With only the addresses above, api, search, jobs and the connector runner stay in Init forever. networkPolicy.egress.kubeApi.serviceIP adds the ClusterIP to the Kubernetes API rule as a /32. It does nothing on GKE and AKS.

  • auto (the default): helm install and helm upgrade look up Service default/kubernetes. helm template and GitOps renderers such as Argo CD can't look it up, so set the address there. If the installing user can't read Services in default, the render fails: set the address, or none.
  • An address: kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}' (dual-stack: "<IPv4>,<IPv6>").
  • none: only kubeApi.cidrs.

When the rule has no ClusterIP, helm install and helm upgrade print a note with the command to add it. On chart 2.0.29 and earlier, add the ClusterIP as a /32 to kubeApi.cidrs instead.

EKS: no ipBlock.except in extraEgress or vectorExtraEgress. The VPC CNI agent applies except ranges as denies that override other policies' allows. A 0.0.0.0/0 rule with except covering the private ranges cuts DNS for every nano pod. List the public destinations as plain cidrs instead. From chart 2.0.30, helm install and helm upgrade warn when they see except in either list.

Pods that use workload credentials also need the metadata endpoint, through extraEgress:

  • EKS Pod Identity: 169.254.170.23/32 on TCP 80.
  • GKE Workload Identity: 169.254.169.252/32 on TCP 988, plus 169.254.169.254/32 on TCP 80 with Dataplane V2.

Turn it on safely. The policies select pods by release, not version, so they apply at once to the running pods as well as new ones.

  1. Turn it on in its own helm upgrade, with the same chart version and only the networkPolicy.egress change.
  2. Read the render first: helm template ... --is-upgrade --show-only templates/networkpolicy-egress.yaml.
  3. Run the upgrade with --wait --timeout 10m --rollback-on-failure (Helm 4) or --atomic (Helm 3). If pods don't become ready, Helm restores the previous revision, which removes the policies.
  4. Check DNS from a pod: kubectl -n nano exec deploy/nano-api -- getent hosts kubernetes.default.svc.

To back out by hand, run helm upgrade with --set networkPolicy.egress.enabled=false, or helm rollback.

ClickHouse network allow-list

From chart 2.0.20, clickhouse.allowedNetworks sets the networks that every nano ClickHouse user accepts connections from. Set it. List your pod CIDRs, and your node CIDRs if a LoadBalancer sits in front of ClickHouse. Entries are IPs, CIDRs or DNS names. An any-address entry such as 0.0.0.0/0 or ::/0 is refused.

This starting value covers the pod networks of the common network plugins and of GKE, EKS and AKS:

clickhouse:
  allowedNetworks: ["127.0.0.1", "::1", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10", "fc00::/7"]

If your pods use other ranges, such as 198.18.0.0/15 or global IPv6, list those too.

While it's unset (the default, and on --reuse-values upgrades), the previous behavior stays: users accept connections from any address, limited only by the ClusterIP Services and the chart's NetworkPolicy. Nothing is locked out on install or upgrade. helm install and helm upgrade print a warning, and SIEM Health reports "ClickHouse users accept connections from any address".

To let a Tenzir or Cribl writer outside the cluster write events directly, add its networks to clickhouse.ingestAllowedNetworks. They are admitted as the ingest user only. With a LoadBalancer in front of ClickHouse, set externalTrafficPolicy: Local, or ClickHouse sees node addresses.

Amazon EKS

Skip this section on other clusters. On EKS, values-aws.yaml exposes nano through two ALBs from the AWS Load Balancer Controller's Gateway API support, with TLS from ACM. Prepare the cluster as follows before you install.

Cluster

A small HA cluster for about 12.5 GB/day: three Availability Zones, a NAT gateway per zone, OIDC for IAM roles for service accounts (IRSA), and two m6i.2xlarge nodes per zone on private subnets, one managed node group per zone. Change the name, region and zones:

# cluster.yaml: eksctl create cluster -f cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: nano
  region: <region>
  version: "1.37"
  tags: {nano-deployment: nano}      # on the CloudFormation stacks, VPC, subnets, NAT gateways and IAM roles
availabilityZones: ["<region>a", "<region>b", "<region>c"]
vpc: {cidr: 10.42.0.0/16, nat: {gateway: HighlyAvailable}}
iam: {withOIDC: true}
addons:
  - name: vpc-cni
    configurationValues: '{"enableNetworkPolicy": "true"}'   # enforce NetworkPolicy
  - name: coredns
  - name: kube-proxy
  - name: eks-pod-identity-agent
  - name: metrics-server
  - name: aws-ebs-csi-driver
    wellKnownPolicies: {ebsCSIController: true}
managedNodeGroups:
  - name: ng-a
    availabilityZones: ["<region>a"]
    instanceType: m6i.2xlarge
    minSize: 2
    desiredCapacity: 2
    maxSize: 2                       # six nodes in all; there is no cluster autoscaler
    tags: {nano-deployment: nano}    # on the instances and their volumes
    volumeSize: 100
    volumeType: gp3
    privateNetworking: true
  # ng-b and ng-c: the same, with availabilityZones ["<region>b"] and ["<region>c"]

The vpc-cni configuration turns on the VPC CNI's network policy agent, which needs VPC CNI v1.14 or newer; see Network policy enforcement. Set it here, at creation.

On an existing cluster, keep the add-on's IAM role when you change its configuration. eksctl creates the vpc-cni add-on with a service account role; an update-addon without --service-account-role-arn drops it, and the new aws-node pods then crash-loop on ec2:DescribeNetworkInterfaces while the add-on sits in UPDATING. A node replaced in that state gets no pod networking. Read the role first, pass it back, and wait for the rollout:

ROLE=$(aws eks describe-addon --cluster-name nano --addon-name vpc-cni \
  --query 'addon.serviceAccountRoleArn' --output text)
echo "$ROLE"    # "None" means the add-on uses the node role; then leave out --service-account-role-arn
aws eks update-addon --cluster-name nano --addon-name vpc-cni \
  --service-account-role-arn "$ROLE" --resolve-conflicts PRESERVE \
  --configuration-values '{"enableNetworkPolicy": "true"}'
aws eks wait addon-active --cluster-name nano --addon-name vpc-cni
kubectl -n kube-system rollout status daemonset/aws-node --timeout=10m
kubectl -n kube-system get pods -l k8s-app=aws-node    # every pod Running, no restarts

If rollout status stalls with an aws-node pod in CrashLoopBackOff, the role was lost: put it back with kubectl -n kube-system annotate sa aws-node eks.amazonaws.com/role-arn=<role> --overwrite, delete the failing pod, and rerun the update-addon above with --service-account-role-arn.

EBS volumes are bound to one Availability Zone, so a ClickHouse replica or Keeper member can only restart on a node in its own zone. With two nodes per zone, a drained or failed node's pods move to the other node in that zone. With one node per zone (three nodes), nano survives the loss of a node with its data safe, but that zone's ClickHouse replica and Keeper member stay Pending until a node returns to the zone, and you have no redundancy meanwhile. One node group spanning the three zones with desiredCapacity: 6 also works, but it doesn't guarantee two nodes in each zone.

Storage: EBS CSI driver and a default StorageClass

The aws-ebs-csi-driver add-on above runs with an IAM role (wellKnownPolicies: {ebsCSIController: true}). On an existing cluster, add it with eksctl create addon and the same policy. EKS has no default StorageClass, so every nano volume stays Pending until you create one:

# gp3.yaml: kubectl apply -f gp3.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  tagSpecification_1: "nano-deployment=nano"   # on every nano EBS volume
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

If you don't want a cluster default, set global.storageClass: gp3 and clickhouse.storage.storageClass: gp3 instead.

AWS Load Balancer Controller with Gateway API

You need controller v2.13 or newer, running with an IAM role (IRSA or Pod Identity), with its Gateway API support switched on. Pick the controller version first; these steps use v3.6.2, the version of our latest test:

  1. Create the controller's IAM policy from its release, and a service account with an IAM role that has it (IRSA, through the cluster's OIDC provider from iam: {withOIDC: true}):

    LBC_VERSION=v3.6.2
    ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
    curl -fsSLo lbc-iam-policy.json \
      "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/$LBC_VERSION/docs/install/iam_policy.json"
    aws iam create-policy --policy-name nano-AWSLoadBalancerControllerIAMPolicy \
      --policy-document file://lbc-iam-policy.json --tags Key=nano-deployment,Value=nano
    eksctl create iamserviceaccount --cluster nano --region <region> \
      --namespace kube-system --name aws-load-balancer-controller \
      --attach-policy-arn "arn:aws:iam::$ACCOUNT:policy/nano-AWSLoadBalancerControllerIAMPolicy" \
      --approve

    Use the iam_policy.json of the controller version you install: newer versions need permissions that older policies lack. When you delete the cluster, delete this policy after it (aws iam delete-policy).

  2. Install the Gateway API standard CRDs, for example v1.6.0:

    kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml
  3. Install the controller's own Gateway CRDs from the same controller release:

    kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/$LBC_VERSION/config/crd/gateway/gateway-crds.yaml"
  4. Install or upgrade the controller with the ALBGatewayAPI feature gate, using the service account from step 1. The chart's version differs from the controller's: helm search repo eks/aws-load-balancer-controller --versions lists each chart version with its controller version (APP VERSION); use the chart version on the row for yours:

    helm repo add eks https://aws.github.io/eks-charts && helm repo update eks
    helm search repo eks/aws-load-balancer-controller --versions | head
    helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
      --version <chart version whose APP VERSION is $LBC_VERSION> \
      --set clusterName=nano --set defaultTags.nano-deployment=nano \
      --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller \
      --set controllerConfig.featureGates.ALBGatewayAPI=true
  5. Create the GatewayClass that values-aws.yaml uses:

    apiVersion: gateway.networking.k8s.io/v1
    kind: GatewayClass
    metadata: {name: aws-alb}
    spec: {controllerName: gateway.k8s.aws/alb}

From chart 2.0.10, the chart renders a TargetGroupConfiguration for each Gateway backend, with ip targets and a /healthz health check, so the controller's default settings work. You don't need defaultTargetType on the controller. The related values:

ValueDefaultPurpose
gateway.aws.targetTypeipinstance also needs edge.service.type and ingest.service.type set to NodePort
gateway.aws.crdApiVersion"" (auto)From chart 2.0.11, the chart uses gateway.k8s.aws/v1 when the cluster serves it (controller v3.x) and v1beta1 otherwise (v2.13 to v2.16). Before 2.0.11 the default was v1beta1, which controller v3.x reports as deprecated on every install and upgrade; set gateway.k8s.aws/v1 there
gateway.appBackendTimeoutSec310The app ALB's idle timeout, so long searches finish

Restrict exposure

By default, both ALBs are internet-facing, and the controller opens ports 80 and 443 to 0.0.0.0/0. From chart 2.0.11, you can limit each ALB to the networks that need it, or make it internal:

ValueDefaultPurpose
gateway.aws.sourceRanges.app, gateway.aws.sourceRanges.ingest[] (0.0.0.0/0)CIDRs allowed to reach that ALB. The chart sets them as sourceRanges on the ALB's LoadBalancerConfiguration, which limits the inbound rules of its security group
gateway.aws.scheme.app, gateway.aws.scheme.ingest"" (tls.aws.scheme, internet-facing)internal makes that ALB reachable only inside the VPC and networks connected to it

For example, the UI only from your office and VPN, and an internal ingest ALB for senders in the VPC or over a VPN:

gateway:
  aws:
    sourceRanges:
      app: [203.0.113.0/24, 10.8.0.0/16]
    scheme:
      ingest: internal

An internal ALB has private addresses, so only senders that can route into the VPC reach it. Point the ingest name at it with a CNAME record as usual.

Certificate and DNS

values-aws.yaml uses tls.mode: cloud-managed, so you need an ACM certificate, in the cluster's region, that covers both the app host and the ingest host, or one certificate for each. Request it for names in a domain you control and validate it before you install. After the install, point both names at the ALBs with CNAME records (see Point DNS at the Gateways).

PostgreSQL on RDS

For production, use RDS for PostgreSQL 16 or newer, Multi-AZ, in the cluster's private subnets, open on port 5432 only to the nodes. With eksctl managed node groups, the nodes use the EKS cluster security group (eks-cluster-sg-nano-…), not the stack's ClusterSharedNodeSecurityGroup; allowing the wrong one leaves the migration waiting for PostgreSQL:

VPC=$(aws eks describe-cluster --name nano --query cluster.resourcesVpcConfig.vpcId --output text)
NODE_SG=$(aws eks describe-cluster --name nano --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
SUBNETS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=tag:kubernetes.io/role/internal-elb,Values=1 \
  --query 'Subnets[].SubnetId' --output text)    # eksctl's private subnets
DB_SG=$(aws ec2 create-security-group --group-name nano-pg --description "nano PostgreSQL" --vpc-id $VPC \
  --tag-specifications 'ResourceType=security-group,Tags=[{Key=nano-deployment,Value=nano}]' --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id $DB_SG --protocol tcp --port 5432 --source-group $NODE_SG
aws rds create-db-subnet-group --db-subnet-group-name nano-pg --db-subnet-group-description "nano" \
  --subnet-ids $SUBNETS --tags Key=nano-deployment,Value=nano
aws rds create-db-instance --db-instance-identifier nano-pg \
  --engine postgres --engine-version 17 --db-instance-class db.m7g.large \
  --allocated-storage 50 --storage-type gp3 --storage-encrypted --multi-az \
  --db-subnet-group-name nano-pg --vpc-security-group-ids $DB_SG --no-publicly-accessible \
  --master-username nanoadmin --manage-master-user-password \
  --backup-retention-period 7 --tags Key=nano-deployment,Value=nano
aws rds wait db-instance-available --db-instance-identifier nano-pg
aws rds describe-db-instances --db-instance-identifier nano-pg --query 'DBInstances[0].Endpoint.Address' --output text

--manage-master-user-password keeps the admin password in Secrets Manager, so it never appears on a command line. See Managed PostgreSQL requirements for the database, user, extensions and password secret, then add the postgresql block from Highly available values to my-values.yaml, with the endpoint as postgresql.external.host.

Google Kubernetes Engine (GKE)

Skip this section on other clusters. On GKE, values-gke.yaml exposes nano through two GKE Gateways of the class gke-l7-global-external-managed, one global external Application Load Balancer for the app and one for ingest, with Google-managed certificates from a Certificate Manager map. Prepare the project and the cluster as follows before you install.

Use a GKE Standard cluster. nano hasn't been tested on Autopilot, and we don't support it yet. The chart expects control over the nodes: from the standard preset up, an ingest node pool labeled and tainted nano.rs/pool=ingest, local SSDs for ClickHouse on large and xlarge, and a known number of nodes in each zone for the zonal ClickHouse and Keeper volumes.

Project, APIs and network

Set the project and region once. Every gcloud command in this section passes --project "$PROJECT":

export PROJECT=<project id> REGION=<region>
gcloud projects describe "$PROJECT" --format='value(projectId,name)'   # check it's the right project

Turn on the APIs that the commands below use: GKE (container), Compute Engine (compute: the VPC, quotas and static addresses), Cloud SQL Admin (sqladmin), Service Networking (servicenetworking: Cloud SQL's private IP) and Certificate Manager (certificatemanager). A new project often has some of them off; sqladmin in particular:

gcloud services enable container.googleapis.com compute.googleapis.com sqladmin.googleapis.com \
  servicenetworking.googleapis.com certificatemanager.googleapis.com --project "$PROJECT"

Create a VPC with one subnet in the region, or use your own (then change nano-vpc and nano-subnet below). GKE adds the Pod and Service ranges to the subnet itself:

gcloud compute networks create nano-vpc --project "$PROJECT" --subnet-mode=custom
gcloud compute networks subnets create nano-subnet --project "$PROJECT" --network=nano-vpc \
  --region="$REGION" --range=10.40.0.0/21 --enable-private-ip-google-access

Labels. Unlike tags on EKS and AKS, GCP labels reach only some of what nano creates. Label what you can with nano-deployment=nano, and find the rest by name:

ResourceHow to label it
GKE cluster, its node VMs and their boot disks--labels nano-deployment=nano on clusters create
nano's volumes (ClickHouse, Keeper, Vector)A StorageClass like standard-rwo with the parameter labels: nano-deployment=nano, set as global.storageClass and clickhouse.storage.storageClass
Certificate Manager DNS authorizations, certificates and map--labels nano-deployment=nano on each create
Cloud SQL instanceThe Cloud SQL Admin API (settings.userLabels), as for the user in Managed PostgreSQL requirements; gcloud sql instances create has no label option
Reserved static IP addressesThe Compute Engine API's setLabels; gcloud compute addresses create has no label option
Not labeled: the VPC, subnet and peering; the load balancers the Gateways create (forwarding rules, proxies, URL maps, backend services, health checks, NEGs); GKE's firewall rulesGCP or GKE creates these without labels. Their names start with gke- or gkegw, or contain the cluster or namespace name

So a label-based asset search doesn't find every nano resource on GKE; search by name as well.

Quota

A new project usually can't hold the HA layout. Check these quotas in the cluster's region before you create anything, and request increases first:

QuotaWhat counts against it
CPUS_ALL_REGIONS (project) and CPUS or N2_CPUS (region)Every node's vCPUs: 48 for six n2-standard-8 nodes
SSD_TOTAL_GB (region)pd-balanced and pd-ssd disks, including each node's boot disk (100 GB pd-balanced by default) and every nano volume. Two 1,500 GiB ClickHouse volumes and six boot disks need about 3,900 GB; the default is often 500 GB

describe has no --filter, so flatten the quotas and filter the output:

gcloud compute project-info describe --project "$PROJECT" --flatten=quotas \
  --format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' | grep -E '^CPUS_ALL_REGIONS,'
gcloud compute regions describe "$REGION" --project "$PROJECT" --flatten=quotas \
  --format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' | grep -E '^(CPUS|N2_CPUS|SSD_TOTAL_GB),'

The documented HA layout needs about 48 vCPUs and 3,900 GB of SSD quota; a new project often has 32 vCPUs and 500 GB. If you can't raise the vCPU quota, six n2-highmem-4 nodes (24 vCPUs, 32 GB each) ran the highly available values at about 12.5 GB/day in our tests; see GKE cluster.

If you can't raise SSD_TOTAL_GB far enough, keep 90 days on the hot disk (clickhouse.storage.size: 500Gi) and turn on the cold tier on Cloud Storage. With two replicas that is still 1,000 GB of pd-balanced, more than the default 500 GB on its own. Two things help: create the nodes with --disk-type pd-standard, which moves their boot disks (600 GB for six nodes) off the SSD quota, and request the increase early. For an evaluation only, clickhouse.storage.size: 150Gi fits the default quota; don't go live on it. With values-standard.yaml (plannedGBPerDay: 100) that disk holds about 4 days, under the chart's 7-day minimum, so a first install refuses it: set sizing.plannedGBPerDay to the volume you'll send, or sizing.allowUndersizedDisk: true (see Upgrades).

GKE cluster

A small HA cluster for about 12.5 GB/day: a regional cluster over three zones with two nodes in each zone (six nodes), the GKE Gateway API controller, Dataplane V2 and Workload Identity, in the VPC from Project, APIs and network. Change the name and zones if you need to:

clusters create adds the cluster to your kubeconfig and makes it the current context. To keep ~/.kube/config and its current context as they are, point KUBECONFIG at a separate file first, in every terminal you use for nano:

export KUBECONFIG=$PWD/nano-kubeconfig
gcloud container clusters create nano --project "$PROJECT" --region "$REGION" \
  --node-locations "$REGION-a,$REGION-b,$REGION-c" --num-nodes 2 \
  --machine-type n2-standard-8 \
  --network nano-vpc --subnetwork nano-subnet --enable-ip-alias \
  --gateway-api=standard \
  --enable-dataplane-v2 \
  --workload-pool="$PROJECT.svc.id.goog" \
  --release-channel regular --labels nano-deployment=nano
FlagWhy nano needs it
--region with --node-locations and --num-nodes 2A regional control plane, and two nodes in each zone. --num-nodes counts per zone. Persistent disks are zonal, so a ClickHouse replica or Keeper member can only restart on a node in its own zone, as on EKS
--gateway-api=standardInstalls the Gateway API CRDs and the GKE GatewayClasses. Without it there is no gke-l7-global-external-managed class, so the install fails or the Gateways never get an address. On an existing cluster, run gcloud container clusters update nano --project "$PROJECT" --region "$REGION" --gateway-api=standard. The class appears a few minutes after the cluster is ready (about 3 minutes in our test); wait for it before you install, as below
--enable-dataplane-v2Enforces the chart's NetworkPolicy. GKE Standard doesn't enforce NetworkPolicy by default. See Network policy enforcement
--workload-poolWorkload Identity, for the cold tier on Cloud Storage without keys
--enable-ip-aliasA VPC-native cluster, which Cloud SQL's private IP needs. It is the default for new clusters

Before you install, wait until the Gateway class exists (it can take a few minutes after the cluster is ready):

for i in $(seq 60); do
  kubectl get gatewayclass gke-l7-global-external-managed >/dev/null 2>&1 && { echo "GatewayClass ready"; break; }
  sleep 10
done

This waits up to 10 minutes and works in any shell, macOS included (macOS has no timeout command).

n2-standard-8 (8 vCPU, 32 GB) matches the HA sizing in Before you start. In our test at about 12.5 GB/day, six n2-highmem-4 nodes (4 vCPU, 32 GB each, 24 vCPUs in total) also ran the highly available values without failures; watch CPU once real data flows.

Private nodes. With --enable-private-nodes, the nodes have no public addresses:

  • Add Cloud NAT to the region's Cloud Router. The cluster must reach api.nano.rs for the license and the registry credential, the nano registry, and the public registries of the ClickHouse, Valkey and refresh images.
  • Make sure you can still reach the control plane. In our test, private nodes turned on authorized networks with an empty list, and kubectl timed out. Either add your network with --enable-master-authorized-networks --master-authorized-networks <your CIDR>, or turn on the DNS-based endpoint with --enable-dns-access and fetch credentials with gcloud container clusters get-credentials nano --project "$PROJECT" --region "$REGION" --dns-endpoint.

Storage. GKE's default StorageClass is standard-rwo: pd-balanced, zonal, WaitForFirstConsumer. It works for nano without changes; check it with kubectl get storageclass. The chart doesn't use regional persistent disks. For large and xlarge, add a node pool with local SSDs and a StorageClass for them, and set clickhouse.storage.storageClass.

Certificates and static IPs

values-gke.yaml uses tls.mode: cloud-managed: both Gateways take their certificates from a Certificate Manager map, set as tls.gke.certificateMap. Create it before you install, with a DNS authorization and a Google-managed certificate for each host name, then one map entry per host name:

  1. Create the DNS authorizations, then add the CNAME record each one shows in your DNS zone:

    gcloud certificate-manager dns-authorizations create nano-app --project "$PROJECT" \
      --domain=nano.example.com --labels nano-deployment=nano
    gcloud certificate-manager dns-authorizations create nano-ingest --project "$PROJECT" \
      --domain=ingest.nano.example.com --labels nano-deployment=nano
    gcloud certificate-manager dns-authorizations describe nano-app --project "$PROJECT" --format='value(dnsResourceRecord)'
    gcloud certificate-manager dns-authorizations describe nano-ingest --project "$PROJECT" --format='value(dnsResourceRecord)'
  2. Create the certificates, the map and its entries:

    gcloud certificate-manager certificates create nano-app --project "$PROJECT" \
      --domains=nano.example.com --dns-authorizations=nano-app --labels nano-deployment=nano
    gcloud certificate-manager certificates create nano-ingest --project "$PROJECT" \
      --domains=ingest.nano.example.com --dns-authorizations=nano-ingest --labels nano-deployment=nano
    gcloud certificate-manager maps create nano --project "$PROJECT" --labels nano-deployment=nano
    gcloud certificate-manager maps entries create nano-app --project "$PROJECT" --map=nano \
      --certificates=nano-app --hostname=nano.example.com
    gcloud certificate-manager maps entries create nano-ingest --project "$PROJECT" --map=nano \
      --certificates=nano-ingest --hostname=ingest.nano.example.com

    Each certificate is ready when gcloud certificate-manager certificates describe nano-app --project "$PROJECT" --format='value(managed.state)' prints ACTIVE. With DNS authorization, that doesn't wait for the load balancers.

  3. Reserve a global static IP address for each Gateway, so the addresses, and the DNS records that point at them, survive an uninstall and reinstall. Create an A record for each host name with its address:

    gcloud compute addresses create nano-app-ip nano-ingest-ip --project "$PROJECT" --global --ip-version=IPV4
    gcloud compute addresses list --project "$PROJECT" --global --filter='name~^nano-'

Then add the map and the addresses to my-values.yaml:

tls:
  gke:
    certificateMap: nano
gateway:
  addresses:
    app: [{type: NamedAddress, value: nano-app-ip}]
    ingest: [{type: NamedAddress, value: nano-ingest-ip}]

Without gateway.addresses, GKE assigns ephemeral addresses, which change if you reinstall. The chart also renders a HealthCheckPolicy for each backend and a GCPBackendPolicy with a 310-second timeout for the app (gateway.appBackendTimeoutSec), so long searches finish. The chart has no setting yet to restrict source addresses on GKE: both load balancers accept traffic from the internet.

PostgreSQL on Cloud SQL

For production, use Cloud SQL for PostgreSQL with regional availability, reached only over a private IP in the cluster's VPC. See Managed PostgreSQL requirements for the version, the user and the password secret. On top of those:

  1. Set up private services access on the VPC once: an allocated range and the peering with Service Networking. The first peering attempt in a project can fail with an authentication error while Google provisions the Service Networking agent; run it again.

    gcloud compute addresses create google-managed-services-nano --project "$PROJECT" --global \
      --purpose=VPC_PEERING --prefix-length=20 --network=nano-vpc
    gcloud services vpc-peerings connect --project "$PROJECT" --service=servicenetworking.googleapis.com \
      --ranges=google-managed-services-nano --network=nano-vpc
    gcloud services vpc-peerings list --project "$PROJECT" --network=nano-vpc   # lists servicenetworking-googleapis-com

    A /20 (4,096 addresses) leaves room for Cloud SQL in several regions and other Google-managed services. A /16 also works but reserves 65,536 addresses of your private range, which may later overlap networks you peer or connect over VPN.

  2. Create the instance with a private IP only and encrypted connections. --ssl-mode=ENCRYPTED_ONLY matches nano's sslmode: require; a mode that requires client certificates would refuse nano's connections. db-custom-2-8192 (2 vCPU, 8 GB) allows enough connections for the HA values. Put the primary and the standby in two of the cluster's node zones with --zone and --secondary-zone; without them, Cloud SQL picks the zones, and in our test it put the primary in a zone with no nodes:

    gcloud sql instances create nano-pg --project "$PROJECT" --database-version=POSTGRES_17 --edition=ENTERPRISE \
      --region="$REGION" --zone="$REGION-b" --secondary-zone="$REGION-c" \
      --tier=db-custom-2-8192 --availability-type=REGIONAL \
      --network="projects/$PROJECT/global/networks/nano-vpc" --no-assign-ip \
      --ssl-mode=ENCRYPTED_ONLY --storage-type=SSD --storage-size=50 --storage-auto-increase \
      --backup-start-time=03:00 --enable-point-in-time-recovery
  3. Create the user and the database with gcloud, as in Managed PostgreSQL requirements. A database created this way is owned by cloudsqlsuperuser, not by nanosiem. That works: the nano user is a member of cloudsqlsuperuser.

  4. Set postgresql.external.host to the private IP: gcloud sql instances describe nano-pg --project "$PROJECT" --format='value(ipAddresses[0].ipAddress)'.

During a failover. In our tests, a Cloud SQL regional failover stalled API requests and sign-ins for about 10 to 25 seconds; on 2.0.18, some API requests and one sign-in took longer than a 15-second client timeout. OTLP writes got HTTP 503 for about 16 seconds on 2.0.18, as on AKS from 2.0.12 (on 2.0.11 they stalled instead). HEC writes slowed by up to about 3 seconds and never failed. Service resumed without a manual step, and no accepted event was lost. Clients must retry: senders on 503 and timeouts, API clients and sign-in monitors on timeouts. An OTLP request that times out at the client may or may not have been stored, so a sender that retries after a timeout can at worst store an event twice.

When you delete the VPC later, gcloud services vpc-peerings delete can refuse while Google still lists the connection as in use. Deleting the peering from the VPC side works: gcloud compute networks peerings delete servicenetworking-googleapis-com --project "$PROJECT" --network=nano-vpc.

Cold tier on Cloud Storage

For long retention, turn on the cold tier with provider: gcs and Workload Identity on the ClickHouse service account. The cluster's --workload-pool above is what makes that work without keys.

Azure AKS with Application Gateway for Containers

Skip this section on other clusters. On AKS, values-azure.yaml exposes nano through Application Gateway for Containers (AGC), with GatewayClass azure-alb-external and certificates from cert-manager. Prepare the cluster as follows before you install. The Azure steps are in Microsoft's Application Gateway for Containers documentation; this section lists what nano needs from them.

AKS cluster

A small HA cluster for about 12.5 GB/day: three availability zones, Azure CNI overlay, the OIDC issuer and workload identity on, and two 8 vCPU, 32 GB nodes per zone (six nodes, for example Standard_D8ds_v5 or Standard_D8ds_v6). New subscriptions often have too little vCPU quota for six such nodes; check the regional and VM-family quotas, and raise them to 48 vCPUs or more before you start.

Create the cluster in a virtual network of your own, with a subnet for the nodes and one for Application Gateway for Containers. Without --vnet-subnet-id, AKS creates its own virtual network in its managed MC_ resource group, and the AGC subnet would then have to go into a network that AKS manages, which Azure advises against. Change the names, region and tag:

export RG=<resource group> LOC=<region> TAGS="nano-deployment=nano"
az network vnet create -g $RG -n nano-aks-vnet -l $LOC --address-prefixes 10.50.0.0/16 --tags $TAGS
az network vnet subnet create -g $RG --vnet-name nano-aks-vnet -n aks --address-prefixes 10.50.0.0/20
az network vnet subnet create -g $RG --vnet-name nano-aks-vnet -n agc --address-prefixes 10.50.16.0/24 \
  --delegations Microsoft.ServiceNetworking/trafficControllers
az aks create -g $RG -n nano -l $LOC --tier standard --tags $TAGS --nodepool-tags $TAGS \
  --zones 1 2 3 --node-count 6 --node-vm-size Standard_D8ds_v6 \
  --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium \
  --vnet-subnet-id "$(az network vnet subnet show -g $RG --vnet-name nano-aks-vnet -n aks --query id -o tsv)" \
  --enable-oidc-issuer --enable-workload-identity --no-ssh-key

--tier standard gives the control plane its uptime SLA; the free tier has none. The cluster's identity needs Network Contributor on a virtual network you bring; AKS grants it when your account may assign roles, otherwise give it before the create with a user-assigned identity (--assign-identity).

--network-dataplane cilium makes Cilium enforce NetworkPolicy; --network-policy azure or calico also works. Without one of them, AKS doesn't enforce it; see Network policy enforcement. On an existing cluster, az aks show -g <resource group> -n nano --query networkProfile.networkPolicy prints none or nothing when nothing enforces it.

AKS spreads one node pool's nodes over the zones on a best-effort basis; to guarantee two in each zone, create one node pool per zone. AKS ships a default StorageClass, default, on the Azure Disk CSI driver (like managed-csi). Azure Disk LRS volumes are zonal, so a ClickHouse replica or Keeper member can only restart in its own zone, as on EKS. Check that a default StorageClass exists with kubectl get storageclass.

Application Gateway for Containers

  1. Install the ALB controller with its Helm chart and workload identity, as in Microsoft's quickstart: a managed identity with a federated credential for the controller's service account in azure-alb-system, and the role assignments the quickstart lists, including AppGw for Containers Configuration Manager.
  2. Use the agc subnet from AKS cluster (delegated to Microsoft.ServiceNetworking/trafficControllers, in the cluster's own virtual network), and give the controller's identity Network Contributor on it.
  3. Choose how AGC is deployed, and tell the chart which one to use. The values go in my-values.yaml:
    • Managed by the controller. Create an ApplicationLoadBalancer resource that names the delegated subnet, in a namespace of your choice. The controller creates the AGC and a frontend per Gateway. Point both Gateways at it:

      gateway:
        annotations:
          alb.networking.azure.io/alb-namespace: alb-infra   # the ApplicationLoadBalancer's namespace
          alb.networking.azure.io/alb-name: nano-alb         # and name
    • Bring your own. Create the AGC, two frontends and the association with the delegated subnet yourself, for example with az network alb. Each frontend's host name is known before you install. Point the Gateways at the AGC and give each its frontend:

      gateway:
        annotations:
          alb.networking.azure.io/alb-id: /subscriptions/<subscription>/resourceGroups/<resource group>/providers/Microsoft.ServiceNetworking/trafficControllers/<AGC name>
        addresses:
          app: [{type: alb.networking.azure.io/alb-frontend, value: <app frontend name>}]
          ingest: [{type: alb.networking.azure.io/alb-frontend, value: <ingest frontend name>}]

Each frontend has a host name under alb.azure.com. Point your two names at the frontends with CNAME records (see Point DNS at the Gateways). For an evaluation without your own domain, use the two frontend host names themselves; see Certificates.

Tags. --tags on your commands doesn't reach what Azure creates for you: the volumes' Azure Disks, in the MC_ resource group, come untagged. Pass --tags to az network alb create and to each az network alb frontend create too. After the install, tag nano's disks:

MC_RG=$(az aks show -g $RG -n nano --query nodeResourceGroup -o tsv)
for id in $(az disk list -g $MC_RG --query "[?tags.\"kubernetes.io-created-for-pvc-namespace\"=='nano'].id" -o tsv); do
  az resource tag --is-incremental --ids "$id" --tags $TAGS -o none
done

To tag disks as they are created instead, create a StorageClass like default with the Azure Disk parameter tags: nano-deployment=nano, and set it as global.storageClass and clickhouse.storage.storageClass.

Certificates

AGC terminates TLS from Kubernetes Secrets, so values-azure.yaml uses tls.mode: cert-manager. Install cert-manager and a ClusterIssuer that can issue certificates for both host names, and set tls.certManager.issuerRef.name. cloud-managed isn't available on Azure; the chart refuses it.

To use certificates you already have, set tls.mode: existingSecret with tls.existingSecret.app and tls.existingSecret.ingest. This is the one case where you set tls.mode yourself: you replace the preset's TLS on purpose, so the caution in Write your values doesn't apply.

Without a domain of your own, for an evaluation, use the two AGC frontends' host names (az network alb frontend show ... --query fqdn -o tsv) as ingress.host and ingest.hostname, with a private certificate authority that cert-manager runs. Don't use Let's Encrypt for *.alb.azure.com names: they share one rate limit with every other AGC user. Browsers and senders must trust the CA certificate:

# private-ca.yaml: kubectl apply -f private-ca.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: selfsigned-bootstrap}
spec: {selfSigned: {}}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: nano-private-ca, namespace: cert-manager}
spec:
  isCA: true
  commonName: nano-private-ca
  secretName: nano-private-ca
  privateKey: {algorithm: ECDSA, size: 256}
  issuerRef: {name: selfsigned-bootstrap, kind: ClusterIssuer}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: nano-private-ca}
spec: {ca: {secretName: nano-private-ca}}

Set tls.certManager.issuerRef.name: nano-private-ca, and give clients the CA certificate: kubectl -n cert-manager get secret nano-private-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > nano-ca.crt.

PostgreSQL on Azure

For production, use Azure Database for PostgreSQL flexible server. It needs the three extensions on its azure.extensions allow-list, and the nano user needs schema public, before you install; see Managed PostgreSQL requirements for both, the private networking and the password secret.

Create the server with private access, zone-redundant HA and a SKU the region offers. Choose the SKU from az postgres flexible-server list-skus -l <region> -o table. A listed SKU can still be refused with SkuNotAvailable (in one test, Standard_D2ds_v5 was refused and Standard_D2ads_v5 worked); try the other family of the same size. On Azure CLI 2.88 and newer, zone-redundant HA is --zonal-resiliency Enabled with --zone and --standby-zone; older versions use --high-availability ZoneRedundant:

az postgres flexible-server create -g $RG -n nano-pg -l $LOC --version 17 \
  --tier GeneralPurpose --sku-name <SKU> --storage-size 128 \
  --zonal-resiliency Enabled --zone 1 --standby-zone 2 \
  --vnet <PostgreSQL VNet> --subnet <delegated subnet> --private-dns-zone <zone> \
  --admin-user nanoadmin --tags $TAGS

Don't pass --admin-password: on the command line it shows in the process list and your shell history. Without it, the CLI generates a password and prints it once in its output; save it to pg-admin-password.txt, readable only by you, for the bootstrap pod. Or create the server in the portal, or with Terraform reading the password from Key Vault. Zone-redundant HA isn't offered in every region or for every subscription; it may need a support request. Same-zone HA (--zonal-resiliency Enabled --allow-same-zone) survives a server failure but not the loss of the zone.

PostgreSQL in another region. Some subscriptions can't create a flexible server in some regions (Provisioning is restricted in this region). Ask Azure for access in the cluster's region if you can. To use another region, the server needs its own virtual network in that region, with a subnet delegated to Microsoft.DBforPostgreSQL/flexibleServers and an address range that doesn't overlap the cluster's. Then:

  1. Peer that network with the cluster's network in both directions (global VNet peering, az network vnet peering create --allow-vnet-access, once from each side).
  2. Create one private DNS zone, for example nano-pg.private.postgres.database.azure.com, and link it to both networks (az network private-dns link vnet create, once for each). Pass it to the server create with --private-dns-zone.

Every query then crosses regions, which adds latency and is billed. In one test (eastus to eastus2), the first migration took about 6 minutes, well within --timeout 30m.

Known limitations on AKS

  • The chart renders no backend timeout policy for AGC, so the app frontend uses AGC's default request timeout. Very long searches may be cut off; this hasn't been tested yet. On EKS and GKE, gateway.appBackendTimeoutSec (310) applies.
  • The chart renders no health-check policy for AGC. AGC uses the pods' readiness probes, which works.
  • The cold tier on Azure Blob storage needs account keys. Managed identity and workload identity aren't supported for it yet.

Managed PostgreSQL requirements

Production installs use a managed PostgreSQL (postgresql.mode: external). Before the migration Job (nano-migrate-r1) applies anything, nano checks the server version, that it can create tables in schema public (from 2.0.12) and that it can create every extension it needs. If something is missing, the Job fails with one message that names it and the fix for your service, and nothing is applied.

RequirementDetail
VersionPostgreSQL 16 or newer. nano is tested on 16, 17 and 18
Extensionspg_trgm, pgcrypto and uuid-ossp. nano creates them itself; they are trusted extensions, so the database owner can. No shared_preload_libraries change is needed
Database and userA database nanosiem owned by the user nanosiem, or set postgresql.database and postgresql.user
Schema publicThe nano user must be able to create tables in schema public of its database. PostgreSQL 15 and newer no longer let every user do that, and on Azure flexible server public belongs to azure_pg_admin, so the first migration fails with permission denied for schema public. Give it the schema as below
ConnectionsAt least 200 (about 20 for each nano pod with the highly available values)
TLSKeep sslmode: require or stricter
PasswordURL-safe, because it goes into the connection URL, and stored in a Kubernetes secret
Amazon RDSGoogle Cloud SQLAzure Database for PostgreSQL flexible server
Minimum version161616
Schema publicUsually nothing to do: the database owner has it. The bootstrap SQL below makes sureUsually nothing to do: a database made with gcloud sql databases create and a user made through Cloud SQL have it. If the check reports it, run GRANT ALL ON SCHEMA public TO nanosiem; as postgres, connected to nanosiemRequired. ALTER SCHEMA public OWNER TO nanosiem; as the admin user, connected to nanosiem (in the bootstrap SQL below)
Allow the extensionsNothing to do, unless your parameter group sets rds.allowed_extensions. Then add pg_trgm,pgcrypto,uuid-ossp to itNo database flags are needed. Create the nano user with gcloud sql users create: those users belong to cloudsqlsuperuser and can create the extensions. Roles created with CREATE ROLE in SQL can'tRequired. Add pg_trgm,pgcrypto,uuid-ossp to the azure.extensions server parameter. It applies without a restart
ParametersNone. The default rds.force_ssl=1 worksNone. On small tiers, raise the max_connections flagazure.extensions. The default require_secure_transport=ON works. Avoid the Burstable B1ms and B1s tiers, which allow 50 connections and have no HA
Private networkingIn the cluster's VPC, not publicly accessible, with a security group that allows 5432 from the nodes' security groupPrivate IP only, through private services access on the cluster's VPC (a VPC-native cluster), with no public IPPrivate access (VNet integration) with a delegated subnet in, or peered with, the AKS VNet and a private DNS zone linked to it, or a private endpoint
High availabilityMulti-AZ DB instance or Multi-AZ DB clusterRegional availability (--availability-type=REGIONAL)Zone-redundant HA. Not every region offers it, and the Burstable tier doesn't; there, use same-zone HA, which survives a server failure but not the loss of the zone

On Azure, read the current value first, then append the three extensions to it. If the current value is empty, leave out <existing>,:

az postgres flexible-server parameter show -g <resource group> --server-name <server> \
  --name azure.extensions --query value -o tsv
az postgres flexible-server parameter set -g <resource group> --server-name <server> \
  --name azure.extensions --value <existing>,pg_trgm,pgcrypto,uuid-ossp

On Cloud SQL, create the database with gcloud, and the user through the Cloud SQL Admin API. gcloud sql users create takes the password only as --password=..., which puts it in the process list and your shell history; it has no option to prompt. Write the password to pg-password.txt (readable only by you), then:

gcloud sql databases create nanosiem --instance=<instance> --project <project>
( umask 077
  printf 'Authorization: Bearer %s\n' "$(gcloud auth print-access-token)" > gcp-auth-header.txt
  python3 -c 'import json,sys; print(json.dumps({"name": "nanosiem", "password": open(sys.argv[1]).read().strip()}))' \
    pg-password.txt > pg-user.json )
curl -fsS -X POST -H @gcp-auth-header.txt -H 'Content-Type: application/json' --data-binary @pg-user.json \
  "https://sqladmin.googleapis.com/v1/projects/<project>/instances/<instance>/users"
curl -fsS -X PATCH -H @gcp-auth-header.txt -H 'Content-Type: application/json' \
  --data '{"settings": {"userLabels": {"nano-deployment": "nano"}}}' \
  "https://sqladmin.googleapis.com/v1/projects/<project>/instances/<instance>"    # optional: label the instance
rm -f gcp-auth-header.txt pg-user.json
gcloud sql users list --instance=<instance> --project <project>    # nanosiem, BUILT_IN

A user created through Cloud SQL (with gcloud or this API) is a member of cloudsqlsuperuser and can create the extensions.

Don't create the Cloud SQL user with CREATE ROLE in SQL: such roles can't create extensions, and the migration stops on pg_trgm.

On RDS and Azure, create the user and database as the admin user in psql. \password prompts for the password, so it doesn't end up in your shell history. On PostgreSQL 16 and newer, an admin who isn't a superuser, such as the admin user on RDS and Azure, must be a member of the new role before creating a database it owns:

CREATE ROLE nanosiem LOGIN;
\password nanosiem
GRANT nanosiem TO <admin user>;
CREATE DATABASE nanosiem OWNER nanosiem;
\connect nanosiem
ALTER SCHEMA public OWNER TO nanosiem;

The last two lines give the nano user schema public in its database. They're required on Azure flexible server, where public belongs to azure_pg_admin, and harmless on RDS, where the database owner already has it. GRANT ALL ON SCHEMA public TO nanosiem; works too if you'd rather keep the owner.

Reaching a private RDS or Azure server. The database has no public address, so psql on your workstation can't reach it. Run it from inside the cluster, in a one-off pod. The passwords travel in a temporary secret, never on a command line. Put the admin password in pg-admin-password.txt (on RDS with --manage-master-user-password, read it from Secrets Manager: aws secretsmanager get-secret-value --secret-id <arn> --query SecretString --output text | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"], end="")' > pg-admin-password.txt) and the new nano password in pg-password.txt, then:

kubectl create namespace nano --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nano create secret generic pg-bootstrap \
  --from-file=admin=./pg-admin-password.txt --from-file=nano=./pg-password.txt
kubectl -n nano run pg-bootstrap --rm -i --restart=Never --image=postgres:17-alpine --overrides='
{"spec": {"containers": [{"name": "pg-bootstrap", "image": "postgres:17-alpine", "stdin": true, "stdinOnce": true,
  "command": ["sh", "-c", "PGPASSWORD=$(cat /s/admin) exec psql -v ON_ERROR_STOP=1 \"host=<endpoint> user=<admin user> dbname=postgres sslmode=require\" -f -"],
  "volumeMounts": [{"name": "s", "mountPath": "/s", "readOnly": true}]}],
  "volumes": [{"name": "s", "secret": {"secretName": "pg-bootstrap"}}]}}' <<'SQL'
\set pw `cat /s/nano`
CREATE ROLE nanosiem LOGIN;
ALTER ROLE nanosiem PASSWORD :'pw';
GRANT nanosiem TO CURRENT_USER;
CREATE DATABASE nanosiem OWNER nanosiem;
\connect nanosiem
ALTER SCHEMA public OWNER TO nanosiem;
SQL
kubectl -n nano delete secret pg-bootstrap

It prints psql's output (CREATE ROLE through ALTER SCHEMA) and exits 0 within a few seconds. Keep "stdinOnce": true: the overrides replace the container kubectl run generates, and without it the pod never sees the end of the SQL, so the command prints nothing, waits about a minute and exits 1, although the SQL ran. If it fails, psql names the statement, for example role "nanosiem" already exists on a second run. kubectl warns that the session is recorded in the container logs, including credentials. Here the passwords are read from the mounted secret, not typed, so they aren't in the logs.

On RDS, <endpoint> is the instance endpoint and <admin user> the master user name. On Azure, <endpoint> is <server>.postgres.database.azure.com and <admin user> the admin you created the server with. The pod resolves the private DNS zone through the cluster's virtual network, so the zone must be linked to it.

After you create the namespace, store the password in a secret from a file that holds only the password, with no trailing newline:

kubectl -n nano create secret generic nano-postgres --from-file=password=./pg-password.txt

The nano-deploy skill checks the version, schema public and the extensions from inside the cluster before helm install.

Install

Pull the chart

Save the registry credential's password to registry-password.txt (readable only by you, for example after umask 077), log in to the registry with it, then pull and unpack the chart:

helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken \
  --password-stdin < ./registry-password.txt
helm pull oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano \
  --version "$NANO_VERSION" --untar

If you have portal access, 1 · Get the installer on the deployment's page shows these commands with your credential filled in.

The install runs from the unpacked chart, so you can log out now. helm registry login may keep the credential in your operating system's credential store, such as the macOS keychain, outside your working directory:

helm registry logout us-central1-docker.pkg.dev

The chart is now in ./nano, with its sizing and exposure presets beside values.yaml. It also holds the release's image lock, nano/images.enterprise.customer.lock, and nano/values-customer-images.yaml, which pins every nano image by digest. From chart 2.0.12 the ClickHouse operator image is pinned by digest too; 2.0.11 and earlier pull it by tag. You only need to regenerate that file if you change the lock:

bash nano/scripts/values-from-lock.sh nano/images.enterprise.customer.lock > nano/values-customer-images.yaml

Create the namespace and registry secret

kubectl create namespace nano

Build the registry secret from registry-password.txt through a Docker config file, so the password never appears on a command line, where other users can see it in the process list and it stays in your shell history:

( umask 077; python3 - registry-password.txt us-central1-docker.pkg.dev oauth2accesstoken \
    > dockerconfig.json <<'PY'
import base64, json, sys
cred, host, user = sys.argv[1:4]
password = open(cred, encoding="utf-8").read().rstrip("\n")
auth = base64.b64encode(f"{user}:{password}".encode()).decode()
print(json.dumps({"auths": {host: {"username": user, "password": password, "auth": auth}}}))
PY
)
kubectl -n nano create secret generic nano-registry \
  --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=dockerconfig.json
rm -f dockerconfig.json

Keep the type kubernetes.io/dockerconfigjson: the refresh CronJob updates the secret in place.

From chart 2.0.10, a CronJob refreshes this secret before the credential expires. See Registry access.

Write your values

Save your enrollment token to license-token.txt, then create my-values.yaml with only your site's values:

nano:
  baseUrl: https://nano.example.com
  deploymentId: <your deployment ID>
  license:
    url: https://api.nano.rs/api/license/status
global:
  imagePullSecrets: [nano-registry]
ingress:
  host: nano.example.com             # UI and API
ingest:
  hostname: ingest.nano.example.com  # senders

Leave tls.mode to the exposure preset. Add only the TLS keys that your preset asks for:

PresetClusterAdd to my-values.yaml
values-aws.yamlEKStls.aws.appCertificateArn and tls.aws.ingestCertificateArn (the same ARN if one certificate covers both hosts)
values-gke.yamlGKEtls.gke.certificateMap
values-azure.yamlAKStls.certManager.issuerRef.name, for example letsencrypt
values-sovereign.yamlAny other cluster (Envoy Gateway)tls.certManager.issuerRef.name, your ClusterIssuer

For example, on EKS:

tls:
  aws:
    appCertificateArn: arn:aws:acm:<region>:<account>:certificate/<id>
    ingestCertificateArn: arn:aws:acm:<region>:<account>:certificate/<id>

Also set clickhouse.allowedNetworks; see ClickHouse network allow-list. With external PostgreSQL, also add the postgresql block from Highly available values. For about 12.5 GB/day, save the 12.5 GB/day example as values-12gb.yaml.

Install

Layer the image values, one sizing preset, one exposure preset, then your own files. Later files win:

helm install nano ./nano -n nano --wait --timeout 30m \
  -f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
  -f values-12gb.yaml -f my-values.yaml \
  --set-file secrets.licenseToken=./license-token.txt

This example is for EKS. Use values-gke.yaml or values-azure.yaml on those clouds, and values-sovereign.yaml (Envoy Gateway) anywhere else. With --wait, Helm reports the release as deployed only once its pods are ready; without it, Helm reports success while pods may still be crash-looping. Keep a note of your -f files and their order: every upgrade needs the same list.

The first install runs the database migration before the app pods start. Against a remote managed PostgreSQL, for example in another region, it can take 10 minutes or more. Follow it with kubectl -n nano logs -f job/nano-migrate-r1. From chart 2.0.11, the Deployments that wait on it allow 30 minutes (migrations.dependentProgressDeadlineSeconds, 1800), so install with --wait --timeout 30m: it matches that deadline, and --wait returns as soon as every pod is ready (about 8 minutes in our 2.0.18 tests). A longer --timeout doesn't help; raise both together if you must. On 2.0.10, see helm install fails with "Progress deadline exceeded".

Point DNS at the Gateways

kubectl -n nano get gateway

Point nano.example.com at the app Gateway's address and ingest.nano.example.com at the ingest Gateway's address. On EKS and AKS, the addresses are host names, so use CNAME records. On GKE, they are IP addresses, so use A records; with reserved static IPs, you created them before the install.

Verify it works

kubectl -n nano get pods        # every pod Running or Completed, every container ready
helm test nano -n nano          # health of every service
curl -fsS https://ingest.nano.example.com/healthz   # 200 from chart 2.0.10

helm test checks that each service answers, not that every pod is ready, so check the pods as well: a pod with fewer ready containers than it has (for example Vector at 1/2) or a rising restart count needs a look with kubectl -n nano describe pod and kubectl -n nano logs. On chart 2.0.9, the ingest host answers /healthz with 404; that is expected.

Then open https://nano.example.com and create the first admin on the one-time setup page. A licensed install shows no license banner.

The setup page asks for the setup token, the setup_token key of nano-secrets (from 2.0.8). Write it to a file that only you can read, rather than printing it to your terminal, where it stays in scrollback and CI logs:

( umask 077; kubectl -n nano get secret nano-secrets -o jsonpath='{.data.setup_token}' | base64 -d > setup-token.txt )

To create the admin without a browser, send the same request the page does, with the token as a bearer token and the password read from a file rather than typed on the command line. curl reads the header from a file, so the token stays out of the process list:

( umask 077; printf 'Authorization: Bearer %s\n' "$(cat setup-token.txt)" > setup-header.txt )
python3 -c 'import json,sys; print(json.dumps({"email": sys.argv[1], "name": sys.argv[2], "password": open(sys.argv[3]).read().strip()}))' \
  admin@example.com "Admin" ./admin-password.txt | \
  curl -fsS -X POST https://nano.example.com/api/setup/initialize \
    -H @setup-header.txt -H 'Content-Type: application/json' --data-binary @-
rm -f setup-header.txt

Once the first admin exists, delete the token files: rm -f setup-token.txt setup-header.txt. The token isn't needed again.

The body is {"email", "name", "password"}; the token is needed only until the first admin exists.

Next steps

Prefer to have a coding agent run the install? The nano-deploy agent skill supports Helm.

Sizing presets

Pick one preset and layer your own values on top. The figures are a starting point; watch ClickHouse CPU and memory once real data flows.

PresetDaily volumeClickHouseVectorPostgreSQL
values-small.yamlUp to ~10 GB2–4 vCPU, 4–6 GiB, 50 Gi1 × 0.5–2 vCPUBundled
values-standard.yaml~50–100 GB4–8 vCPU, 32 GiB, 500 Gi2–4 × 2–4 vCPUBundled, external recommended
values-large.yaml~1 TB8–16 vCPU, 64 GiB, 4 Ti NVMe2–4 × 6–8 vCPU (pre-shaped: 2–3 × 2–4 vCPU)External
values-xlarge.yaml~10 TB2 shards, 28–32 vCPU, 256 GiB, 16 Ti NVMe each4–7 × 28–31 vCPU (pre-shaped: 2–4 × 4–8 vCPU)External

The Vector pools are sized for an average event of 1,000 bytes. From standard up, the ingest edge and Vector run on an ingest node pool labeled and tainted nano.rs/pool=ingest, and the Vector autoscaler needs metrics-server. large and xlarge need a local-NVMe StorageClass for ClickHouse (clickhouse.storage.storageClass).

Planned volume and ingest path

Each preset records its nominal volume in sizing.plannedGBPerDay. Two more values tune the Vector estimate:

ValueWhat it meansDefault
sizing.ingestPathraw for logs nano parses. preshaped for OCSF that arrives already shaped from Tenzir or Cribl: the chart then uses the preset's smaller vector.preshaped poolraw
sizing.avgEventBytesYour average event size, in bytes: about 300 for firewall and flow, 1000 for mixed feeds, 2000 for EDR1000

If sizing.plannedGBPerDay is more than the Vector pool can carry, helm install and helm upgrade print a warning. Firewall-heavy installs need a larger vector.hpa.maxReplicas.

Plan Vector from events per second. Parsing costs about the same per event, whatever its size:

events/s    = GB/day × 10^9 ÷ 86,400 ÷ sizing.avgEventBytes
Vector vCPU = 2 (peak hour) × events/s ÷ events per core

Raw logs are planned at 1,900 events per second per core. 16 benchmarked shipped OCSF parsers measured 1,950–6,500. Per TB/day of raw logs, that is about 41 vCPU at 300 bytes per event, 13 at 1,000 and 7 at 2,000. Pre-shaped OCSF is planned at 45,000 events per second per core, about 0.5 vCPU per TB/day. Vector gets at least 1 GiB of memory per pod, and 8 GiB or more for raw parsing at 1 TB/day and up.

The large-deployment ClickHouse settings ship only in the large and xlarge presets, so small and standard keep their values:

Settingvalues-large.yaml (~1 TB/day)values-xlarge.yaml (≥10 TB/day)
Insert flushAdaptive, 1–5 sAdaptive, 1–10 s
async_insert_max_data_size256 MB256 MB
parts_to_delay_insert / parts_to_throw_insert2,000 / 4,0005,000 / 10,000
merges_mutations_memory_usage_to_ram_ratio0.50.5
Vector sink concurrency612
Prevalence refresh (migrations.prevalenceRefreshMinutes)30 min60 min

Prevalence refresh. The rare and new lists, and the host counts in the prevalence drawer, are refreshed every 10 minutes by default. Refreshing less often saves CPU at volume. On large and xlarge, the lists can lag ingest by up to 30 or 60 minutes. Detections and searches still check a new artifact against the real-time counts, so it is still caught as rare. To set the interval yourself, set migrations.prevalenceRefreshMinutes (10–1440). 0 means the default of 10 minutes; moving back to a smaller preset resets it.

Direct OCSF writers. A Tenzir or Cribl writer next to the release can load its ClickHouse connection from the <release>-direct-ocsf ConfigMap with envFrom. It carries NANO_CH_SHARDS and NANO_CH_TABLE. With more than one shard, the table is the distributed one, so events spread across shards. The ingest user's password stays in the release secret. To admit a writer outside the cluster, see ClickHouse network allow-list.

Example: about 12.5 GB/day

Start from values-small.yaml and give ClickHouse room for enrichment, queries and a year of data:

# values-12gb.yaml
clickhouse:
  resources:
    requests: {cpu: "2", memory: 12Gi}
    limits: {cpu: "4", memory: 16Gi}
  storage:
    size: 1500Gi   # 365 days; use 500Gi for 90 days
  profiles:        # about 50% of the 0.8 x 16 GiB server cap per query
    default: {max_memory_usage: "6800000000", max_memory_usage_for_user: "9000000000", max_threads: "4"}
    rawsql: {max_memory_usage: "6800000000", max_threads: "4"}

See Retention and disk for the retention math.

Highly available values

To survive the loss of a node, add a second ClickHouse replica, three Keepers, two of each app service and an external, highly available PostgreSQL 16 or newer, such as Multi-AZ RDS, regional Cloud SQL or zone-redundant Azure flexible server (see Managed PostgreSQL requirements). Layer this after values-12gb.yaml:

# values-12gb-ha.yaml
clickhouse:
  replicas: 2          # two full copies of the data: about 2x ClickHouse cost
  antiAffinity: hard   # one ClickHouse pod per node
  zoneAware: true      # replicas in different zones; Keeper over 3 zones
  keeper:
    replicas: 3
api: {replicas: 2}
search: {replicas: 2}
jobs: {replicas: 2}
web: {replicas: 2}
edge: {replicas: 2}
connectorRunner: {replicas: 2}   # lease fencing: each feed runs on one runner at a time
vector:
  replicas: 2
ingest:
  edge:
    replicas: 2
    hecAcks: false     # HEC acks need a single ingest edge
postgresql:
  mode: external
  external:
    host: nano-pg.internal.example.com   # on EKS, the RDS endpoint
    sslmode: require
    passwordSecret: {name: nano-postgres, key: password}

Create the PostgreSQL password secret as in Managed PostgreSQL requirements, then install with both files:

kubectl -n nano create secret generic nano-postgres --from-file=password=./pg-password.txt
helm install nano ./nano -n nano --wait --timeout 30m \
  -f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
  -f values-12gb.yaml -f values-12gb-ha.yaml -f my-values.yaml \
  --set-file secrets.licenseToken=./license-token.txt

Plan at least two worker nodes in each of three zones: six nodes with 8 vCPU and 32 GB each. clickhouse.zoneAware: true puts the replicas of a shard in different zones and spreads Keeper over three zones (minDomains: 3), so set it only on a cluster with nodes in three zones. Three nodes, one per zone, also work: nano survives the loss of a node with its data safe, but that zone's ClickHouse replica and Keeper member stay Pending until the node returns. See Cluster on EKS, GKE cluster on GKE and AKS cluster on AKS.

Cloud block storage is zonal, so a ClickHouse replica or Keeper member can only restart in its own zone. On every cloud, give each zone that ClickHouse and Keeper use at least two nodes, and set clickhouse.zoneAware: true and antiAffinity: hard:

CloudVolumeNode layout
EKSEBS gp3, bound to one Availability ZoneOne managed node group per zone, at least 2 nodes each, so the cluster autoscaler scales the right zone
GKEpd-balanced, zonal. Regional persistent disks exist, but the chart doesn't use themOne node pool over three zones with --node-locations and --num-nodes 2 (2 per zone, 6 nodes), or one pool per zone. See GKE cluster
AKSAzure Disk LRS, zonalOne node pool with --zones 1 2 3 and a count of 6 (AKS spreads nodes on a best-effort basis), or one pool per zone to guarantee 2 in each. See AKS cluster

If any sender uses HEC acknowledgements, keep ingest.edge.replicas: 1 and hecAcks: true.

From chart 2.0.10, every component that runs more than one replica gets a PodDisruptionBudget with maxUnavailable: 1: api, search, jobs, web, edge, connector runner, ingest edge and Vector. A node drain then moves one pod at a time.

What to expect during a failure. Measured on one test install each, with these values and two nodes per zone:

EventEffect
A ClickHouse, Valkey or api pod deletedNo failed writes, sign-ins or API requests. After a ClickHouse pod delete on AKS, a few events written in the same second were stored twice (see below)
A node drained, more than an hour after installIts pods were running elsewhere within about a minute, with images pulled using the refreshed registry credential. No requests failed
RDS Multi-AZ forced failover (EKS)HEC writes never failed. API reads stalled for about 15 seconds. No sign-in failed
Cloud SQL regional failover (GKE)HEC writes never failed (up to about 3 seconds slower). API requests and sign-ins stalled for about 10 to 25 seconds; on 2.0.18 some exceeded a 15-second client timeout. OTLP writes got 503 for about 16 seconds on 2.0.18 (they stalled on 2.0.11). No accepted event was lost. An OTLP request that timed out at the client may or may not have been stored, so a retry can at worst store it twice
Azure flexible server forced failover (AKS), same-zone and zone-redundant HASign-ins and API writes failed for about 30 seconds, then resumed, and some API reads failed. No HEC write failed and no event was lost. OTLP writes failed for about 48 seconds while both api pods were unready: on 2.0.11 with HTTP 500 or a 15-second stall. From 2.0.12 they get 503 with Retry-After, which OpenTelemetry exporters retry, within 10 seconds

In every test, every accepted event was searchable afterwards, and no failover needed a manual repair. Senders and clients should retry on 429, 502, 503, 504 and timeouts, as with any HA layout; buffer OTLP in a collector so a failover doesn't drop batches.

Delivery is at least once. An event that nano accepted is never lost, but in rare cases it is stored twice: when a ClickHouse replica goes away while a batch is being written, the batch's acknowledgement can be lost, and the batch is written again. In one AKS test, deleting a ClickHouse pod stored 10 of 150 HEC events and 10 of 150 OTLP events twice, all written in the second after the delete. Counts and detections over that minute can be slightly high. Each copy has the same metadata.uid, so dc(metadata.uid) gives the true count. Nothing reports the duplicates, and nothing needs repairing.

Sign-ins are rate limited. nano allows 100 sign-in requests per client address in 5 minutes, and 10 per account in 10 minutes, successful ones included. Above either, it answers 429 with a Retry-After header (up to 300 seconds). A failover probe that signs in as one user more often than once a minute hits the per-account limit. Sign in once and reuse the session, or use an API key.

After a ClickHouse pod restarts, the nano-deploy skill's health check on 2.0.11 can report distributed DDL entries pending > 10 min for good. A cluster-wide schema statement issued while the replica restarted stayed Inactive on that replica, although the replica ran every later statement and nothing diverged. ClickHouse removes such an entry after 7 days (distributed_ddl.task_max_lifetime); don't delete it from Keeper by hand. From 2.0.12 the check only warns about it.

Upgrades

Pull the new release's chart, with a fresh registry login, then upgrade with the same -f files as the install, in the same order, plus --wait --timeout 20m. The new version comes from helm pull --version. This example matches the HA install above:

export NANO_VERSION=<new version>
helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken \
  --password-stdin < ./registry-password.txt
rm -rf nano && helm pull oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano \
  --version "$NANO_VERSION" --untar
helm registry logout us-central1-docker.pkg.dev
helm upgrade nano ./nano -n nano --wait --timeout 20m \
  -f nano/values-customer-images.yaml -f nano/values-small.yaml -f nano/values-aws.yaml \
  -f values-12gb.yaml -f values-12gb-ha.yaml -f my-values.yaml

The upgrade doesn't need --set-file secrets.licenseToken: the chart keeps every key of its existing secret, the license token included, when you leave it out. Pass it only to replace the token.

Don't use --reuse-values. It renders the new chart with the previous release's values, including the previous chart's defaults, and ignores the new chart's values.yaml. Settings a release adds or changes in its defaults never arrive, and Helm still reports the new chart version. With charts 2.0.12 and earlier it also kept the old images: an upgrade to 2.0.12 with --reuse-values or --reset-then-reuse-values ran 2.0.11. From 2.0.13 the chart pins its own images, so both flags get the new release's images, but --reuse-values still drops new defaults. Pass the same -f files as above. If you no longer have them, --reset-then-reuse-values -f nano/values-customer-images.yaml is safe: it takes the new chart's defaults and keeps the values you set before. Check with --dry-run=server that every nano image is the new version before you upgrade.

To run an image other than the release's on purpose, set images.<name>.tag (and digest). An explicit tag always wins; leave image.tag empty.

Ingest during an upgrade. An ingest-edge or Vector rollout drops nothing only if the load balancer stops sending to an old pod before it exits, and has registered and health-checked the new pod before the old one stops. From chart 2.0.15 the ingest.rollout values handle this:

ValueDefaultWhy
minReadySeconds45A new pod serves this long before an old one stops
edgePreStopSeconds35The ingest edge keeps serving this long after it is told to stop: on EKS, deregistration plus 3 × 10 s failed checks; on GKE, 2 × 5 s
vectorPreStopSeconds40Longer than the 30 s cluster DNS cache plus nginx's 5 s re-read
vectorMaxUnavailable0Vector rolls one extra pod at a time instead of stopping one

GKE: the chart's HealthCheckPolicy checks every 5 s with a 3 s timeout, healthy after 1 success and unhealthy after 2 failures (gateway.gke.healthCheck). EKS: the TargetGroupConfiguration checks /healthz every 10 s with a 5 s timeout, healthy after 2 and unhealthy after 3. Also label the namespace elbv2.k8s.aws/pod-readiness-gate-inject=enabled. With your own load balancer, set edgePreStopSeconds above its check interval × unhealthy threshold, and minReadySeconds above its interval × healthy threshold plus its registration time. On chart 2.0.13 an HA upgrade under traffic answered about 19 retryable 502/503s over 40 seconds; senders that retry lost nothing.

Always pass --wait --timeout 20m: migrations wait for ClickHouse, Helm's default of 5 minutes covers the whole upgrade, and --wait reports failure while the new pods are still not ready instead of printing success. Migrations run before any Deployment changes; if they fail, the previous version keeps serving. Migrations are forward-only, so roll forward instead of using helm rollback across a schema change. Helm never upgrades CRDs: when a release moves to a newer ClickHouse operator, first run kubectl apply --server-side -f nano/crds/clickhouse-operator-crds.yaml.

Upgrading from 2.0.10 or earlier to 2.0.11: a short OTLP gap. Vector now accepts OTLP only through the otlp-guard, which requires the ingest edge's hop token. The ingest edge and Vector roll out together, so until every ingest-edge pod is new, an old edge pod can reach a new Vector pod and get 401 on /otlp. OpenTelemetry exporters don't retry a 401, so those batches are lost. HTTP /ingest and HEC aren't affected. The gap closes when kubectl -n nano rollout status deployment/nano-ingest-edge completes. If OTLP loss matters, pause your OTLP senders for the upgrade. If you bring your own secret (secrets.existingSecret), add vector_otlp_hop_token to it first (see the Secrets list under Advanced), and set vector.native.enabled: true if in-cluster Vector forwarders send to port 6000.

Upgrading to 2.0.27 or later on a smaller ClickHouse disk than the plan. From 2.0.27, with sizing.plannedGBPerDay set (every sizing preset sets it), the chart compares clickhouse.storage.size × shards with the planned volume and the retention.

  • First install: under 7 days, it refuses to render.
  • helm upgrade of a running release: it goes ahead and prints a warning. For example, values-small.yaml (plannedGBPerDay: 13) with a 20Gi ClickHouse disk holds about 5 days, and still upgrades.
  • Renders that count as a first install: helm template without --is-upgrade, helm install --dry-run, and Argo CD (which renders with helm template) refuse it.

Before you upgrade, compare the disk with GB/day × retention days ÷ 5 × 1.15 ÷ 0.70 (see Retention and disk). Then do one of these:

  • Raise clickhouse.storage.size. The StorageClass must allow expansion. GKE standard-rwo and AKS managed-csi do. On EKS, set allowVolumeExpansion: true on your gp3 class; EBS allows one size change per volume every 6 hours.
  • Turn on the cold tier.
  • Set sizing.allowUndersizedDisk: true in the values your renders and GitOps tool use.

The check reads values, not the volume, so it behaves the same on every cloud.

Egress NetworkPolicy DNS changes in 2.0.28. See Egress NetworkPolicy.

Upgrade recovery

An upgrade runs two pre-upgrade hook Jobs before anything else changes:

  1. nano-ch-config-heal-r<revision> (from chart 2.0.10). If ClickHouse is still unhealthy after migrations.clickhouseConfigHeal.graceSeconds (60) and is down on its configuration (no replica answers /ping, or a ClickHouse pod is in CrashLoopBackOff or exited with code 36), it merge-patches only spec.settings of the ClickHouseCluster with the new release's values. A ClickHouse that crash-loops on its configuration gets the fix, and the migration can proceed. On a healthy cluster, or with one replica down for another reason (a node drain, a Pending pod), it does nothing.
  2. nano-migrate-r<revision>. It waits at most migrations.upgradeWaitSeconds (600) for every registered ClickHouse replica, then fails with a hint. A failed hook fails the upgrade, and the previous version keeps serving.

To recover, fix the value that broke ClickHouse and run the same helm upgrade --wait --timeout 20m again. If the first attempt timed out, run it again once the ClickHouse pods are Running.

Find the cause:

kubectl -n nano get pods -l app.kubernetes.io/name=clickhouse-server
kubectl -n nano logs <clickhouse pod> --previous | tail
kubectl -n nano logs job/nano-migrate-r<N>
kubectl -n nano logs job/nano-ch-config-heal-r<N>

Charts before 2.0.10, or with migrations.clickhouseConfigHeal.enabled: false. Patch the offending key on the ClickHouseCluster yourself, wait for the ClickHouse pods to be Running, then run helm upgrade again. For example:

kubectl -n nano patch clickhousecluster nano --type merge \
  -p '{"spec":{"settings":{"extraConfig":{"background_pool_size":"16"}}}}'

First install. The migration Job retries up to migrations.backoffLimit (4) times. If it still fails, app pods stay in Init:Error on wait-for-migrations. Fix the cause, then run helm upgrade with the same values: migrations run as the hook, and the app pods no longer wait on the install Job.

If you fixed the cause while the Job was still retrying and a later attempt succeeded, the failed attempts' pods stay behind in Error until the next helm upgrade replaces the Job. They don't affect nano. Delete them once the install is healthy:

kubectl -n nano delete pod -l job-name=nano-migrate-r1 --field-selector=status.phase=Failed

Server-side apply conflict. If a later helm upgrade reports a conflict with the field manager nano-ch-config-heal, run it again with --force-conflicts.

Known 2.0.9 issues fixed in 2.0.10

  • ClickHouse exits with code 36 on values-small.yaml. Its background_pool_size of 8 is rejected by ClickHouse 26.10. On 2.0.9, set clickhouse.server.background_pool_size: "16". From 2.0.10, the chart refuses to render a pool setting that would stop ClickHouse.
  • With clickhouse.replicas: 2, the migration waits for "database visible on 2/4 nodes". The ClickHouse operator's database sync makes default a Replicated database, which lists a second default cluster. 2.0.10 keeps database sync on and counts only the configured cluster. On 2.0.9 there is no values workaround; upgrade to 2.0.10. As a last resort on 2.0.9 only, an advanced step: patch the ClickHouseCluster with spec.settings.enableDatabaseSync: false, run DROP DATABASE default SYNC; CREATE DATABASE default ENGINE=Atomic on each replica, then run helm upgrade again. Don't do this on 2.0.10; it isn't needed there and isn't recommended.
  • Vector never becomes ready (Permission denied reading the config generations). 2.0.10 runs Vector as the config sidecar's user, uid 100.
  • AWS: "TargetGroup port is empty". 2.0.10 renders a TargetGroupConfiguration with ip targets. On 2.0.9, install the AWS Load Balancer Controller with --set defaultTargetType=ip.
  • The connector runner crash-loops with VRL sandbox self-test failed: VRL worker died: Operation not permitted where the container runtime gives pods a finite stack hard limit (seen on EKS AL2023 with containerd 2.3). The sandboxed VRL worker asked for a larger stack limit than the pod may set. No values workaround; 2.0.10 keeps the inherited limit, with the sandbox (seccomp included) still enforced. Pull feeds are unavailable until then.

Air-gapped clusters

  1. Load the signed image bundle into your internal registry.
  2. Set global.imageRegistry: registry.internal and clickhouse-operator.global.imageRegistry: registry.internal.
  3. On a connected host, helm pull the chart as in the install (without --untar), copy nano-<version>.tgz in, and install it with the same values.

With global.imageRegistry set, the registry credential refresh stays off. See Air-gapped deployment for offline content, licensing and AI.

Advanced

Registry access

nano images come from us-central1-docker.pkg.dev/nano-rs/nanosiem-customer with a pull-only credential that lasts one hour. Pods that start later, for example after a node replacement, can't pull images with an expired credential.

From chart 2.0.10, the chart refreshes it. The CronJob nano-registry-cred-refresh runs every 20 minutes. It requests a new credential from GET https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<id> with your enrollment token (the license_token key of the chart secret, set from secrets.licenseToken), and updates the pull secret in place.

  • registryCredentialRefresh.enabled: auto (the default) turns it on when nano.license.url, nano.deploymentId and global.imagePullSecrets are set and global.imageRegistry is empty.
  • It updates the first entry of global.imagePullSecrets, here nano-registry, or registryCredentialRefresh.secretName. The secret must already exist; create it as in the install.
  • Its own image, curlimages/curl, comes from Docker Hub, not from the registry it refreshes.
  • With secrets.existingSecret, your secret must hold license_token.

Check it, or refresh now:

kubectl -n nano get cronjob nano-registry-cred-refresh
kubectl -n nano create job --from=cronjob/nano-registry-cred-refresh refresh-now
kubectl -n nano logs job/refresh-now

On older charts, or to avoid the dependency on the license API, mirror the images into your own registry once and set global.imageRegistry. To request a credential by hand, for example for helm registry login:

( umask 077; printf 'Authorization: Bearer %s\n' "$(cat license-token.txt)" > license-header.txt )
curl -fsS -H @license-header.txt \
  "https://api.nano.rs/api/license/self-run/registry-token?deploymentId=<your deployment ID>"
rm -f license-header.txt

The response has username (oauth2accesstoken), password and expiresAt. For the same call on a single host, see Reruns, the registry and TLS. ClickHouse, Keeper, Vector, PostgreSQL, Valkey and nginx come from public registries and are digest-pinned in values.yaml.

What the chart deploys
ComponentKindNotes
api, search, jobs, webDeploymentsjobs uses leader election, so several replicas are safe
edgeDeployment (nginx)The app frontend (UI and API)
ingest edgeDeployment, autoscaler, PodDisruptionBudgetThe ingest frontend, with its own hostname and Gateway
VectorDeployment, autoscalerA sidecar delivers parsers, sources and ingest tokens
connector runnerDeploymentPull feeds. Lease fencing makes several replicas safe
ValkeyDeploymentCache and coordination, one replica
PostgreSQLStatefulSet, or externalBundled for small installs; managed for production
ClickHouse and KeeperClickHouseCluster, KeeperClusterThrough the bundled ClickHouse operator
Schema migrationsJobA pre-upgrade hook on upgrades, after the ClickHouse config heal hook
Registry credential refreshCronJobOnline-licensed installs without a mirror

Components with more than one replica also get a PodDisruptionBudget. The chart installs the ClickHouse operator with its admission webhooks off, so it doesn't need cert-manager. If the operator is already installed in the cluster, set clickhouse-operator.enabled=false and pass --skip-crds. Run one operator per cluster.

Values you will set
ValuePurpose
nano.baseUrlPublic URL. An http:// URL turns on dev mode (cookies without Secure)
nano.deploymentId, nano.license.url, secrets.licenseTokenOnline license. Leave nano.license.url empty for the signed offline license
nano.schemaProfileocsf (default). Never change it on an existing install
nano.retention.hotDays / warmDaysRetention policy
nano.extraEnvExtra environment for api, search and jobs
secrets.existingSecretBring your own secret (External Secrets, Vault, sealed-secrets)
global.storageClassStorageClass for nano's volumes; empty uses the cluster default
expose.mode, gateway.*, tls.*Gateway API exposure and TLS (set by the exposure preset)
postgresql.modebundled or external
clickhouse.shards, resources, storageSet by the preset. Plan the shard count up front
clickhouse.replicasHA switch, 1 by default
clickhouse.zoneAwareSpread replicas and Keepers across zones
clickhouse.coldStorage.*Object-storage cold tier
clickhouse.allowedNetworks, clickhouse.ingestAllowedNetworksClickHouse network allow-list
sizing.plannedGBPerDay, sizing.ingestPath, sizing.avgEventBytesPlanned volume and ingest path
migrations.prevalenceRefreshMinutesPrevalence refresh interval. Set by the large and xlarge presets
ingress.hostApp hostname
ingest.hostname, ingest.service.*, ingest.edge.*Ingest frontend
ingest.nodeSelector, tolerations, affinityIngest node pool
registryCredentialRefresh.*Registry credential refresh
migrations.upgradeWaitSeconds, migrations.clickhouseConfigHeal.*Upgrade recovery

ClickHouse needs at least 6 GiB of memory with enrichment on. Per-query memory (clickhouse.profiles.default.max_memory_usage) should be about 50% of the server cap, which is 80% of the pod memory limit; change them together. Replicas add availability, not capacity, and cost about twice the ClickHouse resources. You can add replicas or shards later with helm upgrade, but never shrink a running Keeper ensemble.

On arm64 node pools that are tainted (GKE: kubernetes.io/arch=arm64:NoSchedule), add the toleration with global.tolerations and clickhouse-operator.manager.tolerations.

Secrets

Unless secrets.existingSecret is set, the chart generates nano-secrets (for the release name nano; with another release name, the prefix changes). It keeps it across upgrades and on helm uninstall. If unsure, run kubectl -n nano get secret | grep secrets. Back it up. Losing encryption_key makes every stored credential unrecoverable.

A secret you bring yourself (secrets.existingSecret) must hold the keys the chart would generate. Keep passwords URL-safe.

KeyRequired
encryption_keyYes. 64 hex characters
jwt_secretYes
postgres_passwordWith the bundled PostgreSQL. External PostgreSQL reads postgresql.external.passwordSecret instead
clickhouse_default_password, clickhouse_password, clickhouse_admin_password, clickhouse_ingest_password, clickhouse_vector_password, clickhouse_rawsql_password, clickhouse_rawsql_noaudit_passwordYes
vector_auth_token, otlp_edge_key, platform_ingest_tokenYes
vector_config_delivery_tokenYes (falls back to vector_auth_token before v2.1)
vector_tap_tokenWith vector.tap.enabled
vector_otlp_hop_tokenYes, from 2.0.11. 32 or more characters from A-Z a-z 0-9 _ -. The ingest edge presents it to Vector's otlp-guard; without it, or in another format, the install or upgrade stops with instructions
setup_tokenTo create the first admin on the setup page (from 2.0.8). Without it, setup works only from inside the api pod (loopback)
license_tokenWith an online license. The registry credential refresh needs it too
docs_search_tokenOptional. pivt's grounding in the nano documentation
health_export_tokenOptional. Only with nano.healthExporter.endpoint
The ingest frontend

Ingest never shares a frontend with the app, so a sender flood can't take the UI down. It has its own hostname, Gateway or LoadBalancer Service, edge pods and node pool. The ingest hostname serves:

ProtocolEndpoint
HTTPhttps://<ingest-host>/ingest
HEChttps://<ingest-host>/services/collector/event (and /services/collector/raw); see HEC
OTLP/HTTPhttps://<ingest-host>/otlp/v1/{logs,traces,metrics}, binary protobuf only (Content-Type: application/x-protobuf, the OpenTelemetry SDK and Collector default). OTLP/JSON gets 415; see OTLP

The app hostname answers 404 on these paths. Issue a token per source under Ingestion → Log Sources → Credentials.

nano's HEC endpoint has indexer acknowledgements on, so every request needs a channel: an X-Splunk-Request-Channel: <uuid> header or a ?channel=<uuid> parameter. Without one, it answers 400 with Data channel is missing. Send every event and every /services/collector/ack poll of a channel with the same channel. ingest.edge.hecAcks: false only lets the ingest edge scale out; it doesn't turn channels off. When Vector scales, a moved channel's pending acks are lost and senders re-send.

From chart 2.0.10, the ingest hostname also answers GET /healthz with 200.

Cold tier

The cold tier is optional and off by default (clickhouse.coldStorage.enabled: false). With it on, ClickHouse moves older parts to an object-storage bucket when the local disk passes about 70% full (move_factor 0.3). Moved parts stay searchable through a local cache (cacheSize, 20Gi by default). Only the table's retention (DELETE TTL) removes data. For 365-day retention on a cloud cluster, we recommend turning it on and sizing the hot disk for 30 to 90 days.

clickhouse:
  coldStorage:
    enabled: true
    provider: s3            # s3 | gcs | azure | s3-compatible
    bucket: acme-nano-cold
    region: us-east-1       # the cluster's region
    prefix: ""
    useEnvironmentCredentials: true   # or credentialsSecret: nano-cold
    cacheSize: 200Gi
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/nano-clickhouse-cold

The ClickHouse service account is nano-clickhouse for the release name nano. To check it, run helm template with your values and look for kind: ServiceAccount.

ProviderCredentials
S3 on EKSuseEnvironmentCredentials: true with IRSA or EKS Pod Identity on the ClickHouse service account (clickhouse.serviceAccount.annotations). Limit the IAM policy to s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket and s3:GetBucketLocation on that bucket and prefix
GCS on GKEuseEnvironmentCredentials: true with Workload Identity: annotate iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com, grant the Google service account roles/storage.objectAdmin on the bucket, and bind roles/iam.workloadIdentityUser for <namespace>/nano-clickhouse. Or HMAC keys in credentialsSecret
Azure Blob on AKSAccount keys only: account_name and account_key in credentialsSecret, plus azure.storageAccountUrl and azure.containerName. The container must exist. Managed identity and Workload Identity aren't supported for the cold tier yet; the chart refuses useEnvironmentCredentials with azure
Wasabi, RustFS, Civo Object Storeprovider: s3-compatible plus endpoint, with keys in credentialsSecret (access_key_id, secret_access_key)

Cost. Approximate list prices per GB-month (us-east-1, us-central1; check your region's prices):

CloudBlock diskObject storage
AWSEBS gp3, about $0.08S3 Standard, about $0.023
Google Cloudpd-balanced, about $0.10GCS Standard, about $0.020
AzureManaged disks, about $0.075 (Standard SSD) to $0.12 or more (Premium)Blob Hot, about $0.018 to $0.02

For example, 12.5 GB/day compresses to about 2.5 GB/day on disk, or about 900 GB for 365 days per replica. On gp3 that is about $72 a month per replica, plus headroom. With the cold tier, it is about $21 a month in S3 plus a hot disk for 90 days (about 225 GB of gp3, about $18). With clickhouse.replicas: 2, each replica writes its own copy to the bucket, so double the object storage as well as the disks.

Requests and egress cost little for a SIEM's access pattern, but a bucket in another region adds egress charges: keep it in the cluster's region. Use a standard storage class. Infrequent-access and archive classes (S3 Standard-IA, Glacier) charge per GB read and have minimum storage durations, and ClickHouse reads this tier.

To enable it on an existing install, run this once per large table after the upgrade (add ON CLUSTER 'default' on clusters):

ALTER TABLE nanosiem.ocsf_logs MODIFY SETTING storage_policy = 'tiered'
Backups and uninstall
  • PostgreSQL: use your managed database's backups. In bundled mode, run kubectl -n nano exec nano-postgres-0 -- pg_dump -U nanosiem nanosiem > nano-pg.sql on a schedule.
  • ClickHouse: BACKUP DATABASE nanosiem TO S3(...) per shard, or volume snapshots. With a cold tier, cold parts are already in your bucket.
  • The chart secret (nano-secrets): back it up out of band, encrypted.

helm uninstall nano -n nano removes the release, including the ClickHouseCluster and KeeperCluster; you don't need to delete those yourself. It keeps the PVCs, the chart secret and any hook Jobs that are still there:

  • From chart 2.0.11, the config heal (nano-ch-config-heal-r<N>) and schema sync (nano-ch-schema-sync-r<N>) hook Jobs delete themselves when they succeed. The migration hook Jobs (nano-migrate-r<N>) stay so you can read their logs, and Kubernetes deletes them a day after they finish. Failed hook Jobs also stay for a day.
  • On chart 2.0.10, every hook Job stays after it finishes, and Kubernetes deletes it a day later.

To delete everything, including all stored events, after the uninstall:

kubectl -n nano delete jobs --all
kubectl -n nano delete pvc --all && kubectl -n nano delete secret nano-secrets

Or delete the namespace, kubectl delete namespace nano, which removes all of it. With the default Delete reclaim policy, the cloud disks go with the PVCs. helm uninstall also keeps the vector-source-credentials secret (the ingest credentials Vector uses); the namespace delete removes it, or delete it with kubectl -n nano delete secret vector-source-credentials.

Tearing down the cluster too. Remove nano's load balancers while the controller that owns them is still running, then the controller, then the cluster. On EKS:

  1. helm uninstall nano -n nano deletes the Gateways. Wait until kubectl -n nano get gateway lists none and the ALBs are gone (aws elbv2 describe-load-balancers no longer lists them).

  2. kubectl delete gatewayclass aws-alb while the AWS Load Balancer Controller still runs. Deleted after the controller, it hangs on the gateway.k8s.aws/gatewayclass finalizer.

  3. Delete the namespace, then uninstall the controller (helm uninstall aws-load-balancer-controller -n kube-system).

  4. Delete the RDS instance, its DB subnet group and its security group, in that order, waiting for each. They are in the cluster's VPC, and eksctl can't delete a VPC that still holds them:

    aws rds delete-db-instance --db-instance-identifier <instance> --skip-final-snapshot   # or --final-db-snapshot-identifier <name>
    aws rds wait db-instance-deleted --db-instance-identifier <instance>
    aws rds delete-db-subnet-group --db-subnet-group-name <subnet group>
    aws ec2 delete-security-group --group-id <database security group id>

    If the instance has deletion protection, turn it off first with aws rds modify-db-instance --db-instance-identifier <instance> --no-deletion-protection --apply-immediately. The instance takes 5 to 15 minutes to delete. If delete-security-group reports DependencyViolation, the instance's network interface is still being released: wait a minute and run it again.

  5. eksctl delete cluster -f cluster.yaml --wait, then the controller's IAM policy.

If eksctl delete cluster fails with The vpc ... has dependencies and cannot be deleted, the cluster is already gone, but the VPC stack (eksctl-<cluster>-cluster in CloudFormation) is in DELETE_FAILED, and its NAT gateways keep billing. Delete whatever is left in the VPC, such as the RDS subnet group and security group, then run aws cloudformation delete-stack --stack-name eksctl-<cluster>-cluster and aws cloudformation wait stack-delete-complete --stack-name eksctl-<cluster>-cluster. Running eksctl delete cluster again only says that no cluster was found.

On GKE, uninstall nano and wait for the Gateways to go before you delete the cluster, so the load balancers and their network endpoint groups are removed with them.

Troubleshoot

Need help? Contact support with your deployment ID and the time, with timezone.

Pods stay in ImagePullBackOff at install

The credential in nano-registry was already expired or mistyped when you created the secret. Get a new one and recreate the secret, or mirror the images. See Registry access.

Pods go to ImagePullBackOff after an hour

The pull credential lasts one hour, so pods scheduled later can't pull. From chart 2.0.10, check that kubectl -n nano get cronjob nano-registry-cred-refresh exists and that its last Job succeeded; run one now with kubectl -n nano create job --from=cronjob/nano-registry-cred-refresh refresh-now. If the CronJob is missing, set nano.license.url, nano.deploymentId and global.imagePullSecrets, and leave global.imageRegistry empty. On older charts, refresh the secret by hand or mirror the images. See Registry access.

helm pull fails with 401 or 403

The registry credential expired, or the chart path is wrong. Use oci://us-central1-docker.pkg.dev/nano-rs/nanosiem-customer/charts/nano; the customer credential can't read other paths. Generate a new credential, run helm registry login again, then pull.

ClickHouse or Keeper pod Pending after a drain

kubectl -n nano describe pod shows volume node affinity conflict. Its disk (EBS, persistent disk or Azure Disk) is in one zone, and no schedulable node is left in that zone. Uncordon the node or add one in that zone; plan two nodes per zone. See Cluster, GKE cluster or AKS cluster.

Volumes stay Pending

The cluster has no default StorageClass. Create one, for example the gp3 StorageClass on EKS, or set global.storageClass.

GKE Gateways get no address

kubectl get gatewayclass doesn't list gke-l7-global-external-managed: the cluster was created without the Gateway API. Run gcloud container clusters update nano --project "$PROJECT" --region "$REGION" --gateway-api=standard, wait for the class to appear, then run the same helm upgrade --install again. If the class exists, kubectl -n nano describe gateway names the problem, for example a certificate map that doesn't exist or a reserved address name that doesn't match gateway.addresses. See Google Kubernetes Engine (GKE).

The ALB shows "TargetGroup port is empty"

The AWS Load Balancer Controller created instance targets for ClusterIP Services. Upgrade to chart 2.0.10, which sets ip targets for every Gateway backend. On 2.0.9, reinstall the controller with --set defaultTargetType=ip.

ClickHouse exits with code 36

ClickHouse rejected a setting at startup; its log names it. On 2.0.9 with values-small.yaml, it is background_pool_size: set clickhouse.server.background_pool_size: "16". Then run helm upgrade --wait --timeout 20m again. See Upgrade recovery.

The migration waits for "database visible on N/M nodes"

On 2.0.9 with clickhouse.replicas: 2, the migrator counts nodes from a second default cluster that the ClickHouse operator's database sync adds, so it waits for nodes that don't exist. Upgrade to 2.0.10; see Known 2.0.9 issues for a last-resort step on 2.0.9. If N is lower than the number of ClickHouse pods you run, check those pods instead. See Upgrade recovery.

The license shows as invalid

Check that nano.deploymentId matches app.nano.rs exactly and that the cluster reaches https://api.nano.rs. If the token was lost or exposed, an owner or admin can select Regenerate license token under 2 · License on the deployment's page. The old token stops working at once. Save the new one to license-token.txt, run helm upgrade with the same values and --set-file secrets.licenseToken=./license-token.txt, then restart the api and jobs pods so they read it.

The chart refuses to render

The message names the value. Common causes: nano.license.url without nano.deploymentId, an ingest hostname equal to the app hostname, ingest.edge.replicas above 1 with hecAcks: true, or an empty ACM ARN with values-aws.yaml. A message that clickhouse.storage.size "holds only about N days" is the disk check: see Upgrades.

Pods wait on the migration Job

nano-migrate-r1 waits until every ClickHouse replica and PostgreSQL answer. Follow it with kubectl -n nano logs -f job/nano-migrate-r1. If it fails, see Upgrade recovery.

helm install fails with "Progress deadline exceeded"

On chart 2.0.10 and earlier, helm install --wait can fail after about 10 minutes with Error: resource Deployment/nano/nano-api not ready. status: Failed, message: Progress deadline exceeded, although the pods become ready a few minutes later. The first migration took longer than the Deployments' 10-minute progress deadline (progressDeadlineSeconds, 600), for example against a managed PostgreSQL in another region, or after retries on a missing extension. Nothing is lost and nothing needs cleaning up.

  1. Wait for the migration to finish: kubectl -n nano logs -f job/nano-migrate-r1, then kubectl -n nano get pods.
  2. Once the Job has completed, run the install again as helm upgrade --install nano ./nano -n nano --wait --timeout 30m with the same -f files, in the same order, and the same --set-file. It completes the release. A second helm install refuses the name, because the failed release still exists.

On 2.0.10, a longer --timeout doesn't help: the deadline is set on each Deployment, and Helm 4 (whose --wait reads the Deployment's progress condition) fails as soon as one passes it. From chart 2.0.11, those Deployments allow migrations.dependentProgressDeadlineSeconds (1800, 30 minutes). Install with --wait --timeout 30m, which matches it, or raise both together.

The migration fails with "transaction_timeout"

unrecognized configuration parameter "transaction_timeout" means nano 2.0.10 or earlier on PostgreSQL 16. Upgrade nano to v2.0.11 or later, or use PostgreSQL 17 or newer. See Managed PostgreSQL requirements.

The migration fails with "permission denied for schema public"

The nano user can't create tables in schema public. On Azure flexible server, public belongs to azure_pg_admin in every new database. As the admin user, connected to the nanosiem database (for a private server, from the one-off pod), run ALTER SCHEMA public OWNER TO nanosiem;, or GRANT ALL ON SCHEMA public TO nanosiem;. Then run the same helm upgrade --install again; if the Job is still retrying, its next attempt succeeds. The failed migration applied nothing. From 2.0.12, nano checks this before the migration and names the fix. On 2.0.11 the check passed and the migration failed.

The migration fails with "not allow-listed" or a missing extension

Azure flexible server refuses extensions that aren't in its azure.extensions server parameter. Append pg_trgm,pgcrypto,uuid-ossp to it as in Managed PostgreSQL requirements, then run the same helm upgrade --install again. The failed migration applied nothing, so the database needs no cleanup. On Cloud SQL, permission denied to create extension means the nano user was created with CREATE ROLE in SQL; create it with gcloud sql users create instead.

On this page

Before you startNetwork policy enforcementVector-native client certificatesEgress NetworkPolicy (optional)ClickHouse network allow-listAmazon EKSClusterStorage: EBS CSI driver and a default StorageClassAWS Load Balancer Controller with Gateway APIRestrict exposureCertificate and DNSPostgreSQL on RDSGoogle Kubernetes Engine (GKE)Project, APIs and networkQuotaGKE clusterCertificates and static IPsPostgreSQL on Cloud SQLCold tier on Cloud StorageAzure AKS with Application Gateway for ContainersAKS clusterApplication Gateway for ContainersCertificatesPostgreSQL on AzureKnown limitations on AKSManaged PostgreSQL requirementsInstallPull the chartCreate the namespace and registry secretWrite your valuesInstallPoint DNS at the GatewaysVerify it worksNext stepsSizing presetsPlanned volume and ingest pathExample: about 12.5 GB/dayHighly available valuesUpgradesUpgrade recoveryKnown 2.0.9 issues fixed in 2.0.10Air-gapped clustersAdvancedTroubleshootPods stay in ImagePullBackOff at installPods go to ImagePullBackOff after an hourhelm pull fails with 401 or 403ClickHouse or Keeper pod Pending after a drainVolumes stay PendingGKE Gateways get no addressThe ALB shows "TargetGroup port is empty"ClickHouse exits with code 36The migration waits for "database visible on N/M nodes"The license shows as invalidThe chart refuses to renderPods wait on the migration Jobhelm install fails with "Progress deadline exceeded"The migration fails with "transaction_timeout"The migration fails with "permission denied for schema public"The migration fails with "not allow-listed" or a missing extension