Commstate Docs

Environments & Clusters

Commstate runs on three independent k3s clusters. They share no control plane, no storage and no failure domain — the only thing that crosses between them is telemetry.

live uat dev
host ranger lander endurance
arch x86_64 x86_64 arm64
spec 8 cores / 15 GB (dedicated) 8 cores / 15 GB (dedicated) 14 cores / 31 GB (shared vCPU)
domains commstate.app uat.commstate.app autocom.wexron.io
k3s v1.36.3 v1.36.4 v1.31.5
fed by release/*, v1.2.3 main develop
overlay k8s/overlays/live k8s/overlays/uat k8s/overlays/default
CI runners 4 (capped) 6 4 (arm64, develop only)
also hosts nothing else nothing else GitLab, monitoring, other projects

ranger and lander are matched dedicated root servers with zero CPU steal. endurance carries GitLab, the monitoring stack and several other projects, so it runs at a high load average with noticeable steal — which is exactly why it is the cluster a change is allowed to be wrong on.

It is also the last arm64 machine, and therefore the only reason develop builds arm64 images at all. See Release Flow.

The dev cluster keeps the overlay directory name default because it is genuinely the overlay the other two layer on top of; uat and live are patches over it, not siblings of it.

Reaching them

All three clusters are reached over the tailnet, never over their public IPs — the Kubernetes API is not exposed to the internet on any of them.

kubectl config get-contexts
#   commstate-live   commstate-live   commstate-live   commstate
#   commstate-uat    commstate-uat    commstate-uat    commstate
#   commstate-dev    commstate-dev    commstate-dev    commstate

kubectl --context=commstate-live get pods
k9s --context=commstate-uat

Contexts, clusters and users all follow commstate-<env>, and all three default to the commstate namespace.

How CI reaches each cluster

The deploy jobs use three separate kubeconfigs, and they do not all take the same route:

Job Variable Address Runs on
deploy:live KUBE_CONFIG_LIVE 100.82.39.108 (tailnet) any amd64 runner
deploy:uat KUBE_CONFIG_UAT 100.119.71.25 (tailnet) any amd64 runner
deploy:dev KUBE_CONFIG_DEV 152.53.131.61 (public) endurance only

Job containers on lander and ranger can reach every tailnet address, so those two deploys run wherever there is a free slot.

endurance is the exception. Its API certificate carries the public IP but not its tailnet address, and that public IP is firewalled from both other hosts — so an amd64 runner can reach it by neither route. deploy:dev therefore carries tags: [arm64] and runs on endurance's own runner, which reaches its own API fine.

To normalise it, add the tailnet IP to tls-san on endurance and repoint KUBE_CONFIG_DEV:

# /etc/rancher/k3s/config.yaml
tls-san:
  - 100.100.50.50

then systemctl restart k3s and re-mint the kubeconfig. Until that happens, the arm64 tag on deploy:dev is load-bearing — removing it will schedule the job on a runner that cannot reach the cluster.

If a new cluster's API rejects your certificate

k3s only puts the node IP, localhost and the hostname in its API server certificate. Reaching it over a tailnet address fails TLS verification until that address is added:

# /etc/rancher/k3s/config.yaml
tls-san:
  - 100.82.39.108   # this node's tailnet address
  - ranger

Then systemctl restart k3s — the certificate is regenerated on start.

Overlays

k8s/overlays/
  default/    UAT — the base every other overlay builds on
  live/       production; layers on ../default
  rings/      module ring pinning (stable / canary / edge)

live deliberately layers on default rather than duplicating it, so the two environments cannot drift structurally. It overrides exactly three things:

  • hostnames — commstate.app instead of autocom.wexron.io
  • image tags — immutable v2.0.0 instead of :latest
  • TLS — cert-manager HTTP-01 at ingress-nginx

Everything else is the same manifest set.

Why live pins image tags and UAT does not

:latest cannot be rolled back. kubectl rollout undo re-pulls the same moving tag, and it cannot answer "what is running right now". UAT tolerates that in exchange for fast iteration; production does not.

NEXT_PUBLIC_API_URL is overridden explicitly

Next.js inlines NEXT_PUBLIC_* at build time. Without an explicit override in the live overlay, the production frontend loads perfectly and talks to the UAT API — a failure that surfaces only when data turns up somewhere unexpected. The value is substituted at container start, so overriding it is a config change rather than a rebuild.

Adding an environment

Adding dev is a values file and a namespace, not a server. Each environment costs roughly 2.5 GB of memory and 0.3 CPU cores measured against the running clusters, so a third fits on either host without new hardware.

  1. Create k8s/overlays/dev/kustomization.yaml layering on ../default, overriding hostnames and image tags.
  2. Create the namespace and its out-of-band secrets (below).
  3. Apply the CNPG cluster first and wait for it, then everything else.
  4. Add a commstate-dev kubeconfig context following the same naming.

Put non-production environments on endurance and keep ranger for production alone. The constraint is blast radius, not capacity: a dev mistake that fills a disk or exhausts memory should not sit one namespace away from live.

Secrets are deliberately not in Git

Five secrets are created out-of-band per environment:

secret contents
commstate-secrets APP_KEY, DB_PASSWORD, REDIS_PASSWORD, REVERB_*, AGENT_SHARED_SECRET
commstate-db-app CNPG application role
commstate-db-superuser CNPG superuser
passport-keys OAuth RSA keypair
gitlab-registry image pull credential

REDIS_PASSWORD must be empty. The Redis manifest has no --requirepass, so a non-empty value makes every client fail with ERR AUTH called without any password configured.

Back these up somewhere off the cluster. Several database columns are encrypted with APP_KEY — plugin credentials, module settings, reseller bank details. A dump restored against a different APP_KEY leaves those columns permanently unreadable, and the restore reports success while doing it.

Two things that bite on a fresh cluster

cert-manager cannot issue anything

The namespace runs default-deny-ingress, and cert-manager creates its HTTP-01 solver pods on the fly, so no app-scoped policy covers them. Challenges sit pending, reporting:

Waiting for HTTP-01 challenge propagation: wrong status code '502'

A 502 rather than a connection error is the tell — ingress-nginx routes to the solver correctly and the policy drops the packet, so the solver looks healthy throughout and cert-manager looks at fault. allow-acme-solver in k8s/base/per-ring/network-policies.yaml permits ingress-nginx to reach solver pods on 8089. This affects renewals as well as first issuance, so a cluster that issued certificates before the policy existed will fail silently 90 days later.

Image pushes fail with MANIFEST_BLOB_UNKNOWN

Docker 29 defaults to the containerd image store, which pushes OCI manifests that GitLab's registry (v4.38) rejects. On any host used as a build runner:

// /etc/docker/daemon.json
{ "features": { "containerd-snapshotter": false } }

Restart Docker afterwards. The storage driver should read overlay2, not overlayfs.