ClickStack

ClickStack is the all-in-one observability stack from the HyperDX team — a single Helm chart that bundles the HyperDX UI, an OpenTelemetry collector, and the glue that lands telemetry in ClickHouse. On this platform it provides both the OTLP ingestion endpoint that every app exports to and the search/trace UI that sits on top of the data.

Deployment

Field	Value
Flux path	flux-clusters/stefanzhelev/apps/clickstack
Base path	flux-apps/clickstack
Namespace	clickstack
Sync wave	5
Depends on	clickhouse, external-secrets-config

What it deploys

• HelmRelease for the ClickStack chart >= 1.0.0 from https://clickhouse.github.io/ClickStack-helm-charts
• HyperDX UI (bundled with the chart — search, trace view, dashboards)
• OpenTelemetry collector (OTLP gRPC + HTTP receivers, ClickHouse exporter)
• ExternalSecrets for the ClickHouse connection credentials
• Terraform CR (via Tofu Controller) that provisions the underlying Vault material

Configuration highlights

• External ClickHouse operator. The chart’s bundled ClickHouse is disabled (clickhouse.enabled: false) — ClickStack reuses the shared operator from ClickHouse instead of running a second one
• OTel Collector: explicitly enabled (otel.enabled: true), accepting OTLP gRPC and HTTP
• HyperDX UI: ships with the chart by default — no separate Kustomization needed
• Receives logs, traces, and metrics over OTLP and writes them straight to the ClickHouse instance

Pipeline overview

ClickStack is the middle of the observability pipeline. Applications speak OTLP to its collector, the collector writes into ClickHouse, and the bundled HyperDX UI plus Grafana read back out for search and dashboards.

┌─────────────┐     OTLP      ┌──────────────┐
│ Application │──────────────→│  OTel        │
│ (OTel SDK)  │               │  Collector   │
└─────────────┘               └──────┬───────┘
                                     │
                                     ▼
                              ┌──────────────┐
                              │  ClickHouse  │
                              │  (storage)   │
                              └──────┬───────┘
                                     │
                              ┌──────┴───────┐
                              ▼              ▼
                       ┌──────────────┐ ┌──────────┐
                       │  ClickStack  │ │ Grafana  │
                       │ (HyperDX UI) │ │ (dash)   │
                       └──────────────┘ └──────────┘

What it collects

Signal	Protocol	Examples
Logs	OTLP	Structured application logs, stdout/stderr
Traces	OTLP	HTTP request spans, database queries, external API calls
Metrics	OTLP	Request latency, error rates, queue depths, custom counters

Instrumenting applications

Applications send telemetry using the OpenTelemetry SDK. Point the OTLP exporter at the in-cluster collector:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://clickstack-otel-collector.clickstack.svc.cluster.local:4317"
  - name: OTEL_SERVICE_NAME
    value: "my-service"

Alerting

Alerting lives in Grafana on top of ClickHouse queries:

• Log-based alerts — trigger on error patterns or log volume spikes
• Metric-based alerts — trigger on latency thresholds or error rates
• Trace-based alerts — trigger on slow traces or high error spans

Dashboards as code

Committed dashboards under flux-apps/clickstack/clickstack-dashboards/dashboards/*.json are reconciled into HyperDX every 5 minutes by the clickstack-dashboards-sync CronJob. Edits to the JSON propagate on the next tick; UI edits don’t roundtrip back to git.

Auth model — two distinct credentials

Credential	Vault path	Purpose
Team API key	secret/clickstack.hyperdx-api-key	Gates OTLP ingestion (/v1/{logs,metrics,traces}). No user binding. Auto-generated by the Terraform CR (flux-apps/clickstack/clickstack-vault-secrets) and reconciled into HyperDX’s MongoDB so a single random UUID is the source of truth across Vault and the running cluster.
Personal Access Token (PAT)	secret/hyperdx-dashboards.pat	Gates HyperDX /api/v2/* (dashboards CRUD). User-bound — the username field next to it records which HyperDX account it was minted under. Used only by the dashboards-sync CronJob.

Manual bootstrap (PAT)

HyperDX OSS has no API to mint PATs without an authenticated UI session, so this is one-time manual:

# 1. Sign in to https://hyperdx.internal.stefanzhelev.com as the canonical
#    owner account, then Team Settings → Personal Access Keys → Create.
# 2. Persist both fields side-by-side in Vault:
vault kv put secret/hyperdx-dashboards \
  pat=<the-token> \
  username=<owner-email>

# 3. Skip the 1h refreshInterval and reconcile immediately:
kubectl -n clickstack annotate externalsecret clickstack-dashboards-credentials \
  force-sync=$(date +%s) --overwrite

The username field isn’t read by any code path — it’s there so a future rotator knows which account to log back into. To rotate: mint a new PAT in the UI, overwrite the same Vault entries, optionally re-force-sync the ExternalSecret.

Integrations

• ClickHouse: storage backend
• Vault + Tofu Controller: ClickHouse connection credentials are managed in Vault by Terraform CR
• Grafana: queries the data ClickStack lands in ClickHouse

Key commands

kubectl get pods -n clickstack
kubectl logs -n clickstack -l app.kubernetes.io/name=clickstack
kubectl get configmap -n clickstack -o yaml