Skip to content
Stefan Zhelev

Lightdash

Lightdash is the dbt-native BI tool that exposes the fortune project’s mart layer as explores, charts, and dashboards. The Lightdash project is configured with type: none, so Lightdash holds no git credentials and does no dbt compilation of its own — every schema change is pushed in from CI by lightdash deploy.

Deployment

Section titled “Deployment”
FieldValue
URLhttps://lightdash.internal.stefanzhelev.com (NetBird VPN only)
Flux pathflux-clusters/stefanzhelev/apps/lightdash
Base pathflux-apps/lightdash
Namespacelightdash
Sync wave7
Depends oncloudnative-pg-config, external-secrets-config, authentik-app-lightdash, tofu-controller

Workflow phases

Section titled “Workflow phases”

How a Lightdash-relevant change reaches the live fortune project. Phase 1 is the developer ritual; phases 2–4 are CI.

1. Branch creation (developer ritual)

Section titled “1. Branch creation (developer ritual)”

As soon as a branch is cut for any change that will touch Lightdash (a new tagged model, a metric, a chart, a dashboard), the developer creates a personal preview project named after the branch:

Terminal window
export LIGHTDASH_URL=https://lightdash.internal.stefanzhelev.com
export LIGHTDASH_API_KEY=<your PAT>


lightdash start-preview --name "${BRANCH}"


# `start-preview` doesn't print the new project's UUID. Capture it:
export LIGHTDASH_PROJECT=$(curl -s \
  -H "Authorization: ApiKey ${LIGHTDASH_API_KEY}" \
  "${LIGHTDASH_URL}/api/v1/org/projects" \
  | jq -r --arg n "${BRANCH}" '.results[] | select(.name == $n) | .projectUuid')

Slashes in branch names (feature/foo) must be replaced with - — Lightdash project names don’t accept them.

The preview is the working surface for the rest of the change. Two iteration patterns:

  • Code-first. Edit YAML under dbt/lightdash/charts/ or dbt/lightdash/dashboards/ (or models/**/properties.yml for explores), then push the change to the preview:
    Terminal window
    lightdash upload --force
  • UI-first. Edit a chart or dashboard interactively in the preview UI, then sync just that one back to YAML:
    Terminal window
    lightdash download -c <chart-slug>
    lightdash download -d <dashboard-slug>

Commit the YAML when satisfied.

2. Open a PR

Section titled “2. Open a PR”

On pull_request: opened|synchronize|reopened, .forgejo/workflows/lightdash-preview.yml runs:

dbt deps && dbt compile
lightdash start-preview --name "${BRANCH}" --select tag:lightdash

start-preview is idempotent on --name — if the dev already created the preview in phase 1, this just refreshes its manifest with the latest compiled dbt. Saved charts and dashboards in the preview are preserved across reruns, so the dev’s in-progress work isn’t clobbered.

If the preview is missing (the dev skipped phase 1), CI creates it and posts a comment on the PR with the project URL and a heads-up nudging the dev toward the expected ritual. This is a fallback, not the primary mechanism.

3. Merge

Section titled “3. Merge”

On merge to stefanzhelev, .forgejo/workflows/lightdash-deploy.yml fires on any change under dbt/**:

dbt deps && dbt compile
lightdash deploy --select tag:lightdash       # explores / metrics / dimensions
lightdash upload --force                      # charts + dashboards

Both steps target the live fortune project, making the merge the single source of truth. type: none and hideRefreshButton: true block any other write path. UI edits made directly in fortune are reverted on the next merge unless they’re captured back into YAML via lightdash download -c|-d and committed.

4. PR close

Section titled “4. PR close”

On pull_request: closed, the workflow’s stop-preview job runs lightdash stop-preview --name "${BRANCH}" || true and tears the preview down. The || true covers the edge case where no preview was ever created.

Initial seed

Section titled “Initial seed”

The Job under flux-apps/lightdash/lightdash-project-bootstrap is the first-run seed only. It:

  1. Creates the fortune project via the API with the ClickHouse warehouse connection if it doesn’t exist (PATCHes it back to manifest on re-runs)
  2. Clones the repo, renders profiles.yml, runs dbt deps && dbt compile
  3. Pushes the manifest with lightdash deploy
  4. Writes the project UUID to Vault at secret/lightdash-project.uuid so CI can fetch it without hardcoding

After the first run, day-to-day sync runs from the Forgejo Actions workflows above. Re-run the bootstrap Job only when re-creating the project from scratch:

Terminal window
kubectl -n lightdash delete job lightdash-project-bootstrap

Flux re-applies the Job manifest on the next reconcile.

Authentication

Section titled “Authentication”

Lightdash’s generic OIDC connector points at Authentik. The Authentik OIDC application, group bindings, and the OIDC client secret are all provisioned via a Terraform CR (authentik-app-lightdash) by Tofu Controller, with credentials sourced from Vault through ExternalSecrets.

Secrets

Section titled “Secrets”

CI secrets (LIGHTDASH_API_KEY, LIGHTDASH_PROJECT, LIGHTDASH_CLICKHOUSE_*) are surfaced from Vault into Forgejo Actions by flux-apps/lightdash/lightdash-forgejo-secrets/tofu — a 10-minute reconcile loop. Rotating any of those values in Vault flows into CI on the next reconcile + workflow run; never edit Forgejo secrets by hand.

Integrations

Section titled “Integrations”
  • dbt: the only source of explore metadata — Lightdash deploys nothing that isn’t compiled from dbt/
  • ClickHouse: warehouse target, queried with the dbt user from data-workflows-database
  • CloudNative-PG: Lightdash’s own metadata store
  • Authentik: OIDC SSO via the generic OIDC connector
  • Vault: holds OIDC client secret, admin creds, PAT, and the project UUID
  • Forgejo: runs lightdash-deploy.yml and lightdash-preview.yml