Version skew

During a rolling deploy of auth or web, some users still run an older client bundle in an open tab. Version skew keeps the hashed client assets (_app/immutable/*) from recently used builds inside the new container image so those tabs do not break mid-session.

Applies to:

Image	Ingest platformId	Public URL env (ingest)
auth	auth	AUTH_URL
web-soacaille	soacaille	SOACAILLE_URL
web-kalyone	kalyone	KALYONE_URL

Runtime layout

Each SvelteKit app is built into a versioned directory:

apps/auth/
  build-1.2.3/     # SSR + client for v1.2.3
  build-1.2.4/     # SSR + client for v1.2.4
  build -> build-1.2.4   # symlink to latest

• APP_VERSION selects the active SSR build at startup (scripts/sveltekit-start.ts).
• GET /_versions returns { current, available } from build-* directories (@lib/sveltekit).

Ingest coordination

Ingest exposes a GraphQL query used at image build time:

query Versions($platformId: String!) {
  versions(platformId: $platformId) {
    current
    available
    used
  }
}

VersionService (apps/ingest/src/version/version.service.ts):

1. Fetches {PLATFORM}_URL/_versions from the live deployment.
2. Returns null when the platform is not deployed yet (missing URL env, HTTP 404, or empty current).
3. Adds used from Postgres (version_usage, window VERSION_USAGE_DAYS, default 30 days).

used lists client versions that still had traffic recently so we keep their build-{version} trees in the next image.

Cumulative build algorithm

Triggered for auth, web-soacaille, and web-kalyone when both VERSION_SKEW_ENABLED=1 (default) and PUBLIC_INGEST_URL are set (CI / production). Local k8s:build-images without those variables performs a fresh SvelteKit build (no cumulative base).

Do not disable in production

Keep VERSION_SKEW_ENABLED=1 on every production pipeline. Disabling version skew means rolling deploys can break users who still have an older client bundle open in the browser.

Only disable for a one-off bootstrap (before ingest is reachable) or a documented emergency — then set VERSION_SKEW_ENABLED=0 and remove it again after ingest is live.

flowchart TD
  A[nerdctl build] --> B{VERSION_SKEW_ENABLED and PUBLIC_INGEST_URL?}
  B -- no --> C[BASE_IMAGE = oven/bun:1.4]
  B -- yes --> D[GraphQL versions at ingest]
  D --> E{GraphQL / network error?}
  E -- yes --> F[Abort build]
  E -- no --> G{response null?}
  G -- yes --> C
  G -- no --> H[BASE_IMAGE = registry/image:response.version]
  H --> I[Pull base image]
  I --> J[prepare: copy retained build-* from base]
  J --> K[Vite build → build-APP_VERSION]
  K --> L[finish: prune + merge client immutable assets]
  L --> M[Runner image with all retained builds]

Steps in detail:

1. Query ingest at $PUBLIC_INGEST_URL/graphql. Any network or GraphQL error aborts the build.
2. versions returns null → first deploy: BASE_IMAGE=oven/bun:1.4, no retained builds.
3. Otherwise → BASE_IMAGE is the currently deployed image ($REGISTRY/<image>:<version> in CI, plus261/<image>:<version> locally).
4. Validate that every available entry exists as build-{version} in the base image export. Missing directories abort the build.
5. Retain build-{v} for v ∈ used ∪ {current} (toKeep); delete other build-* from the working tree.
6. Build the new version into build-{APP_VERSION} and symlink build → latest.
7. Merge client/_app/immutable from retained builds into the active build so one Bun process serves all legacy chunks.

Implementation:

Piece	Location
Plan + file ops	libs/sveltekit/src/version-skew.ts
Ingest GraphQL client (host)	devops/scripts/lib/version-skew.ts
Docker prepare/finish	scripts/sveltekit-version-skew.ts
Container image build	devops/scripts/lib/container.ts → buildAppImage
Dockerfile stages	raw-base (export previous app tree) + builder

Environment variables

Variable	Where	Role
VERSION_SKEW_ENABLED	CI build host	1 (default) — cumulative builds on. 0 → warning in log + build:images:app manual.
VERSION_SKEW_BASE_IMAGE_TAG	CI / local build	Optional test override: semver base image tag from registry (e.g. 0.4.0, never latest) without Ingest. Alert + build:images:app manual.
PUBLIC_INGEST_URL	CI build host	Public ingest base URL (e.g. https://ingest.plus261.com). Required when skew is enabled.
REGISTRY	CI build host	Registry prefix for cumulative base image pull
APP_VERSION (injected from git tags)	Build / start	Transport only: Docker ARG/ENV from resolveAppVersion — never set in .env
IMAGE_TAG	Container	Docker registry/local tag only
AUTH_URL, SOACAILLE_URL, KALYONE_URL	Ingest runtime	Live /_versions endpoint per platform
AUTH_EXCHANGE_TOKEN	Ingest → platform	Optional bearer for /_versions
VERSION_USAGE_DAYS	Ingest	Lookback for used (default 30)

CI

Set PUBLIC_INGEST_URL and keep VERSION_SKEW_ENABLED=1 (default in .gitlab-ci.yml) in GitLab CI/CD variables.

Before build:images:app, scripts/ci/version-skew-guard.sh:

1. Warns in the job log if VERSION_SKEW_ENABLED=0 (build continues without skew; job is manual in GitLab)
2. Warns in the job log if VERSION_SKEW_BASE_IMAGE_TAG is set (build forces base image from registry without Ingest; job is manual in GitLab)
3. Fails if skew is enabled and no base tag override is set, but Ingest is unreachable at PUBLIC_INGEST_URL

Testing without Ingest (VERSION_SKEW_BASE_IMAGE_TAG)

To verify cumulative SvelteKit builds without deploying Ingest:

# In GitLab CI: run pipeline with variable VERSION_SKEW_BASE_IMAGE_TAG=0.4.0 (git semver, never latest)
# Locally:
VERSION_SKEW_BASE_IMAGE_TAG=0.4.0 \
REGISTRY=registry.gitlab.com/iraitek/plus261 \
bun --bun run k8s:build-images

• Pulls $REGISTRY/<image>:$VERSION_SKEW_BASE_IMAGE_TAG as base image
• Carries over retained client assets from that image into the new build
• Dangerous in production: like VERSION_SKEW_ENABLED=0, it bypasses Ingest telemetry and freezes the base image. The CI job is manual and prints a prominent warning banner. Do not leave it set in CI/CD variables.

First deploy (bootstrap)

Ingest must be built, deployed, and reachable before app images that use version skew:

1. build:images:ops → push ingest image
2. Deploy ingest to the cluster (deploy:ops:production when enabled)
3. build:images:app — ingest returns null for each platform (nothing deployed yet) → fresh SvelteKit builds; this is expected

Do not skip step 2 and rely on VERSION_SKEW_ENABLED=0 except for a one-off bootstrap before ingest is deployed.

The job already exports REGISTRY and APP_VERSION; nerdctl passes BASE_IMAGE and VERSION_SKEW_B64 into the Dockerfile.

Local production-like test:

export PUBLIC_INGEST_URL=https://ingest.plus261.com
export REGISTRY=ghcr.io/org/plus261
export APP_VERSION=1.2.4
bun --bun run k8s:push-images

Operations

• Rollback: redeploy a previous image tag; cumulative builds are tag-to-tag, not automatic downgrades.
• Observability: compare GET /_versions on the platform with ingest versions(platformId) during a deploy.
• Storage: each retained build-{version} increases image size; used window limits growth.

Related

• Deployment — cluster deploy overview
• CI/CD — build:images pipeline
• Web architecture — SvelteKit client layout