AutoCom Docs

Module Versioning — Overview & Reference

This is the single source-of-truth reference for AutoCom's module versioning system. It explains what the system does, why it exists, how it's wired together, and how to operate it day-to-day.

If you only have 2 minutes, read In Plain Terms. If you're operating a cluster, jump to Operator Runbook. If you're debugging, start with Troubleshooting.

In Plain Terms

Before this system, all 25 modules lived in a folder with no version tracking, no way to ship a specific version to a customer, and no way to roll back if something broke. Now:

  1. Every module has a version receipt. Each module.json declares a semantic version, and a per-ring lock file at modules/manifests/<ring>.lock.json (default ring: stable) records the exact (version, sha256) of every module that ring deploys. Same lock file → same code, every deploy.

  2. Developers can't accidentally skip versioning. When code changes inside a module without bumping its version, CI blocks the merge. Same if someone tries a version regression (1.0.0 → 0.9.0).

  3. Modules build into shippable packages automatically. Tagging v1.4.2 on a module's mirror repo triggers a release pipeline that zips the module, computes a checksum, uploads it to GitLab's Generic Package Registry, and creates a GitLab Release entry — all hands-off.

  4. Install / upgrade / rollback with one command:

    • php artisan module:install orders@1.0.0
    • php artisan module:upgrade orders --to=1.0.1
    • php artisan module:rollback orders The system downloads the right version from the registry, verifies the checksum, swaps files in place, runs new database migrations, and updates the module_versions audit table. Old versions stay on disk for instant rollback.

  5. Modules can refuse to run on incompatible platforms. A module declares compatibility.platform: "^2.0.0" and the loader refuses to register it if the current /VERSION doesn't satisfy that constraint. Dependent modules cascade-fail naturally.

  6. Everything is auditable. The module_versions table records every install/upgrade/rollback with version, hash, source path, and timestamp. You can answer "when did Orders 1.4.2 ship?" with one SQL query.

Why This Was Built

Problem before Solution now
"Which version of Orders is in production?" had no answer php artisan module:list + module_versions table
Deploys drifted between environments manifest.lock.json + module:verify
A module change could ship without anyone realizing CI module-version-check blocks unbumped MRs
No way to ship a hotfix to one module Release pipeline per module, independent cadence
Bad release? Full backend redeploy + git revert module:rollback orders --to=1.0.0
Third-party modules from vendors? Impossible Same package registry pattern works for external publishers

Architecture

Repository Topology

The system uses a monorepo-canonical model with polyrepo distribution targets:

autocommerce/main                    ← canonical development repo
  ├── VERSION                        ← platform version "2.0.0"
  ├── modules/
  │   ├── manifest.lock.json         ← pins every module to (version, sha256)
  │   ├── module.schema.json
  │   └── Orders/, Products/, …
  └── ci/
      ├── templates/
      │   ├── modules-versioning.yml ← CI gate: enforce version bumps + lock validity
      │   ├── module-split.yml       ← auto-mirror main → per-module repos
      │   └── php-module.yml         ← release pipeline (used by mirror repos)
      └── scripts/
          ├── check-module-versions.sh
          ├── validate-module-lock.sh
          ├── check-module-changelogs.sh
          └── split-and-mirror.sh

autocommerce/modules/<alias>         ← per-module mirror (one per module)
  └── (subtree split of modules/<Name>/ from main)
       ├── module.json
       ├── backend/
       ├── frontend/
       ├── CHANGELOG.md
       └── .gitlab-ci.yml            ← references php-module.yml from main

autocommerce/themes/theme-<name>     ← themes follow the same pattern

Developers work in autocommerce/main exactly like before. Mirror repos receive automated pushes from main and exist purely as release targets — never clone them, never push to them directly.

Two-Track Deployment Flow

                       ┌──────────────────────────────────────────┐
                       │  developer commits to autocommerce/main  │
                       │  (bumps modules/Orders/module.json)      │
                       └────────────────┬─────────────────────────┘
                                        │
                       ┌────────────────▼────────────────┐
                       │  CI: module-version-check       │
                       │  CI: module-lock-validate       │
                       │  CI: module-changelog-check     │
                       └────────────────┬────────────────┘
                                        │ MR merged
                       ┌────────────────▼────────────────┐
                       │  CI: module-split job runs      │
                       │  - git subtree split per module │
                       │  - push to mirror repo          │
                       │  - if version changed: tag v*   │
                       └────────────────┬────────────────┘
                                        │ tag triggers
                       ┌────────────────▼────────────────┐
                       │  CI on mirror repo:             │
                       │  - package-module → zip         │
                       │  - release-module → registry    │
                       │  - create-release-entry → API   │
                       └────────────────┬────────────────┘
                                        │
                       ┌────────────────▼────────────────┐
                       │  GitLab Package Registry holds: │
                       │  Orders-1.4.2.zip + sha256 +    │
                       │  MANIFEST.json                  │
                       └────────────────┬────────────────┘
                                        │
                       ┌────────────────▼────────────────┐
                       │  Production cluster operator:   │
                       │  $ php artisan module:install   │
                       │      orders@1.4.2               │
                       └─────────────────────────────────┘

Database Schema

Two tables back the system:

modules

The currently active version of each module. Maintained by ModuleLoaderService::syncModulesToDatabase() and updated by ModuleInstaller::activate().

module_versions

Version history per module. New rows on every install/upgrade. Exactly one row per module has is_active = true.

module_versions
├── id
├── module_id (fk → modules.id)
├── version            semver string
├── source_path        storage/app/module-versions/<alias>/<version>/
├── content_hash       raw sha256 hex (no prefix, varchar(64))
├── is_active          bool — exactly one true per module
├── changelog          text, nullable
├── size_bytes
├── created_by         "module:install", "module:upgrade", etc.
├── created_at
└── updated_at

Inactive versions' source_path is preserved so module:rollback can swap back without re-downloading.

Component Map

Layer Component Purpose
Backend service App\Core\Services\ModuleHasher Deterministic SHA256 over module source (excludes build artifacts)
Backend service App\Core\Services\PlatformCompatibility Reads /VERSION, evaluates compatibility.platform constraints with hand-rolled semver matcher
Backend service App\Core\Services\ModuleRegistryClient Downloads + verifies + extracts artifacts from GitLab Package Registry
Backend service App\Core\Services\ModuleInstaller Atomic-ish swap of module files, updates DB tables, fires platform lifecycle hooks
Backend service App\Core\Services\ModuleLoaderService (extended) Refuses incompatible modules at boot
Backend service App\Core\Services\ModuleManager::runPlatformLifecycleHook() Non-tenant-scoped hook dispatcher for onInstall/onUpgrade/onDowngrade
Artisan command module:lock Generate/update manifest.lock.json
Artisan command module:verify CI gate — filesystem matches lock + compat
Artisan command module:compat Report compat status of every module
Artisan command module:install Install from registry
Artisan command module:upgrade Upgrade installed module
Artisan command module:rollback Roll back to prior version
CI template ci/templates/modules-versioning.yml MR enforcement (3 jobs)
CI template ci/templates/module-split.yml Auto-mirror main → per-module repos
CI template ci/templates/php-module.yml (extended) Release pipeline: package + release + create-release-entry
CI script ci/scripts/check-module-versions.sh Enforce semver bump on code change
CI script ci/scripts/validate-module-lock.sh Inline PHP verifier (CI-equivalent of module:verify)
CI script ci/scripts/check-module-changelogs.sh Advisory: warn on minor/major bump without CHANGELOG
CI script ci/scripts/split-and-mirror.sh Subtree split + push to mirror + auto-tag
CI template ci/templates/backend-tests.yml Runs the 16 Ring PHPUnit contract tests on MRs that touch backend/ or modules/
One-time script bin/seed-module-mirrors.sh Initial seed of mirror repos from main — data-driven: resolves each mirror URL from repository.url in modules/<Name>/module.json, falls back to a small legacy table only for historical off-name repos

Developer Workflow

Making a code change

  1. Work in autocommerce/main exactly like any other change.
  2. Edit files under modules/<Name>/.
  3. Bump the version field in modules/<Name>/module.json:
    • 1.0.0 → 1.0.1 for bug fixes
    • 1.0.0 → 1.1.0 for new features
    • 1.0.0 → 2.0.0 for breaking changes
  4. (Recommended) Add a CHANGELOG.md entry for minor/major bumps.
  5. Run php artisan module:lock to update the lock file.
  6. Commit and push your branch.

What CI enforces in your MR

Job Behavior
module-version-check Fails if module source changed but version didn't bump. Also fails on regressions.
module-lock-validate Fails if manifest.lock.json doesn't match the filesystem. Run php artisan module:lock to fix.
module-changelog-check Advisory only — warns if a minor/major bump has no CHANGELOG.md entry.
backend-ring-tests Runs the 16 PHPUnit tests in backend/tests/Feature/Rings/* that lock in the Deployment Rings contract: each ring reads from its own lock file, tenants resolve to their assigned ring, and ring:promote mutates only the target ring's manifest. Fails the MR if any assertion breaks — this is how the isolation guarantee stays enforced on every merge.

After merge to main

The module-split job runs git subtree split on each changed module and force-pushes to the matching mirror repo. If module.json's version changed, the mirror is also tagged with v<version>, which triggers the release pipeline and produces an artifact in the GitLab Package Registry.

How CI rules work (path-based job triggering)

Every CI job has a changes: filter declaring which file paths it cares about. If your commit doesn't touch any of those paths, the job is silently skipped — no compute spent.

If you change… Jobs that fire
modules/<Name>/backend/... (real module code) All 3 module gates (version-check, lock-validate, changelog-check) + module-split + build-api + build-horizon
modules/<Name>/frontend/... Same as above + build-frontend
modules/<Name>/module.json only All 3 gates + module-split (auto-tags mirror if version changed)
modules/manifests/<ring>.lock.json Nothing — these are explicitly excluded everywhere
backend/... (platform Laravel code) build-api, build-horizon
frontend/... (platform Next.js) build-frontend
docs/... build-docs
composer.json / composer.lock build-api, build-horizon
ci/scripts/... or .gitlab-ci.yml Nothing — pure CI changes don't trigger anything
k8s/... Nothing — manifests are config, not code
Any push to main deploy-k8s (manual; always present, no changes: filter)

This means:

  • A docs-only MR doesn't burn build minutes
  • A CI-script-only MR doesn't burn build minutes (and doesn't trigger module-split)
  • Bumping Orders doesn't rebuild the frontend
  • A typo fix in repo-root README doesn't fire anything

If a pipeline shows only deploy-k8s and nothing else, the right read is: "this commit didn't change anything that needs CI work." deploy-k8s is a manual button that's always available — not a pending action.

Operator Runbook

Installing a fresh module on a running cluster

export MODULE_REGISTRY_URL=https://gitlab.wexron.io
export MODULE_REGISTRY_TOKEN=<personal-or-deploy-token-with-read_package_registry>

php artisan module:install orders@1.0.0

What happens internally:

  1. ModuleRegistryClient downloads Orders-1.0.0.zip, .sha256 sidecar, and MANIFEST.json from the registry
  2. SHA256 verified — mismatch aborts
  3. Extracted to storage/app/module-versions/orders/1.0.0/
  4. ModuleInstaller backs up current modules/Orders/, wipes contents, copies new contents
  5. Central database migrations from the new version are run (unless --skip-migrations)
  6. onInstall platform lifecycle hook fires
  7. module_versions row inserted with is_active = true

Upgrading a module

php artisan module:upgrade orders --to=1.0.1

Same flow as install, plus:

  • Previous module_versions row gets is_active = false (kept for rollback)
  • onUpgrade hook fires with {fromVersion, toVersion, sourcePath, manifest} context

Rolling back

php artisan module:rollback orders                # most recent prior version
php artisan module:rollback orders --to=1.0.0     # specific version

⚠ Schema migrations are NOT auto-reverted. Run php artisan migrate:rollback first if the version you're rolling back to has a narrower schema.

Checking system health

# Are all modules pinned correctly?
php artisan module:verify

# Are all modules compatible with the current platform version?
php artisan module:compat

# What's installed right now?
php artisan module:list

# Show full version history for a module
php artisan tinker --execute='echo \App\Models\ModuleVersion::with("module")->where(["module_id" => \App\Models\Module::where("alias", "orders")->value("id")])->orderBy("id")->get(["id", "version", "is_active", "created_at"])->toJson(JSON_PRETTY_PRINT);'

Atomicity caveat

The swap uses wipe-and-copy with backup, not atomic rename. This is because in container runtimes using overlayfs without redirect_dir=on (default in Docker Desktop / containerd), renaming a directory from a lower layer fails with EXDEV.

There is a brief window during the swap where modules/<Name>/ is being repopulated. This is acceptable for most production deployments since module code is loaded into PHP memory at boot, not re-read on every request.

For full atomicity, mount /app/modules on a PVC — that puts everything in the upper overlay layer where rename works. Then you can switch swapInPlace() to use it.

Per-Tenant Module Versions (Deployment Rings)

Everything described above runs in a single deployment: one set of api/horizon/nginx pods, one set of pinned module versions, every tenant on the same code. That's the day-1 setup and the right default.

When you need different tenants on different module versions — canary releases, opt-in beta, enterprise tenants on long-term-stable, internal QA dogfooding — the architecture extends via deployment rings:

  • Each ring is its own k8s namespace with its own api/horizon/nginx pods
  • Each ring pins its own module versions via modules/manifests/<ring>.lock.json
  • Tenants are routed to a ring at the ingress layer based on tenants.module_ring_id
  • Promoting a module version up the chain is one command + one git commit

The ring system is built on top of the versioning system documented above — same package registry, same install/upgrade/rollback commands, same module.json format. Rings just add the per-process boundary that lets multiple versions coexist on the same cluster.

                ring: stable      ring: edge       ring: canary
                Orders @ 1.4.0    Orders @ 1.5.0   Orders @ 1.6.0-beta
                ──────────────    ──────────────   ──────────────
                k8s ns:           k8s ns:          k8s ns:
                autocom           autocom-edge     autocom-canary

                Tenants:          Tenants:         Tenants:
                acme, foo, …      widgets, bar     internal-qa

Day-1 you have exactly one ring (stable). Adding a second ring is opt-in via:

kubectl apply -k k8s/overlays/rings/canary  # spin up canary pods
php artisan ring:assign internal-qa canary  # move one tenant

Operator commands:

php artisan ring:list                                    # all rings + tenant counts
php artisan ring:show stable --tenants --modules         # ring detail
php artisan ring:assign acme canary --reason="opt-in"    # move a tenant
php artisan ring:promote orders --from=canary --to=edge  # promote a version

See Deployment Rings for the full architecture, k8s overlays, hostname routing, and trade-offs.

Complete CLI Reference

module:lock

php artisan module:lock                         # regenerate the lock file
php artisan module:lock --check                 # exit non-zero if lock is stale
php artisan module:lock --module=Orders         # update one module only

Generates or updates modules/manifest.lock.json so it captures the exact (version, sha256) of every module on disk.

module:verify

php artisan module:verify

CI gate. Checks three things per module: (1) version in module.json matches the lock file, (2) content SHA256 matches, (3) compatibility.platform constraint is satisfied. Non-zero exit on any failure.

module:compat

php artisan module:compat
php artisan module:compat --strict

Tabular report of every module's platform constraint vs the current /VERSION. Non-zero exit if any incompatibility.

module:install

php artisan module:install orders@1.0.0
php artisan module:install orders@1.0.0 --force            # reinstall same version
php artisan module:install orders@1.0.0 --skip-migrations

Refuses if the module is already installed at a different version (use module:upgrade for that path).

module:upgrade

php artisan module:upgrade orders --to=1.0.1
php artisan module:upgrade orders --to=1.0.1 --skip-migrations

module:rollback

php artisan module:rollback orders                # most recent prior version
php artisan module:rollback orders --to=1.0.0     # specific version

module:lock --ring=<name> (per-ring lock files)

php artisan module:lock                           # writes stable.lock.json
php artisan module:lock --ring=stable             # explicit
php artisan module:lock --ring=edge               # writes edge.lock.json
php artisan module:lock --ring=canary --module=Orders

Each ring keeps its own modules/manifests/<ring>.lock.json. module:verify and module:compat accept the same --ring= flag.

ring:list, ring:show, ring:assign, ring:promote

Operator-side commands for the deployment-rings system.

php artisan ring:list                                    # all rings + tenant counts
php artisan ring:show stable --tenants --modules         # ring detail
php artisan ring:assign acme canary --reason="opt-in"    # move a tenant
php artisan ring:promote orders --from=canary --to=edge  # promote a version

See Deployment Rings for the full design and runbook.

Lifecycle Hooks

A module can declare custom platform lifecycle handlers in module.json:

{
  "lifecycle": {
    "onEnable":    "Modules\\MyModule\\App\\Lifecycle\\OnEnable",
    "onDisable":   "Modules\\MyModule\\App\\Lifecycle\\OnDisable",
    "onInstall":   "Modules\\MyModule\\App\\Lifecycle\\OnInstall",
    "onUpgrade":   "Modules\\MyModule\\App\\Lifecycle\\OnUpgrade",
    "onDowngrade": "Modules\\MyModule\\App\\Lifecycle\\OnDowngrade"
  }
}
Hook Scope Fires on Use for
onEnable Per-tenant Tenant enables module Seed tenant data, register tenant webhooks
onDisable Per-tenant Tenant disables module Clean up tenant resources
onInstall Platform-wide module:install Compile binaries, warm shared caches
onUpgrade Platform-wide module:upgrade Migrate central data formats, invalidate caches
onDowngrade Platform-wide module:rollback Restore central data formats

Platform handlers receive array $context with {fromVersion, toVersion, sourcePath, manifest}. They run once per operation regardless of how many tenants exist. Failed platform hooks halt the operation; failed tenant hooks are logged but non-fatal.

Setup Checklist (one-time)

After merging feat/module-versioning to main:

  1. Set MIRROR_PUSH_TOKEN as a CI/CD variable on autocommerce/main:

    • Group access token on autocommerce with Developer role + write_repository scope
    • Mark as Masked and Protected
    • Required for module-split.yml to push to mirror repos

  2. Set MODULE_REGISTRY_TOKEN for production environments:

    • Personal or deploy token with read_package_registry scope on autocommerce/modules/*
    • Add to k8s secret autocom-secrets and reference from api deployment env vars
    • Required for module:install/upgrade to work in prod

  3. Seed mirror repos (one-time, from a dev machine with all gl--modules-* remotes configured):

    bash bin/seed-module-mirrors.sh --dry-run --all   # preview
bash bin/seed-module-mirrors.sh --all             # actually push

    First run is a no-op for any mirror that's already in sync. Already verified to work for Orders (the seed produced the same tip commit as the existing mirror).

  4. (Optional) Mount /app/modules on a PVC in production for true atomic swap. Otherwise the wipe-and-copy strategy works but has a brief non-atomic window.

  5. (Optional) Widen module_versions.content_hash to varchar(80) if you want to store the sha256: prefix natively. Currently we strip it.

CI/CD Variables Reference

Variable Where Type Purpose
MIRROR_PUSH_TOKEN autocommerce/main Group access token, masked, protected write_repository on autocommerce/modules/* and autocommerce/themes/* so split-and-mirror.sh can push to mirrors
CI_JOB_TOKEN mirror repos Automatic, no setup Used by release-module to upload to the project's own Package Registry
MODULE_REGISTRY_URL production env Plain env var Base URL of GitLab (default: https://gitlab.wexron.io)
MODULE_REGISTRY_TOKEN production env Secret env var Personal/deploy token with read_package_registry

Lock File Format

modules/manifest.lock.json is the contract for reproducible deploys:

{
  "$schema": "./manifest.lock.schema.json",
  "platform": "2.0.0",
  "generated_at": "2026-04-12T19:54:19+00:00",
  "modules": {
    "orders": {
      "version": "1.0.0",
      "source": "gitlab:autocommerce/modules/orders",
      "artifact": null,
      "sha256": "sha256:9ad8f12909c541c029379df6307bfd534a3e4859e6f6370618449979a314b635",
      "file_count": 405,
      "size_bytes": 1834521,
      "released_at": null
    }
  }
}

Generated by php artisan module:lock. Verified by php artisan module:verify.

The hash algorithm walks the module directory in sorted order and SHA256s file contents + relative paths, excluding: node_modules/, vendor/, .git/, dist/, build/, .next/, .cache/, coverage/, storage/framework/, storage/logs/, plus lock files and .DS_Store. Two machines producing the same source tree always produce the same hash.

Implementation History

The system was built in 6 phases. Each phase ended with end-to-end verification before moving on.

Phase 1 — Foundation

  • VERSION file at repo root (platform 2.0.0)
  • modules/module.schema.json extended with repository, distribution, compatibility.platform, changelog fields
  • App\Core\Services\ModuleHasher — deterministic content hashing
  • php artisan module:lock — generates manifest.lock.json from filesystem state
  • php artisan module:verify — validates filesystem matches lock file
  • modules/manifest.lock.json initial generation

Verified: all 22 modules hashed cleanly, drift detection works (added a file to Orders, verify failed with clear message), reset.

Phase 2 — CI versioning enforcement

  • ci/templates/modules-versioning.yml — 3 jobs in main repo pipeline
  • version-check: blocks MRs that change module source without bumping version
  • module-lock-validate: runs equivalent of module:verify
  • changelog-check: advisory warning on minor/major without CHANGELOG.md entry
  • All scripts bash-3.2 compatible (run locally on macOS too)

Verified: 4 scenarios — no change passes, modify-without-bump fails, modify+bump passes, regression fails.

Phase 3 — Release pipeline

  • bin/seed-module-mirrors.sh — one-time git subtree split script. Originally shipped with a hardcoded 22-module map; now data-driven: resolves each mirror URL from repository.url in modules/<Name>/module.json, with a small legacy fallback table only for historical off-name repos. Adding a new module requires no script changes — set repository.url in its module.json and the seed / sync both pick it up automatically. All 23 modules + themes successfully seeded.
  • ci/templates/module-split.yml + ci/scripts/split-and-mirror.sh — auto-mirror main changes to per-module repos, auto-tag on version change
  • ci/templates/php-module.yml extended with 3 new jobs:
    • package-module: zip + sha256 + MANIFEST.json
    • release-module: upload to GitLab Generic Package Registry
    • create-release-entry: create GitLab Release via direct API call (avoids release-cli ARM64 panic)

Verified: real release of Orders v1.0.0 — pipeline 307 ran on the Orders mirror, produced Orders-1.0.0.zip (141 KB, 128 files), uploaded to registry, created GitLab Release. SHA256 verified locally against the registry artifact.

Phase 4 — Install / upgrade / rollback

  • App\Core\Services\ModuleRegistryClient — downloads + verifies artifacts
  • App\Core\Services\ModuleInstaller — atomic-ish swap with backup/restore
  • module:install, module:upgrade, module:rollback artisan commands
  • ModuleManager::runPlatformLifecycleHook() — non-tenant-scoped hook dispatcher

Verified: real upgrade chain on the live k3s cluster — installed Orders 1.0.0, upgraded to 1.0.1, rolled back to 1.0.0. DB state and on-disk state both reflected each step. API stayed HTTP 200 throughout.

Phase 5 — Platform compatibility

  • App\Core\Services\PlatformCompatibility — central semver matcher (^, ~, >=, >, <=, <, exact, ranges like >=2.0 <3.0)
  • ModuleLoaderService refuses incompatible modules at boot
  • module:compat reports compatibility status
  • module:verify refactored to delegate to PlatformCompatibility

Verified: set Orders to require ^99.0.0, loader refused with clear error, dependent modules (StoreShopify, Communications, MobileApp) cascade-failed naturally via the dependency graph. Reset, all clean.

Phase 6 — Documentation + final E2E

  • versioning.md — workflow doc
  • cli-commands.md — extended with new commands
  • lifecycle.md — added platform-level hooks
  • 8-step cross-cutting smoke test: lock → verify → compat → install → upgrade → rollback → DB confirmation → API health check

Phase 7 — Deployment Rings (per-tenant module versions)

  • module_rings table + tenants.module_ring_id FK (every existing tenant auto-migrated to stable)
  • ModuleRing model, Tenant.moduleRing() relationship
  • ring:list, ring:show, ring:assign, ring:promote artisan commands
  • modules/manifest.lock.json → modules/manifests/<ring>.lock.json per-ring split
  • module:lock --ring=<name>, module:verify --ring=<name>
  • ModuleLoaderService::ringName() reads RING_NAME env var, picks the right lock file
  • /api/health returns {ring: {name, lock_path}} for operator verification
  • k8s/overlays/rings/{stable,edge,canary}/ Kustomize overlays with strategic merge label patches
  • ci/templates/ring-promote.yml — manual-trigger pipeline for module promotion across rings
  • docs/content/docs/modules/versioning-rings.md — full architecture + operator runbook

Verified: on the live cluster — ring:list returns 3 rings, /api/health reports ring=stable, isolation verified by mutating canary's manifest and confirming stable+edge unaffected, multi-module verified by adding experimental-feature to canary only.

Phase 8 — Cleanup (k8s base split + PSA hardening + ring tests)

  • k8s/base/ split into per-ring/ (api, horizon, nginx, network-policies) and shared/ (frontend, docs, redis, indian-post). Meta kustomization.yaml preserves backward compat for local/production/staging overlays
  • k8s/overlays/shared/ — new dedicated overlay for the shared layer
  • k8s/overlays/rings/_shared-aliases/ — kustomize component with ExternalName services bridging non-stable rings to shared services in autocom
  • PSA restricted compliance: seccompProfile: RuntimeDefault, container-level runAsNonRoot, capabilities: drop ALL everywhere
  • nginx switched from nginx:alpine → nginxinc/nginx-unprivileged:alpine (port 8080) to fix chown failure on read-only root
  • 3 PHPUnit feature test files in backend/tests/Feature/Rings/:
    • RingIsolationTest.php (5 tests)
    • RingMultiModuleTest.php (5 tests)
    • RingPromotionTest.php (6 tests)

Verified: all 7 kustomize overlays still build cleanly; ring overlays went from 7 deployments to 3 each (api+horizon+nginx).

Bugs Found & Fixed During Verification

Real bugs the verification phases caught that would have broken production:

# Phase Bug Fix
1 3 release-cli Go runtime panic on ARM64 (lfstack.push invalid packing) Replaced with direct curl to GitLab Releases API
2 3 create-release-entry had empty dotenv vars, produced HTTP 400 Made it needs: package-module directly to inherit dotenv report
3 4 File::copyDirectory flaky in Octane — rename(): cannot be a directory Replaced with manual recursiveCopy() using RecursiveDirectoryIterator
4 4 rename() EXDEV on overlayfs lower layer Switched to wipe-and-copy strategy with backup-restore
5 4 content_hash varchar(64) overflow with sha256: prefix (71 chars) Strip prefix when storing
6 5 composer/semver listed as transitive constraint but not installed Hand-rolled matcher in PlatformCompatibility::satisfies()
7 post-3 bin/seed-module-mirrors.sh had hardcoded 22-module map; new modules silently skipped Made data-driven — reads repository.url from module.json, falls back to legacy table only for off-name historical mirrors
8 post-3 module-split.yml ran bash as a git subcommand because alpine/git image's ENTRYPOINT was git Override entrypoint: [""] in the image declaration
9 post-3 build-docs CI job failed: COPY package.json couldn't find file at repo root Changed build context from . to docs to match Dockerfile.prod's expectations
10 post-7 Frontend leaked literal __NEXT_PUBLIC_API_URL__ into URLs because entrypoint.sh sed -i failed silently under readOnlyRootFilesystem: true Removed readOnlyRootFilesystem from frontend k8s manifest; documented the trade-off inline
11 7 manifests/ subdir treated as a module by 4 different scanners (verify, lock, compat, validate-lock CI) Added skip in all 6 places — but missed split-and-mirror.sh which broke pipeline 411
12 7 JSON Patch add /metadata/labels/ring failed when base manifest had no metadata.labels block Switched to strategic merge patch via ring-labels.yaml
13 7 nginx:alpine chown failed under PSA restricted + readOnlyRootFilesystem Switched to nginxinc/nginx-unprivileged:alpine (port 8080, no chown step)
14 8 split-and-mirror.sh missing manifests/ skip from #11 — broke main pipeline post-merge of MR !7 Added the same skip pattern to bring it in line with the other 5 scanners

Troubleshooting

MR fails with module-version-check

Cause: You changed code inside modules/<Name>/ but didn't bump module.json's version field. Fix: Bump the version, run php artisan module:lock, commit, push.

MR fails with module-lock-validate

Cause: modules/manifest.lock.json doesn't match the filesystem state of one or more modules. Fix: Run php artisan module:lock locally, commit the updated lock file.

Pipeline succeeds but no artifact appears in registry

Cause: The release-module job uploaded successfully but you're checking with the wrong URL/token. Fix: Visit https://gitlab.wexron.io/<project-path>/-/packages directly. If artifact is there, the issue is auth on your client side. If not, check the release-module job logs.

module:install fails with Fetch failed: HTTP 401

Cause: MODULE_REGISTRY_TOKEN env var is missing, expired, or has the wrong scopes. Fix: Generate a new personal access token with read_package_registry, set it as the env var, retry.

module:install fails with SHA256 mismatch

Cause: Artifact in the registry was tampered with, or there's a transport corruption (rare). Fix: Re-run the release pipeline on the mirror repo. Worst case, delete the artifact from the registry and re-tag.

module:upgrade swap fails with rename(): cross-device link

Cause: Production cluster has /app/modules baked into the Docker image (lower overlay layer), and the swap is trying to use rename(). Fix: This shouldn't happen because swapInPlace() uses wipe-and-copy specifically to avoid this. If it does, check that the latest ModuleInstaller.php is deployed (look for recursiveCopy() method).

Module loader refuses a module with "Incompatible: platform X does not satisfy Y"

Cause: Module's compatibility.platform constraint doesn't match the current /VERSION. Fix: Either bump /VERSION (if the module's constraint is correct), or relax the module's constraint, or downgrade the module to a version compatible with the current platform.

Module disappears after seemingly successful install

Cause: A dependent module is also incompatible, and the dependency graph cascade-failed both. Fix: Run php artisan module:compat to find all incompatibilities at once. Fix the root cause (usually a single module with the wrong constraint).

Pipeline's release-cli job panics on ARM64

Cause: Pre-fix template. The fix is in commit 93ab80d on feat/module-versioning. Fix: Make sure feat/module-versioning (or the merged main) is what the mirror's .gitlab-ci.yml references.

Future Work

Item Why Effort
PVC-mounted /app/modules True atomic swap, eliminates overlay EXDEV workaround Small (k8s manifest change)
Widen module_versions.content_hash to varchar(80) Store the sha256: prefix natively, no stripping Migration + 2-line code change
Multi-version concurrent install Tenants on different versions of the same module Significant — needs per-tenant version pinning, namespace isolation
Public module registry Third-party developers publish modules to a public catalog Significant — auth, moderation, trust layer
module:diff old@new Show what changed between two versions of a module Small — git log between tags
Auto-module:lock on commit (pre-commit hook) Prevent the most common CI failure Tiny — git hook + symlink
Roll-forward: module:upgrade --all from lock file Apply a whole lock file in one command Medium — need transactional ordering
Migration auto-rollback on module:rollback Reverse schema changes when rolling back Hard — Laravel migrations aren't always reversible