K3s Migration Roadmap¶

Borealis will migrate the Engine runtime from Docker Compose into single-node K3s through staged bridge rollout. The existing Engine deployment command shape stays intact while each workload moves only after explicit validation and one-way retirement planning for the old runtime equivalent.

Goal¶

Migrate Borealis Engine from Docker Compose into single-node K3s through staged bridge rollout. Preserve existing Engine.sh --network-mode <public|local> deploy <prod|dev> command shape and idempotent deployment behavior.

Principles¶

• Keep Engine.sh deploy idempotent: reconcile, do not recreate healthy K3s state.
• Start with single-node non-HA K3s.
• Keep Compose Engine authoritative until each workload reaches explicit cutover.
• Introduce borealis-operator as only runtime K3s writer; Engine.sh remains the deploy-time bootstrap/reconcile writer.
• Keep api-backend, job-scheduler, and migrated workloads Kubernetes-blind or operator-API-only.
• Use allowlisted Borealis verbs, never raw YAML/kubectl from runtime services.
• Do not rotate secrets, delete PVCs, or teardown cluster during normal deploy.
• Use Longhorn as the Borealis K3s PVC/persistent volume backend for workloads that need durable container storage.
• Propagate the Engine host timezone into Borealis-managed Compose containers and K3s pods.

Stage 1: K3s Baseline¶

• Add K3s detection/install/reconcile phase inside existing Engine.sh deploy.
  • Install K3s only when missing.
  • Verify K3s service, kubeconfig, node readiness, container runtime.
  • Disable bundled public ingress features until Borealis owns cutover path.
  • Create borealis namespace.
  • Add managed labels/annotations for ownership and config hashes.
• Validation:
  • Run deploy twice back to back and confirm no cluster teardown or pod churn.
  • Confirm existing Compose Engine remains healthy.
  • Confirm K3s API not publicly exposed.

Stage 2: borealis-operator Bridge¶

• Add borealis-operator workload in K3s.
  • Replace future site-worker-orchestrator role.
  • Expose internal HMAC-authenticated Borealis API.
  • Grant namespace-scoped RBAC only.
  • Support read-only status verbs first.
• Allowed verbs:
  • GetClusterSummary
  • ListWorkloads
  • GetWorkloadStatus
  • ListSiteWorkers
• Blocked verbs:
  • raw YAML apply
  • arbitrary pod spec
  • arbitrary image/env/volume/service account
  • hostPath or privileged pod creation outside fixed allowlist
• Validation:
  • Compose api-backend can query operator status.
  • Operator rejects unauthenticated and unsupported requests.
  • Operator RBAC cannot mutate resources outside Borealis namespace.

Stage 3: Workload Rollout Interface¶

• Add restricted operator lifecycle verbs.
  • RolloutKnownWorkload(service_key, image_ref)
  • RestartKnownWorkload(service_key)
  • ScaleKnownWorkload(service_key, replicas)
  • LaunchSiteWorker(site_id, worker_guid, image_ref, resource_profile)
  • RetireSiteWorker(worker_guid, reason)
• Add image allowlist and digest/hash validation.
• Keep build work in Engine.sh, not runtime services.
• Validation:
  • Known service rollout succeeds with valid image.
  • Unknown service/image is rejected.
  • Failed rollout leaves previous healthy workload serving.

Stage 4: First Low-Risk Workloads¶

• Move webui-frontend dev workload into K3s bridge mode.
  • Preserve dev HMR path where practical.
  • Keep Compose Traefik/API authoritative.
• Move remote-desktop-guacd into K3s bridge mode.
  • Keep ClusterIP-only exposure.
  • Preserve existing VNC/Guacamole API behavior.
• Add scoped service-action bridge reconciliation.
  • Reconcile K3s bridge workloads after webui-frontend rebuilds.
  • Reconcile K3s bridge workloads after remote-desktop-guacd rebuilds.
  • Keep Compose routing authoritative while refreshing K3s bridge images.
• Fix DEV-mode Vite dependency cache mounts.
  • Mount node_modules/.vite as writable memory-backed cache for the transitional Compose WebUI path before Stage 6 retirement.
  • Mount node_modules/.vite as writable memory-backed cache for K3s WebUI bridge pods.
  • Keep WebUI runtime source mounts read-only.
• Fix DEV-mode WebUI runtime source staging.
  • Sync staged WebUI source into Engine/Services/webui-frontend/data/web-interface/ on every deploy dev.
  • Sync staged WebUI source during --service webui-frontend rebuild dev.
  • Keep production deploys on seed-if-missing behavior unless BOREALIS_REFRESH_WEBUI_RUNTIME_SOURCE=1.
• Validation:
  • WebUI dev path works.
  • Guacd readiness passes.
  • Existing Compose path remained available until Stage 6 WebUI cutover removed it.
  • Scoped WebUI and guacd rebuilds refresh bridge workloads without full deploy.
  • Unchanged bridge pod UIDs stay stable during scoped bridge reconciliation.
  • DEV Vite dependency modules return JavaScript MIME through public Traefik after scoped WebUI rebuild.
  • DEV bootstrap runtime no longer opens root /socket.io during normal page load or operator-presence sync.
  • /api/server/timezones reaches authenticated Go backend route instead of returning an API 404.
  • DEV HMR source sync verified after next operator redeploy.

Stage 5: Site Worker Migration¶

• Move dynamic site workers from Docker launch to operator-managed K3s pods.
  • Add job-scheduler lifecycle mode switch for bridge validation before K3s site-worker ownership became default.
  • Route K3s site-worker launch/retire/list operations through borealis-operator allowlisted verbs.
  • Preserve one active worker per site in v1 through scheduler reconciliation.
  • Preserve worker idle TTL and stale worker retirement in the K3s bridge path.
  • Preserve non-root UID/GID, dropped capabilities, read-only root, memory-backed /tmp, fixed resource requests/limits, and no ServiceAccount token.
  • Retire legacy Docker site-worker containers through site-worker-orchestrator after K3s worker listing succeeds.
  • Retire terminal K3s site-worker pods so failed pods do not block replacement workers.
  • Restart K3s site-worker containers on nonzero exit while preserving idle TTL clean exits.
  • Treat persistent transient heartbeat deadlocks as skipped heartbeat cycles instead of fatal worker exits.
  • Retry site-worker database registration and heartbeat setup during transient PostgreSQL startup/unavailable windows.
  • Preserve Agent Socket.IO registration after live redeploy validation.
  • Add Agent stale connected Socket.IO detection so worker restarts do not leave Agents stuck on dead control channels.
  • Name new K3s site-worker pods with deterministic sanitized site slugs for operator-readable Kubernetes and Server Info views.
  • Enforce unique, bounded site-worker slugs on site create and rename so K3s worker names do not need UUID suffixes.
  • Keep K3s worker GUIDs deterministic per site so Agent Socket.IO route URLs do not churn across redeploys.
• Replace Traefik route-file behavior with a controlled host-loopback bridge route while PostgreSQL stays localhost-only.
  • Keep route files owned by job-scheduler.
  • Keep worker listeners bound to 127.0.0.1 inside host-network K3s pods.
  • Prune terminal route rows and orphaned site-worker-*.yml files during scheduler reconciliation.
  • Replace temporary host-loopback worker bridge with per-worker ClusterIP routing after API/PostgreSQL cutover.
    • borealis-operator creates one fixed-template ClusterIP Service per site worker.
    • K3s site-worker pods bind on pod networking instead of host loopback.
    • job-scheduler route rows and Traefik route files point at the worker Service ClusterIP or DNS name.
    • Scheduler reconciliation retires old host-loopback worker pods so demand reconciliation can relaunch ClusterIP-backed workers.
• Validation:
  • Focused operator/scheduler unit tests pass.
  • Static Engine.sh and Python worker checks pass.
  • Agents connect to site worker through existing enrollment/token flow.
  • Remote shell works through K3s site workers.
  • File management works through K3s site workers.
  • Process management works through K3s site workers.
  • Service management works through K3s site workers.
  • Connected Devices column reflects K3s worker registrations.
  • Add Metrics Server-backed K3s site-worker CPU/RAM plumbing through borealis-operator.
  • /api/server/workers returns metrics.k8s.io CPU/RAM payloads for every active K3s site worker.
  • K3s resource metrics appear in Sites and Server Info.
  • Stale workers retire cleanly.
  • Redeploy validates K3s site workers survive PostgreSQL pod rollout/restart without disconnecting the fleet.
  • Focused Go tests confirm new K3s site-worker pods are not host-networked and route through a per-worker ClusterIP Service.
  • Live redeploy validates existing host-loopback site-worker pods retire and relaunch through ClusterIP Services without manual Agent service restart.
  • Agent Socket.IO reconnect hardening is covered by Agent runtime tests/docs, and Device Inventory labels API-online/socket-missing devices as Reconnecting.

Stage 6: Production WebUI Cutover¶

• Make K3s webui-frontend authoritative.
  • Add production BOREALIS_WEBUI_TRAFFIC_OWNER=k3s route owner that renders Compose Traefik's core WebUI upstream to the K3s webui-frontend ClusterIP.
  • Route dev mode through the K3s webui-frontend workload as well.
  • Keep Compose Traefik as HTTP/HTTPS, certificate, and watched dynamic-route owner.
  • Remove Compose webui-frontend from compose.yaml.
  • Reject BOREALIS_WEBUI_TRAFFIC_OWNER=docker-compose in Engine.sh.
  • Remove stale borealis-engine-webui-frontend container during deploy.
• Validation:
  • Public network mode serves WebUI through K3s route owner.
  • Local network mode live validation is waived for this PR and remains tracked by Technical Debt #374 because active public-mode agents may reject the local CA trust plane during live testing.
  • Static assets and SPA route handling work in public network mode.
  • Guacamole dynamic Traefik route still works after WebUI cutover.
  • Repeated deploys do not restart unchanged WebUI pod.
  • Compose webui-frontend container is removed after a public-mode redeploy.

Stage 7: API Backend Cutover¶

• Move api-backend into K3s as one replica.
  • Add non-authoritative K3s api-backend bridge Deployment.
  • Bind initial bridge API to host loopback port 5001 while Compose API owned 127.0.0.1:5000.
  • Mirror generated runtime env into borealis-api-backend-runtime-env.
  • Disable API-owned background loops in the bridge pod with BOREALIS_API_BACKGROUND_LOOPS=0 before traffic cutover.
  • Use Recreate rollout strategy for the host-network bridge so single-node K3s does not deadlock on fixed port 5001.
  • Add one-shot K3s api-backend shadow DB validator Job that targets imported K3s PostgreSQL data without moving public API traffic.
  • Make K3s api-backend authoritative for API, Socket.IO, and internal caller traffic on port 5001, initially through host loopback.
  • Enable API-owned background loops in the K3s pod after Compose API retirement.
  • Remove Compose api-backend from compose.yaml and retire stale borealis-engine-api-backend containers during deploy.
  • Route Traefik's /api and /socket.io upstream to the configured K3s API target.
  • Recycle existing K3s site-worker pods once when BOREALIS_INTERNAL_API_BASE_URL changes from Compose API 5000 to K3s API 5001.
  • Add a K3s api-backend ClusterIP Service and render non-API internal callers plus Traefik to api-backend.borealis.svc.cluster.local:5001.
  • Move the authoritative K3s api-backend pod off host networking; it listens on pod networking behind the ClusterIP Service while in-pod health/self-calls use pod loopback.
  • Reconcile K3s Traefik immediately after API rollout so public API and Socket.IO routes stop depending on host loopback.
  • Preserve Aegis bootstrap/unlock behavior.
  • Preserve internal API token behavior.
  • Preserve logs/secrets/cache path contracts through fixed hostPath bridge mounts.
  • Re-evaluate fixed hostPath bridge mounts for Longhorn PVC/Secret/ConfigMap mapping where durable pod-local storage is required.
    • Remove broad api-backend service-root hostPath mounts from API, shadow DB validator, and PostgreSQL schema Job manifests; keep only reviewed subpath mounts or scratch roots.
    • Live redeploy validates api-backend Deployment uses an emptyDir service root with only API cache/config/logs/secrets hostPath subpaths.
    • Final API storage authority split remains tracked by Technical Debt #379 because v1 intentionally keeps API cache/config/logs/secrets host-visible for Backup/Restore, logs, Aegis/Engine secret files, and scheduler/site-worker read contracts.
• Validation:
  • Bridge pod rollout passes.
  • Pre-ClusterIP bridge curl -fsS http://127.0.0.1:5001/health passed.
  • Engine.sh --network-mode public|local --service api-backend shadow-db-validate prod validates Go API bootstrap state against K3s PostgreSQL shadow data.
  • /health passes after traffic-owner deploy.
  • Public WebUI route returns HTTP 200 after K3s API cutover and Compose API retirement.
  • Repeated public-mode deploy keeps K3s API, scheduler, and WebUI pods ready while Compose api-backend, job-scheduler, and webui-frontend containers remain absent.
  • Login, Aegis unlock, enrollment, heartbeat, and API routes pass.
  • Remote shell, file management, process management, service management, registry, and remote desktop pass after API traffic-owner cutover.
  • Backup export passes after API traffic-owner cutover.
  • All agents appear connected to their site workers after API traffic-owner cutover.
  • Realtime SSE works with one replica.
  • Compose API removal plan is documented: K3s API owns traffic before stale Compose API container removal; Compose PostgreSQL remains DB owner until Stage 9.
  • Live redeploy validates api-backend Service DNS routing, non-host-network pod state, Service health, and public API/Socket.IO routing after ClusterIP cutover.

Stage 8: Scheduler Cutover¶

• Move job-scheduler into K3s as one replica.
  • Add fixed K3s job-scheduler Deployment with Recreate rollout strategy.
  • Remove Compose job-scheduler service and retire stale borealis-engine-job-scheduler containers during deploy.
  • Use borealis-operator for K3s workload/site-worker lifecycle where the scheduler can complete the queued action safely.
  • Preserve Postgres work leases and queue behavior through the existing scheduler manager and work-item tables.
  • Keep scheduler Kubernetes-blind: no ServiceAccount token, no kubeconfig, no Docker socket.
  • Keep temporary host-loopback access to K3s API and Compose PostgreSQL until PostgreSQL/API Service cutover cleanup.
  • Move K3s job-scheduler off host networking and target internal API calls through api-backend.borealis.svc.cluster.local:5001.
• Validation:
  • Scheduled job tick creates expected runs after live redeploy.
  • Service actions route through operator for K3s-owned restart paths in focused Go tests.
  • No duplicate scheduler loops in deployment model: Compose service removed, stale container retired before K3s rollout, and K3s Deployment uses Recreate.
  • Live redeploy confirms exactly one K3s scheduler pod and no Compose borealis-engine-job-scheduler container.
  • Scoped job-scheduler rebuild applies the retired-orchestrator cleanup and fresh logs stay quiet for stale site-worker-orchestrator socket probes.
  • Live redeploy confirms job-scheduler is not host-networked and BOREALIS_INTERNAL_API_BASE_URL uses the K3s API Service DNS.

Stage 9: PostgreSQL Cutover¶

• Move postgres-db into K3s StatefulSet with one Longhorn-backed PVC.
  • Add Engine.sh deploy Longhorn reconcile phase before the first PVC-backed workload cutover.
  • Install/reconcile Longhorn from pinned manifest version v1.12.0 if it is enabled and not already present.
  • Reconcile Longhorn host iSCSI prerequisites without deleting existing Longhorn resources or PVCs.
  • Keep K3s local-path default unchanged for now; future Borealis PVC manifests should request BOREALIS_K3S_PVC_STORAGE_CLASS explicitly.
  • Override Longhorn manifest default-StorageClass annotation so Longhorn stays explicit-use only.
  • Add Borealis-owned single-node Longhorn StorageClass for fresh PVCs while preserving existing PostgreSQL PVC StorageClass on redeploy.
  • Add non-authoritative K3s postgres-db shadow StatefulSet with one Longhorn-backed PVC.
  • Set PostgreSQL StatefulSet manifests to use the Longhorn StorageClass.
  • Keep K3s PostgreSQL ClusterIP-only.
  • Preserve profile-managed PostgreSQL startup settings in the K3s StatefulSet manifest.
  • Use PGDATA=/var/lib/postgresql/data/pgdata so Longhorn filesystem metadata does not block initdb.
  • Add retry-safe PostgreSQL PVC ownership init path with narrow init-container ownership capabilities.
  • Add controlled final cutover path.
    • Quiesce K3s API backend, K3s job-scheduler, and K3s site-worker writers before the final Compose snapshot.
    • Import the final Compose PostgreSQL logical snapshot into K3s PostgreSQL before moving runtime DB URLs.
    • Re-render runtime env so API backend, job-scheduler, and site-worker pods use postgres-db.borealis.svc.
    • Run a K3s schema initializer Job after PostgreSQL becomes traffic owner.
    • Retire stale Compose borealis-engine-postgres-db containers and remove postgres-db from Compose.
  • Preserve backup/export semantics.
    • Keep Traefik ACME storage strict and Backup/Restore-readable while allowing Traefik restart/renewal.
    • Add postgres-db shadow-import validation path that restores a logical Compose DB snapshot into the K3s shadow DB without changing live traffic ownership.
    • Operator completed encrypted WebUI configuration export before cutover.
  • No HA/replication in v1 StatefulSet.
  • No automatic PVC deletion.
• Validation:
  • Static Engine.sh syntax passes after Longhorn reconcile addition.
  • First live deploy reconciles Longhorn once.
  • Second live deploy after explicit-only StorageClass guard does not churn Longhorn pods.
  • Longhorn system pods are ready and StorageClass is available.
  • Live validation found upstream Longhorn default annotation drift; Engine reconcile now clears it after manifest apply.
  • Longhorn volume attaches, mounts, detaches, and reattaches cleanly on the single-node Engine host.
  • Fresh Longhorn StorageClass policy creates borealis-longhorn with one replica without mutating existing postgres-data-postgres-db-0.
  • Live deploy creates K3s postgres-db-0 and postgres-data-postgres-db-0 without changing Compose DB traffic owner.
  • K3s PostgreSQL readiness passes on the shadow StatefulSet.
  • Logical backup/restore tested through pg_dump from Compose PostgreSQL and pg_restore into K3s shadow PostgreSQL.
  • Existing Engine data imports cleanly into K3s shadow PostgreSQL.
  • Final cutover imports existing Engine data into authoritative K3s PostgreSQL.
  • API backend, job-scheduler, and site-worker pods resolve BOREALIS_DATABASE_URL to postgres-db.borealis.svc.
  • K3s PostgreSQL PVC is bound to Longhorn.
  • Second live redeploy keeps PostgreSQL on K3s and does not rerun Compose snapshot/import.
  • Encrypted WebUI Backup/Restore analyze path is documented and covered by Go tests without mutating K3s PostgreSQL.
  • Encrypted WebUI Backup/Restore Import live validation passed on a clean/disposable K3s target after rollback, redeploy, restore, Agent reconnect, site-worker reconnect, Remote Desktop smoke, and basic 7-Zip assembly execution on LAB-OPERATOR-01.
  • DB profile settings preserved or documented.

Stage 10: WireGuard Cutover¶

• Move wireguard-tunnel into K3s pinned host-network pod.
  • Preserve /dev/net/tun, NET_ADMIN, NET_RAW, control socket boundary.
  • Preserve firewall chains and peer /32 policy.
  • Route queued WireGuard reconcile/recover actions from K3s job-scheduler through the mounted control socket instead of the Docker helper bridge.
• Validation:
  • Existing agents reconnect.
    • SQL worker audit shows K3s site-worker socket counts match online device counts after redeploy.
    • WireGuard status shows fresh peer handshakes from the K3s host-network pod.
  • Remote shell/desktop over tunnel works.
    • API and K3s site-worker pods resolve and connect to remote-desktop-guacd.borealis.svc.cluster.local:4822.
    • Operator browser smoke after guacd traffic-owner env propagation.
    • WebUI login, Server Info, Device Inventory, Remote Desktop, Remote Shell, File Management, Process Management, and Service Management pass after WireGuard and guacd cutover.
  • Focused scheduler tests confirm K3s WireGuard reconcile sends only predefined reconcile command to the control socket and does not require the helper bridge.
  • Live Server Info Recover Listener action validates queued scheduler-to-control-socket path after redeploy.
  • Quarantine/revocation live peer-removal validation is waived for this PR and tracked by Technical Debt #381; focused tests cover containment access denial and runtime cleanup calls.

Stage 11: Compose Retirement¶

• Remove Compose ownership only after all K3s-owned services are stable.
  • Compose webui-frontend retired.
  • Compose api-backend retired.
  • Compose job-scheduler retired.
  • Compose postgres-db retired.
  • Compose wireguard-tunnel retired.
  • Compose remote-desktop-guacd retired.
  • Move traefik-edge into K3s as the public edge owner.
  • Preserve HTTP/HTTPS host-network ports, ACME/local CA state, and watched dynamic route files.
  • Route traefik-edge reload through K3s job-scheduler and borealis-operator, not a Docker helper.
  • Retire site-worker-orchestrator as a long-running Compose service.
  • Leave compose.yaml as an empty retired manifest.
  • Move Server Overview service rows for K3s-owned workloads off retired Compose container lookups and onto borealis-operator workload status.
  • Expose WebUI restart as an operator-routed K3s action so simple WebUI pod restarts no longer need the helper bridge.
  • Keep WebUI rebuilds CLI-only through Engine.sh --service webui-frontend rebuild prod|dev; reject queued runtime WebUI rebuild helper actions.
  • Route K3s API, WebUI, PostgreSQL, and guacd restarts through borealis-operator without Docker-helper fallback.
  • Route K3s PostgreSQL restart service actions through borealis-operator instead of the Docker helper bridge.
  • Route K3s WireGuard reconcile service actions through the scheduler-mounted control socket instead of the Docker helper bridge.
  • Remove the site-worker-orchestrator service-action helper path after Traefik edge cutover.
  • Remove unused packaged site-worker-orchestrator healthcheck wrapper from the K3s job-scheduler image.
  • Remove remaining Compose bridge service rows after Traefik and orchestrator retirement.
  • Stop Docker metadata reads for K3s site-worker rows once operator metrics are present.
  • Retire Compose docker-proxy after K3s worker metrics and scheduler service snapshots became authoritative.
  • Remove Docker CLI, Docker Compose plugin, and retired compose.yaml from the K3s job-scheduler image.
  • Make empty, auto, and unknown site-worker lifecycle modes resolve to K3s instead of falling back to retired Docker helpers.
  • Reject retired site-worker-orchestrator roles from the K3s job-scheduler entrypoint.
  • Remove BOREALIS_ALLOW_LEGACY_DOCKER_SITE_WORKERS; explicit Docker site-worker lifecycle modes now resolve to K3s.
  • Remove retired site-worker-orchestrator socket env generation and K3s job-scheduler hostPath mount.
  • Reject retired site-worker-orchestrator roles from the api-backend container entrypoint.
  • Remove retired site-worker-orchestrator resource-limit env generation from Engine deploy env.
  • Remove retired Go site_worker_orchestrator source/tests and scheduler Docker lifecycle fallback paths.
  • Remove retired site-worker-orchestrator image alias compatibility logic from deploy manifest generation.
  • Remove unused Docker socket and service-action-helper env generation after Docker helper retirement.
  • Remove unreachable Docker Compose service reconciliation branches from full Engine deploy.
  • Remove retired Compose service-action rebuild fallback, old Compose schema initializer, and unused docker compose up helper.
  • Narrow remaining legacy Docker status code to PostgreSQL snapshot support and shared retired-container cleanup.
  • Stop Engine deploy from installing/requiring Docker Compose plugin and retarget legacy dump import helper to K3s PostgreSQL.
  • Clean stale post-retirement docs that still described Compose as deployment/runtime tooling.
  • Rename operator RBAC from stale readonly naming to lifecycle-scoped controller naming and add old RBAC cleanup.
• Keep migration and recovery docs until stable release.
  • K3s Backup/Restore recovery notes document active K3s PostgreSQL targeting, safe Analyze validation, destructive Import behavior, and post-import redeploy expectations.
  • K3s Backup/Restore docs include a clean/disposable restore-target validation runbook and post-import smoke checks.
• Update Docs/Reference/Core Runtimes/Stack_Breakdown.md, engine-runtime.md, security-whitepaper.md, SBOM if dependencies changed.
• Validation:
  • Fresh install path works.
  • Redeploy path works.
  • Backup/restore path is validated for this PR with export/analyze/test coverage plus destructive clean-target Import live proof on a disposable DEV Engine.
    • Backup export works against authoritative K3s API/PostgreSQL.
    • Backup analyze path is documented and covered by focused Go tests without data mutation.
    • Backup Import live proof on a clean K3s restore target passed after DEV rollback, redeploy, restore, restored Agent reconnect, site-worker reconnect, Remote Desktop, remote tooling, and assembly execution validation.
  • Operator WebUI smoke passes after Compose retirement: login, Server Info, Device Inventory, Remote Desktop, Remote Shell, Files, Process Management, Service Management, Registry, and Backup Export.
  • Narrow Engine tests pass.
  • Compose policy confirms retired services stay out of compose.yaml.
  • Live Docker check confirms retired Compose containers are absent.
  • docker compose config --services returns no service names.
  • Server Overview unit tests confirm retired workloads render as K3s rows and legacy Compose snapshots are ignored.
  • Server action tests confirm WebUI restart uses the operator path and WebUI rebuild is rejected from queued runtime helper paths.
  • Scheduler tests confirm Traefik reload uses the operator path.
  • Orchestrator tests confirm the helper path is retired.
  • Engine.sh --network-mode public --service webui-frontend restart prod rolls the K3s WebUI Deployment without restoring the retired Compose WebUI container.
  • Live deploy table reports K3s Traefik edge ready and Docker Compose retired.
  • Live K3s rollout confirms deployment/traefik-edge healthy.
  • Live redeploy applies borealis-operator-controller RBAC and removes legacy borealis-operator-readonly Role/RoleBinding.
  • Live Docker check confirms stale borealis-engine-traefik-edge and borealis-engine-site-worker-orchestrator containers are absent.
  • Docker cache corruption during required image restore is recoverable through builder-cache prune plus no-cache rebuild.

Open Risks¶

• borealis-operator RBAC must stay narrower than Docker socket power.
  • Live kubectl auth can-i audit confirms operator ServiceAccount is namespace-scoped, cannot mutate Secrets, cannot read Nodes, cannot read kube-system pods, and does not have wildcard Borealis namespace access.
  • Operator RBAC still grants namespace-wide pod and Service create/delete for dynamic site workers; accepted for this PR as narrower than Docker socket power, with narrower lifecycle primitive tracked by Technical Debt #380.
• Host networking must be minimized except WireGuard/edge needs.
  • Live host-network audit identifies required v1 host-network pods: traefik-edge for TCP 80/443/health and wireguard-tunnel for UDP 30000 plus /dev/net/tun.
  • Live host-network audit confirms borealis-operator, postgres-db, remote-desktop-guacd, webui-frontend, and site-worker-* are not host-networked.
  • Code and live redeploy move K3s site-worker-* pods off host networking and onto per-worker ClusterIP Services.
  • Code moves K3s api-backend and job-scheduler off host networking and routes API consumers through the api-backend ClusterIP Service DNS.
  • Live redeploy validates api-backend and job-scheduler are not host-networked, api-backend Service endpoints are ready, Traefik routes API traffic through Service DNS, and scheduler/site-worker runtime env uses the Service URL.
• Runtime service actions must not become raw Kubernetes mutation API.
  • Code audit confirms operator command surface is fixed to named verbs in borealisOperatorAllowedVerbs and executeCommand; unsupported verbs return unsupported Borealis operator verb.
  • Code audit confirms scheduler service actions route supported K3s restart/reload actions through RestartKnownWorkload and WireGuard reconcile through the fixed control-socket command, not raw YAML or kubectl.
  • Retired orchestrator source and Docker lifecycle fallback are removed; remaining references are fail-closed process-role guards, tests, retired-container cleanup, or PostgreSQL shadow-import migration support.
• K3s secrets must not replace Aegis security model.
  • Security whitepaper states K3s Secret encryption at rest is enabled for Kubernetes objects while Aegis remains the Borealis model for protected operator, credential, token, and signing material.
  • Backup/Restore code keeps engine.aegis_cipher_state inside the encrypted payload, derives backup keys from the Aegis Cipher, and clears the in-memory Aegis key after restore.
  • Continue treating generated K3s runtime-env Secrets as deployment plumbing, not durable protected-secret authority.
    • Code audit confirms generated K3s runtime Secrets are rendered from deploy env and consumed through envFrom.secretRef for pods/jobs; durable API secrets remain mounted Engine paths and Aegis/database-managed payloads.
• Future PostgreSQL rollouts must avoid unnecessary site-worker churn and agent disconnects.
  • Code audit confirms normal deploy only recycles K3s site-worker pods for API base URL changes, site-worker runtime Secret hash changes, or timezone mismatch; repeated unchanged deploys should not recycle workers.
  • Code audit confirms PostgreSQL StatefulSet storage/profile hash changes do not directly recycle site workers unless shared runtime env values that workers consume actually change.
• Agents that already hold stale connected Socket.IO state may require Agent release rollout or manual service restart before the new reconnect logic is active.
  • Current Agent code waits for site-worker registration acknowledgement, reconnects after 60 seconds of read-idle socket inactivity, and lets the watchdog restart stale socket states after 90 seconds. WebUI labels API-online/socket-missing devices as Reconnecting while the management socket reattaches.
• Longhorn adds CSI/storage-manager dependencies that must be reconciled idempotently before PVC workloads depend on it.
  • Live audit confirms Longhorn StorageClass exists as explicit-use only, postgres-data-postgres-db-0 is Bound to StorageClass longhorn, and all Longhorn pods report Running.
  • Live audit found single-node Longhorn volume robustness is degraded because upstream StorageClass provisioned numberOfReplicas: 3; tracked by Technical Debt #378.
  • Future fresh installs should use a Borealis-owned single-node Longhorn StorageClass or equivalent policy so v1 non-HA volumes do not start degraded.
  • Existing 3-replica Longhorn PVCs are preserved during normal deploy and require an explicit manual migration if replica policy should change later.
• Stateful data migration must have reversible checkpoints and no automatic Longhorn volume/PVC deletion during normal deploy.
  • Code audit found no deploy-time PVC delete path for PostgreSQL or Longhorn volumes.
  • Server-side dry-run confirms existing longhorn StorageClass replica parameters are immutable, so existing PVC replica-count repair needs a deliberate migration/rebuild workflow rather than normal deploy reconciliation.
• Full Engine WebUI unit lane needs K3s-compatible runtime test-cache staging; accepted as deferred for this PR and tracked by Technical Debt #377.