Service Validation Closure Design¶

Date: 2026-09-22
Status: Approved design
Scope: The 229 HomelabOS roles in 2026-09-09-service-validation-inventory.json

Context¶

The validation campaign currently reports 229 roles as 129 family_validated_with_recorded_limits, seven bounded results, and 93 not run. That rollup is too permissive for the intended pipeline contract. family_validated is partly inferred from prose, ten of the 129 records are explicitly only partially sampled, and a conservative audit flags 25 of the 129 for possible startup or steady-state evidence gaps. A role can therefore appear validated without proving every result an operator needs.

The intended product is stricter: every compatible role is integrated through the real HomelabOS deployment path, functionally tested, documented, and measured for base resource usage during fresh installation and settled steady state. A terminal exception must be concrete and evidenced, not an ambiguous bounded label.

The broader 1,442-entry application discovery tracker and the 242 Authentik integration guides are inputs to future catalog expansion, not the denominator for this closure campaign. This design closes the current 229-role catalog.

Goals¶

Give complete one machine-enforced meaning across all 229 roles.
Fully validate every supportable role and repair role/image defects before considering exclusion.
Record operator-useful cold-install and settled steady-state resource figures.
Use the AMD64 droplet aggressively without accepting misleading measurements.
Reuse sound existing evidence and rerun only missing, stale, contradictory, or invalid dimensions.
Generate current role documentation and catalog reports from structured evidence.
Preserve immutable evidence, retained service data, unrelated workloads, and private secrets.
End with every role either complete or assigned a specific evidenced terminal disposition and retest requirement.

Non-goals¶

Expanding this campaign from 229 roles to the 1,442 missing-app candidate tracker.
Claiming production capacity from one synthetic fixture or idle observation.
Requiring paid native SSO licenses. License-gated native SSO is a documented limitation, not a reason to abandon otherwise complete role validation.
Making active-load resource trials, every historical upgrade path, hardware integration, or third-party production account mandatory for base completion. These remain separate optional or blocked dimensions.
Deleting retained service data, pruning the host globally, or rewriting historical evidence.

Terminology and independent statuses¶

Each role has a catalog disposition and a completion state. They are independent.

Catalog disposition¶

Disposition	Meaning
`eligible`	The role can be supported and must satisfy the complete contract.
`eligible_with_license_limitation`	The role is supportable, but a native Authentik feature requires a paid edition. All other required dimensions still apply.
`repairable`	A missing tag, stale image, role defect, invalid Compose file, or similar defect must be repaired before validation can finish.
`environment_blocked`	A concrete external account, hardware device, network peer, or host capability unavailable to this campaign prevents a required result. The receipt identifies the needed environment.
`incompatible`	Supported deployment on the available AMD64 Linux target is not technically possible after documented source and replacement research.

AMD64-only images are eligible on the droplet. Missing images, broken tags, floating tags, invalid YAML, old topology, and role bugs are repair work, not incompatibility.

Completion state¶

State	Meaning
`not_started`	No accepted execution evidence under the strict contract.
`in_progress`	One or more required dimensions are being produced or repaired.
`complete`	Every required dimension is accepted at the recorded source and image revisions.
`blocked`	A required dimension cannot run because of a typed external or environmental prerequisite.
`failed`	A reproducible application, role, or test result failed and remains unresolved.
`stale`	Previously accepted evidence no longer applies to the current role version, source revision, image digest, or contract version.

bounded is not a terminal completion state. Existing bounded results must become complete, blocked, failed, incompatible, or stale after adjudication.

Strict completion contract¶

A role becomes complete only when a versioned structured record proves every applicable requirement below. Human prose may explain results but cannot promote a role.

1. Identity and provenance¶

The record contains:

role slug and catalog identity;
HomelabOS source revision;
validation contract version;
upstream release or explicit immutable source reference;
every image reference and resolved digest;
host architecture, OS, kernel, Docker, and Compose versions;
deployed container and one-shot-job topology;
evidence directory and hashes for required receipts.

A source revision, role version, or image digest change marks affected dimensions stale unless an explicit compatibility rule proves that evidence remains applicable.

2. Fresh installation and readiness¶

The role must deploy through the real HomelabOS Ansible/systemd path using fresh owned application data. Evidence must show:

deployment exit status and duration;
migrations, one-shot jobs, health transitions, restarts, and OOM state;
first HTTP response separately from usable readiness;
expected status/content or API behavior at readiness;
a usable administrator;
a normal user denied administrator privileges where the product has roles;
one deterministic representative operation appropriate to the application.

An HTTP response alone is not readiness. A setup redirect without completing setup is not a pass.

3. Authentication and Authentik outcome¶

Every role records the integration mode independently from the result:

native OIDC or SAML;
LDAP;
Authentik proxy/forward-auth;
application-native accounts;
stateless/no-native-login application;
license-gated native SSO;
unsupported or unavailable authentication feature.

Applicable tests cover an allowed user, denied user, repeat login, application logout, local administrator recovery, first-user linking/provisioning, and separate API/mobile authentication behavior when browser SSO does not cover it. Proxy tests include bypass routes and spoofed identity headers. Access protection must not be described as native account SSO.

A role with license-gated native SSO may complete as eligible_with_license_limitation after the rest of the contract passes. An Authentik proxy gate is tested where it is technically safe and compatible with application APIs. The documentation states any remaining native login prompt.

4. Duplicate-instance isolation¶

A primary and a source_service duplicate run simultaneously unless the role has an accepted technical not_applicable reason. Evidence proves:

unique writable paths, database identity, secrets, Compose project, ports, scheduled jobs, and fixed names;
separate Authentik clients, callbacks, and identities where applicable;
reciprocal credentials, sessions, and private data do not cross instances;
a secondary-only change leaves the primary unchanged;
stopping the secondary leaves the primary healthy.

A host-wide singleton must document the specific conflict and receive a reviewed not_applicable result. Render-only evidence cannot satisfy this dimension.

5. Idempotence, lifecycle, persistence, and backup verification¶

Evidence proves:

the same configuration can be applied twice without rotating stable credentials, duplicating provider objects, or losing data;
source deployment/restart cycles preserve the representative operation and account state;
direct container recreation/restart preserves state;
a stopped peer does not break the remaining instance;
an isolated backup extraction or equivalent content verification preserves expected records and checksums.

Full-host disaster recovery and every upgrade path remain separate dimensions unless a role change itself requires migration proof.

6. Cold-install resource measurement¶

The cold-install window begins before the fresh deployment starts and ends only after usable readiness. Image pull time is measured separately so a cached pull does not hide installation cost. The accepted record includes:

sample interval and missed-sample/gap information;
deploy duration, container-start time, first response, and usable readiness;
maximum stack RAM and CPU across every app/database/worker/sidecar container;
host CPU convention and logical CPU count;
health transitions, restarts, exit codes, and OOM events;
container count and one-shot jobs;
resolved image logical size and persistent disk delta;
host load, pressure, available memory, swap activity, and concurrent campaign jobs.

A current-version measurement is required. Historical-version values remain visible but do not satisfy the current result.

7. Settled steady-state resource measurement¶

After readiness and representative setup, the stack settles for at least five minutes and is then sampled for at least ten minutes. The record includes:

RAM median, p95, and maximum per container and summed stack;
CPU mean, p95, and maximum per container and summed stack;
actual observation duration and sample count;
health, restart, exit, and OOM states;
persistent disk use and container count;
the exact fixture/data state;
host load, pressure, memory availability, swap activity, and concurrent jobs;
the shared Traefik/Authentik core reported separately and counted once in host summaries.

The result is an observed base range for the stated fixture, not a production minimum or arbitrary-user capacity claim.

Each complete role also publishes a base deployment planning envelope. Its default RAM figure is 125% of the larger of the observed cold-install maximum and settled steady-state maximum, rounded up to the next 64 MiB with a 128 MiB minimum. This is sampling headroom for deploying the tested fixture, not active-user capacity. Adaptive runtimes, upstream hard requirements, or a measured migration peak may require a documented higher override. The envelope also reports observed CPU, unique image bytes, fresh persistent-data bytes, container count, and shared-core cost; it does not invent an unmeasured CPU or disk limit.

8. Documentation and catalog publication¶

The role documentation contains one canonical generated facts block with:

validation date and exact tested version/digest/architecture;
deployment and first-admin steps;
authentication mode, Authentik result, manual configuration, and limitations;
duplicate support and required unique resources;
cold-install and settled steady-state figures;
container/image/storage costs and the calculated base deployment planning envelope;
licensing, hardware, external dependency, migration, and recovery limitations;
direct sanitized evidence links.

Older preliminary or pending sections that contradict accepted evidence are removed or clearly marked historical. The consolidated catalog row is generated from the same structured record.

Evidence model¶

Each role has one current summary record plus immutable step receipts. The summary references evidence rather than copying unverified prose.

Required receipt classes are:

preflight;
fresh_install;
readiness_and_admin;
representative_operation;
authentication;
duplicate_isolation or reviewed duplicate_not_applicable;
redeploy_and_lifecycle;
backup_verification;
cold_install_metrics;
steady_state_metrics;
documentation_projection.

Every receipt contains schema version, role/source/image identity, timestamps, target identity, command or worker identity, result, typed failure reason, evidence files, sanitizer result, and content hashes. Revisions are write-once. Retests produce a new receipt and supersede an older result without deleting it.

Private credentials, browser storage, tokens, certificates, and passwords remain under ignored run-private directories. Published receipts contain only identifiers, redacted command metadata, hashes, and non-secret results.

Pipeline architecture¶

Validation contract¶

A versioned JSON Schema defines dispositions, completion states, dimension states, limitations, measurement windows, evidence references, and required/not-applicable rules. The completion calculation is deterministic and fails closed on missing, unknown, stale, contradictory, or malformed fields.

Inventory auditor¶

The auditor reads the 229-role inventory, current source roles, evidence receipts, and role documentation. It produces:

the strict completion result for each role;
missing and stale dimensions;
contradictory evidence and documentation;
repair and retest queue items;
exact catalog rollups by disposition and completion state.

The initial audit is expected to reduce the current reported completion count. That is a correction, not a regression.

Capacity-aware scheduler¶

The scheduler persists queue and lease state so work can resume after interruption. Jobs request RAM, CPU, disk, port, fixture, browser, shared-state, and measurement-interference tokens. Admission is based on live capacity and declared or observed envelopes.

Known roles use accepted measurements plus safety headroom. Unknown roles run their first cold install alone; the observed envelope then controls later duplicate and lifecycle packing. A failed or expired worker releases its lease but leaves owned remote resources and evidence identifiable.

Role runner¶

The runner generalizes the existing droplet deployment helper and calls the real HomelabOS deployment path. Each job uses an isolated source/settings directory, synthetic credentials, Compose projects, application paths, browser context, and ownership manifest. Shared Traefik or Authentik mutation takes a shared-state lock; role-local deployments may run concurrently after the canary proves there are no settings or unit collisions.

Metrics collector and analyzer¶

The existing collect_metrics.py and analyze_metrics.py remain the measurement core. They gain host pressure, swap, disk delta, concurrent-job, contamination, and usable-readiness metadata. The collector continues to fail on Docker errors and record retries and sample gaps rather than turning failures into zero usage.

Documentation and report generator¶

The generator projects accepted structured records into the per-role facts block, consolidated validation catalog, and gap report. It never promotes status from prose. Generated output and source evidence must agree before a result is accepted.

Droplet capacity and safety¶

The live droplet has 32 vCPUs, 62.8 GiB RAM, no current swap, and 167.7 GB free disk. Docker retains 198.9 GB of images; that cache is valuable and remains intact. Only the shared HomelabOS core is currently running.

The default campaign envelope is:

no more than 28 schedulable vCPUs;
no more than 52 GiB admitted RAM;
at least 8 GiB available host memory reserved;
at least 40 GB free disk reserved;
retained application data and image cache preserved;
no global Docker/systemd/data prune.

Implementation adds a persistent 8 GiB swap file with vm.swappiness=1 as emergency protection. Swap does not increase schedulable memory. Any swap-in or swap-out during a resource window contaminates that measurement and queues a replacement.

Admission halts on OOM, unexpected restarts, reserve breach, low disk, excessive pressure, collector gaps, or unrelated-host degradation. The triggering evidence is retained and classified.

Concurrency and measurement interference¶

The scheduler has independent lanes:

image resolution, pull, render, and preflight: up to eight workers;
isolated role deployment: four initial workers, increased only after collision-free canaries;
fixture, authentication, and lifecycle work: four to eight workers within capacity tokens;
isolated browser contexts per role and instance;
repair and source research concurrent with runtime execution.

Measurements do not require a globally idle host. Low-noise unrelated functional, browser, analysis, and documentation jobs may continue. Pulls, builds, migrations, backups, and active workloads may overlap only while the measurement interference budget remains valid.

Each resource receipt records host CPU, Linux pressure stall information, available memory, swap activity, disk I/O, and concurrent campaign jobs. A window is accepted only while available memory stays at or above 8 GiB, free disk stays at or above 40 GB, swap-in and swap-out deltas remain zero, no OOM or unexpected restart occurs, total host CPU stays at or below 85% over every ten-second interval, /proc/pressure/cpu some avg10 stays at or below 10.00, /proc/pressure/memory full avg10 remains 0.00, and /proc/pressure/io full avg10 stays at or below 2.00. Crossing any threshold labels the window contaminated; the role remains incomplete for that dimension and the scheduler automatically queues a replacement. Changing a threshold requires a contract-version change so results remain comparable. Per-container accounting is necessary but not sufficient when host contention delays readiness or changes cache behavior.

Gap-closing queues¶

Queue 1: Contract audit and reclassification¶

Install the strict schema and auditor, then reclassify all 229 roles before accepting further completion claims. Publish the exact starting counts and every missing gate.

Queue 2: Existing evidence backfill¶

Review all 129 roles currently labeled family-validated. Start with the 25 roles conservatively flagged for possible startup or idle evidence gaps. Reuse accepted evidence and rerun only missing or stale dimensions unless a source/version change invalidates more of the record.

Queue 3: Bounded-result adjudication¶

Process the seven bounded results. Repair and retest where possible. Otherwise record a concrete blocked, failed, incompatible, or stale result with the required retest environment. No role remains terminally bounded.

Queue 4: AMD64-admitted unrun roles¶

Run the complete contract for the remaining 69 AMD64-admitted roles. Lightweight families pack densely; heavy and unknown families start conservatively. Their role fixes feed back into the source branch before final evidence is accepted.

Queue 5: Image and role repair¶

Research and repair the 19 image-issue roles concurrently with runtime work. Replacement images must be supportable, pinned, architecture-correct, and compatible with the role's persistence and migration behavior. A repaired role then enters the full validation queue.

Queue 6: Nonstandard deployment fixtures¶

Build dedicated test paths for the five host/APT/build-based roles. Absence of a conventional Compose image is not itself incompatibility. Exclusion requires documented evidence that no supportable deployment can be validated on the available environment.

Short ready backfills receive priority while long repairs continue asynchronously. A blocked role cannot occupy an execution worker.

Repair-before-exclusion policy¶

Before assigning incompatible, the pipeline records:

current upstream support and release state;
official or maintained replacement images and deployment methods considered;
architecture and host prerequisites;
migration/persistence compatibility with existing HomelabOS installations;
the exact reason no supportable deployment can be produced;
the environment or upstream change that would permit retest.

A role defect discovered during validation is fixed at source and retested at the new revision. A workaround that mutates a container, bypasses authentication, ignores persistence, or changes production data without a migration path is not accepted.

Failure handling and recovery¶

Every failed step has a typed reason:

role_defect;
upstream_or_image_defect;
license_limitation;
external_fixture_missing;
hardware_or_host_requirement;
incompatible_deployment_model;
test_infrastructure_failure;
measurement_contaminated;
application_failure;
capacity_guard.

Retries are bounded and visible. Infrastructure failure never becomes an application failure. A partial trace stays partial even when the application later recovers. Queue state, ownership manifests, and immutable receipts permit safe resume without repeating accepted work or deleting failed state.

Documentation outputs¶

The pipeline publishes:

one current machine-readable role result for every catalog entry;
immutable per-step receipts and raw sanitized measurements;
a generated facts block in every role's docs.md;
a consolidated 229-role table with independent deployment, resources, authentication, duplicate, lifecycle, limitation, and completion fields;
a host-sizing summary that counts shared Traefik/Authentik once;
a gap report listing every incomplete gate and its next executable action;
per-wave summaries of accepted results, fixes, failures, and contaminated reruns.

The headline report always states both the 229-role total and the eligible denominator. License-limited completed roles remain visible as limited, not silently merged into unrestricted roles.

Verification strategy¶

Contract fixtures¶

Automated fixtures prove that:

a fully evidenced eligible role becomes complete;
one missing required receipt prevents completion;
a license-limited role can complete while preserving the limitation;
an incompatible role cannot appear complete;
stale revisions invalidate only affected dimensions;
contradictory or malformed evidence fails closed;
a contaminated measurement cannot satisfy a resource dimension;
generated docs and catalog rollups match the accepted structured record.

Scheduler fixtures¶

Tests prove capacity admission, shared locks, port/fixture exclusivity, lease recovery, bounded retry, reserve enforcement, measurement contamination, swap invalidation, and blocked-job release. The tests use deterministic fake resources rather than asserting implementation wiring.

Live canaries¶

Before opening all queues, validate three existing roles on the droplet:

a lightweight role with native Authentik OIDC;
a multi-container role using application-native accounts;
a heavy stack.

The canaries must prove isolated concurrent deployment, ownership manifests, resume behavior, authentication workers, duplicate/lifecycle checks, cold-install and steady-state accounting, interference detection, documentation projection, and exact completion gating. Failed canaries repair the pipeline before broader execution.

Completion criteria¶

The closure campaign is complete when:

all 229 roles conform to the current result schema;
every eligible and eligible-with-license-limitation role is complete under the strict contract;
every remaining role is blocked, failed, or incompatible with a specific evidenced reason and concrete retest prerequisite;
no role remains bounded, ambiguously validated, or unclassified;
every complete role has current cold-install and settled steady-state measurements;
every role document and consolidated catalog row matches its accepted structured record;
the final report gives exact counts by disposition, completion state, Authentik mode/result, architecture, and missing/excluded reason;
all campaign-owned running workloads are stopped, retained data remains intact, and no unrelated resource was modified or deleted.

Only these criteria authorize a statement that the 229-role catalog has been closed.

Last update: September 27, 2026