Skip to content

Service Validation Catalog Closure Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the permissive validation rollup with a strict evidence-gated pipeline, then use the AMD64 droplet to close every supportable role in the 229-role HomelabOS catalog.

Architecture: A versioned service_validation Python package in the HomelabOS repository owns the result contract, immutable receipts, audit, resource collection, capacity scheduling, remote execution, and generated documentation. The existing inventory and evidence remain inputs, but only typed current-version receipts can produce complete; persistent campaign state and secrets remain in the sibling homelabos-pipeline/.batch-private/ workspace.

Tech Stack: Python 3 standard library, JSON/JSONL, unittest, Ansible through docker_helper_notty.sh, Docker/Compose, systemd, SSH, Linux PSI and swap metrics, existing HomelabOS role/docs conventions.

Approved design: docs/superpowers/specs/2026-09-22-service-validation-closure-design.md

Execution boundaries

  • Source repository: /Users/vincehark/Code/homelabos, branch feat/service-batch.
  • Campaign workspace: /Users/vincehark/Code/homelabos-pipeline.
  • Evidence root: /Users/vincehark/Code/homelabos-pipeline/test-results.
  • Private state: /Users/vincehark/Code/homelabos-pipeline/.batch-private/service-validation-v2.
  • AMD64 target: root@143.198.234.10 using ~/.ssh/id_rsa; deployment user remains homelab through HomelabOS settings.
  • Preserve the existing 198.9 GB Docker image cache, stopped application data, and unrelated resources. Never run host-wide Docker prune, volume prune, service reset, or data deletion.
  • All live fixes belong in roles/<slug>/, shared includes, or shared templates. Never accept a direct edit under /var/homelabos as the source fix.
  • The existing 2026-09-09 inventory remains the 229-role catalog identity source. New v2 results live separately so legacy prose cannot silently promote status.
  • The acceptance baseline is production-readiness at zero user load: real install, readiness/admin, one representative operation, authentication/Authentik, isolation, lifecycle/persistence, backup verification, and measured base resources. Sustained active load, upgrade rehearsal, and a 30-minute soak are not completion gates.

File map

Create

  • service_validation/__init__.py — package version and shared constants.
  • service_validation/schema/role-result-v2.schema.json — portable result contract.
  • service_validation/contract.py — deterministic completion evaluation.
  • service_validation/evidence.py — immutable receipt writes, hashes, loading, and secret rejection.
  • service_validation/collect_metrics.py — versioned direct/nested Docker collector.
  • service_validation/analyze_metrics.py — resource window analysis and contamination result.
  • service_validation/host_metrics.py — CPU, memory, swap, disk, PSI, and interference checks.
  • service_validation/legacy.py — legacy evidence candidate discovery without automatic acceptance.
  • service_validation/audit.py — 229-role strict audit and rollups.
  • service_validation/docs.py — role facts, catalog table, sizing summary, and gap report projection.
  • service_validation/capacity.py — token accounting and admission decisions.
  • service_validation/scheduler.py — persistent leases, queue transitions, and worker dispatch.
  • service_validation/runner.py — isolated controller workspace and step execution.
  • service_validation/host_prepare.py — idempotent swap and live-capacity preparation.
  • service_validation/campaign.py — fixed closure manifest and queue CLI.
  • service_validation/campaigns/catalog-closure-v2.json — exact queues, limits, and target-independent policy.
  • tests/test_service_validation_contract.py
  • tests/test_service_validation_evidence.py
  • tests/test_service_validation_metrics.py
  • tests/test_service_validation_audit.py
  • tests/test_service_validation_docs.py
  • tests/test_service_validation_capacity.py
  • tests/test_service_validation_scheduler.py
  • tests/test_service_validation_runner.py
  • tests/test_service_validation_campaign.py
  • tests/test_service_validation_host_prepare.py
  • docs/development/service-validation-results.json — generated current v2 registry.
  • docs/development/service-validation-gaps.md — generated human gap report.

Modify

  • docs/development/service-validation.md — add marker-owned current catalog summary and retain historical narrative below it.
  • roles/*/docs.md — replace only the marker-owned service-facts block from accepted v2 records; historical preliminary sections are marked superseded when they conflict.
  • tests/README.md — document the validation package tests and strict distinction between offline and live evidence.
  • /Users/vincehark/Code/homelabos-pipeline/RESUME.md — point resumption at the v2 audit and scheduler after the live canaries pass.
  • /Users/vincehark/Code/homelabos-pipeline/README.md — replace legacy fold commands with the v2 audit/campaign workflow.
  • /Users/vincehark/Code/homelabos-pipeline/fold_in.sh — remove after the v2 fold proves byte-stable.
  • /Users/vincehark/Code/homelabos-pipeline/facts_to_inventory.py — remove after strict cutover.
  • /Users/vincehark/Code/homelabos-pipeline/service_facts.py — remove after strict cutover.
  • /Users/vincehark/Code/homelabos-pipeline/update_inventory_from_smokes.py — remove after strict cutover.
  • /Users/vincehark/Code/homelabos-pipeline/docs_from_smokes.py — remove after strict cutover.
  • /Users/vincehark/Code/homelabos-pipeline/test_facts_to_inventory.py — remove with the retired prose classifier.

Retire through clean cutover

After the new audit and documentation projection produce byte-stable output from v2 records, stop invoking facts_to_inventory.py, service_facts.py, update_inventory_from_smokes.py, and docs_from_smokes.py for completion status. Preserve them with historical evidence until the live smoke proves the replacement; then remove their active-call references rather than adding compatibility aliases.

Dependency graph

  1. Tasks 1–3 establish contract, receipts, and metrics.
  2. Tasks 4–7 establish audit, docs, scheduler, and runner.
  3. Task 8 prepares the host and proves three live canaries.
  4. Task 9 publishes the corrected baseline and drains existing-evidence gaps.
  5. Tasks 10 and 11 run concurrently: admitted roles on the droplet while repair/nonstandard work prepares more roles.
  6. Task 12 performs the final closure audit and publication.

Task 1: Implement the strict role-result contract

Files: - Create: service_validation/__init__.py - Create: service_validation/schema/role-result-v2.schema.json - Create: service_validation/contract.py - Create: tests/test_service_validation_contract.py

  • [ ] Step 1: Write contract tests that define observable completion

Create tests/test_service_validation_contract.py with fixtures that prove a complete eligible role passes, one missing dimension prevents completion, license limitation remains visible while allowing completion, incompatible cannot be complete, stale source identity invalidates dimensions, contaminated resources fail, and duplicate not_applicable requires review evidence.

import unittest

from service_validation.contract import ContractError, evaluate


REQUIRED = (
    "preflight", "fresh_install", "readiness_and_admin",
    "representative_operation", "authentication", "duplicate_isolation",
    "redeploy_and_lifecycle", "backup_verification",
    "cold_install_metrics", "steady_state_metrics", "documentation_projection",
)


def passed_record(disposition="eligible"):
    return {
        "schema_version": 2,
        "slug": "example",
        "disposition": disposition,
        "source": {"sha": "a" * 40, "role_version": "1.2.3"},
        "images": [{"reference": "example/app:1.2.3", "digest": "sha256:" + "b" * 64,
                    "architecture": "amd64"}],
        "dimensions": {
            name: {"status": "passed", "source_sha": "a" * 40,
                   "receipt": f"example-{name}-1.json",
                   "receipt_sha256": "c" * 64}
            for name in REQUIRED
        },
        "limitations": [],
    }


class ContractTests(unittest.TestCase):
    def test_complete_requires_every_required_dimension(self):
        outcome = evaluate(passed_record())
        self.assertEqual(outcome.completion_state, "complete")
        self.assertEqual(outcome.missing_dimensions, ())

    def test_missing_dimension_prevents_complete(self):
        record = passed_record()
        del record["dimensions"]["cold_install_metrics"]
        outcome = evaluate(record)
        self.assertEqual(outcome.completion_state, "in_progress")
        self.assertEqual(outcome.missing_dimensions, ("cold_install_metrics",))

    def test_license_limitation_is_complete_but_visible(self):
        record = passed_record("eligible_with_license_limitation")
        record["limitations"] = [{"kind": "license_limitation",
                                  "feature": "native_oidc"}]
        outcome = evaluate(record)
        self.assertEqual(outcome.completion_state, "complete")
        self.assertEqual(outcome.limitations[0]["kind"], "license_limitation")

    def test_incompatible_record_cannot_claim_passed_dimensions(self):
        record = passed_record("incompatible")
        with self.assertRaises(ContractError):
            evaluate(record)

    def test_noneligible_dispositions_cannot_complete(self):
        for disposition, expected in (
                ("repairable", "failed"),
                ("environment_blocked", "blocked")):
            with self.subTest(disposition=disposition):
                outcome = evaluate(passed_record(disposition))
                self.assertEqual(outcome.completion_state, expected)

    def test_empty_dimensions_are_not_started(self):
        record = passed_record()
        record["dimensions"] = {}
        outcome = evaluate(record)
        self.assertEqual(outcome.completion_state, "not_started")

    def test_contaminated_measurement_remains_incomplete(self):
        record = passed_record()
        record["dimensions"]["steady_state_metrics"]["status"] = "contaminated"
        outcome = evaluate(record)
        self.assertEqual(outcome.completion_state, "in_progress")
        self.assertIn("steady_state_metrics", outcome.missing_dimensions)

    def test_duplicate_not_applicable_requires_review_receipt(self):
        record = passed_record()
        record["dimensions"]["duplicate_isolation"] = {"status": "not_applicable"}
        with self.assertRaises(ContractError):
            evaluate(record)
        record["dimensions"]["duplicate_isolation"].update(
            source_sha="a" * 40,
            review_receipt="example-duplicate-review-1.json",
            review_receipt_sha256="d" * 64,
        )
        self.assertEqual(evaluate(record).completion_state, "complete")

    def test_source_change_marks_bound_dimensions_stale(self):
        record = passed_record()
        record["dimensions"]["fresh_install"]["source_sha"] = "e" * 40
        outcome = evaluate(record)
        self.assertEqual(outcome.completion_state, "stale")
        self.assertIn("fresh_install", outcome.stale_dimensions)


if __name__ == "__main__":
    unittest.main()
  • [ ] Step 2: Run the test and verify it fails before implementation

Run:

python3 -m unittest discover -s tests -p test_service_validation_contract.py -v

Expected: import failure for service_validation.contract.

  • [ ] Step 3: Add the JSON Schema and deterministic evaluator

The schema must enumerate the five dispositions, six completion states, dimension statuses, provenance, image digests, limitations, and dimension receipt references. completion_state is generated output and must not be trusted as input.

Implement the evaluator around these constants and result type:

from dataclasses import dataclass

SCHEMA_VERSION = 2
REQUIRED_DIMENSIONS = (
    "preflight", "fresh_install", "readiness_and_admin",
    "representative_operation", "authentication", "duplicate_isolation",
    "redeploy_and_lifecycle", "backup_verification",
    "cold_install_metrics", "steady_state_metrics", "documentation_projection",
)
DISPOSITIONS = {
    "eligible", "eligible_with_license_limitation", "repairable",
    "environment_blocked", "incompatible",
}
PASSING = {"passed", "not_applicable"}
RESOURCE_DIMENSIONS = {"cold_install_metrics", "steady_state_metrics"}


class ContractError(ValueError):
    pass


@dataclass(frozen=True)
class Outcome:
    completion_state: str
    missing_dimensions: tuple[str, ...]
    stale_dimensions: tuple[str, ...]
    blocked_dimensions: tuple[str, ...]
    limitations: tuple[dict, ...]


def evaluate(record: dict) -> Outcome:
    if record.get("schema_version") != SCHEMA_VERSION:
        raise ContractError("unsupported schema_version")
    if record.get("disposition") not in DISPOSITIONS:
        raise ContractError("unknown disposition")
    if record["disposition"] == "incompatible" and any(
            value.get("status") == "passed"
            for value in record.get("dimensions", {}).values()
            if isinstance(value, dict)):
        raise ContractError("incompatible records cannot carry passed dimensions")
    source_sha = record.get("source", {}).get("sha")
    if not isinstance(source_sha, str) or len(source_sha) != 40:
        raise ContractError("source.sha must be a full commit SHA")

    dimensions = record.get("dimensions")
    if not isinstance(dimensions, dict):
        raise ContractError("dimensions must be an object")
    missing, stale, blocked = [], [], []
    for name in REQUIRED_DIMENSIONS:
        value = dimensions.get(name)
        if not isinstance(value, dict):
            missing.append(name)
            continue
        status = value.get("status")
        if status == "passed" and (
                not value.get("receipt") or not value.get("receipt_sha256")):
            raise ContractError(f"{name} passed without an evidence receipt")
        if status == "not_applicable" and (
                name != "duplicate_isolation"
                or not value.get("review_receipt")
                or not value.get("review_receipt_sha256")):
            raise ContractError("duplicate not_applicable requires reviewed evidence")
        if status == "stale" or value.get("source_sha") != source_sha:
            stale.append(name)
        elif status == "blocked":
            blocked.append(name)
        elif status not in PASSING:
            missing.append(name)

    limitations = tuple(record.get("limitations") or ())
    if record["disposition"] == "eligible_with_license_limitation" and not any(
            row.get("kind") == "license_limitation" for row in limitations):
        raise ContractError("license-limited disposition requires limitation details")
    if stale:
        state = "stale"
    elif record["disposition"] in {"environment_blocked", "incompatible"} or blocked:
        state = "blocked"
    elif record["disposition"] == "repairable":
        state = "failed"
    elif len(missing) == len(REQUIRED_DIMENSIONS):
        state = "not_started"
    elif missing:
        state = "in_progress"
    else:
        state = "complete"
    return Outcome(state, tuple(missing), tuple(stale), tuple(blocked), limitations)

The schema and Python constants must use the same enum values. Add a test that loads the schema and compares those enums so drift fails offline.

  • [ ] Step 4: Run the contract tests

Run:

python3 -m unittest discover -s tests -p test_service_validation_contract.py -v

Expected: all contract tests pass.

  • [ ] Step 5: Commit the contract
git add service_validation tests/test_service_validation_contract.py
git commit -m "feat(validation): define strict completion contract"

Task 2: Add immutable, sanitized evidence receipts

Files: - Create: service_validation/evidence.py - Create: tests/test_service_validation_evidence.py

  • [ ] Step 1: Write behavioral tests for receipt immutability and sanitization

Cover exclusive creation, stable SHA-256, refusal to overwrite a revision, rejection of secret-bearing keys/values, allowance for digest fields such as receipt_sha256, and relative-path enforcement under the selected evidence root.

import json
import tempfile
import unittest
from pathlib import Path

from service_validation.evidence import EvidenceError, EvidenceStore


class EvidenceStoreTests(unittest.TestCase):
    def test_write_once_receipt_returns_hash(self):
        with tempfile.TemporaryDirectory() as tmp:
            store = EvidenceStore(Path(tmp))
            result = store.write("run-1/example-preflight-1.json", {
                "schema_version": 2, "slug": "example", "status": "passed"})
            self.assertEqual(len(result.sha256), 64)
            self.assertEqual(json.loads(result.path.read_text())["slug"], "example")
            with self.assertRaises(EvidenceError):
                store.write("run-1/example-preflight-1.json", {
                    "schema_version": 2, "slug": "example", "status": "passed"})

    def test_secret_fields_are_rejected(self):
        with tempfile.TemporaryDirectory() as tmp:
            store = EvidenceStore(Path(tmp))
            for payload in ({"password": "value"}, {"nested": {"access_token": "value"}},
                            {"authorization": "Bearer value"}):
                with self.subTest(payload=payload), self.assertRaises(EvidenceError):
                    store.write("run-1/bad.json", payload)

    def test_hash_metadata_is_not_treated_as_secret(self):
        with tempfile.TemporaryDirectory() as tmp:
            store = EvidenceStore(Path(tmp))
            store.write("run-1/good.json", {"receipt_sha256": "a" * 64,
                                             "secret_sha256": "b" * 64})

    def test_path_cannot_escape_evidence_root(self):
        with tempfile.TemporaryDirectory() as tmp:
            with self.assertRaises(EvidenceError):
                EvidenceStore(Path(tmp)).write("../escape.json", {"status": "passed"})
  • [ ] Step 2: Run the evidence tests and verify failure
python3 -m unittest discover -s tests -p test_service_validation_evidence.py -v

Expected: import failure for service_validation.evidence.

  • [ ] Step 3: Implement exclusive receipt creation

Use os.open(..., os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600), canonical JSON (sort_keys=True, two-space indent, trailing newline), fsync, and SHA-256 of the exact published bytes. Reject absolute paths, .., symlink escapes, raw tokens/cookies/passwords/private keys, and strings containing Authorization: or JWT-shaped values. Hash-only fields ending _sha256 are permitted.

The public API is:

@dataclass(frozen=True)
class StoredReceipt:
    path: Path
    relative_path: str
    sha256: str


class EvidenceStore:
    def __init__(self, root: Path):
        self.root = root.resolve()

    def write(self, relative_path: str, payload: dict) -> StoredReceipt:
        """Validate, sanitize, atomically create, fsync, and hash one receipt."""

    def load(self, relative_path: str, expected_sha256: str | None = None) -> dict:
        """Load only a file below root and optionally verify its exact hash."""
  • [ ] Step 4: Run the evidence tests
python3 -m unittest discover -s tests -p test_service_validation_evidence.py -v

Expected: all evidence tests pass.

  • [ ] Step 5: Commit the evidence store
git add service_validation/evidence.py tests/test_service_validation_evidence.py
git commit -m "feat(validation): add immutable evidence receipts"

Task 3: Version and extend the resource collectors

Files: - Create: service_validation/collect_metrics.py - Create: service_validation/analyze_metrics.py - Create: service_validation/host_metrics.py - Create: tests/test_service_validation_metrics.py - Modify: tests/README.md

  • [ ] Step 1: Port the tested collector and analyzer into the versioned package

Copy the current implementations without behavior changes:

cp ../homelabos-pipeline/testenv/collect_metrics.py service_validation/collect_metrics.py
cp ../homelabos-pipeline/testenv/analyze_metrics.py service_validation/analyze_metrics.py

Change imports only as required for package execution. Keep the existing Docker selection, unit conversion, lifecycle retry, readiness, I/O, and malformed-input behavior intact.

  • [ ] Step 2: Port the existing collector/analyzer tests before changing behavior

Copy the current test bodies into tests/test_service_validation_metrics.py, importing service_validation.collect_metrics and service_validation.analyze_metrics. Run:

python3 -m unittest discover -s tests -p test_service_validation_metrics.py -v

Expected: the ported collector/analyzer tests pass before extensions.

  • [ ] Step 3: Add failing tests for direct Docker and host interference

Add tests that prove:

  • DockerClient(target=None) invokes docker ps, not docker exec <target> docker ps;
  • every sample carries a host snapshot and sorted concurrent-job identifiers;
  • available memory below 8 GiB contaminates the window;
  • any swap I/O contaminates the window;
  • host CPU above 85%, CPU some avg10 above 10.00, memory full avg10 above 0.00, I/O full avg10 above 2.00, free disk below 40 GB, OOM, or unexpected restart contaminates the window;
  • accepted windows preserve host threshold evidence;
  • the planning envelope rounds 1.25 * max(startup_max, steady_max) to 64 MiB with a 128 MiB floor.
  • the steady-state contract rejects a window shorter than 10 minutes or lacking a preceding uninterrupted five-minute settle period;

Use this value object and threshold function:

from dataclasses import dataclass

GIB = 1024 ** 3


@dataclass(frozen=True)
class HostSnapshot:
    cpu_percent_10s: float
    available_memory_bytes: int
    swap_in_bytes: int
    swap_out_bytes: int
    free_disk_bytes: int
    cpu_some_avg10: float
    memory_full_avg10: float
    io_full_avg10: float
    concurrent_jobs: tuple[str, ...]


def contamination_reasons(samples: list[HostSnapshot]) -> tuple[str, ...]:
    reasons = set()
    for row in samples:
        if row.available_memory_bytes < 8 * GIB:
            reasons.add("available_memory_below_8_gib")
        if row.free_disk_bytes < 40_000_000_000:
            reasons.add("free_disk_below_40_gb")
        if row.swap_in_bytes or row.swap_out_bytes:
            reasons.add("swap_io_observed")
        if row.cpu_percent_10s > 85:
            reasons.add("host_cpu_above_85_percent")
        if row.cpu_some_avg10 > 10:
            reasons.add("cpu_psi_above_10")
        if row.memory_full_avg10 > 0:
            reasons.add("memory_full_psi_observed")
        if row.io_full_avg10 > 2:
            reasons.add("io_full_psi_above_2")
    return tuple(sorted(reasons))
  • [ ] Step 4: Extend collection and analysis

Add --direct-docker, --host-metrics-command, and repeatable --concurrent-job arguments. Exactly one of --target and --direct-docker is required. The default live droplet path uses direct Docker.

The cold-install phase starts before the fresh deploy and continues through declared readiness so its maximum includes pulls, extraction, migrations, and startup. The steady-state phase begins only after five uninterrupted minutes at ready, then records ten uninterrupted minutes. swap_in_bytes and swap_out_bytes are deltas from the phase baseline, not host-lifetime counters; CPU is a rolling 10-second value.

Each sample includes:

{
  "host": {
    "cpu_percent_10s": 0.0,
    "available_memory_bytes": 0,
    "swap_in_bytes": 0,
    "swap_out_bytes": 0,
    "free_disk_bytes": 0,
    "pressure": {
      "cpu_some_avg10": 0.0,
      "memory_full_avg10": 0.0,
      "io_full_avg10": 0.0
    },
    "concurrent_jobs": []
  }
}

The analyzer emits measurement_status: accepted|contaminated, contamination_reasons, and the host maxima/minima used for the decision. It never drops raw samples.

Add:

def planning_memory_mib(startup_max_mib: float, steady_max_mib: float,
                        override_mib: int | None = None) -> int:
    observed = max(startup_max_mib, steady_max_mib)
    calculated = max(128, math.ceil((observed * 1.25) / 64) * 64)
    return max(calculated, override_mib or 0)
  • [ ] Step 5: Run the metrics tests
python3 -m unittest discover -s tests -p test_service_validation_metrics.py -v

Expected: all original and new metrics behaviors pass.

  • [ ] Step 6: Update offline-test documentation and commit

Add the exact validation test command to tests/README.md:

python3 -m unittest discover -s tests -p 'test_service_validation_*.py' -v

State that these tests prove contract/tool behavior but not a live deployment.

git add service_validation tests/test_service_validation_metrics.py tests/README.md
git commit -m "feat(validation): record host-aware resource windows"

Task 4: Build the strict auditor and legacy candidate importer

Files: - Create: service_validation/legacy.py - Create: service_validation/audit.py - Create: tests/test_service_validation_audit.py - Create: docs/development/service-validation-results.json

  • [ ] Step 1: Write audit tests over a synthetic three-role catalog

The fixture must contain one valid v2 result, one legacy-only role, and one malformed result. Assert:

  • catalog count is preserved;
  • only the valid v2 record is complete;
  • legacy files become evidence candidates, never accepted dimensions;
  • malformed records fail closed and appear in the gap report;
  • rollup keys distinguish disposition, completion, Authentik mode, and missing dimension;
  • source/version mismatch reports stale rather than complete.
class AuditTests(unittest.TestCase):
    def test_legacy_prose_never_promotes_completion(self):
        report = audit_catalog(self.inventory, self.results, self.evidence_root,
                               current_source_sha="a" * 40)
        self.assertEqual(report["catalog_total"], 3)
        self.assertEqual(report["completion_counts"]["complete"], 1)
        legacy = next(row for row in report["roles"] if row["slug"] == "legacy")
        self.assertEqual(legacy["completion_state"], "not_started")
        self.assertTrue(legacy["legacy_candidates"])
        self.assertIn("fresh_install", legacy["missing_dimensions"])
  • [ ] Step 2: Run the audit tests and verify failure
python3 -m unittest discover -s tests -p test_service_validation_audit.py -v

Expected: import failure for the audit module.

  • [ ] Step 3: Implement legacy candidate discovery

Map filenames only to candidate dimensions:

CANDIDATE_HINTS = {
    "cold_install_metrics": ("startup",),
    "steady_state_metrics": ("idle",),
    "authentication": ("sso", "auth"),
    "duplicate_isolation": ("duplicate", "pair-lifecycle"),
    "redeploy_and_lifecycle": ("pair-lifecycle", "lifecycle"),
    "backup_verification": ("backup", "restore", "persist-after-shutdown"),
    "fresh_install": ("deploy",),
}

The importer reads referenced JSON to reject malformed files and captures filename, hash, top-level status, version/date fields, and missing provenance. It never turns a candidate into passed. Acceptance requires a v2 receipt created by a current live run or an explicit reviewed adjudication receipt with source/version/digest identity.

  • [ ] Step 4: Implement the 229-role audit CLI

CLI:

python3 -m service_validation.audit \
  --inventory docs/superpowers/plans/2026-09-09-service-validation-inventory.json \
  --results docs/development/service-validation-results.json \
  --evidence-root ../homelabos-pipeline/test-results \
  --source-root . \
  --output ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json

audit_catalog() must require exactly 229 unique slugs for this campaign, locate every roles/<slug>/service.yml and docs.md, evaluate v2 records, collect legacy candidates, and emit exact missing/stale/blocked dimensions. Exit 0 means the audit ran; --require-closed exits 2 while any eligible role is incomplete.

Initialize docs/development/service-validation-results.json as:

{
  "schema_version": 2,
  "catalog": "homelabos-229",
  "roles": {}
}
  • [ ] Step 5: Run audit tests
python3 -m unittest discover -s tests -p test_service_validation_audit.py -v

Expected: all audit tests pass.

  • [ ] Step 6: Commit the auditor
git add service_validation/legacy.py service_validation/audit.py \
  tests/test_service_validation_audit.py \
  docs/development/service-validation-results.json
git commit -m "feat(validation): audit catalog from structured evidence"

Task 5: Generate canonical role and catalog documentation

Files: - Create: service_validation/docs.py - Create: tests/test_service_validation_docs.py - Create: docs/development/service-validation-gaps.md - Modify: docs/development/service-validation.md - Modify: roles/*/docs.md only through marker-owned blocks during live result folds

  • [ ] Step 1: Write projection tests

Test that one accepted result produces:

  • a single <!-- service-facts:start --> / <!-- service-facts:end --> block;
  • tested version/digest/architecture;
  • Authentik mode and license limitation;
  • duplicate result;
  • install maximum and settled median/p95/max;
  • the calculated RAM planning envelope;
  • image bytes, fresh data bytes, container count, and shared core separately;
  • evidence links and hashes;
  • no private path, address, password, token, cookie, or browser storage;
  • a stale ## Local ARM validation section relabeled ## Historical preliminary validation (superseded) once, not duplicated;

  • deterministic consolidated table and gap report ordering.

  • [ ] Step 2: Run projection tests and verify failure

python3 -m unittest discover -s tests -p test_service_validation_docs.py -v

Expected: import failure for service_validation.docs.

  • [ ] Step 3: Implement marker-owned projections

Use fixed markers:

FACTS_START = "<!-- service-facts:start -->"
FACTS_END = "<!-- service-facts:end -->"
CATALOG_START = "<!-- validation-catalog-v2:start -->"
CATALOG_END = "<!-- validation-catalog-v2:end -->"

render_service_facts(result) must read only accepted structured fields. update_marked_block() must refuse mismatched or duplicate markers. mark_preliminary_superseded() changes only the exact heading ## Local ARM validation for a complete role and leaves its historical text intact.

render_gap_report() lists each role, disposition, state, missing/stale dimensions, blocker, and next executable queue action. render_catalog() includes the total 229 denominator and the eligible denominator.

  • [ ] Step 4: Add dry-run and write CLI modes
python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json \
  --source-root . \
  --dry-run

--dry-run prints changed paths and exits 1 if generated files differ. --write updates them atomically. It must never modify role prose outside the facts block and exact historical-heading relabel.

  • [ ] Step 5: Run projection tests
python3 -m unittest discover -s tests -p test_service_validation_docs.py -v

Expected: all projection tests pass.

  • [ ] Step 6: Commit the generator
git add service_validation/docs.py tests/test_service_validation_docs.py \
  docs/development/service-validation-gaps.md docs/development/service-validation.md
git commit -m "feat(validation): generate strict catalog documentation"

Task 6: Implement resource-aware scheduling and persistent leases

Files: - Create: service_validation/capacity.py - Create: service_validation/scheduler.py - Create: tests/test_service_validation_capacity.py - Create: tests/test_service_validation_scheduler.py

  • [ ] Step 1: Write capacity admission tests

Cover the approved host envelope: 28 CPU tokens, 52 GiB RAM tokens, 120 GB working disk, 8 GiB available-memory reserve, 40 GB free-disk reserve, exclusive port/fixture/shared-state locks, unknown cold installs running alone, known light jobs packing, and blocked jobs releasing workers.

@dataclass(frozen=True)
class ResourceRequest:
    cpu: float
    memory_bytes: int
    disk_bytes: int
    locks: frozenset[str] = frozenset()
    unknown_cold_install: bool = False


class CapacityTests(unittest.TestCase):
    def test_unknown_cold_install_runs_alone(self):
        pool = CapacityPool(cpu=28, memory_bytes=52 * 1024**3,
                            disk_bytes=120_000_000_000)
        self.assertTrue(pool.can_admit(ResourceRequest(2, 2 * 1024**3,
                                                       4_000_000_000,
                                                       unknown_cold_install=True)))
        pool.admit("unknown", ResourceRequest(2, 2 * 1024**3, 4_000_000_000,
                                               unknown_cold_install=True))
        self.assertFalse(pool.can_admit(ResourceRequest(1, 128 * 1024**2,
                                                        1_000_000_000)))
  • [ ] Step 2: Write scheduler transition tests

Prove pending -> leased -> passed|repairable|blocked|contaminated, lease expiry recovery, attempt history, bounded retries, contaminated measurement requeue, dependency release, priority order, and atomic resume after process interruption.

  • [ ] Step 3: Run scheduler tests and verify failure
python3 -m unittest discover -s tests \
  -p 'test_service_validation_capacity.py' -v
python3 -m unittest discover -s tests \
  -p 'test_service_validation_scheduler.py' -v

Expected: import failures for capacity and scheduler modules.

  • [ ] Step 4: Implement capacity tokens

CapacityPool must expose can_admit, admit, release, and snapshot. It rejects duplicate job IDs, overcommit, conflicting locks, and any new job while an unknown cold install is active. It never counts swap as memory.

  • [ ] Step 5: Implement the persistent scheduler

Store state at an explicit --state path using write-to-new-file, fsync, and os.replace. Each job contains stable ID, slug, step, dependencies, priority, requested resources, locks, attempt cap, status, lease owner/expiry, and receipt path. Workers claim through an exclusive state-file lock.

CLI:

python3 -m service_validation.scheduler status \
  --state ../homelabos-pipeline/.batch-private/service-validation-v2/state.json
python3 -m service_validation.scheduler drain \
  --state ../homelabos-pipeline/.batch-private/service-validation-v2/state.json \
  --workers 16 --deploy-workers 4 --preflight-workers 8 --browser-workers 8

The scheduler asks host_metrics for a fresh capacity snapshot before every admission. Reserve breach stops new admission but does not kill running jobs. OOM or unexpected restart pauses the queue and emits capacity_guard.

  • [ ] Step 6: Run capacity and scheduler tests
python3 -m unittest discover -s tests \
  -p 'test_service_validation_capacity.py' -v
python3 -m unittest discover -s tests \
  -p 'test_service_validation_scheduler.py' -v

Expected: all tests pass.

  • [ ] Step 7: Commit scheduler foundations
git add service_validation/capacity.py service_validation/scheduler.py \
  tests/test_service_validation_capacity.py \
  tests/test_service_validation_scheduler.py
git commit -m "feat(validation): schedule work by live capacity"

Task 7: Add isolated role execution and the fixed campaign manifest

Files: - Create: service_validation/runner.py - Create: service_validation/campaign.py - Create: service_validation/campaigns/catalog-closure-v2.json - Create: tests/test_service_validation_runner.py - Create: tests/test_service_validation_campaign.py

  • [ ] Step 1: Write runner tests around a fake process boundary

Prove identifier validation, write-once logs/receipts, isolated controller paths, exact Ansible arguments, timeout classification, shared lock selection, ownership manifest contents, no secret values in public receipts, stopped-data retention, and that the family state machine cannot publish before all eleven dimensions have receipts.

The deploy command must be assembled as an argument list:

def deploy_command(source: Path, overlay: Path, services: list[str]) -> list[str]:
    return [
        str(source / "docker_helper_notty.sh"), "ansible-playbook",
        "--extra-vars=@settings/config.yml",
        "--extra-vars=@settings/additional_services_config.yml",
        "--extra-vars=@settings/vault.yml",
        f"--extra-vars=@{overlay.relative_to(source)}",
        "--extra-vars=" + json.dumps({"services": services}, separators=(",", ":")),
        "-i", "inventory", "-t", "deploy", "playbook.homelabos.yml",
    ]
  • [ ] Step 2: Run runner tests and verify failure
python3 -m unittest discover -s tests -p test_service_validation_runner.py -v

Expected: import failure for service_validation.runner.

  • [ ] Step 3: Implement isolated controller workspaces

For each family job:

  1. create controllers/<job-id>/source as a detached worktree at the selected source SHA;
  2. copy only the synthetic base settings and the job overlay into that worktree;
  3. give primary and duplicate unique names, domains, paths, projects, credentials, browser contexts, and ownership entries;
  4. run role-local deploys concurrently only when their declared locks do not intersect;
  5. require shared:authentik for provider/outpost mutations, shared:traefik for core changes, and specific port:*, fixture:*, or device:* locks;
  6. stop campaign-owned units after receipts are saved while retaining their data.

The ownership manifest records controller path, remote units, Compose projects, data paths, networks, browser contexts, evidence files, and whether each resource is retained. It is the only source cleanup may use.

Every family executes the same ordered state machine:

  1. resolve source SHA, role version, image references/digests, architecture, and required fixtures;
  2. install a fresh primary from an absent campaign-owned data path while collecting cold-install metrics through readiness;
  3. verify the admin surface and one role-specific representative operation, persisting a unique sentinel when the service supports writes;
  4. verify the declared Authentik route and application authorization in an isolated browser context; if native OIDC is license-gated, record eligible_with_license_limitation and use a safe proxy gate when the application supports it;
  5. install and exercise a simultaneous duplicate with unique ports, paths, projects, domains, and credentials, or attach a reviewed not_applicable receipt when the product cannot have a second instance;
  6. redeploy and restart both instances, verify readiness and sentinel persistence, then stop/restart the duplicate independently;
  7. create the documented backup/export, verify its manifest and sentinel by restoring into a campaign-owned scratch path or disposable instance, and never overwrite the live validation data;
  8. wait five uninterrupted ready minutes, record ten uninterrupted steady-state minutes, calculate the planning envelope, project docs, and stop owned units with data retained.

HTTP/API-capable checks may use direct probes. UI-only admin, operation, and authentication checks use isolated browser contexts and save sanitized screenshots plus structured assertions; successful HTTP status alone never substitutes for the actual user-visible path.

  • [ ] Step 4: Create the exact campaign manifest

service_validation/campaigns/catalog-closure-v2.json contains these fixed queues:

{
  "schema_version": 2,
  "catalog_total": 229,
  "capacity": {
    "cpu_tokens": 28,
    "memory_bytes": 55834574848,
    "working_disk_bytes": 120000000000,
    "available_memory_reserve_bytes": 8589934592,
    "free_disk_reserve_bytes": 40000000000
  },
  "bounded": [
    "chowdown", "excalidraw", "folding_at_home", "grownetics",
    "homebox", "homedash", "ittools"
  ],
  "admitted_not_run": [
    "archisteamfarm", "authelia", "authentik", "drone", "duckdns",
    "erpnext", "factorio", "gluetun", "invidious", "invoiceninja",
    "keycloak", "kibitzr", "langsmith", "matomo", "matterbridge",
    "minecraft", "minecraftbedrockserver", "mybb", "netbird", "netdata",
    "nzbget", "octoprint", "odoo", "ollama", "opencode", "opengsd",
    "openldap", "openvpn", "overseerr", "paperless", "paseo", "pixelfed",
    "portainer", "postgresql", "privoxyvpn", "prometheus", "qbittorrent",
    "quakejs", "restic", "samba", "searxng", "seat", "shinobi", "snibox",
    "speedtest", "speedtest_tracker", "statping", "stirlingpdf", "sui",
    "taisun", "teedy", "thespaghettidetective", "tick", "tiddlywiki",
    "tubearchivist_jf", "turtl", "vikunja", "wallabag", "watchtower",
    "webdavserver", "webtrees", "webvirtmgr", "wekan", "workadventure",
    "xfinityusageinfluxdb", "xteve", "zammad", "ztncui", "zulip"
  ],
  "image_repair": [
    "barcodebuddy", "clawdbot", "graylog", "hlos_dash", "hubzilla", "lidarr",
    "mailu", "mayan", "minio", "ombi", "ownphotos", "peertube", "phpbb",
    "pleroma", "readarr", "sabnzbd", "searx", "simplyshorten", "transmission"
  ],
  "nonstandard": ["cockpit", "jenkins", "matrix", "nut", "unofficial_ddns"],
  "fixture_locks": {
    "duckdns": ["fixture:dns"],
    "factorio": ["fixture:game"],
    "gluetun": ["fixture:vpn"],
    "netbird": ["fixture:mesh"],
    "octoprint": ["fixture:printer"],
    "privoxyvpn": ["fixture:vpn"],
    "quakejs": ["fixture:game"],
    "thespaghettidetective": ["fixture:printer"],
    "watchtower": ["fixture:docker-daemon"],
    "unofficial_ddns": ["fixture:dns"]
  }
}

Add tests/test_service_validation_campaign.py to join the manifest against the 229-role inventory and assert:

  • every listed slug exists exactly once in its fixed queue;
  • the three not-run queues total 93;
  • counts are 69 admitted, 19 image-repair, and five nonstandard;
  • bounded count is seven;

  • catalog total remains 229.

  • [ ] Step 5: Implement campaign enqueue/status commands

python3 -m service_validation.campaign audit
python3 -m service_validation.campaign enqueue --queue existing-backfill
python3 -m service_validation.campaign enqueue --queue bounded
python3 -m service_validation.campaign enqueue --queue admitted-not-run
python3 -m service_validation.campaign enqueue --queue image-repair
python3 -m service_validation.campaign enqueue --queue nonstandard
python3 -m service_validation.campaign status

All commands require explicit --workspace, --source-root, --evidence-root, and --state or read them from one private runtime configuration file. No absolute developer path is embedded in source.

  • [ ] Step 6: Run runner and manifest tests
python3 -m unittest discover -s tests -p test_service_validation_runner.py -v
python3 -m unittest discover -s tests -p test_service_validation_campaign.py -v

Expected: all runner and manifest tests pass.

  • [ ] Step 7: Commit runner and campaign
git add service_validation/runner.py service_validation/campaign.py \
  service_validation/campaigns/catalog-closure-v2.json \
  tests/test_service_validation_runner.py \
  tests/test_service_validation_campaign.py
git commit -m "feat(validation): add isolated catalog campaign runner"

Task 8: Prepare the droplet and prove live canaries

Files: - Create: service_validation/host_prepare.py - Create: tests/test_service_validation_host_prepare.py - Create privately at runtime: ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json - Produce: ../homelabos-pipeline/test-results/20260922-closure-canary/

  • [ ] Step 1: Write idempotent host-preparation tests

Test command generation and parsed verification for an absent swap file, an already-correct swap file, a wrong-size inactive file, duplicate /etc/fstab entries, wrong swappiness, insufficient disk, and nonzero swap I/O. Never execute privileged commands in unit tests.

  • [ ] Step 2: Implement host preparation

The remote operation is idempotent and fails closed:

sudo test -e /swapfile || sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo file /swapfile | grep -q 'swap file' || sudo mkswap /swapfile
sudo swapon --show=NAME --noheadings | grep -qx /swapfile || sudo swapon /swapfile
sudo sed -i '\|^/swapfile none swap sw 0 0$|d' /etc/fstab
printf '/swapfile none swap sw 0 0\n' | sudo tee -a /etc/fstab
printf 'vm.swappiness=1\n' | sudo tee /etc/sysctl.d/99-hlos-validation-swap.conf
sudo sysctl --system

Before creating it, require at least 50 GB free disk. Afterward verify exact size, mode 0600, active status, one fstab entry, vm.swappiness=1, no swap I/O delta during a 10-second quiet observation, and unchanged running containers.

Before mutation, inspect /swapfile size, type, mode, and active state. A wrong-size inactive file is removed and recreated; a wrong-size active file fails closed rather than calling swapoff under live load. The implementation executes the displayed commands only for the state transitions they represent, so an already-correct host is unchanged.

  • [ ] Step 3: Run host-preparation tests
python3 -m unittest discover -s tests -p test_service_validation_host_prepare.py -v

Expected: all tests pass.

  • [ ] Step 4: Create the private runtime configuration

Write mode 0600 JSON under the private workspace with source/evidence/state paths, SSH target, key path, public sslip domain, deployment user, and worker ceilings. Do not store passwords, Authentik tokens, cookies, or vault contents in this file.

  • [ ] Step 5: Apply and verify swap on the live host
python3 -m service_validation.host_prepare \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --ensure-swap --verify

Expected output includes:

cpus=32
memory_total_bytes>=66000000000
swap_bytes=8589934592
swappiness=1
free_disk_bytes>=40000000000
running_container_set_unchanged=true

If the running-container set changes unexpectedly, stop and investigate before any canary.

  • [ ] Step 6: Enqueue exact canary families with fresh aliases

Use existing roles but new owned instances so retained historical data is untouched:

  • wallosclosure / wallosclosure2 from wallos — lightweight native OIDC;
  • guacamoleclosure / guacamoleclosure2 from guacamole — multi-container app-native authentication;
  • elkstackclosure / elkstackclosure2 from elkstack — heavy stack and license-limited native SSO.
python3 -m service_validation.campaign enqueue-canaries \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --run-id 20260922-closure-canary \
  --family wallos:wallosclosure:wallosclosure2 \
  --family guacamole:guacamoleclosure:guacamoleclosure2 \
  --family elkstack:elkstackclosure:elkstackclosure2
  • [ ] Step 7: Drain canaries under the approved scheduler
python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue canary --workers 12 --deploy-workers 4 \
  --preflight-workers 8 --browser-workers 6

Expected terminal state:

  • three family results evaluate complete;
  • no measurement has swap I/O, OOM, unexpected restart, or threshold breach;
  • any contaminated first window has a later accepted replacement receipt;
  • all six canary units end inactive with data retained;
  • existing wallos, guacamole, and elkstack retained data is unchanged;

  • ownership manifests enumerate every created remote resource.

  • [ ] Step 8: Fold canary results and run focused verification

python3 -m service_validation.audit \
  --inventory docs/superpowers/plans/2026-09-09-service-validation-inventory.json \
  --results docs/development/service-validation-results.json \
  --evidence-root ../homelabos-pipeline/test-results \
  --source-root . \
  --output ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json
python3 -m unittest discover -s tests -p 'test_service_validation_*.py' -v

Canary aliases are harness resources, not extra catalog entries; the audit must still report catalog_total=229.

  • [ ] Step 9: Commit host tooling and accepted canary-backed source fixes

Commit only versioned tooling and any role/shared-source fixes proven by canaries. Do not commit private runtime state or secrets.

git add service_validation tests docs/development roles
git commit -m "feat(validation): prove strict live canaries"

Task 9: Publish the corrected baseline and close existing-evidence gaps

Files: - Modify: docs/development/service-validation-results.json - Modify: docs/development/service-validation-gaps.md - Modify: docs/development/service-validation.md - Modify: affected roles/*/docs.md - Modify: affected role/shared source files when a retest exposes a defect - Modify: /Users/vincehark/Code/homelabos-pipeline/RESUME.md - Remove after verified cutover: /Users/vincehark/Code/homelabos-pipeline/{fold_in.sh,facts_to_inventory.py,service_facts.py,update_inventory_from_smokes.py,docs_from_smokes.py,test_facts_to_inventory.py} - Modify: /Users/vincehark/Code/homelabos-pipeline/README.md

  • [ ] Step 1: Run the strict 229-role baseline audit
python3 -m service_validation.campaign audit \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --write-results --write-gap-report

Expected invariants:

  • catalog total exactly 229;
  • legacy family_validated prose does not determine v2 completion;
  • every role has a state and missing-dimension list;
  • seven bounded roles appear in the bounded queue;
  • 93 not-run roles split 69/19/5;

  • the corrected v2 complete count is allowed to be lower than 129 and is published without euphemism.

  • [ ] Step 2: Commit the corrected baseline before backfill

git add docs/development/service-validation-results.json \
  docs/development/service-validation-gaps.md \
  docs/development/service-validation.md
git commit -m "docs(validation): publish strict catalog baseline"
  • [ ] Step 3: Enqueue the 25 known resource-gap priorities

Priority slugs:

archivebox audiobookshelf baserow beets docmost dozzle duplicati elkstack
freshrss funkwhale ghost gitlab guacamole homarr huginn karakeep langfuse
monicahq mstream n8n nodered ntfy outline uptimekuma wallos

python3 -m service_validation.campaign enqueue \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue existing-backfill \
  --priority-slugs archivebox,audiobookshelf,baserow,beets,docmost,dozzle,duplicati,elkstack,freshrss,funkwhale,ghost,gitlab,guacamole,homarr,huginn,karakeep,langfuse,monicahq,mstream,n8n,nodered,ntfy,outline,uptimekuma,wallos

The queue also includes every other existing role whose strict audit reports a missing or stale dimension. Accepted current receipts are reused; missing cold-install or idle evidence is rerun on fresh owned aliases.

  • [ ] Step 4: Enqueue and adjudicate all seven bounded roles
python3 -m service_validation.campaign enqueue \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue bounded

Required terminal outcomes:

  • excalidraw, homebox, and ittools: complete after strict resource/provenance backfill if their current roles remain supportable;
  • chowdown, folding_at_home, grownetics, and homedash: repair and retest, or receive a typed blocked/failed/incompatible result with source research and a concrete retest prerequisite;

  • no v2 record remains bounded.

  • [ ] Step 5: Drain existing-backfill and bounded queues

python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue existing-backfill,bounded \
  --workers 16 --deploy-workers 4 --preflight-workers 8 --browser-workers 8

During accepted measurements, unrelated jobs may continue only under the encoded interference thresholds. Any contaminated result is automatically requeued.

  • [ ] Step 6: Fold results, generate docs, and verify gates
python3 -m service_validation.campaign audit \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --write-results --write-gap-report
python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json \
  --source-root . --write
python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json \
  --source-root . --dry-run

Expected: final dry-run reports no changed paths.

  • [ ] Step 7: Retire the prose-driven pipeline after verified cutover

Search the pipeline workspace for active references to facts_to_inventory.py, service_facts.py, update_inventory_from_smokes.py, and docs_from_smokes.py. Replace the README, RESUME.md, and MR-description commands with the v2 audit/campaign/docs commands. Remove fold_in.sh, the four superseded scripts, and test_facts_to_inventory.py only after the v2 audit and docs dry-run above succeed. Keep every historical JSON receipt and run directory.

  • [ ] Step 8: Run affected tests and commit the backfill wave

Run the validation package tests plus targeted role tests for every source file changed in this wave. Then:

git add roles tests docs/development service_validation
git commit -m "test(validation): close existing catalog evidence gaps"

Do not update the legacy inventory's old aggregate names to imply v2 completion; label those fields historical and point readers to the v2 registry.

Task 10: Validate all 69 AMD64-admitted not-run roles

Files: - Modify: affected roles/<slug>/ source and docs - Modify: targeted tests/test_<slug>*.py - Modify: docs/development/service-validation-results.json - Modify: docs/development/service-validation-gaps.md - Produce: immutable run directories under ../homelabos-pipeline/test-results/

  • [ ] Step 1: Enqueue the admitted queue from the fixed manifest
python3 -m service_validation.campaign enqueue \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue admitted-not-run

Expected: exactly 69 unique family jobs are present and no role from image-repair or nonstandard queues is admitted accidentally.

  • [ ] Step 2: Run preflight/pull/render at high parallelism
python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue admitted-not-run --steps preflight,image,pull,render \
  --workers 16 --preflight-workers 8 --deploy-workers 0 --browser-workers 0

Each role records exact images/digests, architecture, required locks, predicted container topology, release support, fixture needs, and initial resource request. Unknown heavy stacks remain marked unknown_cold_install and will run alone.

  • [ ] Step 3: Drain full role validation adaptively
python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue admitted-not-run \
  --workers 16 --deploy-workers 4 --preflight-workers 8 --browser-workers 8

Each supportable role must produce all eleven required receipt classes. When a role defect appears, its job becomes repairable, releases the worker, and emits the exact failing evidence. A role-specific worker then changes source under roles/<slug>/, adds a targeted observable regression where the bug is plausibly recurrent, commits the fix, and requeues the role at the new SHA. Infrastructure faults requeue without being called application failures.

  • [ ] Step 4: Handle fixture-locked roles with controlled fixtures

The manifest serializes shared fixture classes:

  • DNS: duckdns;
  • game protocol: factorio, quakejs;
  • VPN: gluetun, privoxyvpn;
  • mesh: netbird;
  • printer/hardware simulator: octoprint, thespaghettidetective;
  • controlled Docker daemon: watchtower.

Fixtures are synthetic, campaign-owned, and included in ownership manifests. They must not change host routing, real DNS, real VPN credentials, real printers, or the host Docker daemon used by HomelabOS.

  • [ ] Step 5: Publish accepted waves without waiting for all 69

At every accepted result boundary:

python3 -m service_validation.campaign audit \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --write-results --write-gap-report
python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/audit.json \
  --source-root . --write

Commit each reviewed wave with its role fixes, tests, docs, and current v2 registry. Do not batch unrelated unreviewed role changes into one commit.

  • [ ] Step 6: Verify the admitted queue terminal state
python3 -m service_validation.campaign status \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue admitted-not-run

Expected: zero pending, leased, contaminated, or ambiguous jobs. Every role is complete or has a typed blocked/failed/incompatible disposition and concrete retest requirement.

Task 11: Repair 19 image-defect roles and validate five nonstandard roles

Files: - Modify: exact affected roles/<slug>/ source and docs - Modify/Create: targeted role regression tests - Modify: service_validation/campaigns/catalog-closure-v2.json only if evidence changes a fixture lock, never to remove a role - Modify: generated v2 results and reports

This task can run concurrently with Task 10 after Tasks 1–8 pass.

  • [ ] Step 1: Dispatch one independent repair worker per image-defect role

Roles:

barcodebuddy clawdbot graylog hlos_dash hubzilla lidarr mailu mayan minio ombi
ownphotos peertube phpbb pleroma readarr sabnzbd searx simplyshorten transmission

Each worker must:

  1. inspect current role source and existing evidence;
  2. check the official project release and deployment documentation first;
  3. identify a maintained image or supported build path with linux/amd64 support;
  4. pin an exact release/digest for the validation run;
  5. preserve existing data paths and fail closed on unsafe migration;
  6. add a targeted regression for the defect repaired;
  7. pass preflight and then enter the full live contract.

Missing latest, abandoned third-party images, invalid prefixes, and discontinued projects are not enough to declare incompatibility. An incompatibility result requires the six-item source review from the approved design and a reviewed incompatibility receipt.

  • [ ] Step 2: Enqueue repaired image roles as each source fix lands
python3 -m service_validation.campaign enqueue \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue image-repair
python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue image-repair \
  --workers 12 --deploy-workers 4 --preflight-workers 8 --browser-workers 6

The scheduler skips blocked research jobs and continues ready repaired roles.

  • [ ] Step 3: Build dedicated fixtures for nonstandard roles

Roles and required execution models:

  • cockpit: disposable VM/host fixture for APT/systemd service behavior;
  • jenkins: reproducible image build, digest capture, and container deployment;
  • matrix: current supported homeserver topology with explicit database and federation boundaries;
  • nut: simulated UPS endpoint or reviewed hardware-blocked result with host integration isolated;
  • unofficial_ddns: controlled DNS provider fixture with no real-zone mutation.

Absence of a conventional Compose template is not incompatibility. Each fixture must still emit the same v2 receipts or a reviewed not_applicable/blocked receipt for dimensions the product cannot support.

  • [ ] Step 4: Drain the nonstandard queue
python3 -m service_validation.campaign enqueue \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue nonstandard
python3 -m service_validation.scheduler drain \
  --runtime ../homelabos-pipeline/.batch-private/service-validation-v2/runtime.json \
  --queue nonstandard \
  --workers 8 --deploy-workers 2 --preflight-workers 5 --browser-workers 4
  • [ ] Step 5: Fold, verify, and commit each repair family

For each accepted source repair, run its targeted offline test and live v2 contract before commit. Then regenerate audit/docs. A repair commit includes one role family and its test/evidence-backed docs; shared fixes may include all directly affected roles with targeted regression coverage.

  • [ ] Step 6: Verify both queues have concrete terminal states

Expected: all 24 roles are complete or have reviewed blocked/failed/incompatible results. Zero roles remain merely image-issue, no-renderable-image, or bounded.

Task 12: Close and publish the 229-role catalog

Files: - Modify: docs/development/service-validation-results.json - Modify: docs/development/service-validation-gaps.md - Modify: docs/development/service-validation.md - Modify: accepted roles/*/docs.md - Modify: CHANGELOG.md - Modify: /Users/vincehark/Code/homelabos-pipeline/RESUME.md - Modify: /Users/vincehark/Code/homelabos-pipeline/mr/801-batch-description.md

  • [ ] Step 1: Run the final fail-closed audit
python3 -m service_validation.audit \
  --inventory docs/superpowers/plans/2026-09-09-service-validation-inventory.json \
  --results docs/development/service-validation-results.json \
  --evidence-root ../homelabos-pipeline/test-results \
  --source-root . \
  --output ../homelabos-pipeline/.batch-private/service-validation-v2/final-audit.json \
  --require-closed

Expected exit 0 and these invariants:

  • catalog_total == 229;
  • every eligible and eligible_with_license_limitation role is complete;
  • every remaining role has an evidenced blocked, failed, or incompatible disposition and a concrete retest prerequisite;
  • zero bounded, unclassified, not-run, in-progress, leased, or contaminated terminal results;
  • every complete role has accepted current-version cold-install and settled steady-state measurements;

  • all evidence hashes resolve and sanitizer checks pass.

  • [ ] Step 2: Regenerate all marker-owned documentation

python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/final-audit.json \
  --source-root . --write
python3 -m service_validation.docs \
  --results docs/development/service-validation-results.json \
  --audit ../homelabos-pipeline/.batch-private/service-validation-v2/final-audit.json \
  --source-root . --dry-run

Expected: the second command reports no changes.

  • [ ] Step 3: Verify actual final surfaces

Run the validation package suite:

python3 -m unittest discover -s tests -p 'test_service_validation_*.py' -v

Run targeted role tests for every source file changed since the last accepted wave. Run the Go service sanity test once after all role changes:

go test ./...
go run main.go test

Expected: all commands pass with no skipped prerequisites in the Python validation suite. The Go sanity command must report all service roles without worker leaks or recursive watchdog failure.

  • [ ] Step 4: Verify the live target is retained and quiet

Use the scheduler ownership manifests to stop campaign-owned units only. Then verify:

  • only the shared HomelabOS core remains running unless a documented fixture must remain;
  • every campaign application unit is inactive;
  • retained data paths still exist;
  • existing pre-campaign retained data is unchanged;
  • swap remains configured with vm.swappiness=1 and no active swapping;

  • no unrelated resource was stopped, removed, or rewritten.

  • [ ] Step 5: Update changelog, resume, and MR description

Record the exact final counts by disposition, completion, Authentik mode/result, architecture, and blocker reason. State explicitly that resource figures are tested base deployment envelopes, not active-user capacity. Replace old 129 validated / seven bounded / 93 not run resume wording with the strict v2 rollup while retaining the historical figures as labeled history.

  • [ ] Step 6: Commit final catalog closure
git add service_validation tests roles docs CHANGELOG.md
git commit -m "docs(validation): close the 229-role catalog"

Do not publish or push MR !801 until glab is authenticated as vincehark, as recorded in RESUME.md.

Final acceptance checklist

  • [ ] The v2 schema/evaluator is the only path that produces complete.
  • [ ] The strict audit always reports the full 229-role denominator and eligible denominator.
  • [ ] All supportable roles have real HomelabOS deploy, operation, auth, duplicate, lifecycle, backup, cold-install, steady-state, and docs receipts.
  • [ ] License-gated native SSO remains visible without blocking otherwise complete roles.
  • [ ] Repairable image/role defects were investigated and fixed before exclusion.
  • [ ] No bounded, not_run, prose-inferred, contaminated, or stale result appears complete.
  • [ ] Every complete role publishes an observed base range and calculated RAM planning envelope.
  • [ ] Scheduler concurrency stayed within live CPU/RAM/disk/lock constraints; swap never counted as capacity.
  • [ ] Evidence is immutable, hash-verified, sanitized, and linked from generated docs.
  • [ ] Campaign-owned workloads are stopped with data retained; unrelated resources are untouched.
Last update: September 27, 2026