Target Designs & Coverage

Declare the experiment matrix you intend to run and check whether current runs actually cover it

Introduction

Project Organization showed how to tag runs with attributes — the factors a run is about (scenario, treatment, site). Attributes answer “what is this run?”. This guide answers a different question:

Is my experiment matrix complete? For every factor combination I intended to run, is there a current (active) run that covers it?

That is what a TargetDesign declares. It is a standing, persisted expectation — a set of Requirements, each an attribute conjunction plus a min_active count — and check_design() reports coverage against the runs that actually exist. Crucially, completeness is asserted over attributes, not run hashes: it makes no claim on how a run was produced (a SweepManager sweep, a one-off dispatch, a colleague’s job), only that enough active runs carry the required attributes. Superseded and bad runs never count.

Setup: a couple of attributed runs

We run the tutorial model under two treatments, declaring the factors we care about on the JobConfig itself via attributes. Those attributes are forwarded to the registry when the job is registered, so there is no separate tagging step after the run — you say what a job is for at the point you define it. (baseline.jshc and high_growth.jshc play the role of two treatments here — real runs, not fabricated rows.)

from pathlib import Path
from joshpy.jobs import JobConfig
from joshpy.sweep import SweepManager
from joshpy.cli import JoshCLI
from joshpy.jar import JarMode
from joshpy.registry import RunRegistry, TargetDesign, Requirement

SOURCE = Path("../../examples/tutorial_sweep.josh")
CONFIGS = {
    "spinup": Path("../../examples/configs/baseline.jshc"),
    "no_spinup": Path("../../examples/configs/high_growth.jshc"),
}
REGISTRY = "target_designs_demo.duckdb"
cli = JoshCLI(josh_jar=JarMode.DEV)

run_hashes = {}
for treatment, config_path in CONFIGS.items():
    config = JobConfig(
        source_path=SOURCE, config_path=config_path,
        simulation="Main", replicates=1,
        # Declare *what this job is for* on the config itself. These attributes
        # are forwarded to the registry at registration — no post-run tagging.
        # They describe intent, not config values: the config need not know
        # which requirement it fulfills.
        attributes={"scenario": "historical", "treatment": treatment},
    )
    manager = (
        SweepManager.builder(config)
        .with_registry(REGISTRY, experiment_name="coverage_demo")
        .with_cli(cli)
        .build()
    )
    result = manager.run(quiet=True)
    # The attributes are already recorded; we just keep the hash to reference
    # the run later when we mark one bad.
    for job, _ in result:
        run_hashes[treatment] = job.run_hash
    manager.cleanup()
    manager.close()

print(run_hashes)
{'spinup': '36d7b5d939aa', 'no_spinup': 'a1ea83982e5a'}

Declare the design and check coverage

A TargetDesign names the cells you intend to fill. Here the matrix is scenario=historical crossed with the two treatments:

registry = RunRegistry(REGISTRY)

design = TargetDesign("jotr_historical", requirements=[
    Requirement({"scenario": "historical", "treatment": "spinup"}),
    Requirement({"scenario": "historical", "treatment": "no_spinup"}),
])
registry.register_design(design)   # persisted; re-registering replaces it

report = registry.check_design("jotr_historical")
print("complete:", report.complete)
complete: True

Both cells are covered by an active run, so the design is complete. The report is per-requirement — satisfied, how many active runs were found vs the min_active target, and the matching hashes:

def show(report):
    for rc in report.requirements:
        mark = "OK " if rc.satisfied else "GAP"
        print(f"  [{mark}] {rc.attributes}  found={rc.found}/{rc.min_active}"
              f"  active={rc.active_run_hashes}")

show(report)
  [OK ] {'scenario': 'historical', 'treatment': 'spinup'}  found=1/1  active=['36d7b5d939aa']
  [OK ] {'scenario': 'historical', 'treatment': 'no_spinup'}  found=1/1  active=['a1ea83982e5a']

A Requirement matches by superset: a run tagged with extra attributes (say site=JOTR001) still satisfies a requirement that only names scenario/treatment. Requirements are as specific as you write them.

Currency: a bad run leaves a gap — but a visible one

Mark the no_spinup run bad (say its external data was wrong). It immediately stops counting toward coverage, so the design is no longer complete. But the report doesn’t just say “empty” — it surfaces the retired run, so you can tell “nothing ran” from “ran, but unusable”:

registry.mark_run(run_hashes["no_spinup"], "bad", reason="wrong forcing data")

report = registry.check_design("jotr_historical")
print("complete:", report.complete)
for rc in report.unmet:
    print("  unmet:", rc.attributes,
          "| non-active present:", rc.non_active_matches)
complete: False
  unmet: {'scenario': 'historical', 'treatment': 'no_spinup'} | non-active present: [('a1ea83982e5a', 'bad')]

Re-running that treatment with corrected inputs produces a fresh, active run for the cell (a corrected input changes the content hash, so it dispatches as a new run) — and the design goes green again. min_active lets a cell demand more than one active run when you need replication across independent runs; replicate counts within a run stay the job of check_consistency.

Designs travel with the registry

A TargetDesign is stored in the registry (in target_designs / target_requirements), so it is part of what push_to_s3 / pull_from_s3 publish and restore. Rehydrate a registry on another machine and its designs — and therefore check_design — come back with it:

registry.push_to_s3(bucket="my-bucket")             # design included

fresh = RunRegistry("rehydrated.duckdb")
fresh.pull_from_s3(bucket="my-bucket", mode="restore")
fresh.check_design("jotr_historical").complete       # recomputed locally

Manage them with list_designs(), get_design(name), and delete_design(name).

Where this fits

  • Attributes (Project Organization) are the factors a run is about.
  • Status (active / superseded / bad) is whether a run should be used; coverage counts only the current ones.
  • Target designs assert that your intended matrix is covered by current runs — a read-only check that never dispatches. Feeding its unmet list into the next sweep is up to you.