Create a self-contained bottle archive from a registered run.
By default, copies data files into the archive and raises if any are missing. Use omit_jshd=True for lightweight archives when the recipient has the data locally.
Detect run<->analysis drift between the execution and data tables.
Guards the principle that the registry (analysis) must reflect what running produced. Checks (scoped to run_hash if given, else all runs):
data_without_config — cell_data for a run_hash with no job_configs row (orphaned data; error).
orphan_cell_data — cell_data whose run_id is absent from job_runs (the FK should prevent this; error if seen).
duplicate_replicate — a (run_hash, replicate) present under more than one run_id (row inflation from a pre-fix re-ingest; error).
ran_not_ingested — a run_hash with a succeeded job_runs row but no cell_data (ran but results never loaded; warning).
Parameters
Name
Type
Description
Default
run_hash
str | None
Limit the check to one run, or None for the whole registry.
None
strict
bool
If True, raise RuntimeError when any issue is found.
False
Returns
Name
Type
Description
list[ConsistencyIssue]
A list of :class:ConsistencyIssue (empty if consistent).
Raises
Name
Type
Description
RuntimeError
If strict and any issue is found.
check_design
registry.RunRegistry.check_design(design)
Check a target design’s coverage against current runs.
Completeness is asserted over attributes, not run_hashes: a requirement is satisfied when at least min_active distinct active runs carry its attributes (as a superset). Superseded/bad runs never count toward satisfaction but are surfaced per requirement so an unmet cell that has runs reads as “present but retired”, not “nothing ran”. See REGISTRY_PROVENANCE.md §12.
Parameters
Name
Type
Description
Default
design
str | TargetDesign
A registered design name, or a :class:TargetDesign to check ad hoc without persisting it.
Check that current runs’ expected outputs actually landed in the bucket.
The bucket-aware sibling of :meth:check_consistency: for every active run, the registered export URIs (the expected set) are checked against what is actually present in the bucket. See REGISTRY_PROVENANCE.md §8c.
Issue kinds:
missing_remote_output (error) — a registered active URI absent from the bucket (a replicate that never landed).
remote_count_mismatch (error) — fewer present files than the run’s registered output count.
orphan_remote_output (warning) — a bucket file in a tracked folder that belongs to a non-active run (or no registered run): the stale/superseded files the manual workaround filters by hand.
Parameters
Name
Type
Description
Default
output_type
str
Export type (see :meth:get_output_uris).
'patch'
current_only
bool
Reserved for symmetry; the expected set is always the active runs. Non-active runs are used only to explain orphans.
True
endpoint
str | None
S3 endpoint; falls back to MINIO_ENDPOINT.
None
access_key
str | None
S3 access key; falls back to MINIO_ACCESS_KEY.
None
secret_key
str | None
S3 secret key; falls back to MINIO_SECRET_KEY.
None
use_ssl
bool | None
Use HTTPS (see :func:configure_s3).
None
Returns
Name
Type
Description
list[ConsistencyIssue]
A list of :class:ConsistencyIssue, most-actionable first (errors
list[ConsistencyIssue]
before warnings).
check_sparsity
registry.RunRegistry.check_sparsity()
Check for sparse columns in cell_data.
Sparse columns (>50% NULL by default) often indicate that different simulation types are being mixed in the same registry, which hurts query performance.
Returns
Name
Type
Description
SparsityReport
SparsityReport with statistics for each variable column.
Job configuration containing simulation, template, and sweep info. Must have simulation, template_path, and to_dict() attributes (typically a JobConfig from joshpy.jobs).
required
experiment_name
str | None
Name for the experiment. Defaults to config.simulation.
None
session_id
str | None
Optional externally-provided session ID. If None, generates a UUID. This allows the frontend/API layer to manage session IDs (e.g., using project IDs).
total_jobs and total_replicates are computed from the JobSet after job expansion. Use job_set.total_jobs and job_set.total_replicates.
delete_design
registry.RunRegistry.delete_design(name)
Delete a registered design and its requirements.
Returns
Name
Type
Description
bool
True if a design was deleted, False if none existed.
describe_labels
registry.RunRegistry.describe_labels()
Human-readable listing of all labeled runs.
Convenience wrapper around :func:joshpy.inspect.format_labels.
Returns
Name
Type
Description
str
A formatted table of labels with run_hash and creation time.
describe_run
registry.RunRegistry.describe_run(label_or_hash)
Human-readable detail for a single run.
Convenience wrapper around :func:joshpy.inspect.format_run_info.
Parameters
Name
Type
Description
Default
label_or_hash
str
Label or run_hash of the run to describe.
required
Returns
Name
Type
Description
str
A multi-section detail string (parameters, data files, replicates,
str
and per-run results).
Raises
Name
Type
Description
KeyError
If the label or hash is not found.
describe_sessions
registry.RunRegistry.describe_sessions()
Human-readable listing of all sweep sessions.
Convenience wrapper around :func:joshpy.inspect.format_sessions.
Returns
Name
Type
Description
str
A formatted table of sessions with experiment name, status, and
str
run counts.
describe_summary
registry.RunRegistry.describe_summary()
Human-readable overview of everything in the registry.
Convenience wrapper around :func:joshpy.inspect.format_summary.
Returns
Name
Type
Description
str
A high-level data summary for the whole registry.
drop_run
registry.RunRegistry.drop_run(label_or_hash)
Delete all registry state for a run, so its config can be redone.
This is the only operation that deletes or replaces existing run data. All other registry writes are append-only (runs, cell_data) or metadata (labels, session status). Use this to clear a run before re-running it from scratch — e.g. when a sweep’s outputs are bad and you want a clean slate rather than pooling more replicates onto them.
Removes, in foreign-key order, everything tied to the run’s hash: cell_data, run_outputs, job_runs, config_parameters, session_configs, and the job_configs row (including its label). The owning session row is left intact (it may hold other runs).
Parameters
Name
Type
Description
Default
label_or_hash
str
Label or run_hash of the run to drop.
required
Returns
Name
Type
Description
A
DropSummary
class:DropSummary with per-table deleted-row counts.
The run-level, currency-aware counterpart of :meth:find_tagged. “Current” is exactly status = 'active' (NULL read as active); superseded and bad runs are excluded by default. See REGISTRY_PROVENANCE.md.
Parameters
Name
Type
Description
Default
key
str
The attribute key to match on (e.g. "site").
required
value
Any
The value to match (compared as text).
required
current_only
bool
When True (default), only active runs are returned. Pass False to include superseded/bad runs.
True
Returns
Name
Type
Description
list[str]
List of matching run_hashes, in no particular order.
Find current runs by attribute, resolved with label and attributes.
Like :meth:find_by_attribute (active only), but returns fully resolved :class:AttributeMatch rows – (run_hash, label, attributes) – so callers never re-implement the label/attribute cross-reference. See REGISTRY_PROVENANCE.md §7.
Parameters
Name
Type
Description
Default
key
str
The attribute key to match on (e.g. "site").
required
value
Any
The value to match (compared as text).
required
Returns
Name
Type
Description
list[AttributeMatch]
List of :class:AttributeMatch, in no particular order.
Find every key in scope whose tags[tag_key] == value.
The reverse of get_tags_by_* / get_custom_tags: given a tag value (e.g. a site code), recover every key (e.g. run_hash) that carries it – the many-to-one direction, since several keys can share the same tag value.
Parameters
Name
Type
Description
Default
tag_key
str
The tag key to match on (e.g. "site").
required
value
Any
The value to match (compared as text).
required
scope
str
Scope to search. Defaults to "run_hash".
'run_hash'
Returns
Name
Type
Description
list[str]
List of matching keys (e.g. run_hashes), in no particular order.
Note
This is the general, scope-parameterized primitive and does not filter by run status. For the run-level attributes slice, prefer :meth:find_by_attribute, which defaults to current (active) runs only.
Resolve registered export URIs for runs, jar-free from run_outputs.
The registry records the resolved export URI of every registry-attached run (on any load_results setting), so it is the current-URI index — no filename regex, no hand-maintained label→hash dict. See REGISTRY_PROVENANCE.md §8a.
Parameters
Name
Type
Description
Default
label_or_hash
str | None
Restrict to one run (by label or hash). None (the default) returns outputs across all matching runs.
None
output_type
str
Export type. A bare name ("patch") maps to the stored export.<name>; pass a dotted name ("debug.organism") to target that namespace directly.
'patch'
current_only
bool
When True (default), only active runs are included.
True
Returns
Name
Type
Description
list[OutputURI]
A list of :class:OutputURI, ordered by run_hash then URI.
Get the number of distinct replicates for a run hash from cell_data.
This is the source-of-truth count, derived from actual loaded data rather than from job_runs metadata. Returns 0 if no data has been loaded yet.
The replicate index is the identity of a replicate: counting distinct replicate values gives the number of replicates loaded for this run, regardless of which execution (run_id) produced each one. Pooled runs dispatch fresh, non-colliding indices, so distinct replicate == total replicates; re-ingesting an already-loaded index is a no-op (see :meth:loaded_replicates).
Parameters
Name
Type
Description
Default
run_hash
str
The run hash to count replicates for.
required
Returns
Name
Type
Description
int
Number of distinct replicates in cell_data.
get_run
registry.RunRegistry.get_run(run_id)
Get run information by ID.
Parameters
Name
Type
Description
Default
run_id
str
The run ID to look up.
required
Returns
Name
Type
Description
RunInfo | None
RunInfo if found, None otherwise.
get_run_info
registry.RunRegistry.get_run_info(label_or_hash)
Get aggregated structured detail for a single run.
Combines :meth:get_config_by_hash, :meth:get_runs_for_hash, and :meth:get_replicate_count into one :class:RunDetail. This is the structured, data-layer counterpart to :meth:describe_run, which formats this same information as a human-readable string.
Parameters
Name
Type
Description
Default
label_or_hash
str
Label or run_hash of the run.
required
Returns
Name
Type
Description
A
RunDetail
class:RunDetail aggregating the config, recorded runs, and
Assign a human-readable label to a run configuration.
A label is a pure alias — the one handle you resolve to a run. It is not where currency or run attributes live: currency is :meth:mark_run/status and attributes are tags (see REGISTRY_PROVENANCE.md). Labels are unique within a registry; on collision the behavior depends on force and on_collision:
Default: raise ValueError.
force=True: silently drop the old label and reassign, leaving the old run’s status untouched. For “I mislabeled it, just fix it.”
on_collision="supersede": the old run releases the label (label = NULL) and is marked status = 'superseded' with superseded_by = run_hash and the given reason; the new run takes the label. This records real provenance instead of mangling the old label with a timestamp suffix (the retired "timestamp" mode).
If True, reassign the label even if already taken.
False
on_collision
str | None
Collision strategy. "supersede" archives the old run via supersession. Mutually exclusive with force.
None
reason
str | None
Free-text explanation stored on the superseded run. Only meaningful with on_collision="supersede".
None
Raises
Name
Type
Description
KeyError
If run_hash does not exist.
ValueError
If label is already assigned to a different run and neither force nor on_collision is set, or if both force and on_collision are set, or if on_collision has an invalid value.
list_attribute_keys
registry.RunRegistry.list_attribute_keys()
List all distinct run-level attribute keys in use.
The run-level (run_hash scope) counterpart of :meth:list_tag_keys.
Returns
Name
Type
Description
list[str]
Sorted list of attribute keys (e.g. ["biome", "site"]).
list_config_columns
registry.RunRegistry.list_config_columns()
List all parameter column names in config_parameters.
Returns the dynamically-added parameter columns. Column names preserve original names with special characters (e.g., ‘soil.moisture’).
List all export variable names from simulation outputs.
These are the variables exported by Josh simulations, stored as typed columns in the cell_data table. Variable names preserve original .josh names (e.g., ‘avg.height’).
When session_id is provided, only returns variables that have at least one non-NULL value for runs in that session.
Parameters
Name
Type
Description
Default
session_id
str | None
Optional session ID to filter by. If provided, only returns variables with data in that session.
Load debug messages for a run from registered debug output files.
Parameters
Name
Type
Description
Default
label_or_hash
str
Run label or run_hash.
required
run_id
str | None
Optional explicit run execution ID. If omitted, uses latest.
None
entity_types
list[str] | None
Optional debug entity types to include.
None
existing_only
bool
If True, only load files that currently exist.
True
Returns
Name
Type
Description
Any
DebugMessageStore with messages merged across all selected files.
Raises
Name
Type
Description
KeyError
If run/run execution is not found.
ValueError
If no matching debug files are available.
FileNotFoundError
If existing_only=False and any file is missing.
loaded_replicates
registry.RunRegistry.loaded_replicates(run_hash)
Return the set of replicate indices already loaded for a run hash.
The single source of truth for “what’s already ingested”. Ingestion skips any replicate index already in this set (idempotent re-ingest); the replicate index is the dedup identity.
Parameters
Name
Type
Description
Default
run_hash
str
The run hash to inspect.
required
Returns
Name
Type
Description
set[int]
Set of distinct replicate indices present in cell_data.
status is a closed enum — "active", "superseded", or "bad" — and is the single source of truth for whether a run should be used. “Current” is exactly status == "active" (see REGISTRY_PROVENANCE.md); there is no separate currency flag.
Parameters
Name
Type
Description
Default
run_hash
str
The run to mark. Also accepts a label (resolved first).
required
status
str
One of {"active", "superseded", "bad"}.
required
superseded_by
str | None
The run_hash that replaces this one. Required when status == "superseded" and rejected otherwise. The target must exist.
None
reason
str | None
Free-text explanation, stored alongside the status.
None
Raises
Name
Type
Description
KeyError
If run_hash (or superseded_by) does not exist.
ValueError
If status is not in the enum, if superseded_by is given without status="superseded" (or vice versa), or if a run is superseded by itself.
Pull a registry’s Parquet export from S3 back into this registry.
Parameters
Name
Type
Description
Default
bucket
str
S3/MinIO bucket the registry was published to.
required
prefix
str | None
S3 key prefix. Defaults to run-registries/<name>, matching :meth:push_to_s3’s default. Required if this registry is ":memory:".
None
mode
str
"restore" (default) or "merge": - "restore": this registry’s tables are emptied and replaced with exactly what’s at prefix. Use this to rehydrate a local registry from its own prior push_to_s3 export – e.g. on a new machine, or after deleting the local file. The table schemas (constraints, the cell_id sequence) are preserved; only the rows change. - "merge": adds rows from prefix that this registry doesn’t already have (by primary key, or for cell_data by (run_hash, replicate) since cell_id is a local surrogate with no meaning across registries); existing local rows are never touched. Only use this to bring in more runs of the same experiment (e.g. a teammate’s machine or a batch worker publishing to the same prefix) – not to combine genuinely different experiments into one registry. Per the one-registry-per-experiment guidance, merging unrelated experiments risks the same hash-collision ambiguity a reused registry already warns about. To analyze multiple experiments together without merging their data, use :func:open_s3_registries instead.
'restore'
endpoint
str | None
S3 endpoint. Falls back to MINIO_ENDPOINT env var.
None
access_key
str | None
S3 access key. Falls back to MINIO_ACCESS_KEY.
None
secret_key
str | None
S3 secret key. Falls back to MINIO_SECRET_KEY.
None
use_ssl
bool | None
Use HTTPS. See :func:configure_s3.
None
tables
tuple[str, …] | None
Tables to pull. Defaults to all of :data:REGISTRY_SYNC_TABLES.
None
Returns
Name
Type
Description
dict[str, int]
Dict mapping table name -> number of rows loaded from S3 (for
dict[str, int]
"merge", only the newly-inserted rows; for "restore",
Publish this registry’s tables to S3 as Parquet, one file per table.
Overwrites whatever is currently at the destination prefix with this registry’s current state. This publishes this registry only – it does not read or touch any other registry. Pair with :meth:pull_from_s3 (same registry, restoring or adding replicates) or :func:open_s3_registries (querying several published registries together, read-only, without merging them).
Parameters
Name
Type
Description
Default
bucket
str
S3/MinIO bucket to publish under.
required
prefix
str | None
S3 key prefix. Defaults to run-registries/<name> where <name> is this registry’s own filename stem – matching the “one registry file per experiment, named after it” convention already used locally. Required if this registry is ":memory:".
None
endpoint
str | None
S3 endpoint. Falls back to MINIO_ENDPOINT env var.
None
access_key
str | None
S3 access key. Falls back to MINIO_ACCESS_KEY.
None
secret_key
str | None
S3 secret key. Falls back to MINIO_SECRET_KEY.
None
use_ssl
bool | None
Use HTTPS. See :func:configure_s3.
None
tables
tuple[str, …] | None
Tables to publish. Defaults to all of :data:REGISTRY_SYNC_TABLES.
This provides direct access to DuckDB for custom queries beyond the pre-built methods. Use this when you need to run complex queries or explore the data in ways not covered by the API.
Parameters
Name
Type
Description
Default
sql
str
SQL query with ? placeholders for parameters.
required
params
list | None
List of parameter values.
None
Returns
Name
Type
Description
Any
DuckDB relation (call .df() for DataFrame, .fetchall() for tuples).
Examples
>>># Get DataFrame>>> df = registry.query(... "SELECT * FROM cell_data WHERE step BETWEEN ? AND ?",... [0, 10]... ).df()
>>># Get raw results>>> rows = registry.query(... "SELECT COUNT(*) FROM cell_data WHERE run_hash = ?",... ["abc123"]... ).fetchone()
Aggregate a variable directly against bucket-resident CSVs, no ingest.
Builds a URI manifest of the current runs (:meth:get_output_uris), scans exactly those CSVs once with DuckDB, and attaches provenance by joining on the (registry-supplied) filename — never by parsing the path. Nothing touches cell_data; this is the read path for the load_results=False batch (REGISTRY_PROVENANCE.md §8b).
Parameters
Name
Type
Description
Default
variable
str
CSV column to aggregate (e.g. "treeCount").
required
agg
str
One of mean/avg/sum/min/max/count/median/stddev.
'mean'
group_by
Sequence[str]
Columns to group by. label/run_hash come from the registry; any other name is a CSV column (step, replicate, position, …).
('label', 'step', 'replicate')
output_type
str
Export type (see :meth:get_output_uris).
'patch'
current_only
bool
Aggregate active runs only (default). Archived URIs are simply absent from the manifest, so stale files in the same folder are excluded for free.
True
label_or_hash
str | None
Restrict to one run.
None
where
str | None
Optional raw SQL predicate on the CSV columns (a passthrough, like :meth:query).
None
cache
str | Path | None
Optional path to also write the result to (.parquet or .csv by extension).
None
endpoint
str | None
S3 endpoint; falls back to MINIO_ENDPOINT.
None
access_key
str | None
S3 access key; falls back to MINIO_ACCESS_KEY.
None
secret_key
str | None
S3 secret key; falls back to MINIO_SECRET_KEY.
None
use_ssl
bool | None
Use HTTPS (see :func:configure_s3).
None
Returns
Name
Type
Description
Any
A pandas DataFrame: the group_by columns, value (the
Any
aggregate), and n_rows.
Raises
Name
Type
Description
ValueError
If agg is unknown or no output URIs are found.
register_design
registry.RunRegistry.register_design(design)
Persist (or replace) a :class:TargetDesign by name.
Idempotent on design.name: re-registering replaces the stored requirements wholesale. See REGISTRY_PROVENANCE.md §12.
Parameters
Name
Type
Description
Default
design
TargetDesign
The design to store.
required
Raises
Name
Type
Description
ValueError
If the design has no name, no requirements, a requirement with no attributes, or a min_active < 1.
MD5 hash of josh + config + file_mappings (12 chars).
required
josh_path
str
Path to the .josh script file.
required
config_content
str
Full text of the rendered configuration.
required
file_mappings
dict[str, dict[str, str]] | None
Dict mapping names to {“path”: “…”, “hash”: “…”}.
required
parameters
dict[str, Any]
Parameter values used to generate this config.
required
josh_content
str | None
Rendered .josh source content (optional).
None
reset_run
registry.RunRegistry.reset_run(label_or_hash)
Clear a run’s executions/results and reactivate it, keeping its config.
The in-place redo primitive behind the bad -> re-dispatch rule (REGISTRY_PROVENANCE.md §11.1). Deletes the run’s cell_data, run_outputs and job_runs — so a re-dispatch replaces rather than appends — and resets status to active (clearing any bad / superseded marks).
Unlike :meth:drop_run, the job_configs row (and config_parameters / session_configs) is kept. A re-dispatch of the identical config produces the same run_hash and reuses it; keeping the row also preserves referential integrity when a sweep has already registered the config up front, so the fresh executions can attach to it.
Locate the original .jshc file on disk and check if it still matches.
Looks up the session metadata to find the original config_path, then checks whether the file exists and whether its content has changed since it was registered.
Parameters
Name
Type
Description
Default
run_hash
str
The run hash to look up.
required
Returns
Name
Type
Description
A
ConfigSourceInfo
class:ConfigSourceInfo describing the file’s status.
Locate the original .josh file on disk and check if it still matches.
Compares the file at josh_path against the stored josh_content.
Parameters
Name
Type
Description
Default
run_hash
str
The run hash to look up.
required
Returns
Name
Type
Description
A
ConfigSourceInfo
class:ConfigSourceInfo describing the file’s status.
resolve_label
registry.RunRegistry.resolve_label(label)
Get the run_hash for a labeled run.
Parameters
Name
Type
Description
Default
label
str
The label to look up.
required
Returns
Name
Type
Description
str
The run_hash associated with the label.
Raises
Name
Type
Description
KeyError
If no run has this label.
run_history
registry.RunRegistry.run_history(label_or_hash)
Walk the supersession chain for a run, newest (current) first.
Replaces the retired resolve_latest() prefix-matching. Starting from the given run, follows superseded_by backwards to reconstruct the lineage. Because supersession points from old → new, the chain is recovered by finding, at each step, the run that the previous entry supersedes.
Parameters
Name
Type
Description
Default
label_or_hash
str
Label or run_hash of any run in the lineage. Typically the current (active) run.
required
Returns
Name
Type
Description
list[RunDetail]
A list of :class:RunDetail, current run first, then each run it
list[RunDetail]
superseded, oldest last. Length 1 when nothing was superseded.
Attach analysis attributes (factors, join keys) to a run_hash.
Attributes are the sanctioned home for what a run is about – the factors carried into analysis (site, treatment, scenario) and the join keys linking registry runs to Josh’s CSV output. They are the run-level slice of the registry’s tag store, keyed on the validated run_hash scope; the general :meth:tag_by_session_id / :meth:tag_by_run_id / :meth:tag_custom facility remains for non-run metadata. See REGISTRY_PROVENANCE.md.
Joins job_configs.run_hash / config_parameters.run_hash / job_runs.run_hash / cell_data.run_hash – all the same value. DiagnosticQueries.get_parameter_comparison(variable, param_name) picks these up automatically for any param_name that isn’t a declared sweep parameter.
Calling this again for the same run_hash merges into the existing attributes (like dict.update) rather than replacing them.
Parameters
Name
Type
Description
Default
run_hash
str
The run to attribute.
required
**attributes
Any
Attribute values to set, in whatever shape you like.
>>># Nested with time filter>>>with registry.spatial_filter(geojson=park_boundary):... with registry.time_filter(step_range=(0, 50)):... df = queries.get_timeseries("height", run_hash="abc123")
Mark run_hash as the replacement for an existing run.
Convenience wrapper over :meth:mark_run for the partial-rerun story: the replaced run becomes status='superseded' pointing at run_hash. If the replaced run held a label, that label is released so it no longer resolves to the stale run — attach it to the new run with :meth:label_run if desired.
Parameters
Name
Type
Description
Default
run_hash
str
The new, replacing run.
required
replaces
str
Label or run_hash of the run being retired.
required
reason
str | None
Free-text explanation stored on the superseded run.
Attach free-form JSON metadata to a specific run execution.
Distinct from set_attributes (run-level): a run_hash is what was run and can have several run_ids (e.g. under the pool collision policy); a run_id is which execution produced a given replicate. Joins job_runs.run_id / cell_data.run_id / run_outputs.run_id (the latter via job_runs).
Attach free-form JSON metadata under a synthetic scope.
For groupings that don’t correspond to any column in the schema – e.g. a site code shared by many run_hashes. Unlike set_attributes etc., there’s no existence check (nothing to check against) and no automatic join anywhere: recover matching keys with find_tagged(), then use them as a plain filter (e.g. run_hash IN (...)) against whatever table you’re actually querying.
Parameters
Name
Type
Description
Default
key
str
The key to tag (e.g. "JOTR001").
required
scope
str
Name for this synthetic grouping (e.g. "site"). Must not be one of the validated scopes (run_hash, session_id, run_id) – use the matching tag_by_* method for those instead.
required
**tags
Any
Tag values to set, in whatever shape you like.
{}
Raises
Name
Type
Description
ValueError
If scope is a validated scope, or no tags are given.