ROBIN

Reusable Open
Bioacoustic
INfrastructure

A Vision and Design Overview

ROBIN provides a common, open source foundation for running bioacoustic models, comparing their performance, and using expert-provided annotations to create better models. It is entirely storage and compute agnostic.

Explore the brief Download PDF

1What does ROBIN do, and why do we need it?

A shared foundation lets monitoring programs spend more time on science and less on rebuilding pipelines.

Bioacoustic monitoring programs collect tens or hundreds of terabytes of audio, far more than people can listen to. To find species in that much audio, programs use foundation models like BirdNET or Perch, which automatically find species calls. But it is hard to run these models at scale, keep track of thousands of runs, compare their outputs, and fine-tune models to perform better in the local ecoregions monitoring programs care about. Separate researchers, labs, and institutions each build software for this ad hoc, around both particular models and the platforms they use for storage and compute.

{"hint": "Hover or tap a part of the diagram.", "parts": [{"title": "Without shared infrastructure", "body": "Every program maintains its own scripts: one for BirdNET, one for Perch, one to copy audio, one to submit jobs, spreadsheets of labels, and folders like results_v2_final. They depend on each other in ways nobody wrote down.", "boxes": [[15, 185, 465, 630]]}, {"title": "Inference", "body": "Runs any model the same way and writes scores, detections and embeddings in one standard format. Adding a model means one small adapter, not a new pipeline.", "boxes": [[601, 263, 632, 104]]}, {"title": "Comparison", "body": "Tests models against expert labels and picks the clips experts should label next, so labelling effort goes where it improves models most.", "boxes": [[601, 407, 632, 104]]}, {"title": "Fine-tuning", "color": null, "body": "Trains a small classifier head on a backbone model using the program's own labels, to do better on local species.", "boxes": [[601, 551, 609, 104]]}, {"title": "Catalog port", "body": "Turns a selection into a list of recordings: from SoundHub, or simply a folder layout.", "boxes": [[601, 691, 120, 150]]}, {"title": "Storage port", "body": "Reads and writes audio and results, on Amazon S3 or a lab server.", "boxes": [[729, 691, 120, 150]]}, {"title": "Compute port", "body": "Runs the work on SageMaker, a Slurm cluster, or a laptop.", "boxes": [[857, 691, 120, 150]]}, {"title": "Inference models port", "body": "Connects the core to model-specific code: BirdNET, Perch, the Spotted Owl model, or a fine-tuned head.", "boxes": [[985, 691, 120, 150]]}, {"title": "Governance port", "body": "Checks whether a use is permitted and stamps results with their provenance.", "boxes": [[1113, 691, 120, 150]]}, {"title": "Labeling tools", "body": "Outside the core. Experts listen to the clips comparison picks, in SoundHub or JupyterBioacoustics, and their labels come back to fine-tuning.", "boxes": [[1377, 287, 248, 470], [1236, 420, 140, 155], [1205, 562, 171, 36]]}, {"title": "The new head joins the models", "body": "A fine-tuned head is registered as another model, so it runs through inference like BirdNET or Perch.", "boxes": [[985, 850, 360, 90]]}]} Reusable Open Bioacoustic Infrastructure Without shared infrastructure each program writes its own scripts for each step birdnet_run.py perch_batch.py s3_sync.sh submit_slurm.sh results_v2_final/ labels.xlsx train_head.ipynb compare.R results_v3/ rerun_site4.sh With ROBIN three modules in one core; one adapter for each outside system ROBIN's Core Modules Inference Runs any model the same way Comparison Tests models against expert labels; picks optimal clips to label Fine-tuning Trains a small head on the program's labels Catalog SoundHub or a folder layout Storage Amazon S3 or a lab server Compute SageMaker or a Slurm cluster Inference models BirdNET Perch · OWL Governance Provenance Tracking Head v2 the new head joins the models Labeling tools outside the core SoundHub JupyterBioacoustics Experts listen to the clips comparison picks and label them. clips to label labels Why ROBIN has three modules
Figure 1Without shared infrastructure, every program rebuilds the same pipeline. ROBIN provides one core that every program can reuse.

To best do bioacoustic monitoring, programs need to do more than just run existing models. Foundation models such as BirdNET and Perch are trained from recordings collected around the world. A monitoring program usually cares about a smaller set of local species, recorded at its own sites with its own equipment, and global foundation models can miss these species or confuse them with others. Monitoring programs can develop better models through an iterative cycle: run models on their catalog of recordings, quantify how reliable model predictions are for each species of interest, use this information to prioritize which clips to validate, and use these expert-validated audio clips to improve models for the taxa they are trying to monitor and evaluate the updated model on separate labeled recordings to check whether performance improves.

Without shared infrastructure, every step in this cycle is a separate script. A program trying to implement this cycle today manually maintains code for each step: one script to run BirdNET, another for Perch, another to copy audio from S3, another to send jobs to a cluster, Excel files with manual labels, a notebook for fine-tuning, and folders of results with names like results_v2_final_final. These pieces depend on each other in ways nobody wrote down, and adding a new model, a new storage system, or running on a different cluster breaks them.

Building and maintaining this code distracts ecologists from doing the science that large-scale bioacoustic data has the potential to enable.

ROBIN exists to solve this problem, by providing a common, open-source foundation for running bioacoustic inference at scale, validating and comparing model performance, and fine-tuning models for specific taxa — all entirely agnostic to where data is stored and what compute platform it is running on.

2Design overview

One reusable core connects to each program’s tools through a small set of adapters.

ROBIN has a core consisting of several parts, each of which are separate Python modules:

  1. Shared contracts define the files and data formats that every part of ROBIN uses: what a model must report about itself, what a run request contains, and what a result contains.
  2. The run manager keeps the record of all work: what was requested, on which recordings, with which settings, and what came out. The record makes every result traceable and prevents duplicate work.
  3. The inference engine runs any model in the same way. It reads audio, splits it into windows, runs the model, checks the output, and writes scores, detections and embeddings in a standardized format. A new model gets all of this work for free, so adding a model means writing one small adapter, not a new pipeline.
  4. The comparison module measures how well models perform against expert labels, and compares models against one another. It also finds the clips that should be a priority for experts to label next to best improve models using active learning criteria.
  5. The fine-tuning engine trains small classifiers on top of a backbone model (BirdNET, Perch) using a program’s own labels, with the aim of improving performance on local species of interest.
{"hint": "Hover or tap a part of the diagram.", "parts": [{"title": "Who drives ROBIN", "body": "People submit runs from the CLI today; the API will give web apps like SoundHub the same operations.", "boxes": [[345, 25, 245, 180]]}, {"title": "The record", "body": "The run manager keeps every run, task and outcome in PostgreSQL: on a laptop in Docker, or on Amazon RDS.", "boxes": [[700, 95, 250, 160]]}, {"title": "Run manager", "body": "Keeps the record of all work: what was requested, on which recordings, with which settings, and what came out. It makes every result traceable and prevents duplicate work.", "boxes": [[53, 354, 916, 104]]}, {"title": "Inference engine", "body": "Runs any model the same way: reads audio, splits it into windows, runs the model, checks the output, and writes scores, detections and embeddings.", "boxes": [[53, 522, 244, 410]]}, {"title": "Comparison module", "body": "Measures how well models perform against expert labels, compares models against one another, and picks the clips experts should label next.", "boxes": [[389, 522, 244, 410]]}, {"title": "Fine-tuning engine", "body": "Trains a small classifier head on a backbone model (BirdNET, Perch) using a program's own labels. The new head runs like any other model.", "boxes": [[725, 522, 244, 410]]}, {"title": "Shared contracts", "body": "The files and formats every part of ROBIN agrees on: what a model reports about itself, what a run request contains, and what a result contains.", "boxes": [[53, 996, 916, 104]]}, {"title": "ROBIN core", "body": "The same core runs everywhere. It only talks to the outside world through its five ports, so a result means the same thing on a laptop as on AWS.", "boxes": [[1417, 482, 316, 316]]}, {"title": "Catalog port", "body": "Resolves which recordings to process: from a folder layout, an S3 bucket prefix, a sheet, or the SoundHub API (via the small soundhub-robin adapter).", "boxes": [[1378, 30, 403, 164], [1500, 194, 150, 180], [1450, 354, 250, 131]]}, {"title": "Storage port", "body": "Reads and writes audio, work files and results, on local disk, S3, or other cloud storage.", "boxes": [[1085, 30, 274, 164], [1160, 194, 190, 432], [1298, 449, 154, 237]]}, {"title": "Governance port", "body": "Checks whether a proposed use is permitted and records what each recording is used for, following the rules of the deployment: fully open, CDFW, or an Indigenous partner's agreement.", "boxes": [[1805, 30, 309, 164], [1800, 194, 190, 440], [1697, 449, 148, 237]]}, {"title": "Compute port", "body": "Launches jobs, stops them and checks their status: on a local machine, AWS SageMaker, a Slurm cluster, or Kubernetes.", "boxes": [[1145, 1050, 360, 164], [1280, 810, 200, 240], [1336, 717, 215, 186]]}, {"title": "Inference models port", "body": "Connects the inference engine to model-specific code: BirdNET, Perch, the PNW Spotted Owl model, or a custom classifier head.", "boxes": [[1645, 1050, 360, 164], [1670, 810, 210, 240], [1600, 717, 215, 186]]}]} User CLI API PostgreSQL Runs · tasks SQL Inside the core Run manager Prepares tasks · starts workers · records outcomes Inference What is in the recordings? Runs any model the same way. Writes scores, detections, and embeddings. Comparison Which model can we trust? Tests models against expert labels. Picks the clips worth labelling next. Fine-tuning How do we do better here? Trains a small head on local labels. The new head runs like any model. results expert labels on the priority clips Shared contracts Work and result files · one format for every model · GBIF species names Catalog ROBIN core Governance Inference models Compute Storage soundhub-robin resolver Provenance Indigenous Partner BirdNET adapter SageMaker adapter Amazon S3 adapter Storage · where files live Local S3 GCS Catalog · how recordings are organized SoundHub Folders Bucket Sheet Governance · what is allowed Fully Open CDFW Indigenous Partner Compute · where work runs Local SageMaker Slurm k8s Models · what runs BirdNET Perch OWL Head One core, three modules, five ports
Figure 2ROBIN’s core modules sit inside five ports. Adapters translate between each port and an outside system.

ROBIN separates its core from every outside system (e.g. S3 storage, a Slurm cluster). The core talks to the external world through five fixed interfaces, called ports, for storage, compute, recording catalogs, inference models, and governance. For each outside system, an adapter translates communication with the outside system into the data types ROBIN’s core expects. There are five ports that surround ROBIN’s core:

  1. Compute launches jobs, stops them, and checks their status on a local machine, AWS SageMaker, a Slurm cluster, or a Kubernetes cluster.
  2. Storage reads and writes audio, work files, and results from the local disk, S3, or other cloud platforms.
  3. Catalog resolves the URIs of the recordings to process: for example, from a folder layout, an S3 bucket prefix, or the SoundHub API.
  4. Inference models connect the core inference engine to the model-specific code for e.g. BirdNET, Perch, the PNW Spotted Owl model, or custom classifier heads.
  5. Governance connects ROBIN to a set of rules for governance, data usage, and provenance tracking. This enables ROBIN to check whether a proposed use is permitted, and track what specific recordings are being used for downstream. This will be the least covered in this document; the governance appendix explains the motivation in more detail.

This design means support for a new environment is one adapter, not a complete rewrite. And because the core is the same everywhere, a result means the same thing on a laptop as on AWS.

3One core, different adapter stacks for different users

The same core fits different infrastructure, budgets, and data-governance needs.

Shared infrastructure is only useful if it can effectively be used by monitoring programs that are very different from each other. A state agency, a university lab, and an Indigenous community have different budgets, different computers, and different rules about who may hold their data. If each of them needs its own version of the software, we’re back to separate, ad hoc systems. Each deployment fits ROBIN to its setting with a deployment profile: a YAML file that picks one adapter for each port and names the host of the run manager database.

{"hint": "Hover or tap a row or a profile.", "parts": [{"title": "Models", "body": "Chosen for each run, in any deployment. Every model plugs into the same inference models port, so the same model can run under any profile.", "boxes": [[25, 40, 1885, 95], [345, 135, 340, 90]]}, {"title": "ROBIN core", "body": "The same run manager, inference engine and shared contracts in every deployment. Only the adapters change.", "boxes": [[55, 185, 450, 100]]}, {"title": "Catalog row", "body": "Each deployment finds its recordings its own way: the SoundHub API through soundhub-robin, a CSV list, or recorder-card folders.", "boxes": [[55, 309, 1830, 106]]}, {"title": "Storage row", "body": "The same storage port reads from an S3 bucket, a mounted lab server, or a partner's own disks.", "boxes": [[55, 421, 1830, 106]]}, {"title": "Compute row", "body": "Work runs as SageMaker jobs, on a campus Slurm cluster, or on a partner's own computers.", "boxes": [[55, 533, 1830, 106]]}, {"title": "Governance row", "body": "A shared provenance service, simple stamps beside the outputs, or a registry that applies the partner's own terms.", "boxes": [[55, 645, 1830, 106]]}, {"title": "Database row", "body": "Always PostgreSQL. Only where it lives changes: Amazon RDS, or Docker on a lab server or partner computer.", "boxes": [[55, 757, 1830, 106]]}, {"title": "CDFW with SoundHub", "body": "The profile in use today. People work in the SoundHub web app, which calls ROBIN's API; operators also use the CLI. Everything runs on AWS.", "boxes": [[695, 190, 340, 100]]}, {"title": "University lab", "body": "A lab that keeps recordings on its own server and runs work on the campus cluster, driven from the CLI.", "boxes": [[1127, 190, 340, 100]]}, {"title": "Indigenous partner", "body": "Everything stays on the partner's own disks and computers, and their registry decides what each use may do.", "boxes": [[1559, 190, 340, 100]]}]} MODELS · CHOSEN FOR EACH RUN, IN ANY DEPLOYMENT BirdNET 2.4 BirdNET 3 Perch 1.8 Perch 2 PNW Owl Model Custom classifier head Inference models · one per run ROBIN core Run manager · inference engine · shared contracts Catalog Storage Compute Governance Database Always PostgreSQL PROFILE · IN USE CDFW with SoundHub SoundHub web app → API · operator CLI SoundHub resolver soundhub-robin Amazon S3 Cloud bucket SageMaker AWS jobs Provenance service Shared registry and graph Amazon RDS Kept by AWS PROFILE University lab Operator CLI CSV list recordings.csv Local files Mounted lab server Slurm Campus cluster Files Stamps beside outputs PostgreSQL in Docker On the lab server or PROFILE Indigenous partner Partner interface → API Folder-layout resolver Recorder card folders Local files Partner disks Local Partner computers Partner registry The partner's own terms PostgreSQL in Docker On a partner computer or One core, many deployments
Figure 3Three deployments, one core. Each profile picks an adapter for every port.

Users create their ROBIN deployments in a repo, called e.g. soundhub-robin, where they define their deployment profile in YAML. ROBIN ships adapters for most common use cases, so most programs set up a deployment by writing YAML, not code.

Different users and programs will store and organize their recordings in different ways. Most catalogs fit a built-in adapter, like defined nested folder layouts or an S3 prefix. However, a larger system like SoundHub may need its own small adapter. In this case, soundhub-robin would also include an adapter, so CLI requests query the SoundHub API for all the recordings from e.g. a specific project, or all the recordings for a specific deployment within a certain year.

{"hint": "Hover or tap a row.", "parts": [{"title": "Platform API \u00b7 soundhub-robin", "body": "A custom adapter, shipped in the soundhub-robin repo. It asks the SoundHub API for a project's recordings using the user's own SoundHub credentials, and identifies each one by SoundHub's recording ID.", "boxes": [[31, 91, 1457, 208]]}, {"title": "Nested folders \u00b7 folder-layout", "body": "Built in. Reads recordings straight off recorder cards or disks; the recorder and start time come from the file names, following a layout set in YAML.", "boxes": [[31, 317, 1457, 208]]}, {"title": "Bucket prefix \u00b7 bucket-prefix", "body": "Built in. Lists everything under a prefix in an S3 bucket. The object key is the recording's identity, so files must never be renamed.", "boxes": [[31, 543, 1457, 208]]}, {"title": "Spreadsheet \u00b7 csv-list", "body": "Built in. Takes the rows of a CSV, with the ID and audio columns named in YAML. The simplest way to hand ROBIN an exact list.", "boxes": [[31, 769, 1457, 208]]}, {"title": "ROBIN core", "color": null, "body": "Every adapter produces the same kind of list: a namespace, an ID and an audio location. The run keeps that list, so it never asks the source again.", "boxes": [[1560, 91, 349, 886]]}]} CATALOG ADAPTERS · RUN BY THE CLI OR THE API, AT SUBMIT THE USER ASKS WHERE THE RECORDINGS LIVE THE CATALOG ADAPTER WHAT ROBIN RECEIVES { } Platform API “All of project 123” CDFW with SoundHub GET /projects/123/recordings/ [{"id": 48213, "path": "s3://…/48213.flac"}, {"id": 48214, "path": "s3://…/48214.flac"}, … 14,203 records, page 1 of 29 soundhub-robin CUSTOM SELECTION {"project": 123} NEEDS the user's SoundHub credentials IDENTITY SoundHub's recording ID COMES FROM written by SoundHub soundhub 48213 s3://sh-audio/p123/48213.flac namespace ID ↑ audio location Nested folders “Site 4, this season” Indigenous partner /media/cards site04/ 2026-05-01/ S4A07_20260501_053000.wav S4A07_20260501_060000.wav 2026-05-02/ … site05/ folder-layout BUILT IN SELECTION {"site": 4, "season": 2026} NEEDS read access to the disk IDENTITY recorder + start time in the name COMES FROM layout set in YAML cards S4A07_20260501_053000 file:///media/cards/site04/2026-05-01/… Bucket prefix “Everything under ARU-12” Any deployment on S3 s3://lab-audio/ 2025/ARU-03/0001.flac 2026/ARU-12/0001.flac 2026/ARU-12/0002.flac 2026/ARU-12/0003.flac 2026/ARU-14/0001.flac bucket-prefix BUILT IN SELECTION {"prefix": "2026/ARU-12/"} NEEDS permission to list the bucket IDENTITY the object key: never rename COMES FROM bucket set in YAML lab-audio 2026/ARU-12/0001.flac s3://lab-audio/2026/ARU-12/0001.flac Spreadsheet “The rows in my sheet” University lab recordings.csv id audio site 2026-05-01-a …/a.wav 4 2026-05-01-b …/b.wav 4 2026-05-02-a …/a.wav 5 2026-05-02-b …/b.wav 5 csv-list BUILT IN SELECTION {"file": "recordings.csv"} NEEDS read access to the file IDENTITY the sheet's ID column COMES FROM file and columns set in YAML lab 2026-05-01-a file:///srv/audio/2026-05-01/a.wav ROBIN core One kind of list, from every source. Keeps each list. A run never asks the source again. ITS FIVE PORTS Catalog Storage Compute Models Governance Many kinds of source, one kind of list
Figure 4Catalog adapters turn a user’s selection into an explicit list of recordings.

4Run lifecycle

Every request becomes traceable work, from the selected recordings to the final results.

At the scale of tens or hundreds of terabytes, one request becomes thousands of jobs that run for hours or longer. Some jobs fail. Some recordings are damaged. People rerun work and change settings. Without a careful record, a program loses track of what has run, pays to process the same audio again, and cannot say which result is the most recent one. ROBIN’s run lifecycle is designed so it’s always known what ran, what failed, and which result is current.

A run is one request to apply one model to a set of recordings. A user submits the request through a web app, the CLI, or the API. A catalog adapter turns the user’s selection into a recording list, and the run manager freezes that list and divides eligible recordings into tasks. For each task, it writes a work file to storage, which says which model to run, with which settings, on which recordings. The compute adapter then launches one worker for each task. Each worker reads its work file, runs the inference engine, and writes its outputs and a results file. The run manager reads each result file, checks it, and records the outcome in the database (Figure 5).

{"hint": "Hover or tap a step.", "parts": [{"title": "1 \u00b7 Resolve the recordings", "body": "The catalog adapter turns the user's selection (a SoundHub project, a folder, a bucket prefix, a sheet) into an explicit list of recordings.", "boxes": [[15, 120, 1045, 70]]}, {"title": "2 \u00b7 Submit the list", "body": "The run manager freezes that list. From here on the run never asks the source again, so the run always means the same recordings.", "boxes": [[15, 210, 400, 60]]}, {"title": "3 \u00b7 Check the terms", "body": "The governance adapter asks whether this use of these recordings is allowed before any work starts.", "boxes": [[15, 285, 1045, 60]]}, {"title": "4 \u00b7 Record the run", "body": "The run, its recording list and its tasks go into PostgreSQL. Recordings that already have a current result for the same settings are skipped.", "boxes": [[15, 360, 420, 60]]}, {"title": "5 \u00b7 Write the work files", "body": "One work.json per task goes to storage: which model, which settings, which recordings, and where to write outputs.", "boxes": [[15, 436, 1045, 60]]}, {"title": "6 \u00b7 Launch the workers", "body": "The compute adapter starts one job per task, passing the worker image's digest and the path to its work file.", "boxes": [[15, 500, 765, 98]]}, {"title": "7 \u00b7 Each worker runs its task", "body": "Outside the core, with files only and no database. A worker reads its work file, runs the inference engine through the model's adapter, and writes outputs plus one outcome for every recording.", "boxes": [[15, 615, 770, 70], [784, 517, 618, 311], [1110, 440, 300, 70]]}, {"title": "8 \u00b7 Read the result files", "body": "The run manager reads each task's result.json from storage and checks it against the work file it wrote.", "boxes": [[15, 825, 1045, 60]]}, {"title": "9 \u00b7 Record the outcomes", "body": "In one transaction: the task's outcome, each recording's outcome, and its output files. Valid results become current.", "boxes": [[15, 900, 420, 60]]}, {"title": "10 \u00b7 Stamp outputs", "body": "The governance adapter puts a provenance stamp on every result, with the terms that apply to it.", "boxes": [[15, 976, 1045, 60]]}]} How a run moves through ROBIN ROBIN CORE ADAPTERS OUTSIDE ROBIN Resolve the recordings Submit the list Check the terms Record the run Write the work files Launch the workers Each worker runs its task Read the result files Record the outcomes Stamp outputs 2 3 4 5 6 7 8 9 10 1 Catalog Catalog adapter Source system SoundHub, folders, a bucket, a sheet explicit list, frozen here Governance Governance adapter Authorization and Consent Is this use allowed? PostgreSQL run · list · tasks Storage Storage adapter Storage one work.json per task Compute Compute adapter Compute platform SageMaker Slurm Kubernetes Local launch a job image digest + work file path Image registry robin-worker@sha256:9f3c… pulled by digest Worker task 0007 · a copy of the image work.json INFERENCE ENGINE windows › model › checks result.json Perch BirdNET OWL Head model adapters One outcome for every recording, written beside the work file. Worker task 0008 Worker task 0009 … one copy per task Worker task 0010 7 runs outside the core: files only, no database Storage Storage adapter Storage result.json for each task PostgreSQL each outcome Governance Governance adapter Provenance tracking a stamp on every result How a run moves through ROBIN
Figure 5How a run moves through ROBIN, from resolving recordings to stamping outputs.

Three rules make this reliable at scale:

  1. Only the run manager changes the record. Workers never connect to the database; they only exchange files with the run manager.
  2. Each recording gets its own outcome. One damaged recording does not fail the whole task. The run manager tracks which recordings succeeded and which failed, and why.
  3. Avoiding duplicate, unnecessary runs. ROBIN keeps at most one valid result for each recording and unique work request (model adapter + selected custom settings). Users can directly ask questions like “Which models ran on this recording, and with which settings?” and “Which result is the current one?”

5Looking forward

Inference and run management are developed; comparison and fine-tuning are next.

ROBIN is still in development. The core inference module, run manager, and the adapters for the storage and compute used by the legacy soundhub_model_runner (AWS S3 for storage, SageMaker for compute) are developed. The development of the CLI is ongoing. The first major milestone will be using ROBIN for a run of Perch on the entire historical catalog of CDFW in SoundHub. After that, development on the comparison and fine-tuning modules will be the next step.

Further reading

The brief continues in three appendices: integration, governance, and technical design.

Appendix 1Integrating ROBIN into SoundHub

This appendix describes how ROBIN works with SoundHub, and the ways the two can be joined to work together better later.

The present

In the present, ROBIN operates entirely independently, outside of a thin wrapper repository soundhub-robin that defines a deployment configuration and the custom adapter for the catalog port that resolves recording URIs using the SoundHub API. Inference runs are then submitted via the ROBIN CLI, which talks to a very lightweight AWS Aurora RDS instance as its database. When ROBIN finishes an inference run, a path to the S3 URI of the resulting detection Parquet is stored in the database. The soundhub_api_demo implementation that uses api_dock can then query that database to get the path to the Parquet file to return when SoundHub requests the detections for a particular model and recording.

ROBIN never replaces a file it has written, so every run keeps its own outputs. To give readers one place to look, the controller also copies the current detections for each configuration into a single dataset, partitioned by recording. A read for a whole project then reads one hive-partitioned dataset. The database still decides which result is current; the dataset is a copy of that decision.

{"hint": "Hover or tap a step.", "parts": [{"title": "1 \u00b7 Choose the recordings", "body": "An operator on our team picks what to run, for example all of SoundHub project 123.", "boxes": [[120, 232, 1030, 80]]}, {"title": "2 \u00b7 Resolve the list", "body": "The soundhub-robin catalog adapter asks the SoundHub API for the project's recordings, using the operator's own SoundHub token. ROBIN never stores it.", "boxes": [[120, 336, 1030, 80], [425, 440, 215, 75]]}, {"title": "3 \u00b7 Record the run", "body": "robin runs submit freezes the list and records the run, the list and the claims in PostgreSQL, so nothing already done is run twice.", "boxes": [[120, 440, 300, 80], [800, 410, 400, 110]]}, {"title": "4 \u00b7 Launch the jobs", "body": "The controller (robin reconcile --loop) starts one SageMaker worker job per task. Jobs read SoundHub's audio from S3 with their own role.", "boxes": [[120, 544, 1400, 80], [1240, 762, 200, 52]]}, {"title": "5 \u00b7 Write the outputs", "body": "Each worker writes its result.json and Parquet outputs to Amazon S3.", "boxes": [[120, 648, 1400, 80], [1140, 684, 110, 112]]}, {"title": "6 \u00b7 Record the outcomes", "body": "The controller checks each result file and records the outcomes in the database.", "boxes": [[120, 752, 1030, 80]]}, {"title": "7 \u00b7 Show the detections", "body": "api-dock reads ROBIN's published views to find the right Parquet file, and SoundHub's web app shows the detections by recording ID. SoundHub's code is unchanged.", "boxes": [[120, 856, 1030, 80]]}]}How SoundHub runs a model today Our team runs ROBIN for SoundHub. SoundHub's code is unchanged; its web app reads the results. SOUNDHUB OUR TEAM · ROBIN AWS 1 Choose the recordings 2 Resolve the list 3 Record the run 4 Launch the jobs 5 Write the outputs 6 Record the outcomes 7 Show the detections Operator our team · picks project 123 soundhub-robin catalog adapter SoundHub API projects and recordings PostgreSQL run · list · claims Controller robin reconcile --loop SageMaker one worker job per task Amazon S3 result.json · Parquet Controller checks each result, records it api-dock reads the published views Web app shows detections by recording ID lookup list robin runs submit launch detections The lookup uses the operator's own SoundHub token. ROBIN never stores it. Jobs read SoundHub's audio from S3 with their own role.
Figure 6Today: ROBIN runs beside SoundHub, and api_dock reads results from ROBIN’s database.

Later

Option 1. A user clicks, for example, “Run BirdNET on this project”. SoundHub’s backend checks the user’s access, builds the explicit list of URIs, and sends POST /v1/runs with the work request as the JSON body to ROBIN’s API server. SoundHub’s backend is a client of that server; it does not host ROBIN code.

This is, in effect, a small microservice architecture. In practice, the likeliest way to make this smooth is to extend the current way api-dock is used to act as a proxy that handles parsing the request from SoundHub and constructing the proper JSON body to send to ROBIN’s API.

{"hint": "Hover or tap a part of either option.", "parts": [{"title": "Option 1 \u00b7 SoundHub", "body": "SoundHub's backend checks the user's access and builds the explicit list of recordings. It installs no ROBIN code.", "boxes": [[92, 324, 288, 312]]}, {"title": "Option 1 \u00b7 ROBIN as a service", "body": "Our team runs ROBIN's API server, run manager, controller and database as a separate service that other programs could share.", "boxes": [[484, 324, 288, 312]]}, {"title": "Option 1 \u00b7 POST /v1/runs", "body": "SoundHub's backend calls ROBIN's API over HTTP with the work request as JSON. Open question: how the API trusts SoundHub's backend and names the user.", "boxes": [[355, 340, 150, 145]]}, {"title": "Option 1 \u00b7 Detections via api-dock", "body": "api-dock, run by our team, reads ROBIN's published views and serves detections back to SoundHub.", "boxes": [[76, 630, 345, 80]]}, {"title": "Option 1 \u00b7 AWS", "body": "The controller launches SageMaker jobs that read audio from S3. Jobs still need their own access to SoundHub's audio.", "boxes": [[76, 748, 712, 88], [560, 670, 90, 30]]}, {"title": "Option 1 \u00b7 Trade-offs", "body": "Clean separation and independent releases, at the cost of one more service and one more database for our team to run.", "boxes": [[90, 875, 700, 210]]}, {"title": "Option 2 \u00b7 SoundHub's web app and backend", "body": "The backend checks its own users and calls ROBIN directly. There is no API to secure.", "boxes": [[928, 380, 228, 136]]}, {"title": "Option 2 \u00b7 ROBIN as a library", "body": "SoundHub installs ROBIN unchanged. The run manager is called in the same process, and the controller runs as a background worker.", "boxes": [[1188, 380, 380, 192], [1085, 520, 75, 30]]}, {"title": "Option 2 \u00b7 One PostgreSQL", "body": "ROBIN's tables live in their own schema inside SoundHub's database, so there is only one database to run.", "boxes": [[928, 596, 640, 104]]}, {"title": "Option 2 \u00b7 AWS", "body": "The controller launches SageMaker jobs with SoundHub's own AWS role.", "boxes": [[892, 748, 712, 88], [1378, 712, 135, 30]]}, {"title": "Option 2 \u00b7 Trade-offs", "body": "One deployment and no API to secure, but SoundHub takes on ROBIN's dependencies and migrations, and its releases are tied to ROBIN's.", "boxes": [[906, 875, 700, 210]]}]}Two ways to bring ROBIN into SoundHub Both keep the same run record and duplicate checks. They differ in who runs what, and where. OPTION 1 · MICROSERVICE A separate ROBIN service SoundHub's backend calls ROBIN's API over HTTP. + SoundHub installs no ROBIN code and no ROBIN dependencies. + ROBIN releases on its own; other programs can share the service. + Credentials stay with people and roles, as designed. − One more service and one more database for our team to run. − Open: how the API trusts SoundHub's backend and names the user. − Jobs still need their own access to SoundHub's audio in S3. SOUNDHUB Web app Backend checks the user, builds the list SoundHub DB ROBIN SERVICE · OUR TEAM API server Run manager ROBIN DB published views Controller POST /v1/runs HTTP api-dock our team detections AWS SageMaker Amazon S3 launches jobs OPTION 2 · ONE CODEBASE ROBIN inside SoundHub SoundHub's backend installs ROBIN and calls it directly. + One deployment and one database for SoundHub to run. + No API to secure: the backend checks its users and calls ROBIN. + ROBIN's code is unchanged; SoundHub installs it as a library. − SoundHub takes on ROBIN's dependencies and its migrations. − The controller needs SoundHub's own AWS role to launch jobs. − SoundHub's releases are tied to ROBIN's releases. SOUNDHUB · ONE CODEBASE, ONE DEPLOYMENT Web app Backend checks the user ROBIN, AS A PYTHON LIBRARY Run manager called in the same process Controller a background worker Python call One PostgreSQL ROBIN's tables in their own schema soundhub robin AWS SageMaker Amazon S3 SoundHub's AWS role
Figure 7Later: SoundHub submits runs to ROBIN’s API through api-dock.

Option 2. SoundHub could integrate the run management system and database into its own PostgreSQL database. This saves a separate DB deployment. Further, it could also integrate the API into its own server, so the deployment of ROBIN that serves SoundHub doesn’t run as a separate service.

For an application this size, a monolithic application may be preferable and cheaper to host, but there is a cost/benefit analysis here as to how much refactoring the SoundHub backend would require to support this.

Appendix 2Governance and provenance tracking

A major goal of DSE’s work is to build tools that enable Indigenous Data Sovereignty (IDSov). As part of that work, the Scoping Plan Draft pitches a system for data governance and provenance tracking that follows CARE and OCAP principles, and enables FAIR principles within the limits they set.

This system has two parts:

  1. A governance model that translates a data governance agreement into rules that software can check: who can use which data, for what purpose, and what rules apply to the results.
  2. A provenance standard for tracking how biodiversity data products are produced, so a result can be traced back to its sources, and a source can trace forward everything it contributed to producing. It extends W3C’s PROV standard.

This system is a separate project; ROBIN connects to it through its governance port. This appendix explains what this system does, why it is separate from ROBIN, and how ROBIN connects to it.

What the system does

Governance

A governance agreement states what a community permits its data to be used for. The scoping plan divides it into axes:

  • Usage consent. Which uses are permitted, and for what purpose.
  • Access tiers. Who can see the data and results derived from it.
  • Ownership and attribution. Who owns the data and its downstream products, and who must be credited for its use.
  • Location sensitivity. Whether the place a recording was made can be shown, and at what precision.

Provenance tracking

Provenance answers two questions:

← Backward

What produced this result? Which records, model, settings, under what consent terms, and when.

Forward →

What did this recording help produce? For a recording or a whole deployment, which detections, fine-tuned models, and downstream analyses used it?

Vocabulary

This standard is an extension of W3C PROV, which has three fundamental classes:

Entity

A thing.

Activity

Makes or changes entities.

Agent

The person or system responsible for an activity.

Why the system is separate from ROBIN

  1. It is not specific to bioacoustics. Most of the standard applies equally to camera traps, eDNA, and museum collections. It should be its own published standard with its own implementations, and it should know nothing about ROBIN or audio.
  2. ROBIN must work without it. Many ROBIN users will not need governance. ROBIN’s own record stays the source of truth for what ROBIN did, and ROBIN never needs the standard to do its work correctly. The consent rules stay out of ROBIN’s core.
  3. Different groups need different rules. One deployment can follow CDFW’s rules on sharing recording locations; another can follow a community’s consent agreement. A custom adapter for each is simpler than one large system with many settings.
  4. One place for the rules across many tools. SoundHub, labelling tools, exports, and ROBIN can all ask the same registry and report to the same graph. Each rule is written once, and every governed action is in one audit log.

The last point also shows where each rule is enforced. ROBIN checks a requested use before it starts work. A rule about who may read results, such as hiding recording locations, is enforced by the tool that serves the results, such as api-dock. ROBIN’s part is to put the terms that apply on every stamp, so the reader can enforce them.

How ROBIN connects

The governance port has two operations:

  • Report. ROBIN sends a provenance record for each thing it does: a run accepted, a task finished, an output written. Records go out through an outbox that the controller delivers, so a slow or unavailable registry never blocks a run.
  • Check. Before ROBIN accepts a run, it asks whether the proposed use is permitted. A refusal stops the run before any work starts.

This likely will look like two separate interfaces:

  1. Provenance out: a sink (ROBIN -> standard). When ROBIN does something, it needs to report what it did. The same database transaction that records an outcome could also write the outgoing provenance records to a provenance_outbox table.
  2. Governance in: a gate (standard -> ROBIN). ROBIN asks whether a use is permitted, and obeys the answer. Unlike the sink, this can block work, so it runs at a clear point and fails clearly.

At this point, designing this interface is not the goal, and will come later after the bulk of ROBIN is built.

In the first SoundHub deployment there is no governance adapter at all (governance: none): the IAM permissions that SoundHub sets for each operator and for the job role decide what can be read, and ROBIN keeps only its own record.

Appendix 3Technical design

APackages

ROBIN consists of a small ecosystem of Python packages.

Package Job May import Today
robin_contracts File formats, port interfaces, shared value types, canonical JSON and digests, and checks that more than one package needs nothing Built
robin_inference_engine Runs one model over a list of recordings and writes outputs contracts Built
robin_models Model adapters (BirdNET, Perch, OWL, and an ONNX runtime for classifier heads) and the output-version table contracts Built
robin_adapters Storage adapters (local disk and S3, through ondio); later the compute adapters contracts Storage built, Compute in progress
robin_worker The robin-worker command: one work file in, one result file out contracts, engine, adapters, models Built
robin_run_manager The record, its SQL, the operations, the profile loader, and the inference configuration contracts; adapters only when a profile is built with compose In progress
robin_cli The robin command run manager In progress
robin_api The web API run manager Later
robin_comparison, robin_fine_tuning Comparing models against expert labels; training heads not designed Later
{"hint": "Hover or tap a package or an image family.", "parts": [{"title": "robin_api", "body": "The web API, planned. A standalone FastAPI server that gives web apps like SoundHub the same operations as the CLI.", "boxes": [[48, 133, 268, 140]]}, {"title": "robin_cli", "body": "The robin command operators use today. It calls the run manager's operations and loads deployment packages like soundhub-robin by name.", "boxes": [[348, 133, 268, 140]]}, {"title": "soundhub-robin", "body": "Lives outside ROBIN, in its own repo: SoundHub's deployment profile and its catalog adapter. The CLI finds it by name; nothing in ROBIN imports it.", "boxes": [[748, 133, 368, 100]]}, {"title": "robin_run_manager", "body": "The record, the rules and the operations: submit, prepare, launch, record, cancel, withdraw, retry. The only package that touches the database.", "boxes": [[48, 293, 568, 100]]}, {"title": "robin_worker", "body": "What runs inside a worker image: one work file in, one result file out, for any kind of work its image holds. Never touches the database.", "boxes": [[748, 293, 1128, 100]]}, {"title": "robin_adapters", "body": "The built-in storage, compute and catalog adapters. The run manager only composes them; it never reaches past their interfaces.", "boxes": [[348, 473, 288, 100]]}, {"title": "robin_inference_engine", "body": "Runs one model the same way for every model: windows the audio, calls the model adapter, checks the output, writes Parquet.", "boxes": [[698, 473, 278, 100]]}, {"title": "robin_models", "body": "The model adapters (BirdNET, Perch, the Spotted Owl model, classifier heads) and the version table the worker reads.", "boxes": [[1068, 473, 238, 100]]}, {"title": "robin_comparison", "body": "Planned. Scores models against expert labels and picks the clips worth labelling next.", "boxes": [[1378, 473, 238, 100]]}, {"title": "robin_fine_tuning", "body": "Planned. Trains classifier heads on a program's own labels.", "boxes": [[1648, 473, 228, 100]]}, {"title": "robin_contracts", "body": "File formats, port interfaces and shared types. Every package depends on it, and it imports nothing from ROBIN.", "boxes": [[48, 653, 1828, 100]]}, {"title": "TensorFlow family", "body": "The worker image every base model shares today: TensorFlow 2.21 with BirdNET, Perch and OWL.", "boxes": [[48, 905, 433, 276]]}, {"title": "ONNX family", "body": "A light image for classifier heads, which read stored embeddings instead of raw audio.", "boxes": [[513, 905, 433, 276]]}, {"title": "Comparison family", "body": "Planned. Its own image with its own libraries, so comparison never shares dependencies with models.", "boxes": [[978, 905, 433, 238]]}, {"title": "Training family", "body": "Planned, likely PyTorch, for fine-tuning heads.", "boxes": [[1443, 905, 433, 238]]}]} robin_api Web API (later) robin_cli The robin command soundhub-robin A resolver, outside ROBIN loads robin_run_manager The record, the rules, the operations robin_worker One work file in, one result file out, for any kind of work its image holds robin_adapters Storage and launchers robin_inference_engine Runs one model robin_models BirdNET, Perch, OWL, heads robin_comparison Planned: scores models robin_fine_tuning Planned: trains heads robin_contracts File formats, port interfaces, shared types. Imports nothing from ROBIN. only compose version table only robin.models robin.tasks (proposed) Worker images: one for each runtime family The release names each model's family. The launcher starts each task in that family's image. An image refuses a work file for a model or kind of work that it does not hold. TensorFlow family Today every base model shares it tensorflow 2.21 robin_models: BirdNET, Perch, OWL robin_inference_engine robin_worker · robin_adapters robin_contracts ONNX family Heads read stored embeddings onnxruntime robin_models: ONNX heads robin_inference_engine robin_worker · robin_adapters robin_contracts Comparison family Planned its own libraries robin_comparison robin_worker · robin_adapters robin_contracts Training family Planned PyTorch, likely robin_fine_tuning robin_worker · robin_adapters robin_contracts Each package imports only what it needs.
Figure 8The ROBIN package ecosystem.

BContracts and files

The different components of ROBIN exchange files (not function calls), wherever they run on different computers. Different components communicate using a shared set of versioned contracts defined in robin_contracts, a package consisting only of Pydantic data classes. These contracts are composed into two primary files which workers use to run inference and describe their results.

The work file (work.json) is everything a worker needs for one task: an ID, where to write outputs, the version of the code the worker must have, and a description of the work itself. The work names the model, its files with digests, the recordings, the settings, and the outputs that are requested.

{"hint": "Hover or tap a class.", "parts": [{"title": "InferenceTask \u00b7 work.json", "body": "The top of the work file the run manager writes: the task's ID, where to write outputs, which output versions are current, and the work itself.", "boxes": [[96, 171, 328, 191]]}, {"title": "ModelOutputVersion", "body": "Which version of a model's outputs this task writes, so a result can be matched to the settings that made it.", "boxes": [[96, 391, 328, 116]]}, {"title": "InferenceWork", "body": "The work itself: the recordings, the model, whether the input is audio or stored embeddings, settings, resources and requested outputs. Its digest is computed, never stored.", "boxes": [[506, 171, 358, 266]]}, {"title": "RecordingRef", "body": "One recording: a namespace and ID that identify it, where its audio is, its duration, and its stored embeddings if a head will read them.", "boxes": [[946, 171, 328, 191]]}, {"title": "InputArtifact", "body": "A stored input such as an embeddings file, given by URI and checksum.", "boxes": [[1326, 171, 298, 116]]}, {"title": "OutputRequest", "body": "One requested output (scores, detections or embeddings), the contract it must follow, and how long to keep it.", "boxes": [[946, 391, 328, 141]]}, {"title": "PinnedModel \u00b7 used by both files", "body": "The exact model: its model or head card and its files. The work file asks for it and the result repeats it, so a result always names what made it.", "boxes": [[506, 601, 358, 116]]}, {"title": "PinnedFile", "body": "One model file, by URI, SHA-256 digest and size, so a worker can prove it loaded the right weights.", "boxes": [[946, 601, 328, 141]]}, {"title": "InferenceTaskResult \u00b7 result.json", "body": "The top of the result file the worker writes: the task, the digest of the work file it read, the ROBIN release, timings, and one of four outcomes.", "boxes": [[96, 821, 328, 241]]}, {"title": "TaskRefusal", "body": "The worker would not start, because the work file was misspecified or asked for something its image does not hold.", "boxes": [[96, 1101, 328, 141]]}, {"title": "WorkerError", "body": "An umbrella for any other unexpected failure.", "boxes": [[96, 1271, 328, 116]]}, {"title": "InferenceCompleted", "body": "The task ran: the pinned model, the window geometry, the output artifacts, and an outcome for every recording, covered or failed.", "boxes": [[506, 821, 358, 291]]}, {"title": "InferenceFailure", "body": "A problem that affected the whole task, such as corrupted model weights.", "boxes": [[506, 1161, 358, 166]]}, {"title": "ArtifactRecord", "body": "One output file: its kind, contract, the recording it covers, where it is, its checksum, size and row count.", "boxes": [[946, 821, 328, 266]]}, {"title": "RecordingCoverage", "body": "A recording that was covered: how many windows completed and how many rows each output got, or why it had zero windows.", "boxes": [[946, 1121, 328, 191]]}, {"title": "RecordingFailed", "body": "A recording that failed, with the failure. One damaged recording never fails the whole task.", "boxes": [[946, 1351, 328, 141]]}, {"title": "FailureReport", "body": "What went wrong: a code, the stage it happened in, the window if there was one, and detail.", "boxes": [[946, 1531, 328, 166]]}]} work.json robin.inference-task/1 · written by the run manager result.json robin.inference-task-result/1 · written by the worker InferenceTask v schema_version contract id task_id uuid output_root text output_version object work object ModelOutputVersion model text version int ≥ 1 InferenceWork v schema_version contract recordings list model object or input audio | embeddings settings map resources map outputs list id work_digest computed, not stored RecordingRef id namespace text id value text audio_uri text duration_seconds float? embeddings object? InputArtifact uri text checksum sha256 OutputRequest or kind scores | … contract_id contract policy / retention per kind PinnedModel card ModelCard | HeadCard files map PinnedFile uri text digest sha256 size_bytes int InferenceTaskResult v schema_version contract id task_id uuid? work_file_digest sha256? robin_release text started_at datetime finished_at datetime or result one of four TaskRefusal or outcome "refused" code text detail text WorkerError or outcome "error" detail text InferenceCompleted or outcome "completed" v schema_version contract work_digest sha256 recipe object model PinnedModel window_geometry object artifacts list coverage list failed list InferenceFailure or outcome "failure" v schema_version contract work_digest sha256 failure object ArtifactRecord kind text contract_id contract id namespace text id value text uri text checksum sha256 size_bytes int rows int ≥ 1 RecordingCoverage id namespace text id value text windows_completed int …_rows int zero_window_reason text? RecordingFailed id namespace text id value text failure object FailureReport code text stage text window_start_s float? detail text Parquet file PinnedModel Used by both files exactly one zero or one one or more zero or more v names the contract and version id identifies the task, work or recording or chooses the variant The contracts.
Figure 9The contracts: two JSON files for each task, each with a named, versioned contract.

The result file (result.json) says how the task ended:

  • Completed. With an outcome for each recording, covered or failed.
  • Failure. A problem that affected the entire work, e.g. a corrupted model weights file.
  • Refused. The worker would not start due to a misspecification in the work file.
  • Worker error. An umbrella around all other unexpected failures.

Outputs. There are three types of outputs, each of which are Parquet files:

  • Embeddings. The output prior to the last layer from a foundation model (Perch, BirdNET) for a particular window.
  • Scores. The set of scores for a particular window, associating each taxon with the output score from the model.
  • Detections. A set of detections in a window derived from scores, by a threshold or by keeping the top k labels.

Both scores and detections for each taxon are associated with the GBIF taxon through each model’s taxa registry.

A request says how many scores to keep: all of them (full), those above a floor (thresholded), or the highest k in each window (top_k). The choice matters at scale. Full BirdNET scores are about 7.8 million rows for each hour of audio (1,200 windows × 6,522 labels), so most runs keep fewer.

Failures. The engine names the stage at which something failed, and each stage has one scope:

  • Whole work. Validating the request, fetching the model files, loading the taxa registry, and building the model. These run before the first recording, and a failure here ends the task with a failure result.
  • One recording. Fetching the audio, reading an input embeddings file, running the model, checking a window, and writing an output. A failure here is recorded against that recording, and the worker continues with the next one.

A recording too short for even one window is not a failure: it is covered, with zero windows and a reason.

Storage layout. Each run has its own run folder, associated with a UUID, and each task that belongs to its run has its own subfolder within it:

<root>/runs/<run>/tasks/<task>/work.json
<root>/runs/<run>/tasks/<task>/result.json
<root>/runs/<run>/tasks/<task>/outputs/...

CModels, cards, and heads

ROBIN runs two kinds of model. A backbone, such as BirdNET or Perch, reads audio and produces embeddings, scores, or both. A head is a small classifier that reads a backbone’s saved embeddings and produces scores. Fine-tuning will produce heads. (A model such as the PNW Spotted Owl model goes straight from audio to scores and has no embeddings.)

Model cards. Each model has a card: a short YAML file that says what the model is. It gives the window length, the sample rate, how audio is downmixed, resampled and padded (in the case of a final window shorter than the full window size), the width of its embeddings, what kind of scores it emits, and the digest of the taxa registry that labels its scores. A card is identified by the hash of its contents. A head’s card names the exact backbone card whose embeddings it reads, and ROBIN refuses to pair a head with any other backbone.

Pinned files. The work file (work.json) pins each of the model’s files by location, SHA-256 digest and size. The engine checks every digest and size before it builds the model. ROBIN never rehosts model weights. They come from either public repositories (Kaggle, HuggingFace, Zenodo) or a user’s S3.

Taxa registries. A registry gives each output position of a model its meaning: a label, and where possible a scientific name, a common name and a GBIF taxon key. A label with no GBIF match is published as unresolved (15 of Perch v1.8’s 10,932 labels) and is never silently dropped.

Heads. A head is three files: a head card, an ONNX graph and a taxa registry. It runs on the CPU with onnxruntime, so scoring stored embeddings needs no TensorFlow. The graph must include any normalisation it was trained with. ROBIN cannot detect a head that leaves it out; it runs, and gives plausible wrong scores.

Output versions. Each model has an integer output version. It goes up in any change that changes that model’s outputs, whatever the cause: changes in the adapter, the engine, preprocessing, bug-fix, or a library such as TensorFlow. A change that leaves the outputs the same keeps the version.

Images. Models whose libraries can share an environment form a runtime family, and each family has its own worker image. PyTorch for fine-tuning, or a model pinned to an older TensorFlow, gets its own family instead of breaking everyone else’s.

DProfiles, deployments, and credentials

A deployment profile is a YAML file which describes the settings for one deployment of ROBIN: where the database lives, and which adapters to use for each port.

Profiles also refer to the environment variables needed for storage and compute adapters to access their systems. ROBIN has no access of its own; it uses the credentials of the person who ran the command. ROBIN doesn’t store any of these credentials, ever.

{"hint": "Hover or tap a section of the profile.", "parts": [{"title": "storage \u2192 S3 storage adapter", "body": "Writes every work file, result file and output under s3://robin-outputs/prod.", "boxes": [[33, 105, 640, 75], [1133, 98, 420, 52]]}, {"title": "compute \u2192 SageMaker compute adapter", "body": "Starts each task in its model family's image, on that model's instance type, with at most 20 running at once. The role comes from an environment variable.", "boxes": [[33, 175, 640, 255], [1133, 167, 420, 70]]}, {"title": "governance \u2192 none yet", "body": "In the first pass SoundHub's IAM policies decide what the job role can read, and ROBIN keeps its own record.", "boxes": [[33, 428, 640, 50], [1133, 420, 420, 70]]}, {"title": "tasks \u2192 controller settings", "body": "Not an adapter: how many recordings go into each task when a run is split up.", "boxes": [[33, 474, 640, 50], [1133, 504, 420, 52]]}, {"title": "resolver \u2192 soundhub-robin", "body": "Names a module and function. At submit, the CLI imports it and calls build(settings) to get the SoundHub catalog adapter.", "boxes": [[33, 520, 640, 100], [1133, 570, 420, 70]]}, {"title": "prod.yaml \u2192 database", "body": "The production environment replaces whole sections of config.yaml. Here: PostgreSQL on RDS, connecting with an IAM token made from the person's AWS login, so no password is stored.", "boxes": [[33, 669, 640, 233], [1133, 706, 420, 70]]}, {"title": "compose", "body": "Runs once at startup. Merge prod.yaml over config.yaml, fill each ${NAME} from the environment (a missing one stops startup), check the combination makes sense, then build the adapters.", "boxes": [[753, 61, 300, 841]]}]} config/config.yaml shared by every environment 1 storage: 2 kind: s3 3 root: s3://robin-outputs/prod 4 compute: 5 kind: sagemaker 6 role_arn: ${ROBIN_SAGEMAKER_ROLE} 7 max_in_flight: 20 8 instance_type: ml.m5.xlarge 9 models: 10 perch/v8: 11 instance_type: ml.m5.2xlarge 12 images: # proposed 13 default: …/robin-worker:tf 14 onnx: …/robin-worker:onnx 15 governance: # first pass 16 kind: none 17 tasks: 18 recordings_per_task: 200 # illustrative 19 resolver: # read by the CLI 20 code: soundhub_robin.resolver:build 21 api_url: https://<soundhub-api> 22 session_token: ${SOUNDHUB_SESSION_TOKEN} config/prod.yaml replaces whole sections 1 database: 2 host: robin.….rds.amazonaws.com 3 dbname: robin 4 user: ${USER} 5 auth: rds_iam # a token, no password 6 region: us-west-2 7 sslrootcert: /etc/robin/rds-ca.pem ROBIN_PROFILE=config ROBIN_ENV=prod selects config.yaml, then prod.yaml compose runs once, at startup Merge prod.yaml replaces whole sections of config.yaml. Fill Each ${NAME} comes from the environment. A missing one stops startup. Check SageMaker compute needs S3 storage: a job cannot read local files. Build One adapter for each port. Only compose imports them. S3 storage adapter Writes every file under s3://robin-outputs/prod SageMaker compute adapter Starts each task in its family's image, on its model's instance type; at most 20 at once No governance adapter yet First pass: SoundHub's IAM policies decide what the job role (role_arn) can read. ROBIN keeps its own record. Controller settings Not an adapter: how the work is split into tasks passes by compose The SoundHub resolver (soundhub-robin) code names a module and function. At submit, the CLI imports it and calls build(settings) PostgreSQL, with an RDS IAM token Made from the person's AWS login for each connection; no password is stored From profile to adapters.
Figure 10A deployment profile selects one adapter per port.

ERun manager database design

The run manager keeps its records in a PostgreSQL database.

Note

PostgreSQL is the only database choice, either on a laptop or lab server (in a Docker container) or in a larger-scale production environment (Amazon RDS). There may be reasonable objections to this, arguing that the database should itself follow the port/adapter pattern, but this has been deferred for now.

{"hint": "Hover or tap a table.", "parts": [{"title": "runs", "body": "One row per request: what kind of run, its state, who submitted it, and when it was cancelled or withdrawn and by whom. Nothing is ever deleted.", "boxes": [[74, 163, 338, 291]]}, {"title": "tasks", "body": "One row per task: its run, the task it retries, its state, the work file and its digest, the result file, the launcher's job ID, instance type, release and timings.", "boxes": [[74, 523, 338, 391]]}, {"title": "inference_runs", "body": "What an inference run asked for: the configuration digest, current output versions, the frozen recording list, and how the selection was made.", "boxes": [[484, 163, 388, 191]]}, {"title": "recording_outcomes", "body": "One row per recording in a run: its state, why it was skipped, which task handled it, and any failure. A recording can hold only one active claim per configuration, which is how duplicate work is avoided.", "boxes": [[484, 403, 388, 316]]}, {"title": "output_files", "body": "Every output file a worker wrote, by run and recording: its kind, contract, URI and checksum.", "boxes": [[484, 783, 388, 266]]}, {"title": "configurations", "body": "A model plus its settings, identified by a digest. Two runs with the same digest asked for the same thing.", "boxes": [[944, 163, 288, 141]]}, {"title": "presets", "body": "Named configurations people can reuse, like a saved recipe.", "boxes": [[944, 343, 288, 141]]}, {"title": "models", "body": "Each registered model: its card, its pinned files, and the current version of its outputs.", "boxes": [[1294, 163, 288, 166]]}, {"title": "current_files \u00b7 view", "body": "For every recording and configuration, the output files that are current, and whether they are up to date. This is what tools like api-dock read.", "boxes": [[944, 603, 288, 291]]}, {"title": "current_datasets \u00b7 view", "body": "One dataset per configuration that gathers its current results.", "boxes": [[1294, 603, 288, 116]]}]} runs PK run_id uuid kind text state text submitted_by text created_at timestamptz cancelled_at timestamptz cancelled_by text withdrawn_at timestamptz withdrawn_by text tasks PK task_id uuid FK retry_of uuid FK run_id uuid state text work_uri text work_digest sha256 result_uri text launcher_ref text instance_type text robin_release text created_by text started_at timestamptz finished_at timestamptz inference_runs PK FK run_id uuid FK configuration_digest text output_versions jsonb recordings jsonb selection_provenance jsonb recording_outcomes PK FK run_id uuid PK namespace text PK value text UQ configuration_digest text UQ output_versions jsonb state text skip_reason text FK task_id uuid failure jsonb FK input_file_id bigint output_files PK file_id bigint FK run_id uuid FK namespace text FK value text kind text contract_id text uri text checksum sha256 configurations PK digest text FK model_id text spec jsonb presets PK name text FK configuration_digest text created_at timestamptz models PK model_id text card jsonb pinned_files jsonb output_version integer current_files view namespace, value configuration_digest output_versions up_to_date bool kind contract_id uri checksum run_id current_datasets view configuration_digest dataset_uri Every kind of work Inference What a run computes Published views, for readers such as api-dock exactly one zero or one one or more zero or more PK primary key FK foreign key UQ unique while active The record.
Figure 11Tables in the run manager database.

Models, configurations, and presets

An operator registers a model from a YAML file that holds its card and pinned files (robin models register FILE). A model is identified by its name and version. Registering the same model again with the same card and files returns the existing row. The same name and version with different content is refused with model_conflict, because a changed model needs a new version.

A configuration is what a run computes, without its recordings: the model, what it reads (the recordings’ audio, or the embeddings that a named backbone configuration saved), its settings, and the outputs requested. An operator writes it as YAML. ROBIN resolves it against the registered model and checks it with the same request checks the inference engine uses (they live in robin_contracts), so a bad configuration is refused when it is created, not later in a worker.

A configuration is identified by a SHA-256 digest of its resolved contents: a format tag, the model’s name, version and card digest, the digest of each of the model’s files, the input type, the settings, and the output requests. Two runs with the same values have the same configuration, whether or not they came from a preset. File locations are left out, so moving a model’s files does not create a new configuration. A test pins the digest of a fixed example, so a new release of ROBIN cannot silently change how configurations are identified.

A head’s configuration names the backbone configuration whose embeddings it reads. ROBIN refuses it unless that backbone configuration saves embeddings, the head was trained on that backbone, and the embeddings are in the shape and format the head expects.

A preset is a fixed name for one configuration (robin presets create NAME --configuration FILE). A preset cannot be edited, and never needs to be: when a new release of ROBIN changes a model’s outputs, it raises the model’s output version (Output versions), which is not part of the configuration, so the preset still names the right thing. Creating the same preset again returns it. The same name for a different configuration is refused with preset_conflict.

No work is done twice

A recording has at most one valid result for each configuration and set of output versions. The database enforces this across all runs, including work that is still in progress: a unique index covers every recording outcome that is pending or covered, keyed by the recording (identified by a namespace and a value within it, such as a SoundHub recording ID), the configuration, and the output versions. So a double-click, a network retry, or two overlapping runs never compute the same thing twice.

ROBIN does not try to recognize a repeated request. Each request makes a new run, and the index stops the run from repeating work: when a run is prepared, it claims each of its recordings (Preparing a run), and a recording that is already covered or in progress elsewhere is recorded as skipped instead. A run whose recordings are all covered or in progress elsewhere finishes with no tasks.

Each recording in a run has one outcome. Only pending and covered hold a claim.

  • Skipped. Already covered, or in progress in another run. The reason is recorded.
  • Pending. Claimed by this run, and not yet finished.
  • Covered. Finished, with its output files.
  • Failed. Not finished, with the stage and reason. The claim is released, so another run can try again.
  • Released. Given up by a cancel before the work started.
  • Withdrawn. Removed from the current results by a withdrawal.

Current results

A published view, current_files, gives one row for each current file, by recording, configuration and kind: its location, its checksum, the run that made it, and whether it is up to date. A result is up to date when its output versions are the highest recorded for its model. Readers read exactly the files this view names, and never match paths with a pattern, because a pattern would also match withdrawn and failed runs.

FThe controller loop

The controller is the one process that makes sure that ROBIN’s records in the run manager database match the state of the outside world. It prepares runs, launches tasks via the compute port, asks the compute port what happened, and records run outcomes.

ROBIN needs a control loop because a run spans two kinds of systems that may not necessarily be able to transact. The record is in a PostgreSQL database, and the work happens on a compute system (SageMaker, k8s, Slurm).

The design of the controller loop is derived from the following requirements:

  • Workers only exchange files. They never touch the database and hold no database credentials, so they cannot record their own outcomes. The controller reads each result, checks it, and records it.
  • Outcomes come from asking, not being told. Every compute adapter can answer “describe this job”. Not every compute platform is capable of sending events to an external process. So the controller polls describe on the compute adapter, which updates the run manager on the status of the run.
  • Credentials belong to the machine the deployment is made from. ROBIN doesn’t store other people’s credentials, because that creates a new origin for security vulnerabilities.

The cost of this design is latency. A result can wait up to one polling interval (30 to 60 seconds) before it is recorded, and the current dataset can be about a minute behind the database. For jobs that run for minutes or hours, and for systems without the capacity to send events back, this was accepted.

One pass

One pass of the controller does the following:

  1. Stop cancelled work and settle unconfirmed launches. Ask the compute adapter to stop the started tasks of runs that are listed as cancelled or withdrawn in the database.
  2. Record ended tasks. Ask the compute adapter for the state of each running task, and record the outcome of each one that has ended (see Recording an outcome).
  3. Prepare waiting runs. Claim recordings and make tasks for new runs (see Preparing a run).
  4. Launch queued tasks, oldest first, up to the limit of running tasks.
  5. Update the “current results” view for the recordings whose results just changed.
  6. Wait for the next poll.

Preparing a run

Submit stores the run with its frozen recording list and configuration. Preparation for a run happens in one transaction, which claims each recording with INSERT ... ON CONFLICT DO NOTHING against the unique index from No work is done twice. A claim is a pending outcome row that says “this run is processing this recording with this configuration”. A recording whose claim fails is written as skipped, along with the reason. Finally it splits the claimed recordings into queued tasks and marks the run as ready.

Launching a task

A deployment profile’s MAX_IN_FLIGHT caps the total number of tasks that can run at once, so a large run does not go over the compute system’s quota. A task counts toward the limit if it is in the state submitting, submitted, or running.

Starting a task takes three steps, because the call to the compute system cannot be inside a single database transaction:

  1. In a short transaction, check that the limit is not reached and that the task’s output versions are current, and move the task from queued to submitting.
  2. Outside the transaction, write work.json, and ask the compute adapter to start the job.
  3. In a second transaction, save the job’s ID and mark the task submitted.

Unconfirmed launches. The controller can stop between steps 2 and 3: the job may have started, but the database does not know it. Each job’s name comes from its task’s ID, and every compute adapter must make a repeated start of the same task return the job that already exists, not start a second one. So the next pass simply repeats step 2. If the adapter cannot tell what happened, the task keeps its place under the limit and ROBIN reports it as uncertain, rather than risk running it twice.

A task moves through these states:

  • Queued. Waiting for a free place under the limit.
  • Submitting. A place is reserved; the job may or may not have started.
  • Submitted. The compute system has accepted the job.
  • Running. The compute system reports that the job has started.
  • Finalizing. The job has ended. ROBIN still has to read and record its result. The task no longer counts toward the limit.
  • Succeeded. A valid result is recorded. Some recordings may still have failed.
  • Failed. The launch, the whole work, or the result file failed.
  • Cancelled. Cancelled before launch, or stopped without a result.

Recording an outcome

When the compute system reports that a job has ended, the task moves to finalizing and no longer counts toward the limit. The controller then reads result.json and checks three things:

  1. It names the SHA-256 of the work file that the controller wrote.
  2. It gives an outcome for every recording in the work.
  3. Every output file it names is inside the task’s folder.

Then, in one database transaction, the controller locks the task and records the task’s outcome, each recording’s outcome, and the output files. Recordings with valid results become current when the transaction commits.

Cancels, withdrawals, and retries

These commands change the database all at once. The controller does the outside work on its next pass. Preparing, starting, and recording each lock the run and read it again, so all three see a cancel or a withdrawal in the same order.

  • Cancel. Frees the claims of queued tasks at once. The next pass stops the tasks that have started. If a valid result arrives after the cancel, ROBIN still records it.
  • Withdraw. Does the same, and also takes the run’s results out of the current results, in the same transaction. A result that arrives later never becomes current.
  • Retry. Makes a new task for the run’s failed recordings, with retry_of pointing to the old task.

Nothing is deleted, and each command is safe to repeat.

Running the controller

The controller runs via the CLI command robin reconcile. Without any options it does a single pass of the controller loop and exits. With --loop it will repeat passes until it gets a SIGINT or SIGTERM from the operating system. The controller loop is designed to be run on the web server that hosts the API. Until then, it should be run while runs submitted via the CLI have not concluded.

GCLI design

Operators use the robin command from the CLI. It calls the run manager’s operations and connects to PostgreSQL using an operator’s specific credentials. Every command has the form:

robin [--profile DIR] [--env ENV] [--json] <noun> <verb> [arguments]
{"hint": "Hover or tap a command group.", "parts": [{"title": "robin runs", "body": "Everything about runs: submit a list of recordings, follow its progress down to each recording and task, and cancel, withdraw or retry it. Each command is safe to repeat.", "boxes": [[170, 100, 870, 330]]}, {"title": "robin models", "body": "Register a model card and its files, and look up what is registered.", "boxes": [[360, 455, 680, 120]]}, {"title": "robin presets", "body": "Give a configuration (a model plus settings) a fixed name people can reuse.", "boxes": [[360, 600, 680, 120]]}, {"title": "robin reconcile", "body": "One pass of the controller loop: stop cancelled work, record ended tasks, prepare waiting runs, launch queued tasks. With --loop it repeats until stopped.", "boxes": [[360, 742, 680, 40]]}, {"title": "robin db migrate", "body": "Create or update the run manager's tables.", "boxes": [[360, 802, 680, 40]]}, {"title": "robin whoami", "body": "Show the database login your commands will use.", "boxes": [[360, 862, 680, 40]]}, {"title": "Global options", "body": "--profile picks the deployment and --env its environment (there is no default, so you never hit production by accident). --json prints one JSON object for scripts.", "boxes": [[1050, 90, 400, 185]]}, {"title": "Every list command", "body": "Pages with --limit and --cursor, or --all for everything.", "boxes": [[1050, 310, 300, 75]]}, {"title": "No API in between", "body": "CLI commands call the run manager directly, with your own database login.", "boxes": [[1050, 425, 310, 80]]}]} robin runs submit start a run on a list of recordings list list runs, newest first show one run's status and counts recordings each recording's outcome tasks tasks and their failure detail cancel stop work that has not finished withdraw take its results out of current retry run failed recordings again models register add a model card and its files list list registered models show one model presets create give a configuration a fixed name list list presets show one preset reconcile [--loop] move all work forward; --loop repeats db migrate create or update the tables whoami show your database login Global options --profile DIR which deployment --env ENV which environment; no default --json one JSON object on stdout --help, --version need no profile Every list command --limit N --cursor C | --all Commands call the run manager directly, with your own database login. They do not go through the API. The robin command.
Figure 12The robin command tree.

What happens when a command runs

Every command goes through the same seven steps before it does its own work. A failure at any step stops the command, and nothing changes in the database.

  1. Choose the deployment. The profile folder comes from --profile or, failing that, ROBIN_PROFILE. The environment comes from --env or ROBIN_ENV. There is no default environment, and ROBIN never runs on config.yaml alone, so every command names the deployment it acts on. If the folder, the environment, or the environment’s file is missing, the command exits with code 2. --help and --version skip all of this.
  2. Load the profile (Section D). The environment file replaces whole sections of config.yaml, then each ${VAR} is filled in from the shell. An unset variable or a malformed file is refused with invalid_profile. From here on, every command prints the profile and environment it is using on stderr, so a stale ROBIN_ENV left in a shell is easy to spot.
  3. Test the login. The CLI opens one connection to the database and closes it, so a failure has a precise reason: not_authenticated (the login was refused), database_not_found, server_not_verified (the server’s certificate failed the check), or unavailable. The server’s own error text is never shown, because it can contain the password.
  4. Open a connection pool. The pool’s size and time-outs come from the profile. With auth: rds_iam, each new connection gets a fresh IAM token, made on the operator’s own computer from their AWS credentials.
  5. Ask the database who is running the command. This is the actor: the name ROBIN records as responsible for an action, such as registered_by on a model or submitted_by on a run. The CLI does not take it from a flag or from the computer’s user name. It asks PostgreSQL, with SELECT session_user, which returns the database login that PostgreSQL has just verified. Because the operator cannot simply type a name, a run is attributed to whoever’s credentials actually logged in. This is a short read-only query, made once per command, before the schema check, so it works even on an empty database (which is why robin whoami works before robin db migrate).
  6. Check the schema. Every command except db migrate and whoami is refused with schema_not_current unless the database has exactly this release’s migrations: the same versions, names, and SHA-256 checksums. If the database is only behind, the message says to run robin db migrate. If it has different migrations, the message says to use the release that migrated it.
  7. Run the operation. Each operation is one function that owns its own transaction. The CLI passes the actor from step 5 to it as an argument, and the operation writes it into the record. The database never fills in the actor by itself. This matters for the API: it will connect as one database login on behalf of many people, so session_user would name the API, not the person, and the API will pass its own actor instead. When the operation finishes, the CLI closes the pool.

The actor is also what ties a run to its submitter’s credentials. robin reconcile only launches, stops, and records the outcomes of runs that the current actor submitted, so one operator’s credentials are never used for another operator’s run. Attribution is only as strong as the database logins behind it: each person needs their own login, and an IAM policy that lets a person log in only as themselves.

HAPI design

The API gives web interfaces (like SoundHub) the same operations as the CLI. It is still in the planning phase. It will be ROBIN’s own server: a standalone FastAPI application run by a given deployment. It supports the same operations as the CLI, with the exception of robin reconcile (see Running the controller), which is intentionally run as a background process by the web server.

{"hint": "Hover or tap a group of routes.", "parts": [{"title": "runs", "body": "The same operations as robin runs. Submitting returns 202 right away with the run in preparing; the controller loop on the server does the rest.", "boxes": [[70, 145, 1060, 335]]}, {"title": "presets", "body": "Read and create named configurations.", "boxes": [[70, 530, 1060, 90]]}, {"title": "models", "body": "Read and register models.", "boxes": [[70, 660, 1060, 90]]}]} runs POST /v1/runs submit; returns 202 and the run in preparing GET /v1/runs list runs, newest first GET /v1/runs/{id} status, counts, failure summary GET /v1/runs/{id}/recordings each recording's outcome GET /v1/runs/{id}/tasks tasks and full failure detail POST /v1/runs/{id}/cancel stop unfinished work POST /v1/runs/{id}/withdraw remove results from current POST /v1/runs/{id}/retry retry failed recordings presets GET /v1/presets[/{name}] read presets POST /v1/presets create a preset models GET /v1/models[/{id}] read models POST /v1/models register a model The routes.
Figure 13The planned API surface.