SURE Harness is the control plane for Pi/Codex-style TUI agents. You state intent in slash commands; the agent plans and executes; deterministic gates refuse to let a run finish until every artifact is on disk and legal. No silent fallbacks, no invented models.
From a model URL to a metric report — with a file on disk for every step in between.
Real sessions, replayed. The second worth watching is the one where a gate says no.
“Agent decides scope, scripts enforce format and execution.”
“Agent decides scope, scripts enforce format and execution.”
The onboarding run below hits a failing contract gate and repairs it — the retry budget is three per unit.
The agent decides what to do: which source, which datasets, which target, how to route. Where each artifact lands, what format it takes and whether its values are legal is decided by deterministic scripts and hook gates — the state machine advances only once the current unit's artifact is fully compliant, and three failed retries mark the unit FAILED instead of looping. Judgment stays with the model; discipline lives in code.
Gates are not suggestions — they are preconditions for a run to finish.
execution=vc must produce real VC submission evidence; if vc is unavailable the run fails outright rather than quietly dropping to local. The final report records both execution_path_requested and execution_path_actual, so where the run went is always auditable.
With device=auto and visible host CUDA, the run is CUDA-first; CPU fallback passes the gate only after at least three recorded CUDA environment repairs (cuda_repair_attempts_before_cpu, default 3) plus a non-empty fallback_reason. Installing a CPU-only torch does not count as a repair attempt.
Smoke fixtures exist for bounded validation only: 1–5 samples proving the model runs end to end. They are never treated as benchmark evidence; formal metrics go through the full route plan and dataset contract.
The pre_finish hook checks terminal artifacts: /sure_onboard must leave a verdict.json, /sure_eval a main_agent_run_report.json, and the state machine must have reached its terminal unit. Anything missing, and the run can only finish as incomplete or failed.
Four layers, and only the top one is allowed to decide anything.
Slash commands + planner. The human states intent; the agent breaks it into an executable order.
Hooks & state machines, deterministic scripts, schemas & contracts. Every unit's artifact is accepted before the next step is released.
Tracked as a submodule gitlink at sure/external/sure-evaluation, so the parent repo records the verified engine commit without vendoring its files. Route nodes and metric capabilities are read at runtime; the harness keeps no static support matrix.
Local, Docker, VC cluster. Where formal inference runs is user-controlled, and the decision is recorded in the artifacts.
Route chains are resolved at runtime by the engine and written into evaluation_route_plan.json — metric support is never inferred inside harness code. See every route in the Pipeline Atlas →
Every skill is self-contained: its own hooks, deterministic scripts and gate validators under scripts/, and artifact contracts under schemas/. The three workflow skills run a hooks/state-machine.ts state machine with retries capped at 3 per unit.
Starting from a model URL or a search, it runs strong-plus-weak task matching, metadata collection, MODEL_INPUT synthesis and rank-and-select, then publishes the handoff. The gates are match_task, synthesize_model_input and rank_and_select. Red line: every match carries a match_source; failed discovery emits an empty candidate list — never invented models.
Turns a model repo into a reproducible inference unit under sure/models/<model>/: discover, classify, pick a backend, build the env, fetch weights, generate the wrapper, then pass four validation stages — import/load/infer/contract. It ends in an internally consistent verdict.json. Device policy is CUDA-first with evidence-backed CPU fallback.
Orchestrates the evaluation of an onboarded model: dataset metadata decides task routing, the execution surface may only be materialized from scripts/templates/, and formal submission waits for a passing smoke test. Execution-surface isolation (red line 1) and the execution-policy check (vc_check.py) both live here.
Takes a results dir, a run dir or a bare predictions/ folder and recomputes the metrics through the current engine — useful when the normalizer, the scorer or an exact pipeline_id is what you want to change, not the model. Red lines: it never starts a model server or calls inference, and it never copies the old evaluation_payload.json, report.jsonl or metrics/ across; evaluation_only is always true and old_evaluation_reused always false.
Both locations can be overridden by environment variable instead: SURE_EVALUATION_HOME to point at an engine checkout outside the submodule, SURE_EVAL_DATASETS_ROOT for a dataset root containing sure_benchmark/jsonl.
Intent stays with the human. Execution goes to the agent. Evidence answers to the gates.