SURE-EVAL
Speech Evaluation · Reproducible by Design

Speech-eval scores that are
reproducible & traceable

SURE-EVAL declares every metric as a versioned pipeline of nodes. Same route, same inputs, same locked environment — the same score, with every step on the record.

Explore the pipeline atlas GitHub Repo ↗
Design

One metric, one declarative route

An evaluation is not a black-box call but a fully declared pipeline: describe the route first, then run it into a report. Scoring nodes wrap the community standards — sclite, sacrebleu, meeteval, WeNet — so the numbers reconcile with the toolchain you already trust. Take Chinese ASR character error rate (CER) —

Input
hyp + ref
A whole test set of transcripts and references, aligned by key
→
Normalize
wetext_norm
Inverse text normalization — spoken forms to written, both sides alike
→
Score
wenet_cer
Corpus-level edit distance with per-character S/D/I counts
→
Report
report.json
Score + full pipeline description, both written to disk
→
Result
CER 10.53%
2 substitutions in 19 characters — every point recomputable
$ git clone https://github.com/PigeonDan1/sure-evaluation && pip install -e .　# install once — base text metrics ready; heavy nodes build their own envs
$ sure-eval agent plan asr --language zh --metric cer --json　# agent-facing: routes & env readiness
$ sure-eval metric describe asr --language zh --metric cer --output asr.json　# describe the route
$ sure-eval metric run --pipeline asr.json --ref-file ref.txt --hyp-file hyp.txt　# run & report
✓ report.json　"score": 0.10526315789473684　"all": 19, "cor": 17, "sub": 2, "ins": 0, "del": 0
Pipeline Atlas

The whole framework in one picture

Each colored ribbon is an auditable evaluation declaration: a task passes through frontend, transcription, normalization and scoring nodes before landing in a report. Click a route to open its Route JSON; enter from a task card below and the atlas keeps only that task’s real paths, with no unrelated shadows.

⌕
Filters
Language
Node stage
Focus task
Capabilities

Start with a task. Follow its exact routes.

Choose what you evaluate, then inspect the nodes it actually runs, the contracts it pins and the JSON it produces. The catalog spans recognition, translation, synthesis, conversion, enhancement, target-speaker extraction and speaker analysis.

Agent Workbench

SURE Harness: an agent runs the whole pipeline

SURE Harness plugs this framework into a Pi-style terminal agent: you state intent through slash commands, the agent plans and executes, and artifact gates keep every step reproducible and auditable — four commands from model discovery to final report, with no silent execution fallbacks.

Configure
/sure_init
Pick the agent, wire auth, run environment checks
→
Discover
/sure_feed
Find and classify candidates from ModelScope / HuggingFace / GitHub
→
Prepare
/sure_onboard
Build a reproducible local inference unit with a gated verdict
→
Evaluate
/sure_eval
Execute the deterministic route plan and score the outputs
→
Artifacts
Reports & audit trail
Run reports, metric payloads and a full artifact manifest, all on disk
Explore SURE Harness GitHub branch ↗
Evidence model

Comparable means jointly specified.

Same checkpoint is not enough. Same predictions are not enough.

A
Execution varianceSame checkpoint, different inference protocol
same utterance→serial number
WER (%)0.0

Same checkpoint ≠ same executable system

B
Scoring variancePredictions fixed, normalization changes
Model A48.1
Model B3.7

Model B ranks first

A scoring contract can reverse the apparent ranking

RCallable toolentry point + version
ΠInference protocoldecoding + fallback
KNormalizationtext contract
EEvaluatormetric + aggregation
Comparableonly when all four match
Versioned nodes + manifestsRebuildable isolated runtimesReport + pipeline written together

Make your next evaluation report stand up to scrutiny.

Explore the pipeline atlas GitHub Repo ↗
$ sure-eval metric run --pipeline asr.json --ref-file ref.txt --hyp-file hyp.txt