Talk to us
Shimmy docs

The Report

What Tuning found, per Workflow — and the file you ship it in.

What’s in it

Open a Workflow in the dashboard and choose Report. For each Step:

  • The winner — the cheapest model that held the Step’s quality — with its price per million tokens and how much cheaper it is than the Baseline.
  • The Backup chain — the best passing runner-up on a different allowed provider, then the Baseline. What Production serves when the winner can’t.
  • Every model tried — samples, how often it matched, its mean grade, its price, its real cost per call, how it compares to the winner, and whether it passed, was eliminated, or was still being tested.
  • The answers — every Tuning response is kept for the life of the Report, so you can read what each model actually said for your inputs, next to what the Baseline said.

The Workflow’s Overview tab draws the same Steps as a graph, colored by status.

Ready, versions, and finalizing

The Report fills in as Steps settle and is ready when every Expected Step has. A new version is stored when it becomes ready, when a Step’s answer changes (a prompt edit re-opened it, or drift re-searched it), and when you finalize it. Pick a version from the Report’s version menu; older versions don’t change.

Finalize early stores the Report as it stands, with any unsettled Steps marked as such — for when a rarely-run Step shouldn’t hold up a decision.

shimmy.lock

The Report exports as a lockfile mapping each Step to its model, provider and Backup chain. It’s built from the newest stored version, so it names a Report that will read the same tomorrow.

bash 
export SHIMMY_API_KEY=sk-opt-…  SHIMMY_WORKFLOW_ID=…
npx shimmy recordings pull    # backend Recordings → shimmy/recordings/
npx shimmy recordings diff --exit-code
npx shimmy lock pull          # the Report's shimmy.lock

Or Download shimmy.lock on the Report tab. Commit it: passthrough Production reads it, and its diff in a pull request shows exactly which Steps changed model. The format is in files and CLI.

Next