Recordings

In the Dev Phase, each Step’s call is made for real once and replayed after that. Building and testing an AI app or agent stops costing money, stops depending on the network, and stops being flaky because a model answered differently this time.

How a call is answered

Every call in the Dev Phase is matched to a Recording by its Step and a request hash — a hash of the parts of the request that change the answer: each message’s role, text and tool calls, the names of the tools offered, and whether JSON mode is on. Sampling parameters and the model name are not part of it, so pinning a different Baseline doesn’t invalidate your Recordings.

strict is the default in CI under a test runner: a test whose input changed should fail loudly, not pass on a stale answer.

Recording is a real call through Shimmy on its keys, drawn from your free Dev allowance. Replays are free.

Where Recordings come from

The Dev source decides where replays are read from:

• local — files in your repo, under shimmy/recordings/. A recording made locally is written there too. The default under a test runner: tests run with no network and no key.
• remote — Shimmy serves them from the backend. The default outside a test runner, so SHIMMY_PHASE=dev against a staging environment just works.

Set either with SHIMMY_DEV_SOURCE or in code:

const shimmy = new Shimmy({
  phase: 'dev',             // 'dev' | 'tuning' | 'production'
  devSource: 'local',       // Dev: replay from the repo, or 'remote' from Shimmy
  recording: 'auto',        // Dev: 'auto' | 'record' | 'strict'
  production: 'managed',    // Production: 'managed' | 'passthrough'
  recordingsDir: 'shimmy/recordings',
  lockfile: 'shimmy.lock',
});

Pulling Recordings into the repo

Recordings made through the backend — by a teammate, in staging, in CI — are pulled with the shimmy command, shipped with the Python and TypeScript SDKs:

export SHIMMY_API_KEY=sk-opt-…  SHIMMY_WORKFLOW_ID=…
npx shimmy recordings pull    # backend Recordings → shimmy/recordings/
npx shimmy recordings diff --exit-code
npx shimmy lock pull          # the Report's shimmy.lock

diff --exit-code exits non-zero when a pull would change anything, which makes “are the committed Recordings current?” a CI check.

Re-recording

When a Step’s prompt changes, its Recordings no longer match. In auto the SDK replays the Step’s latest Recording and warns once — enough to keep a test suite running while you work. To refresh:

SHIMMY_RECORDING=record pytest     # or: pytest --shimmy-record

The file format and the request hash are specified in files and CLI; all three SDKs and the backend compute the same hash, so a Recording made from Python replays in TypeScript.

Next

• Testing — assertions on Steps, and making them fail on purpose.