Talk to us
Shimmy docs

Migrating

From a base_url swap to the SDK — and when not to bother.

When not to bother

If your traffic is plain request/response — a support bot, a classifier, a RAG endpoint with no multi-step structure — the base_url swap is most of the benefit and the SDK adds little:

const openai = new OpenAI({
  baseURL: 'https://rfa-labs.com/v1',
  apiKey: process.env.SHIMMY_API_KEY,
});

Caching, routing and the quality gate all work. What you are missing is structure the optimizer can only guess at, and if your calls have no structure worth declaring, there is nothing to gain.

When it is worth it

Any of these means the edge is currently getting your structure wrong:

  • Concurrent work. Parallel sub-calls are recorded as a chain, and as a phantom loop when they share a step shape.
  • Context compaction. The moment history is trimmed, one run silently becomes two.
  • A summarizing agent. It never builds a prefix, so runs never link.
  • More than one replica. The run tracker is process-local; a five-step run can fragment into five one-step runs.
  • Prompt tuning. Editing a system prompt resets that step’s converged model search to zero.

And regardless of structure: reporting outcomes removes a paid LLM judge from the loop.

The migration, one rung at a time

Nothing here is all-or-nothing. A call with no annotations behaves exactly as it did before the SDK existed.

1. Wrap the client

const shimmy = new Shimmy();
const openai = shimmy.wrap(new OpenAI());   // instead of setting baseURL

Same routing you had. You can now read optimizerMeta(res) for per-call cost accounting, which the raw swap discards.

2. Scope your runs

Wrap the outermost handler. One line, and run identity stops being inferred.

await shimmy.run('handle-ticket', async () => { /* existing code */ });

3. Name your steps

Wherever you make a call, say what it is. This is where prompt edits stop resetting your accumulated optimization.

await shimmy.step('classify', { kind: 'classification' }, () => /* … */);

4. Report outcomes

The rung that pays for itself. You almost certainly already have the check:

const parsed = await shimmy.verify(() => JSON.parse(text));

Environment variables

OPTIMIZER_API_KEY and OPTIMIZER_BASE_URL are still honored, so nothing breaks on upgrade. SHIMMY_* takes precedence when both are set.

What does not change

  • Your provider keys. Still yours, still per-request.
  • The wire field. Annotations ride under optimizer, unchanged.
  • model: "auto". Still how you let the optimizer route; a pinned model is still an instruction that disables routing. model: "discover" goes further: it searches each step for the cheapest model that holds quality, on platform keys, once discovery is enabled on the agent.
  • Your client. wrap() returns the object you passed. Every OpenAI feature the SDK has never heard of keeps working.

Rolling back

Delete the run/step scopes and the SDK stops declaring anything; the edge returns to inference. Remove wrap() and set baseURL yourself, and you are exactly where you started. There is no stored state to unwind — annotations describe calls, they do not configure your account.