Settled charges
Read the final billing amount rather than estimating a total from the cheapest catalog row.
Change one thing. Measure what happens.
Prepare a controlled comparison of prompts and model behavior. Review cost and output quality together before deciding what to change.
Experiment / Workflow
From a question to an evaluation.
Create a control and treatment in your workspace. Save a draft, then start it to assign matching requests using weighted random or sticky per-user allocation.
Only requests with your X-CI-Experiment header enter an experiment. Pausing stops assignment; promotion sends future matching requests to your chosen winner. Existing observations stay available.
- Define a meaningful difference between variants
- Choose a representative workload
- Measure settled cost and token use
- Review output quality before expanding
One comparison / Four steps
Evidence before expansion.
- Create 2–10 weighted variants with one control.
- Compare prompts, models, or explicit technique selections.
- Review recorded charges, token use, failures, and 1–5 quality ratings.
- Pause to review, or promote the variant you choose.
New experiments start without samples. No savings or statistical confidence is invented; use representative traffic and your own quality criteria.
Open experimentsCost belongs beside quality.
Use the request history to compare charges and token use. Keep your own quality criteria explicit so a lower bill does not conceal an unsuitable answer.
Representative outputs
Evaluate the kinds of prompts, tool calls, and edge cases your application actually uses.
Observed reliability
Review request status and timing, then decide whether the evidence supports a wider rollout.