04 · golden sets · offline + online eval · CI gate
evals before users find the bugs
harnesses, golden sets, regression gates in ci.
from $12k · 2–4 weeks·harness + CI gate, one surface
30 min · no workshop · no deck
// for who
technical founders who shipped a prompt and prayed — and now need a gate before the next model swap breaks production.
// not for
benchmark vanity · rfp theater · strategy decks · stamps without systems
// what you leave with
- a golden set that matches the real product surface
- offline eval in ci that can block a merge
- a thin online sample so regressions show up after ship too
// what you get
- failure taxonomy: wrong, unsafe, silent, slow, expensive
- golden set shaped like the product, not public leaderboards
- offline suite that can fail the build
- thin online sample / canary after ship
- cost and latency budgets you can defend
- pins so you can reproduce a failure
- kill path when the model provider has a bad friday
// out
- prove sota on public benchmarks as the whole job
- compliance stamp shopping (we leave artifacts; we don't sell stamps)
- greenfield feature with no path yet (start product-ai / agents / rag)
- multi-surface iron for the whole product under one band
// how it runs
- 01
tell us where it breaks
wrong answers that hurt, not abstract quality.
- 02
fail the happy path first
golden cases that would have caught last week's bug.
- 03
put the gate in
ci red means no merge. budgets visible.
- 04
keys + drill
you can rerun, pin, and kill without us.
// the stack
golden sets · offline + online eval · CI gate
named tools. no stack theater. if it isn't on the critical path for this line, it isn't listed.
// the band
from $12k · 2–4 weeks
harness + CI gate, one surface
bands filter the room. exact number after a call — not a 40-page sow.
// faq
- is this just unit tests for prompts?
- no. unit tests are part of it. production iron is the rest: golden clusters that hurt, offline + online signals, a gate that can fail, guardrails, cost and latency ceilings, pins, a kill path.
- do you train models?
- rarely. we make the system around the model survivable. if fine-tunes are in play, they still get pins, suites, and budgets.
- we already have an eval vendor. why you?
- tools are fine. most teams still lack a product-shaped golden set, a real merge gate, failure taxonomy, and budgets tied to product behavior. we wire your tools or ours — the deliverable is the iron.
- how do you price this?
- from $12k · 2–4 weeks for harness + CI gate on one product surface. exact number after a call.
// adjacent lines
book if you're not sure which. these two sit next to this line.
tell us what you're building.
30 min · your clock · no deck·hello@nightshiftglow.studio · we answer at 3am