Every tool call, retry and model call — recorded span by span, with the tokens and dollars attached. Silent failures surface in seconds, not on next month's invoice.
run_9f3c2a41 · support-triage v2.5.1 · eu-west-1
Watching production agents at
A retry loop burned $1,840 in 40 minutes before anyone noticed.
The refunds API started returning 429s at 14:02. The agent's retry policy was exponential with no jitter and no cap on total attempts, so it kept firing — and after every failure it re-sent the entire failure transcript to GPT-4o to decide what to do next. Context grew from 1.2K to 5.7K tokens per attempt.
Nothing crashed. The dashboard was green. The only signal was the bill.
| Retry attempts | 3,118 |
| Tokens re-sent as context | 17.8M |
| Time to detection (before Nocturne) | 40 min |
| Time to detection (with loop rule) | 38 s |
Every node below is a real span from run_9f3c2a41. Expand the failing tool call to see all six attempts. Click any span for its inputs, outputs, tokens and cost.
Keyboard: Tab to walk the tree, ← / → to collapse and expand.
The second hump past 20 s is not noise — it is every run that hit the retry ceiling. Nocturne clusters it automatically and links straight to the traces.
| Tenant | Runs | Tokens | Cost |
|---|---|---|---|
| acme-eu | 41,208 | 318M | $2,984.10 |
| northwind | 28,640 | 204M | $1,776.44 |
| lumen-health | 12,904 | 96M | $918.02 |
| hexa-freight | 8,116 | 61M | $540.71 |
| pilot-sandbox | 2,044 | 9M | $74.30 |
Tag a run once with tenant_id and every downstream span inherits it — including retries you never wrote code for.
loop guard enabled 09:14 ↑
Rules run on the ingest path, not on a 5-minute cron, so a loop is caught mid-run. Median alert latency across our fleet is 38 seconds from the first repeated call.
The SDK patches the clients you already use — OpenAI, Anthropic, LangGraph, LlamaIndex, the standard requests and httpx sessions — and emits OTel-compatible spans. If you already run an OTel collector, point it at us instead and skip the SDK entirely.
# pip install nocturne
import nocturne
nocturne.init(api_key=os.environ["NOCTURNE_KEY"], service="support-triage")
# everything below is traced — no other changes
with nocturne.run(tenant_id=customer.id) as run:
result = agent.invoke({"input": message})
run.tag(outcome=result.status)// npm i @nocturne/sdk
import { init, run } from "@nocturne/sdk";
init({ apiKey: process.env.NOCTURNE_KEY!, service: "support-triage" });
await run({ tenantId: customer.id }, async (span) => {
const result = await agent.invoke({ input: message });
span.tag({ outcome: result.status });
});# send spans straight from an OTel collector
curl -X POST https://ingest.nocturne.dev/v1/traces \
-H "Authorization: Bearer $NOCTURNE_KEY" \
-H "Content-Type: application/x-protobuf" \
--data-binary @spans.pb
# 202 Accepted · {"accepted":418,"rejected":0}p50 overhead per span, in-process
p99 overhead — measured, not modelled
Average payload per traced run
Median time to alert on a loop
"We had logs. Logs told us the agent finished. They didn't tell us it finished by giving up, twice, at a cost of forty cents a run."
TL Tomas Lindqvist — Platform Lead, Orrery
"Replay is the part I didn't expect to need. We re-ran 400 failed traces against a patched prompt before shipping it, offline, for about eleven dollars."
DW Dana Whitfield — Head of AI, Fathom Retail
Free forever under 50,000 spans a month. No card, no sales call, no "book a demo to see pricing".