Tools · Production readiness Agent production readiness A demo that works in a keynote is not an agent that works at 2am. This is the checklist I use to tell the difference. Score your own system against it. The gaps are the work.
The executable version, a Claude skill, is on GitHub↗ . This is the ship gate in the two gates .
Evaluation Cost Observability Guardrails Oversight Reliability Governance 0 / 28 Score your system Copy summary Reset
Evaluation 0/4
An evaluation set built from real user inputs, not hand-picked happy paths. Evals run automatically when the prompt, tools, or model change. You measure grounding and accuracy, not just whether a reply was produced. Someone reviews a sample of real conversations on a set cadence. Cost 0/4
A token or credit budget per conversation that you actually enforce. The model is right-sized for the task, not the largest one by default. Cost is monitored, with an alert before the bill surprises you. You know the cost per resolved conversation, not just per call. Observability 0/4
Every conversation is traced end to end, tools included. Tool-call success and failure rates are tracked, not assumed. Latency is measured at the tail, not just the average. You can pull up any past conversation and see what happened. Guardrails 0/4
Inputs are validated before they reach the model. The agent stays in scope and declines work outside it. Sensitive data is handled deliberately, not passed through by accident. Outputs are checked before they reach a customer or a system of record. Human oversight 0/4
There is a clear path to a human when the agent should step back. High-risk actions require confirmation before they run. Low-confidence answers are caught rather than sent anyway. The people on the hook for the agent know how it behaves. Reliability 0/4
Prompts and agent config are versioned like code. You can roll back a bad change in minutes, not days. The agent degrades gracefully when a tool or model is down. You have tested what happens under load and under failure. Governance 0/4
Every consequential action leaves an audit trail. Data handling and retention are written down and followed. Access to the agent and its tools is controlled and reviewed. Changes go through review before they reach production. This is my checklist, not a standard. Adapt it to your own system. The score is a prompt for a conversation, not a certificate.