AI Agent Book: a study map for agent engineering
Measuring agent experiments: completion, correctness and tokens
Avoid turning one sample or a book claim into a general performance number.
What you will learn
- Pre-register the question
- Keep several outcome columns
- Report uncertainty honestly
Before you start
- Python environment basics
- Access to a supported model provider for live experiments
Turn one context lesson into a documented experiment and design decision.
Key takeaways
- Control the provider and task when comparing modes.
- Contract failure invalidates a performance comparison.
- Sample counts belong beside every score.
Pre-register the question
State which context mode should change which behavior. Use several tasks that need different tools and repeat each mode. Hold provider, model, task text and temperature fixed where possible.
A single successful run may reflect sampling luck. A failed API call is not evidence that one context strategy is worse.
Keep several outcome columns
The runner records completion and correctness checks as well as token usage. Add retries, tool errors and provider latency to your own notebook. Separate contract failure from task failure.
If no_tool_results does not actually remove visible tool results for a provider, stop before comparing its score with full context.
Report uncertainty honestly
Give sample counts and raw outcomes. The book’s 109 experiments span very different domains; one chapter-1 trial cannot establish a general agent architecture winner.
No model calls or benchmark were run for these articles. This section supplies a measurement plan, not measured results.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Write the hypothesis and task set first.
- 2
Repeat each mode and retain raw trajectories.
- 3
Report contract checks, outcomes, tokens and failures.
Copy-ready example
task,mode,contract_ok,completed,correct,tokens,tool_error,latency_msFrequently asked questions
Which mode is fastest?
That depends on model, task and implementation; this review has no measurement.
Can a token reduction be called an improvement?
Only if task quality and hidden costs remain acceptable.
Sources
- AI Agent Book / chapter1/context/README.mdSource checked 2026-09-26
- AI Agent Book / chapter1/context/run_experiment_1_1.pySource checked 2026-09-26