i-have-adhd: readable responses, adapters and evidence
Measure i-have-adhd without confusing shorter answers with lower cost
Account for injected context, output, retries and judging, and interpret the upstream failed release gate without cherry-picking improvements.
What you will learn
- Define what you are trying to save
- Read the upstream evaluation as a bounded report
- Keep the baseline clean and the release gate fixed
Before you start
- Basic Git and command-line knowledge
- Ability to separate observed behavior from an untested claim
Explain the inspected mechanism, plan a reversible trial and interpret evidence without treating style as correctness.
Key takeaways
- Reading effort, latency and cost need separate measurements.
- The upstream report records a failed release gate.
- A contaminated baseline cannot establish an improvement.
Define what you are trying to save
Reading effort, response latency and provider charges are different outcomes. A shorter answer can still omit a prerequisite and create another turn. An injected ruleset adds input context; whether the provider caches it and how that affects billing depends on the actual service. Neither repository size nor the number of presentation rules establishes a savings percentage.
Build a per-task ledger for input tokens, output tokens, retries, judge calls and elapsed time. Keep the model, host version, task and configuration constant between conditions. Record unmeasured values as unknown. Our offline adapter tests make no provider calls and measure neither model latency nor task completion time for real readers.
Read the upstream evaluation as a bounded report
The pinned RESULTS.md describes fourteen cases and three trials, with the same model family producing and judging answers. It reports a higher weighted candidate score but also a failed release gate because blocking findings remain. The report identifies a tool-disabled case that cannot satisfy its action requirement and a partial-success regression involving an unsupported asserted cause.
Those findings weaken any simple claim that the skill makes every answer better. A favorable aggregate can coexist with failures in a task you care about. We did not reproduce the upstream generation or judging runs, verify provider invoices or validate the reported model identifier. Treat the table as the maintainers’ report, not our independently measured benchmark.
Keep the baseline clean and the release gate fixed
The evaluation documentation warns that a personal always-on setting can inject the candidate rules into the baseline. Isolation therefore belongs in the experiment design, not just installation hygiene. Record the configuration sources used by each runner, pin the model explicitly and verify that both conditions contain the same task set before comparing results.
Choose acceptance criteria before inspecting the scores: preserve required details, avoid unsupported causal claims and reject blocking mistakes even if average readability improves. If a case cannot be passed under its permitted tools, repair the experiment and rerun both conditions rather than silently dropping inconvenient rows. No paid evaluation was launched for this article.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Specify one outcome and its measurement method.
- 2
Pin tasks, models, versions and configuration sources.
- 3
Record input, output, retries and judging separately.
- 4
Review blockers before the average score.
Copy-ready example
{
"taskId": "same-task",
"inputTokens": null,
"outputTokens": null,
"retries": null,
"judgeCost": null,
"readerCompletionSeconds": null,
"independentlyMeasured": false
}Frequently asked questions
How much money does it save?
No independent savings measurement is available in this series.
Can I quote only the improved weighted score?
That omits the failed gate, blockers and experiment limitations needed to interpret the result.
Sources
- i-have-adhd / evals/README.mdSource checked 2026-09-12
- i-have-adhd / evals/RESULTS.mdSource checked 2026-09-12
- i-have-adhd / evals/rubric.mdSource checked 2026-09-12