Spec Kit: from testable intent to traceable acceptance
Measure Spec Kit by accepted behavior and rework, not document volume
Design a fair pilot that accounts for specification effort, agent usage, review and repeated convergence without inventing productivity gains.
What you will learn
- Count the whole feature cycle
- Distinguish coverage from implemented correctness
- Set a stopping rule for an evaluation
Before you start
- Basic requirements, Git and testing concepts
- Understanding of local development versus application deployment
Trace a small feature from intent to evidence and distinguish workflow contracts from verified behavior.
Key takeaways
- Measure the entire accepted-feature cycle.
- Requirement-to-task coverage is not implementation correctness.
- Repeated convergence needs a bounded evaluation policy.
Count the whole feature cycle
A useful comparison includes requirement clarification, planning, task generation, implementation, validation and review. The alternative should solve the same bounded feature under the same acceptance conditions. Comparing a Spec Kit prototype with an unreviewed direct prompt hides different definitions of done.
Record elapsed time separately from human effort and model usage. Keep rejected changes, reopened requirements and unsuccessful iterations in the record. More generated pages can mean better traceability or unnecessary ceremony; document count alone cannot distinguish those outcomes.
Distinguish coverage from implemented correctness
The analyze template defines coverage as requirements associated with at least one task and limits its findings table to fifty rows with overflow summarized. That is a reporting contract, not a benchmark result or a bound on every model request. Full task coverage does not prove all tasks are correct or implemented.
Converge inspects the current code against artifacts and can append more work. Track which acceptance scenarios actually pass and why new findings recur. If the specification changes between runs, report that change rather than attributing all differences to model performance.
Set a stopping rule for an evaluation
Choose a small feature, a review budget and explicit acceptance checks before starting a pilot. Stop and reassess when repeated iterations produce the same unresolved gap, broaden scope without approval or consume the agreed budget. A command’s suggestion to iterate is not evidence that unlimited retries are economically sensible.
This series reports no speedup, token savings or pricing estimate. The proposed worksheet leaves unknown measurements null. A later team trial can compare accepted outcomes and rework while documenting model, integration, workflow revision and feature complexity.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Fix feature scope and acceptance conditions.
- 2
Record human time, model usage and failed attempts.
- 3
Track tested behavior rather than task checkmarks alone.
- 4
Stop repeated unproductive iterations and review the cause.
Copy-ready example
{
"pilotProposal": true,
"acceptedScenarios": null,
"humanMinutes": null,
"agentUsage": null,
"repeatedFindings": null,
"requirementsChanged": false,
"benchmarkExecuted": false
}Frequently asked questions
Does 100% task coverage mean the feature works?
No. It means requirements have task mappings under that report’s definition.
Was a productivity improvement measured here?
No. This series proposes a measurement method without claiming a benchmark.
Sources
- Spec Kit / templates/commands/analyze.mdSource checked 2026-09-14
- Spec Kit / templates/commands/converge.mdSource checked 2026-09-14