Matt Pocock skills
Measuring skills overhead: context, human decisions and maintenance
Separate model context from human navigation and update work; use a controlled evaluation instead of claiming token savings from shorter instructions.
What you will learn
- There are several different costs
- Compare equivalent tasks and acceptance conditions
- Budget for maintaining your chosen distribution
Before you start
- Basic repository, issue-tracker and test concepts
- An understanding that instructions and permission are different
Choose an adoption model and trace its files, authority boundaries and verification evidence.
Key takeaways
- Model usage, human decisions and maintenance are separate costs.
- Equivalent acceptance conditions matter more than shorter transcripts.
- Inventory counts are not token or performance measurements.
There are several different costs
The skill-mechanics reference frames discoverability as a tradeoff: automatically reachable descriptions consume context, while human-only entry points require the person to remember when to use them. A router can reduce that navigation burden. The exact runtime loading behavior and token accounting remain host-specific; counting Markdown characters is not a measurement of billed model usage.
A large installed catalogue is not necessarily the content loaded for one task. Conversely, a short entry point can lead to long supporting references, repeated clarification or multiple review passes. Measure the complete task trace you are permitted to observe, and separate model input, output, elapsed time and human decision time rather than collapsing them into one efficiency score.
Compare equivalent tasks and acceptance conditions
Use the same starting repository and explicit acceptance condition for a baseline and a selected-skill trial. Keep the model, host version, tool access and task difficulty comparable, and repeat the comparison across several examples. Record whether the result works, not just whether the session ended sooner. These are proposed evaluation controls, not a benchmark already performed by this review.
Include costly failure paths: a clarification that prevents implementing the wrong behavior, a test that exposes a defect, or a review that reopens an incomplete change. A longer successful session may be preferable to a short incorrect one. Report unresolved work and reviewer corrections alongside any timing or usage numbers.
Budget for maintaining your chosen distribution
Editable copies create review and merge work when upstream instructions change. Managed distribution moves some update mechanics elsewhere, but still requires checking what revision and inventory arrived. The pinned source’s 25 promoted entries, 33 potential development links and 37 listed skill files describe different scopes; none is a benchmark of runtime speed.
The version-sync probe demonstrates a small deterministic maintenance check, not end-to-end agent efficiency. This series has no measured token reduction, throughput advantage or monetary saving to advertise. Keep unknown values null in evaluation records until you have actual measurements and a clear account of what was included.
Implementation steps
- 1
Fix the task, model, host and available tools.
- 2
Record baseline and selected-skill runs with the same acceptance condition.
- 3
Include corrections, failures and human intervention time.
- 4
Publish measured values and leave unavailable metrics unknown.
Copy-ready example
{
"evaluationStatus": "proposed",
"modelInputTokens": null,
"modelOutputTokens": null,
"humanDecisionMinutes": null,
"acceptedChanges": null,
"costSavingPercent": null
}Frequently asked questions
Can I infer savings from a smaller description?
No. Loaded references, repeated turns, model behavior and the host’s accounting all affect actual usage.
Are 25 skills faster than 37 skills?
Those are distribution inventories, not comparable runtime measurements. A task-specific experiment is required.
Sources
- .claude-plugin/plugin.jsonSource checked 2026-09-08
- scripts/link-skills.shSource checked 2026-09-08
- scripts/list-skills.shSource checked 2026-09-08
- skills/productivity/writing-for-agents/SKILL-MECHANICS.mdSource checked 2026-09-08