reverse-skill
reverse-skill Performance and Cost: Measure Routing, Tool Startup, and Evidence Work
A reproducible benchmark plan for reverse-skill route latency, tool discovery, case initialization, artifact processing, report generation, and analyst time.
What you will learn
- Design a safe reverse-skill benchmark
- Allocate infrastructure and review cost
- Interpret speed versus evidence quality
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- Benchmark complete authorized cases and separate routing from tool startup.
- Include infrastructure, optional tool, storage, CI, and analyst-review cost.
- Never trade away scope, evidence, or review gates for speed.
Choose safe benchmark units
Measure a complete authorized case, not only the route script: hint-to-route latency, tool-index refresh, case initialization, read-only artifact inspection, evidence hashing, report generation, and human review minutes. Use public toy fixtures and keep active network operations out of the benchmark.
Pin OS, shell, Node/Python/Java versions, reverse-skill commit, tool-index state, fixture size, and client. Cold and warm runs answer different questions; label caches, downloads, and skipped tools explicitly.
Separate routing from tool cost
Record p50/p95 route time, rule count, index parsing, selected skill load, process startup, artifact bytes, hash time, report size, and failure/retry rate. A slow result may come from a fresh tool index or JVM startup rather than routing logic.
For multi-tool scenarios, report critical-path and parallelizable work separately. Never optimize by removing authorization, evidence, or review gates; those are correctness controls, not incidental overhead.
Price analyst and infrastructure time
Total cost includes host resources, optional licensed tools, storage, backups, network egress, CI minutes, and analyst triage/review. Calculate cost per accepted report or resolved finding, not per command, and keep estimates distinct from metered values.
Compare a pinned baseline with a candidate route or script change using the same fixture and expected evidence. A faster route that produces weaker provenance or more manual cleanup is not an improvement.
Publish an evidence-backed report
Store raw timings, tool-index snapshot, fixture hash, environment, route decision, output hashes, and review notes. Set budgets for route latency, disk retention, CI minutes, and case review time, then canary changes through the 173-case regression suite.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Pin host, commit, tool index, fixture, and client.
- 2
Measure cold/warm routing, index, case, artifact, evidence, and report paths.
- 3
Calculate p50/p95, bytes, failures, CI, and human review cost.
- 4
Canary changes and publish raw receipts with acceptance criteria.
Copy-ready example
hint -> route p50/p95 -> tool/index startup
case-init -> artifact/hash -> report
resources + analyst review -> cost per accepted reportFrequently asked questions
Is route latency the only performance metric?
No. Include tool discovery, case setup, artifact/evidence processing, report generation, failures, and analyst time.
Can I benchmark live pentesting targets?
Only with explicit authorization and a written scope; article fixtures should remain harmless and local.
Sources
- reverse-skill README (captured 2026-08-31)Source checked 2026-08-31
- reverse-skill repositorySource checked 2026-08-31