Paperclip explained: an operating layer for teams of AI agents
Measuring Paperclip: throughput, cost and unfinished work
Build a small evaluation that separates agent effort from board overhead.
What you will learn
- Count useful outcomes
- Reconcile the cost ledger
- Make failure visible
Before you start
- Node.js and agent runtime basics
- A test company with non-sensitive tasks
Turn the source review into a bounded experiment with stop conditions.
Key takeaways
- Completed heartbeats are a weak success metric.
- Approval delay can dominate total lead time.
- A policy ledger needs invoice reconciliation.
Count useful outcomes
Track issues accepted by a reviewer, not only heartbeats triggered. Record rework, abandoned runs and time waiting for an approval. Those measures reveal whether coordination helps the team.
Use the same small task set before and after adding a second agent. A faster run that requires more reviewer cleanup is not an improvement.
Reconcile the cost ledger
Paperclip records token and cost events by scopes described in the README. Compare its monthly view with provider invoices, then inspect when the warning and hard-stop thresholds fire.
Set a low ceiling in a synthetic company and record the policy window, spending events and pause state. Do not extrapolate from a tiny test to a production savings claim.
Make failure visible
A missed heartbeat, duplicate task pickup or late approval creates unfinished work. Report these cases alongside completed outputs.
This series has not run a benchmark. The proposed metrics are an evaluation design based on documented behavior, not measured Paperclip performance.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Define five representative non-sensitive issues.
- 2
Record accepted output, rework, waiting time and cost per issue.
- 3
Compare policy events with provider billing.
Copy-ready example
issue,accepted,rework_minutes,approval_wait_minutes,recorded_cost,provider_costFrequently asked questions
What is the right budget amount?
Set it from your workload and acceptable loss; this source review cannot supply a universal number.
Are the savings measured here?
No. The article provides a measurement plan only.
Sources
- Paperclip / README.mdSource checked 2026-09-26
- Paperclip / server/src/services/budgets.tsSource checked 2026-09-26