OpenMAIC
OpenMAIC Performance and Cost: Measure Agents, Media, Persistence, and Review
Build a reproducible OpenMAIC benchmark that separates model tokens, material parsing, tool calls, media generation, storage, export time, and human review.
What you will learn
- Design a complete-session benchmark
- Separate latency and cost surfaces
- Optimize while preserving lesson quality
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- Benchmark complete sessions with fixed materials, providers, media, and exports.
- Separate model, tool, persistence, media, export, and reviewer costs.
- Optimize for accepted lesson quality, not raw generation speed.
Define the unit of work
Measure a complete lesson session, not just time to first token. Fix the OpenMAIC commit, material hash and size, learning objective, provider/model, enabled media, persistence backend, and export format. Run a small lesson, a medium lesson, and a resume-after-restart case so interactive latency and batch cost are not mixed together.
The README lists slides, quizzes, simulations, PBL, images, video, voices, and exports. Each adds a different cost surface: model context, tool calls, rendering, storage, bandwidth, or review. Record which capabilities are enabled for every sample.
Capture p50 and p95 evidence
Log queue wait, first response, agent-step latency, tool duration, material parsing, artifact rendering, persistence save, and export time. Capture input/output tokens where the provider exposes them, along with retries, rate limits, and generated byte size. Report p50 and p95 across repeated runs; one successful demo is not a benchmark.
Compare cloud and local routes only when the task, quality rubric, and hardware are held constant. A faster local response can still cost more operator time if an export or speech capability is missing, while a cheaper model can increase human review time through corrections.
Budget durable sessions and media
Durable sessions and incremental saves improve recovery but add database writes and serialization. Measure writes per agent step, session size, restart recovery time, and cleanup of abandoned runs. For media, measure generation time, file size, cache hit rate, and whether a retry creates duplicate assets.
Use a cost ledger with variable fields rather than invented prices: model tokens, provider requests, storage, egress, runner minutes, and reviewer minutes. Fill prices from the deployment's actual contracts and date them separately from the engineering measurements.
Optimize without hiding quality loss
Start with bounded materials, selective skills, caching of immutable source extraction, and parallel work only where provider limits allow it. Re-run the same acceptance rubric after every optimization: factual coverage, accessibility, artifact correctness, citation/source trace, and reviewer edits.
Publish slow and failed runs beside fast successes. The useful question is cost per accepted lesson, not cost per generated paragraph; generated educational content still requires a subject-matter and audience review gate.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Pin commit, fixture, provider/model, media, persistence, and export.
- 2
Measure p50/p95 latency, tokens, retries, bytes, writes, and recovery.
- 3
Build a dated cost ledger from actual contracts and reviewer minutes.
- 4
Re-run factual, accessibility, artifact, and review checks after optimizations.
Copy-ready example
{
"fixture_hash": "<sha256>",
"provider": "<name>",
"p95_ms": { "agent": 0, "render": 0, "export": 0 },
"tokens": { "input": 0, "output": 0 },
"review_minutes": 0
}Frequently asked questions
Can I compare two providers by response time alone?
No. Hold task and quality constant and include tool, media, persistence, export, retries, and reviewer time.
Where do prices belong in the benchmark?
Record measured usage first, then join dated prices from the actual provider, storage, and runner contracts.
Sources
- OpenMAIC README (captured 2026-08-31)Source checked 2026-08-31
- OpenMAIC repositorySource checked 2026-08-31