OpenMAIC
OpenMAIC: rendimiento y coste de agentes, multimedia, persistencia y revisión
Construye un benchmark reproducible de sesión completa y separa tokens, herramientas, medios, almacenamiento, exportación y minutos humanos.
Qué aprenderás
- Design a complete-session benchmark
- Separate latency and cost surfaces
- Optimize while preserving lesson quality
Antes de empezar
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Conclusiones clave
- Benchmark complete sessions with fixed materials, providers, media, and exports.
- Separate model, tool, persistence, media, export, and reviewer costs.
- Optimize for accepted lesson quality, not raw generation speed.
Define the unit of work
Mide una sesión completa con commit, material, proveedor, modelo, multimedia, persistencia y export fijados. Registra p50/p95, tokens, reintentos, bytes, escrituras, recuperación y revisión; el coste útil es por lección aceptada, no por párrafo generado.
The README lists slides, quizzes, simulations, PBL, images, video, voices, and exports. Each adds a different cost surface: model context, tool calls, rendering, storage, bandwidth, or review. Record which capabilities are enabled for every sample.
Capture p50 and p95 evidence
Log queue wait, first response, agent-step latency, tool duration, material parsing, artifact rendering, persistence save, and export time. Capture input/output tokens where the provider exposes them, along with retries, rate limits, and generated byte size. Report p50 and p95 across repeated runs; one successful demo is not a benchmark.
Compare cloud and local routes only when the task, quality rubric, and hardware are held constant. A faster local response can still cost more operator time if an export or speech capability is missing, while a cheaper model can increase human review time through corrections.
Budget durable sessions and media
Durable sessions and incremental saves improve recovery but add database writes and serialization. Measure writes per agent step, session size, restart recovery time, and cleanup of abandoned runs. For media, measure generation time, file size, cache hit rate, and whether a retry creates duplicate assets.
Use a cost ledger with variable fields rather than invented prices: model tokens, provider requests, storage, egress, runner minutes, and reviewer minutes. Fill prices from the deployment's actual contracts and date them separately from the engineering measurements.
Optimize without hiding quality loss
Start with bounded materials, selective skills, caching of immutable source extraction, and parallel work only where provider limits allow it. Re-run the same acceptance rubric after every optimization: factual coverage, accessibility, artifact correctness, citation/source trace, and reviewer edits.
Publish slow and failed runs beside fast successes. The useful question is cost per accepted lesson, not cost per generated paragraph; generated educational content still requires a subject-matter and audience review gate.
Cómo elegir
| Criterio | Opción A | Opción B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Pasos de implementación
- 1
Pin commit, fixture, provider/model, media, persistence, and export.
- 2
Measure p50/p95 latency, tokens, retries, bytes, writes, and recovery.
- 3
Build a dated cost ledger from actual contracts and reviewer minutes.
- 4
Re-run factual, accessibility, artifact, and review checks after optimizations.
Ejemplo para copiar
{
"fixture_hash": "<sha256>",
"provider": "<name>",
"p95_ms": { "agent": 0, "render": 0, "export": 0 },
"tokens": { "input": 0, "output": 0 },
"review_minutes": 0
}Preguntas frecuentes
Can I compare two providers by response time alone?
No. Hold task and quality constant and include tool, media, persistence, export, retries, and reviewer time.
Where do prices belong in the benchmark?
Record measured usage first, then join dated prices from the actual provider, storage, and runner contracts.
Fuentes
- OpenMAIC README (captured 2026-08-31)Fuente verificada 2026-08-31
- OpenMAIC repositoryFuente verificada 2026-08-31