OpenMAIC
OpenMAIC 性能与成本：测量代理、媒体、持久化与人工复核
建立可复现的完整课堂会话基准，区分模型 token、工具、媒体、存储、导出和审核成本。
你将学会
- Design a complete-session benchmark
- Separate latency and cost surfaces
- Optimize while preserving lesson quality
开始前需要
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
先看结论
- Benchmark complete sessions with fixed materials, providers, media, and exports.
- Separate model, tool, persistence, media, export, and reviewer costs.
- Optimize for accepted lesson quality, not raw generation speed.
Define the unit of work
OpenMAIC 的性能单位应是完整课堂会话，而不是首 token：固定材料、Provider、模型、媒体、持久化和导出，记录 p50/p95、token、重试、文件大小、数据库写入、恢复时间和人工审核分钟数，并以每个被接受的课堂产物计算成本。
The README lists slides, quizzes, simulations, PBL, images, video, voices, and exports. Each adds a different cost surface: model context, tool calls, rendering, storage, bandwidth, or review. Record which capabilities are enabled for every sample.
Capture p50 and p95 evidence
Log queue wait, first response, agent-step latency, tool duration, material parsing, artifact rendering, persistence save, and export time. Capture input/output tokens where the provider exposes them, along with retries, rate limits, and generated byte size. Report p50 and p95 across repeated runs; one successful demo is not a benchmark.
Compare cloud and local routes only when the task, quality rubric, and hardware are held constant. A faster local response can still cost more operator time if an export or speech capability is missing, while a cheaper model can increase human review time through corrections.
Budget durable sessions and media
Durable sessions and incremental saves improve recovery but add database writes and serialization. Measure writes per agent step, session size, restart recovery time, and cleanup of abandoned runs. For media, measure generation time, file size, cache hit rate, and whether a retry creates duplicate assets.
Use a cost ledger with variable fields rather than invented prices: model tokens, provider requests, storage, egress, runner minutes, and reviewer minutes. Fill prices from the deployment's actual contracts and date them separately from the engineering measurements.
Optimize without hiding quality loss
Start with bounded materials, selective skills, caching of immutable source extraction, and parallel work only where provider limits allow it. Re-run the same acceptance rubric after every optimization: factual coverage, accessibility, artifact correctness, citation/source trace, and reviewer edits.
Publish slow and failed runs beside fast successes. The useful question is cost per accepted lesson, not cost per generated paragraph; generated educational content still requires a subject-matter and audience review gate.
如何选择
| 比较维度 | 方案 A | 方案 B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
实施步骤
- 1
Pin commit, fixture, provider/model, media, persistence, and export.
- 2
Measure p50/p95 latency, tokens, retries, bytes, writes, and recovery.
- 3
Build a dated cost ledger from actual contracts and reviewer minutes.
- 4
Re-run factual, accessibility, artifact, and review checks after optimizations.
可复制示例
{
"fixture_hash": "<sha256>",
"provider": "<name>",
"p95_ms": { "agent": 0, "render": 0, "export": 0 },
"tokens": { "input": 0, "output": 0 },
"review_minutes": 0
}常见问题
Can I compare two providers by response time alone?
No. Hold task and quality constant and include tool, media, persistence, export, retries, and reviewer time.
Where do prices belong in the benchmark?
Record measured usage first, then join dated prices from the actual provider, storage, and runner contracts.
资料来源
- OpenMAIC README (captured 2026-08-31)来源核查 2026-08-31
- OpenMAIC repository来源核查 2026-08-31