MiniMind
MiniMind 性能与成本：测量训练、评估和服务
从数据吞吐、显存、训练阶段到评测质量、推理延迟和实验总成本，建立可复现基准。
你将学会
- Design reproducible MiniMind benchmarks
- Interpret training and serving trade-offs
- Calculate decision-ready experiment cost
开始前需要
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
先看结论
- Treat the README's SFT time/cost as one measured setup, not a universal promise.
- Benchmark each training stage with fixed data, tokenizer, seed, hardware, and evaluation.
- Report cost per accepted checkpoint or response, including failures and review.
Define the experiment before quoting a number
README 的 SFT 时间与费用是特定 3090、数据和配置下的测量，不是通用承诺。应固定代码、数据、tokenizer、seed、硬件和 checkpoint，分阶段记录 tokens/s、显存、恢复、固定评测、服务 p50/p95 与每个被接受结果的成本。
Use separate scorecards for pretraining, full SFT, LoRA, preference optimization, RL, distillation, and serving. Each stage has a different data contract and objective, so a faster wall-clock time can mean less data, shorter context, a smaller model, or omitted evaluation.
Measure training as a pipeline
Record data-loading time, samples and tokens per second, GPU utilization and peak memory, CPU and disk throughput, optimizer steps, checkpoint bytes, and resume time. Report cold and warm runs, gradient accumulation, effective batch, precision, and distributed topology. For DDP or DeepSpeed, show scaling efficiency rather than only aggregate throughput.
Track loss with held-out data and inspect qualitative samples at fixed checkpoints. A lower loss or faster step is not automatically a better assistant; tokenizer coverage, prompt template, repetition, safety, and formatting can regress even when the training chart looks smooth.
Connect quality, latency, and cost
The README references C-Eval, CMMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, and Social-IQA through lm-evaluation-harness. Pin the harness revision, task templates, few-shot settings, decoding, and checkpoint. Add a small target-use prompt set and human review so benchmark deltas are not mistaken for universal capability.
For serving, measure first-token and completion latency, tokens per second, concurrency, peak memory, request failures, and output acceptance. Include data preparation, failed runs, storage, electricity or cloud GPU time, and reviewer minutes in cost per accepted checkpoint—not only the successful training job.
Publish evidence instead of a headline
A useful report bundles the environment lock, run configuration, data/checkpoint hashes, raw metrics, evaluation outputs, limitations, and rollback checkpoint. Compare only rows with the same fixture, and mark estimates separately from metered values.
Set budgets for GPU memory, wall time, checkpoint storage, evaluation drift, and serving latency. Canary a new checkpoint behind the same API, replay the held-out set, and roll back when quality, safety, or resource use leaves the approved envelope.
如何选择
| 比较维度 | 方案 A | 方案 B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
实施步骤
- 1
Freeze code, environment, data, tokenizer, seed, checkpoint, and hardware.
- 2
Measure loader, training, checkpoint, evaluation, and serving paths separately.
- 3
Run pinned benchmarks plus target-use prompts and safety/format review.
- 4
Publish raw evidence, budgets, canary results, and rollback criteria.
可复制示例
run fixture -> tokens/s + peak VRAM + checkpoints
-> pinned eval + human review
-> serving p50/p95 -> cost per accepted result常见问题
Can I reuse the two-hour figure for any MiniMind run?
No. It is tied to a specific SFT measurement and hardware/cost assumption; reproduce the full fixture before comparing.
Which performance metric matters most?
Use a set: quality on fixed tasks, latency/throughput, memory, failure rate, and total cost per accepted outcome.
资料来源
- MiniMind README (captured 2026-08-31)来源核查 2026-08-31
- MiniMind repository来源核查 2026-08-31