MiniMind
MiniMind Performance and Cost: Benchmark Training, Evaluation, and Serving
A reproducible MiniMind measurement plan for data throughput, GPU memory, training stages, benchmark quality, inference latency, and total experiment cost.
What you will learn
- Design reproducible MiniMind benchmarks
- Interpret training and serving trade-offs
- Calculate decision-ready experiment cost
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- Treat the README's SFT time/cost as one measured setup, not a universal promise.
- Benchmark each training stage with fixed data, tokenizer, seed, hardware, and evaluation.
- Report cost per accepted checkpoint or response, including failures and review.
Define the experiment before quoting a number
MiniMind's README includes a memorable SFT example—one epoch on one NVIDIA 3090 in roughly two hours at a stated electricity-cost assumption. That is a measured configuration, not a promise for pretraining, a different dataset, sequence length, precision, accelerator, or post-training stage. Preserve the exact command, commit, data hash, seed, tokenizer, checkpoint, and hardware before comparing results.
Use separate scorecards for pretraining, full SFT, LoRA, preference optimization, RL, distillation, and serving. Each stage has a different data contract and objective, so a faster wall-clock time can mean less data, shorter context, a smaller model, or omitted evaluation.
Measure training as a pipeline
Record data-loading time, samples and tokens per second, GPU utilization and peak memory, CPU and disk throughput, optimizer steps, checkpoint bytes, and resume time. Report cold and warm runs, gradient accumulation, effective batch, precision, and distributed topology. For DDP or DeepSpeed, show scaling efficiency rather than only aggregate throughput.
Track loss with held-out data and inspect qualitative samples at fixed checkpoints. A lower loss or faster step is not automatically a better assistant; tokenizer coverage, prompt template, repetition, safety, and formatting can regress even when the training chart looks smooth.
Connect quality, latency, and cost
The README references C-Eval, CMMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, and Social-IQA through lm-evaluation-harness. Pin the harness revision, task templates, few-shot settings, decoding, and checkpoint. Add a small target-use prompt set and human review so benchmark deltas are not mistaken for universal capability.
For serving, measure first-token and completion latency, tokens per second, concurrency, peak memory, request failures, and output acceptance. Include data preparation, failed runs, storage, electricity or cloud GPU time, and reviewer minutes in cost per accepted checkpoint—not only the successful training job.
Publish evidence instead of a headline
A useful report bundles the environment lock, run configuration, data/checkpoint hashes, raw metrics, evaluation outputs, limitations, and rollback checkpoint. Compare only rows with the same fixture, and mark estimates separately from metered values.
Set budgets for GPU memory, wall time, checkpoint storage, evaluation drift, and serving latency. Canary a new checkpoint behind the same API, replay the held-out set, and roll back when quality, safety, or resource use leaves the approved envelope.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Freeze code, environment, data, tokenizer, seed, checkpoint, and hardware.
- 2
Measure loader, training, checkpoint, evaluation, and serving paths separately.
- 3
Run pinned benchmarks plus target-use prompts and safety/format review.
- 4
Publish raw evidence, budgets, canary results, and rollback criteria.
Copy-ready example
run fixture -> tokens/s + peak VRAM + checkpoints
-> pinned eval + human review
-> serving p50/p95 -> cost per accepted resultFrequently asked questions
Can I reuse the two-hour figure for any MiniMind run?
No. It is tied to a specific SFT measurement and hardware/cost assumption; reproduce the full fixture before comparing.
Which performance metric matters most?
Use a set: quality on fixed tasks, latency/throughput, memory, failure rate, and total cost per accepted outcome.
Sources
- MiniMind README (captured 2026-08-31)Source checked 2026-08-31
- MiniMind repositorySource checked 2026-08-31