MiniMind Source-Code Analysis: From Data Loader to Loss, Checkpoint, and Inference

Begin with a tiny JSONL fixture

Create a handful of records that exercise plain text, instruction turns, `<think>`, and a tool-call marker only when needed. Pin the tokenizer and template, print token IDs and masks, and verify truncation before opening the model. This gives every later tensor a known origin.

The README presents MiniMind as a from-scratch teaching codebase, so prefer its native PyTorch path for source study before comparing wrapper integrations. Keep the fixture synthetic or openly licensed and record its hash.

Trace forward, loss, and optimizer

Follow batch collation into embeddings, attention, MLP or expert routing, logits, labels, masking, and loss. Record tensor shapes, dtype, device, and whether gradients are enabled. Then trace optimizer construction, scheduler, gradient accumulation, clipping, logging, and checkpoint cadence.

A one-batch forward/loss test catches tokenizer, dimension, mask, and device errors earlier than a multi-hour run. For MoE or long-context experiments, add a separate fixture so routing and sequence-length assumptions are visible.

Inspect save/resume and inference seams

Locate the checkpoint fields for model, optimizer, scheduler, step, and configuration, and test an interrupted run followed by resume. Compare generated text only with fixed decoding settings; also compare structural fields such as tool calls or reasoning markers when the script exposes them.

Trace `serve_openai_api.py` and `web_demo.py` from request parsing to tokenizer, generation, streaming, and error response. Keep serving code read-only and separate from training writes so an API request cannot mutate a checkpoint.

Turn observations into contributions

Good first contributions include a tokenizer/template regression, a deterministic save/resume fixture, a clearer tensor-shape assertion, an evaluation configuration lock, or a redacted API error test. State whether the behavior is observed in source, documented in the README, or still a hypothesis, and measure any optimization with the same fixture.