Arquitectura de MiniMind: tokenizer, Transformer, etapas de entrenamiento y servicio

Six layers to keep distinct

MiniMind se entiende como capas de tokenizer/plantillas, modelo Dense/MoE, datos JSONL, Pretrain/SFT/LoRA/RL/Agentic RL, evaluación, checkpoints y API/WebUI. Cada frontera tiene un contrato distinto; fija configuración, datos y hardware antes de comparar etapas.

Each layer has a different contract. A tokenizer change alters sequence statistics, a dataset change alters supervision, a LoRA run changes parameter ownership, and a serving wrapper changes the integration surface. Do not attribute a result to the model architecture until those variables are fixed.

Follow one token through the model

Start with a text sample and trace normalization, special markers such as `<think>` or tool-call templates, token IDs, embeddings, attention/MLP blocks, logits, and loss. For MoE variants, record routing configuration and effective active parameters rather than treating the checkpoint as a generic Dense model.

The training loop adds batching, masking, mixed precision, optimizer updates, logging, and checkpoint cadence. Capture where each field is created and consumed; this is more reliable than inferring internals from a high-level framework command.

Separate learning and serving graphs

Pretraining, SFT, DPO, PPO/GRPO/CISPO, tool use, agentic RL, and distillation reuse pieces but ask different data and evaluation questions. Checkpoint names and directories should make the stage explicit, and evaluation must load the matching tokenizer and prompt template.

Serving wraps a checkpoint with generation settings and an API or Streamlit UI. Keep the model directory read-only, isolate tool calls, and record `reasoning_content`, `tool_calls`, or `open_thinking` fields as optional protocol features rather than assumptions about model cognition.

Architecture review fixtures

Use a tiny JSONL fixture to test tokenizer coverage, a one-batch forward/loss pass, save/resume, one evaluation prompt, and one API request. Add a MoE or tool-call fixture only after the dense path is understood. Contributions should include expected tensor shapes, checkpoint fields, and a reproducible command.