Colibri explained: streaming MoE experts across storage, RAM and VRAM
Build a Colibri benchmark notebook that preserves negative results
Create a reproducibility ledger before attempting another inference optimization.
What you will learn
- Start with records, not model execution
- Validate missing and misleading fields
- Integrate real measurements only afterward
Before you start
- Basic Python and command-line knowledge
- A documented hardware and storage inventory
Create a reproducibility ledger before attempting another inference optimization.
Key takeaways
- The notebook is a proposed extension.
- Failures and retractions belong in the record.
- Unknown measurements must remain unknown.
Start with records, not model execution
Build a local report from synthetic CSV rows describing model revision, cache state, I/O shape and latency. Mark invented values as fixtures so a screenshot cannot be mistaken for a real benchmark.
This is a proposed learning project around the documented protocol. It does not claim that Colibri already provides the exact notebook or that this series ran the model.
Validate missing and misleading fields
Reject a throughput row without model format or cache state. Keep failed runs and exclusion reasons in a separate visible section. A chart should not quietly remove every result that makes an optimization look worse.
Include a retraction record linked to the earlier hypothesis. The upstream alignment correction is a useful design example: preserving changed explanations makes later conclusions auditable.
Integrate real measurements only afterward
After schema and display tests pass, import one authorized run with its exact command and redacted raw logs. Compare alternating baseline and candidate trials under a fixed model and quality test.
The acceptance condition is traceability: every chart point reaches its input record, method and quality result. A missing measurement stays unknown; the notebook must not manufacture a savings claim from incomplete data.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Create clearly labeled synthetic records.
- 2
Test missing metadata and retained failures.
- 3
Link each chart point to method and raw evidence.
Copy-ready example
{"trial":"fixture-only","cache_state":"unknown","throughput":null,"publishable_benchmark":false}Frequently asked questions
Must the first version download weights?
No. Synthetic records are enough to test the reporting contract.
What is the main acceptance check?
Every displayed value traces to its method, source record and quality evidence.
Sources
- Colibri / docs/benchmarking.mdSource checked 2026-09-23
- Colibri / README.mdSource checked 2026-09-23