Colibri explained: streaming MoE experts across storage, RAM and VRAM
Benchmarking Colibri: cold cache, expert slabs and honest comparisons
Use the upstream protocol to avoid mistaking page-cache speed for model performance.
What you will learn
- Describe the operating point
- Use an independent control
- Report cost and quality together
Before you start
- Basic Python and command-line knowledge
- A documented hardware and storage inventory
Create a reproducibility ledger before attempting another inference optimization.
Key takeaways
- Warm cache needs an explicit label.
- I/O shape affects interpretation.
- A retracted mechanism must stay retracted.
Describe the operating point
Record model ID, container format, commit, hardware, prompt and token count. State how cache was made cold or label the run warm. A warm measurement cannot be presented as raw storage bandwidth.
Match request size, alignment, queue depth and buffered or direct I/O to the engine workload. A small-block QD1 test and large expert-slab traffic measure different operating points on the same drive.
Use an independent control
The benchmark protocol asks for a sanity metric that should remain stable. Background builds, a second engine process or thermal drift can create believable but misleading throughput differences.
It also retains a retracted alignment explanation after a controlled rerun failed to reproduce it. Do not repeat that discarded mechanism as a proven optimization or turn one host’s result into a general storage rule.
Report cost and quality together
Alternate baseline and candidate runs, change one variable, and include failures and spread. Measure first-token latency, throughput, bytes read and a quality check. Include hardware and electricity costs only from your actual measured setup.
No benchmark was executed for this series. The worksheet deliberately leaves numbers empty; upstream results remain attributed observations, not performance guarantees for your machine.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Record model, hardware and cache state.
- 2
Match I/O shape and add an independent control.
- 3
Repeat one-variable trials and retain negative results.
Copy-ready example
trial,model,format,cache_state,request_size,queue_depth,ttft,tokens_per_second,bytes_read,quality
baseline,,,,,,,,,Frequently asked questions
Can SSD headline bandwidth predict generation speed?
Not reliably without matching workload, cache and compute conditions.
Are the worksheet values measured?
No. They are fields for a future controlled experiment.
Sources
- Colibri / docs/benchmarking.mdSource checked 2026-09-23
- Colibri / README.mdSource checked 2026-09-23