Colibri explained: streaming MoE experts across storage, RAM and VRAM
Colibri explained: streaming MoE experts across storage, RAM and VRAM
Understand why a model can fit without promising that it will generate quickly on your hardware.
What you will learn
- Capacity and speed are different questions
- Separate the C engine and Python tools
- Read upstream measurements as scoped evidence
Before you start
- Basic Python and command-line knowledge
- A documented hardware and storage inventory
Create a reproducibility ledger before attempting another inference optimization.
Key takeaways
- Fitting a model does not guarantee interactive speed.
- Python support tools surround the C engine.
- Weights and engine have separate requirements.
Capacity and speed are different questions
Colibri is an inference engine for supported mixture-of-experts models. Its documented design treats storage, RAM and VRAM as placement tiers, moving expert weights instead of requiring every expert to fit in fast memory.
This changes the capacity question, but storage reads still take time. A machine that can load the chosen model may deliver unacceptable interactive latency. Begin with a latency target and a hardware inventory, not a parameter-count headline.
Separate the C engine and Python tools
The repository describes a pure C engine, while its launcher, API gateway and preparation tools use Python. The pip entry point delegates to c/coli. A statement about engine dependencies does not describe the whole user-facing toolchain.
Model weights are a separate download with their own format, revision and license. The engine’s Apache-2.0 license does not grant rights to every supported model or make a hundreds-of-gigabytes download suitable for every trial.
Read upstream measurements as scoped evidence
The README presents performance and research claims for particular hosts and workloads. This series inspects commit 9d5d05d and the launcher and benchmark protocol; it does not reproduce those model runs.
No weights were downloaded, no engine was built and no inference benchmark was executed here. Use the later worksheet to collect your own throughput, first-token latency and quality evidence.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
List hardware and a latency requirement.
- 2
Select a documented model format and review its license.
- 3
Keep upstream measurements separate from local evidence.
Copy-ready example
evaluation:
engine_revision: 9d5d05d
model_revision: record-separately
hardware: inventory-first
latency_target: define-before-downloadFrequently asked questions
Is a GPU mandatory?
The quickstart documents a CPU-only path; suitability still depends on the model and hardware.
Were the advertised speeds reproduced?
No. This series is source-based analysis without a model benchmark.
Sources
- Colibri / README.mdSource checked 2026-09-23
- Colibri / docs/quickstart.mdSource checked 2026-09-23
- Colibri / colibri/cli.pySource checked 2026-09-23