Colibri explained: streaming MoE experts across storage, RAM and VRAM
Colibri architecture: follow weight placement without changing the model
Separate memory-tier policy, inference computation and the Python control surface.
What you will learn
- Follow the tiers
- Locate the control boundary
- Measure the actual bottleneck
Before you start
- Basic Python and command-line knowledge
- A documented hardware and storage inventory
Create a reproducibility ledger before attempting another inference optimization.
Key takeaways
- Placement and quantization are different variables.
- The pip wrapper is not a kernel.
- Hit rate alone does not prove faster decoding.
Follow the tiers
The README describes expert placement across VRAM, RAM and storage, with routing history, caching and prefetch as optimization policies. Hot experts may avoid repeated storage reads, while misses still require movement.
Placement is not a license to silently alter precision or router semantics. The repository states that such changes need explicit quality and correctness evidence. Distinguish a placement experiment from a different quantized model.
Locate the control boundary
The inspected Python package entry point locates the sibling c directory and runs c/coli in-process with runpy. That small wrapper connects package installation to the original launcher; it does not implement tensor kernels.
The launcher and API tools are the control surface around the C engine. This series does not trace every C backend, GPU kernel or model-family implementation, so it avoids claiming a complete kernel architecture audit.
Measure the actual bottleneck
A cache hit-rate improvement can coexist with slower generation if another stage becomes expensive. Storage traffic, dense computation and synchronization need separate measurements beside end-to-end latency.
Use the documented benchmark protocol to match access patterns and record cache state. A fast synthetic SSD number alone cannot explain a workload that reads expert slabs with a different queue depth.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Map weights across storage and fast memory.
- 2
Separate launcher control from engine computation.
- 3
Measure stage costs beside end-to-end results.
Copy-ready example
model container -> placement policy -> RAM / VRAM / storage
Python entry point -> c/coli -> engine workflow
measurement -> latency + bytes read + qualityFrequently asked questions
Does more RAM necessarily change model quality?
Placement policy should be distinguished from explicit precision or routing changes; verify the actual configuration.
Was every backend inspected?
No. The source trace here covers the Python entry point and documented architecture.
Sources
- Colibri / README.mdSource checked 2026-09-23
- Colibri / colibri/cli.pySource checked 2026-09-23
- Colibri / docs/benchmarking.mdSource checked 2026-09-23