Heretic
Heretic Performance and Cost: Benchmark Experiments, Not Just GPU Minutes
Build a reproducible benchmark for Heretic transforms, Optuna trials, quantization, residual plots, storage, and human review.
What you will learn
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- The meaningful unit is an accepted, reproducible checkpoint.
- GPU transform and CPU research plots have different bottlenecks.
- Cost includes review, storage, and failed experiments.
Choose the right unit
Measure a complete accepted experiment: baseline load, transform, optimization trials, evaluation, artifact writing, and review. A single runtime number hides prompt count, model size, VRAM, retries, quantization, and whether CPU PaCMAP research was enabled.
Freeze model revision, prompt manifests, trial budget, PyTorch/CUDA, GPU, batch policy, quantization, and output format. Report p50/p95 wall time, peak VRAM, CPU hours, storage, download bytes, trial count, and failure rate.
Separate pathways
Benchmark baseline inference, transformation, built-in evaluation, chat smoke tests, and research plots separately. The README's 20–30 minute RTX 3090 example is useful context for Qwen3-4B, while larger models or PaCMAP can change the cost profile dramatically.
Calculate cost per accepted checkpoint, including GPU time, CPU plotting, storage, review minutes, and re-runs after failed or unsafe outputs. Quantization may lower VRAM but should be measured for quality drift.
Optimization levers
Tune batch size only within memory and reproducibility limits; cap Optuna trials, cache immutable model weights, and schedule CPU-heavy plots asynchronously. Establish canaries and stop thresholds for capability regression or anomalous refusal changes.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Freeze workload, stack, hardware, and trial budget.
- 2
Measure cold/warm, transform, evaluation, and plot paths.
- 3
Record VRAM, CPU, bytes, failures, and review time.
- 4
Set canary and rollback thresholds before scaling.
Copy-ready example
manifest -> baseline p50/p95
transform + Optuna trials -> VRAM/time
eval + plots + review -> accepted checkpoint cost
canary drift -> stop / rollbackFrequently asked questions
Is the README's 20–30 minute figure a guarantee?
No. It is a hardware- and model-specific example; reproduce it with matching conditions.
Should I maximize Optuna trials?
No. Set a budget and stop when quality, cost, or safety evidence no longer improves.
Sources
- Heretic README (captured 2026-08-31)Source checked 2026-08-31
- Heretic repositorySource checked 2026-08-31