Heretic Deployment: GPU Environments, Quantization, and Model Provenance

Treat deployment as an experiment runner

Heretic is documented as a command-line research tool rather than a multi-tenant serving platform. A practical deployment is an isolated GPU worker with a read-only model cache, a controlled output volume, and a manifest for every run.

Python 3.10+ and PyTorch 2.2+ are the baseline. The README warns that some models need later PyTorch features, such as `torch.accelerator` for MXFP4 gpt-oss models; pin the exact stack instead of assuming the minimum is sufficient.

Capacity and artifacts

The program benchmarks the machine to choose batch size. On the documented RTX 3090 example, Qwen3-4B takes roughly 20–30 minutes, but treat that as a workload-specific reference. Bitsandbytes `bnb_4bit` can reduce VRAM while adding a quantization variable to the evidence.

Persist the source model revision, lockfile, TOML configuration, hardware inventory, logs, evaluation outputs, checkpoint hash, and license. Keep upload credentials outside the worker and make export an explicit, reviewed step.

Recovery

Use a disposable output directory, checkpoint space and a known-good baseline. Test interrupted runs, insufficient VRAM, corrupted downloads, failed uploads, and cleanup before scheduling batches.