Breaking the 1.58-bit Barrier for Ternary LLMs

Ternary LLM weights are mostly zeros, so storing them by zero-ness beats 1.58 bits.

1.485

bits per weight

Storage cost on sparsest model

Below the log2 3 ≈ 1.585 bits usually treated as the floor

How it works

Packing ignores zeros
Five-trit-per-byte packing costs 1.625 bits per weight and treats -1, 0, +1 as equally common. Yet zeros reach 51.5%.
Bitmap plus signs
Store one presence bit per weight and one sign bit only per non-zero. Cost is 2 minus zero density.
Smaller, faster decode
Beats five-trit packing on 26 of 29 models. Decode runs up to 1.18x faster on CPUs, 1.27x on GPUs.

From the paper