FixedToLogQuantized clamped its input to |value| < 0.5 (Q16.16) before a
table lookup, so every weight with |value| >= 0.5 was forced to 0.5 and
many small weights collapsed to exactly 0 -- catastrophic for real GPT-2
weights (~N(0, 0.05)). The log format already spans magnitudes up to ~120,
so no clamp is needed: encode directly with a bounded MSB scan (O(1)),
reusing the existing (exponent << EXPONENT_SHIFT) | mantissa packing.
Validated in the real DOS runtime (DJGPP fbc + dosbox-x DPMI, agreeing
bit-for-bit with a 32-bit Linux fbc reference): on 50k N(0,0.05) weights
RMS drops 0.02757 -> 0.001409 (19.6x, within 6% of an exhaustive
nearest-code search) with no weights collapsed to zero, at ~36x lower cost
than that search.
Problem
FixedToLogQuantized clamps its input to [-32768, 32767] (i.e. |value| < 0.5 in Q16.16) before a table lookup:
Two consequences, both bad for real weights:
Fix
Encode directly with a closed form — no clamp, no lookup table — using the same packing ((exponent << EXPONENT_SHIFT) | mantissa) that InitQuantizationTables and LogQuantizedToFixed already dequantize. It's O(1): a bounded MSB scan (<= ~20 iterations), not the 511-entry nearest-code search one might otherwise reach for.
Evidence (measured in the real DOS runtime)
Cross-compiled with DJGPP FreeBASIC and run under DPMI (dosbox-x); numbers agree bit-for-bit with a 32-bit Linux fbc reference.
Accuracy — RMS reconstruction error:
→ 19.6x lower error on realistic weights, within 6% of the exhaustive optimum. Original collapses 1867/50000 weights to 0; this fix collapses none.
Cost — per call, cycles=max:
→ constant-time; ~36x faster than a nearest-code search at matching accuracy, a small constant over the (incorrect) original.
Checks
Note
QuantFixedLookup (and its fill loop in InitQuantizationTables) is now unused by the encode path. I left it in place to keep this diff to one function — happy to remove it as a follow-up if you'd prefer.