Quantization explained: how giant models fit on a normal laptop

Quantization is essentially compression for models — it allows giant models to fit on normal laptops. The word sounds more technical than it needs to. If the full model is the giant version, the quantized model is the version that can actually fit on a normal machine. You might lose a little quality, but suddenly the thing magically runs. You'll see formats like Q4 and Q8. As a beginner rule, Q4 is usually easier to run, and Q8 keeps more quality but needs more memory. If you're just getting started, Q4 is a reasonable place to begin.