What Is Model Quantization? A Guide for Mac Users

If you have been following our series on running large language models on Apple Silicon, you have seen benchmark numbers like “DeepSeek XYZ, 25 tokens per second.” Those numbers are close to meaningless until you know one thing: the quantization.

Quantization is the single biggest lever you have for making a model fit in memory. It determines whether a model needs one Mac Studio, two of them, or a whole rack of Mac minis. This post explains what quantization actually is, what it costs you in quality, and how to use it when planning local AI hardware.

The one-sentence version

Quantization makes an AI model smaller (and usually faster) by storing its billions of numerical weights with less precision.

That is it. Everything else is detail. But the details matter a lot, so let’s build up from the basics.

What is inside a language model?

Strip away the mystique and an LLM is an enormous pile of learned numbers called weights. A 100-billion-parameter model has roughly 100 billion of them. During training, those numbers get nudged trillions of times until the model produces useful output. During inference (when you actually use the model), those numbers are read from memory over and over for every token generated.

Each weight has to be stored somewhere, and how much space it takes depends on how precisely you store it. That precision is measured in bits.

Here is what that looks like for a hypothetical 100-billion-parameter model:

The math is simple: parameters multiplied by bytes per parameter. At 16 bits (2 bytes), 100 billion parameters need about 200 GB. Cut that to 4 bits (half a byte) and you are at 50 GB. That is a 75% reduction from a single decision.

Real inference adds overhead on top of this (the KV cache for context, activations, the operating system, and whatever else is running), so these are not total system-memory requirements. But the relationship holds, and it is the dominant term.

What is actually being compressed?

Imagine training produced weights like these:

0.782314
-1.249370
0.003817
2.983210

Sixteen-bit storage preserves those values quite accurately. Quantization makes a bet: the model probably does not need that much precision on every single number to produce good output. So it stores them approximately:

0.782314   ->   0.78
-1.249370  ->  -1.25
0.003817   ->   0.00
2.983210   ->   2.98

The model still “knows” roughly the same things. The relative relationships between weights, which is what actually drives the output, are mostly preserved.

Real quantization algorithms are considerably smarter than rounding. They analyze which weights matter most, group weights into blocks with shared scaling factors, and in some cases run calibration data through the model to decide where precision can be spent and where it can be sacrificed. But conceptually, “store the numbers with fewer digits” is the right mental model.

Why not quantize everything down to 1 or 2 bits?

Because eventually you throw away too much information.

The best analogy is image compression. Start with an original RAW photo. Save it as a high-quality JPEG and it is hard to tell the difference. Save it as a medium JPEG and it is still fine for the web. Compress it heavily and it turns blocky, smeared, and obviously degraded.

LLMs behave the same way. Going from 16-bit to 8-bit is nearly free in quality terms. Going to 4-bit usually costs a little, often less than you would expect. Going to 2-bit can make a model noticeably dumber: worse reasoning, more hallucination, more repetition.

The tradeoff in one line:

• More bits: larger model, more memory, generally better fidelity
• Fewer bits: smaller model, less memory, often faster, higher risk of quality loss

This is exactly why 4-bit quantization has become the default sweet spot for local inference. For most models and modern quantization methods, Q4 gives you a quarter of the memory footprint while keeping the large majority of the model’s capability.

Why quantization also makes models faster

This surprises people. Fewer bits does not just save memory, it often increases tokens per second.

On Apple Silicon (and most inference hardware), generating each token means reading essentially every weight in the model out of memory. The bottleneck is not arithmetic, it is memory bandwidth. A Mac Studio with 800 GB/s of memory bandwidth can only pull so many bytes per second through the chip.

If your model is 800 GB at BF16, each token requires moving 800 GB. If it is 200 GB at Q4, each token moves 200 GB. Same hardware, roughly four times less data per token, and a meaningful speedup as a result.

That is why comparing benchmarks without knowing the quantization is misleading:

DeepSeek XYZ  BF16   10 tok/s
DeepSeek XYZ  Q4     25 tok/s

Those two lines are not a hardware comparison. One system is moving four times more weight data per token than the other.

Decoding the names: Q4, Q8, BF16, and friends

In benchmarks and model repositories you will see a jungle of labels. Here is the decoder ring.

Baseline precision formats

• FP16: 16-bit floating point. The standard “full precision” for inference.
• BF16: Bfloat16, also 16 bits but with a different split between range and precision. Common for models trained on modern GPUs. For memory planning, treat FP16 and BF16 as identical.
• FP8 / INT8 / Q8: 8-bit. Half the size of 16-bit with almost no measurable quality loss.

The Q-number shorthand

• Q8: roughly 8 bits per weight
• Q6: roughly 6 bits
• Q5: roughly 5 bits
• Q4: roughly 4 bits
• Q3, Q2: aggressive, use with caution

The suffixes and alphabet soup

You will also see formats like Q4_K_M, Q5_K_S, IQ2_XXS, MXFP4, AWQ, and GPTQ. These describe how the quantization is done, not just how many bits:

• Q4_K_M, Q5_K_S, etc.: the “K-quant” family used by llama.cpp and the GGUF format. The letter after K (S, M, L) indicates small, medium, or large, meaning some layers are kept at higher precision. Q4_K_M is the most common “just use this one” choice.
• IQ2_XXS, IQ3_XS, etc.: “importance-aware” quantization that uses a calibration step to protect the weights that matter most. These get surprisingly good results at very low bit counts.
• MXFP4: a block floating-point 4-bit format gaining traction for hardware acceleration.
• AWQ, GPTQ: GPU-oriented quantization methods that use calibration data to minimize quality loss. Common on Nvidia hardware; less common in the Mac world, where GGUF and MLX formats dominate.

The critical takeaway: two different “4-bit” formats are not equivalent. They can differ in real memory footprint, in speed, and especially in output quality. A Q4_K_M and an IQ4_XS of the same model are different things.

How to read a quantized model’s size

A rough rule for planning: take the parameter count in billions and multiply.

Then add headroom. For a comfortable setup, budget an additional 10 to 20% of the model size for the KV cache and activations at moderate context lengths, and much more if you plan to run 100K+ token contexts. Leave a few gigabytes for macOS itself.

Why this changes everything for Mac hardware planning

Now the interesting part. Suppose you are looking at a hypothetical 400-billion-parameter model. At BF16 it is roughly 800 GB. At 8-bit, roughly 400 GB. At 4-bit, roughly 200 GB.

At BF16, that model does not fit on any single Mac. You are into cluster territory, distributing the model across several machines connected over Thunderbolt or Ethernet.

At Q4, the same model fits on a single 256 GB or 512 GB Mac Studio with room for context.

That distinction matters more than it looks, because distributed inference has a real cost. When a model is split across machines, every token requires the machines to exchange intermediate data over the interconnect. Thunderbolt 5 is fast, but it is nowhere near the 800 GB/s of unified memory bandwidth inside a single Mac Studio. For the decode phase (generating tokens one at a time), a model that fits on one machine will very often beat the same model split across several.

Example: the Studio vs. mini cluster decision

Say you are choosing between:

• 1 x Mac Studio with 512 GB unified memory
• 8 x Mac mini with 64 GB each = 512 GB aggregate

Same total memory. You look at a 600 GB model and conclude you need the cluster, since 600 does not fit in 512.

But that 600 GB figure might be BF16. At Q4 it becomes roughly 300 GB. Now it fits on the Studio with 200 GB to spare, no interconnect overhead, and decode speed that the cluster cannot match.

The reverse also happens. Scale up to a 1-trillion-parameter model:

Here quantization decides whether you are looking at one maxed-out Studio (Q4, tight), two Studios, or a genuine cluster. Same model. The bit count sets the shopping list.

The quality question: how much do you actually lose?

This is where honest answers get uncomfortable, because “it depends” is the true one. A few reliable patterns:

• Q8 is effectively lossless for practical purposes. If you have the memory, there is little reason not to use it.
• Q6 and Q5 are nearly indistinguishable from Q8 for most tasks.
• Q4 (especially Q4_K_M) is the workhorse. Most users cannot tell the difference in casual chat. Careful evaluation shows small drops in reasoning-heavy benchmarks and code correctness.
• Q3 and below start to show visible degradation: more factual errors, weaker instruction following, occasional incoherence. Importance-aware formats (IQ3, IQ2) soften this but do not eliminate it.
• Larger models tolerate quantization better than small ones. A 70B model at Q4 is usually far better than a 13B model at Q8, even though they use similar memory. When memory-constrained, prefer a bigger model at lower precision over a smaller model at higher precision.

The right approach is to test with your actual workload. Quality loss from quantization is task-dependent, and the only benchmark that matters is yours.

The six variables every benchmark needs

If you take one practical thing from this post, make it this. A multi-Mac LLM benchmark is only meaningful if it reports all six of these:

1. Model (name and parameter count)
2. Quantization (the specific format, not just “4-bit”)
3. Context length (the KV cache grows with it and eats memory and speed)
4. Hardware (which chip, how much memory, how many machines)
5. Parallelism (single machine, or split across nodes and how)
6. Tokens per second (and whether that is prompt processing or generation)

Leave any one of them out and the number can mislead you into buying the wrong hardware. We now track all six for every benchmark in this series.

Frequently asked questions

Does quantization affect training or only inference?
Almost always inference. Models are trained at high precision and quantized afterward. There is research into training at lower precision, but for anyone running local models, quantization is a deployment step.

Can I quantize a model myself?
Yes. Tools like llama.cpp and Apple’s MLX framework include quantization utilities. In practice most people download pre-quantized versions from model repositories, where community members have already produced a range of formats for popular models.

Which quantization should I download?
For GGUF models on a Mac, Q4_K_M is the default recommendation when memory is tight, and Q8_0 or Q6_K when it is not. For MLX models, 4-bit and 8-bit variants are the standard options.

Is a bigger model at Q4 better than a smaller model at Q8?
Usually yes, when both fit in your memory budget. Parameter count tends to matter more than precision until you drop below roughly 4 bits.

Why does my 4-bit model use more memory than the math predicts?
Three reasons: K-quants keep some layers at higher precision, the KV cache scales with context length, and the runtime needs working memory. Always leave headroom.

Does quantization make responses faster on every machine?
On memory-bandwidth-bound hardware like Apple Silicon, generally yes. On hardware with dedicated low-precision compute units, the gains can be even larger.

The bottom line

Quantization is not a footnote in local AI. It is a first-order design decision that sets your memory requirement, shapes your speed, and determines whether one Mac is enough. Before you compare benchmarks, before you spec hardware, and certainly before you buy a cluster, ask the question: what quantization?