Quantization & Compression — Advanced

Method internals and trade-offs: GPTQ/AWQ/SmoothQuant/LLM.int8, NF4 and double quant, FP8, SparseGPT vs Wanda, KV-cache quant, extreme low-bit, distillation objectives.

Hard50 questions

Sample questions

GPTQ builds on Optimal Brain Quantization. What information does it use to compensate for the error of each quantized weight?

The running mean of activations gathered during a warm-up phase
The gradient of the loss with respect to each individual weight
The variance of the weights within each quantization group
The layer's Hessian, used to cancel each weight's induced error

Why

GPTQ uses second-order (Hessian) information from a calibration set to update the not-yet-quantized weights so they absorb the error introduced by each rounding step. It is not a simple activation mean, a first-order gradient, or a per-group variance statistic.

GPTQ HF: GPTQ

In GPTQ, what does enabling 'act-order' (activation-order / desc_act) change?

Activations rather than weights become the quantized tensors
The group size is doubled to reduce the number of scales
The calibration set is shuffled before every layer
Columns are quantized in order of activation importance, improving accuracy

Why

act-order processes weight columns from most to least important (by the diagonal of the Hessian) so error accumulates on the least significant columns, usually improving accuracy at some speed cost. It does not switch to activation quantization, change group size, or reshuffle calibration data.

GPTQ HF: GPTQ

AWQ avoids second-order optimization. How does it decide how much to scale each weight channel?

From activation magnitudes, via a small grid search
By backpropagating a reconstruction loss through the whole layer
By the variance of the weights in each output channel
By pruning channels below a magnitude threshold first

Why

AWQ measures which channels are salient from activation statistics and searches for a per-channel scale that protects them before round-to-nearest — no gradient descent. It is not backprop-based, not driven by weight variance, and does not prune channels.

AWQ HF: AWQ

LLM.int8() detects outlier feature dimensions using a magnitude threshold (about 6.0). Why is a per-column (vector-wise) treatment necessary?

Every column has the same range, so one shared scale would suffice
Outliers appear only in the weight matrix, never in activations
Outliers concentrate in specific feature dimensions, not uniformly across the tensor
The threshold must be recomputed for each token at generation time

Why

Emergent outliers cluster in a few specific hidden dimensions, so those columns are split out to FP16 while the rest use vector-wise INT8 scales. A single shared scale is exactly what fails; outliers are in activations, and the decomposition is structural, not per-token recomputation.

LLM.int8()SmoothQuant

SmoothQuant introduces a per-channel smoothing factor. What is the migration strength alpha trading off?

The ratio of INT8 to INT4 bits used across the layer
How much quantization difficulty is shifted from activations onto weights
The fraction of channels kept in FP16 as outliers
The learning rate applied during a short fine-tuning pass

Why

Alpha controls how aggressively the equivalent transform moves range from activations (hard to quantize) to weights (easy); too high overloads the weights, too low leaves activation outliers. It is not a bit-allocation ratio, an outlier fraction, or a learning rate — SmoothQuant is training-free.

SmoothQuant LLM.int8()

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack