Quantization & Compression — Core

Numeric formats, scale and zero-point, PTQ vs QAT, weight vs activation quantization, LLM.int8/GPTQ/AWQ/NF4, distillation, and pruning.

Medium50 questions

Sample questions

What is the primary goal of quantizing a neural network's weights to a lower-precision type like INT8?

Reduce memory and speed up inference with fewer bits per value
Increase the model's accuracy on held-out validation data
Add more trainable parameters so the model can fit harder tasks
Guarantee the outputs are identical to the full-precision model

Why

Quantization stores each value in fewer bits, cutting memory and often speeding up compute; it is a compression/efficiency technique. It does not add parameters, and it typically loses a little accuracy rather than improving or exactly preserving it.

HF: Quantization concepts PyTorch Quantization docs

How much GPU memory do the weights of a 7-billion-parameter model occupy in FP16?

About 7 GB
About 28 GB
About 14 GB
About 3.5 GB

Why

FP16 uses 2 bytes per parameter, so 7B x 2 bytes is about 14 GB. 7 GB would be 1 byte (INT8), 3.5 GB would be 4-bit, and 28 GB would be FP32 (4 bytes).

HF: Quantization concepts The case for 4-bit precision

In affine (asymmetric) uniform quantization, what do the scale and zero-point encode?

The learning rate and the weight-decay coefficient used during quantization
The step size between levels and the integer that maps to real zero
The mean and variance of the activations across the calibration set
The number of bits and the clipping threshold selected by the user

Why

A real value r maps to an integer q via r = scale x (q - zero_point): the scale is the quantization step, and the zero-point is the integer representing r=0. The other pairs describe optimizer settings, activation statistics, or unrelated hyperparameters.

PyTorch Quantization docs HF: Quantization concepts

What distinguishes symmetric quantization from asymmetric quantization?

Symmetric pins zero-point at 0; asymmetric lets it shift to fit range
Symmetric uses INT8 while asymmetric always uses INT4 for the same tensor
Symmetric quantizes weights while asymmetric quantizes only activations
Symmetric requires calibration data whereas asymmetric never does

Why

Symmetric quantization centers the range at zero (zero-point = 0), which suits roughly zero-centered weights; asymmetric adds a nonzero zero-point to cover skewed ranges like post-ReLU activations. Bit-width, weight-vs-activation, and calibration are independent choices.

PyTorch Quantization docs HF: Quantization concepts

Why is BF16 often preferred over FP16 for training large models despite having fewer mantissa bits?

It stores twice as many significant digits as FP16 for higher precision
It keeps FP32's 8-bit exponent, so its dynamic range resists overflow and underflow
It uses 32 bits internally and only rounds to 16 bits when saving to disk
It removes the need for any loss scaling by using integer arithmetic

Why

BF16 allocates the same 8 exponent bits as FP32, giving a wide dynamic range that avoids the overflow/underflow FP16 suffers (FP16 has only 5 exponent bits). The cost is fewer mantissa bits, so BF16 has less precision, not more, and it is still a 16-bit float.

Mixed Precision Training FP8 Formats for Deep Learning

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack