Quantization (signal processing)

The simplest way to quantize a signal is to choose the digital amplitude value closest to the original analog amplitude. This example shows the original analog signal (green), the quantized signal (black dots), the signal reconstructed from the quantized signal (yellow) and the difference between the original signal and the reconstructed signal (red). The difference between the original signal and the reconstructed signal is the quantization error and, in this simple quantization scheme, is a deterministic function of the input signal.

In mathematics and digital signal processing, quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements. Rounding and truncation are typical examples of quantization processes. Quantization is involved to some degree in nearly all digital signal processing, as the process of representing a signal in digital form ordinarily involves rounding. Quantization also forms the core of essentially all lossy compression algorithms.

The difference between an input value and its quantized value (such as round-off error) is referred to as quantization error, noise or distortion. A device or algorithmic function that performs quantization is called a quantizer. An analog-to-digital converter is an example of a quantizer.

Δ can be expressed as

Q(x)=Δ⋅⌈xΔ−12⌉.

(The notation Δ=1 or with 10⋅log10(1/4) ≈ −6 dB.

Because the set of possible output values of a quantizer is countable, any quantizer can be decomposed into two distinct stages, which can be referred to as the classification stage (or forward quantization stage) and the reconstruction stage (or inverse quantization stage), where the classification stage maps the input value to an integer quantization index yk that is the output approximation of the input value. For the example uniform quantizer described above, the forward quantization stage can be expressed as

Q(x)=Δ⋅⌊xΔ+12⌋,

Mid-riser quantization involves truncation. The input-output formula for a mid-riser uniform quantizer is given by:

yk=Δ⋅(k+12).

Note that mid-riser uniform quantizers do not have a zero output value – their minimum output magnitude is half the step size. In contrast, mid-tread quantizers do have a zero output level. For some applications, having a zero output signal representation may be a necessity.

In general, a mid-riser or mid-tread quantizer may not actually be a uniform quantizer – i.e., the size of the quantizer's classification intervals may not all be the same, or the spacing between its possible output values may not all be the same. The distinguishing characteristic of a mid-riser quantizer is that it has a classification threshold value that is exactly zero, and the distinguishing characteristic of a mid-tread quantizer is that is it has a reconstruction value that is exactly zero.

A dead-zone quantizer is a type of mid-tread quantizer with symmetric behavior around 0. The region around the zero output value of such a quantizer is referred to as the dead zone or deadband. The dead zone can sometimes serve the same purpose as a noise gate or squelch function. Especially for compression applications, the dead-zone may be given a different width than that for the other steps. For an otherwise-uniform quantizer, the dead-zone width can be set to any value rk is a reconstruction offset value in the range of 0 to 1 as a fraction of the step size. Ordinarily, k in general and can be chosen to fulfill the optimality condition described below, it is often simply set to a constant, such as w=Δ and Δ relative to the signal strength) with smooth PDFs.

Additive noise behavior is not always a valid assumption. Quantization error (for quantizers defined as described here) is deterministically related to the signal and not entirely independent of it. Thus, periodic signals can create periodic quantization noise. And in some cases, it can even cause limit cycles to appear in digital signal processing systems. One way to ensure effective independence of the quantization error from the source signal is to perform dithered quantization (sometimes with noise shaping), which involves adding random (or pseudo-random) noise to the signal prior to quantization.

In the typical case, the original signal is much larger than one least significant bit (LSB). When this is the case, the quantization error is not significantly correlated with the signal and has an approximately uniform distribution. When rounding is used to quantize, the quantization error has a mean of zero and the root mean square (RMS) value is the standard deviation of this distribution, given by 13LSB. Although rounding yields less RMS error than truncation, the difference is only due to the static (DC) term of SQNR=20log10(2Q)≈6.02⋅Q dB

where Q is the number of quantization bits.

The most common test signals that fulfill this are full amplitude triangle waves and sawtooth waves.

For example, a 16-bit ADC has a maximum signal-to-quantization-noise ratio of 6.02 × 16 = 96.3 dB.

When the input signal is a full-amplitude sine wave the distribution of the signal is no longer uniform, and the corresponding equation is instead

M non-overlapping intervals {bk}k=1M−1, such that b0=−∞ and Ik are associated with the same quantization index yk which implements the mapping k with a binary codeword M-level quantizer and an associated set of codewords for communicating its index values requires finding the values of {yk}k=1M which optimally satisfy a selected set of design constraints such as the bit rate S produces random variables pk that the random variable falls within a particular quantization interval R, in units of average bits per quantized value, for this quantizer can be derived as follows:

R depends on the decision boundaries D depends on the decision boundaries D≤Dmax, minimize the bit rate D

Often the solution to these problems can be equivalently (or approximately) expressed and solved by converting the formulation to the unconstrained problem {yk}k=1M affect only the distortion – they do not affect the bit rate – and that each individual D=∑k=1Mdk

where

yk can be optimized separately to minimize its contribution to the distortion yk within each interval {k}k=1M, such that effectively

{pk}k=1M.

In some designs, rather than optimizing for a particular number of classification regions M approaches infinity.

Neglecting the entropy constraint: Lloyd–Max quantization

edit

In the above formulation, if the bit rate constraint is neglected by setting M-level quantizer can be coded using a fixed-length code using R is 8 bits/symbol. For this reason, such a quantizer has sometimes been called an 8-bit quantizer. However, using an FLC eliminates the compression improvement that can be obtained by use of better entropy coding.

Assuming an FLC with f(x) and the constraint that the quantizer must use only {yk}k=1M to minimize the resulting distortion

∂D/∂yk=0, as follows:

x∈[−Xmax,Xmax] and 0 elsewhere. The step size N bits, N=8 bits, M=65536 and SQNR = 16×6 = 96 dB. The property of 6 dB improvement in SQNR for each extra bit used in quantization is a well-known figure of merit. However, it must be used with care: this derivation is only for a uniform quantizer applied to a uniform source. For other source PDFs and other quantizer designs, the SQNR may be somewhat different from that predicted by 6 dB/bit, depending on the type of PDF, the type of source, the type of quantizer, and the bit rate range of operation.

However, it is common to assume that for many sources, the slope of a quantizer SQNR function can be approximated as 6 dB/bit when operating at a sufficiently high bit rate. At asymptotically high bit rates, cutting the step size in half increases the bit rate by approximately 1 bit per sample (because 1 bit is needed to indicate whether the value is in the left or right half of the prior double-sized interval) and reduces the mean squared error by a factor of 4 (i.e., 6 dB) based on the