The document discusses scalar quantization and the Lloyd-Max algorithm. It provides examples of using the Lloyd-Max algorithm to design scalar quantizers for Gaussian and Laplacian distributed signals. The algorithm works by iteratively calculating decision thresholds and representative levels to minimize mean squared error. At high rates, the distortion-rate function of a Lloyd-Max quantizer is approximated. The document also discusses entropy-constrained scalar quantization and an iterative algorithm to design those quantizers.

1. Quantization
Input-output characteristic of a scalar quantizer
x ˆ
x
Q Sometimes, this
Output ˆ
x convention is used:
ˆ
xq  2
M represen-
tative levels
x Q q
ˆ
xq 1
ˆ
xq -1
t q+2 q Q x
ˆ
tq t q+1
input signal x
M-1 decision
thresholds
Bernd Girod: EE398A Image and Video Compression Quantization no. 1

2. Example of a quantized waveform
Original and Quantized Signal
Quantization Error
Bernd Girod: EE398A Image and Video Compression Quantization no. 2

3. Lloyd-Max scalar quantizer
 Problem : For a signal x with given PDF f X ( x) find a quantizer with
M representative levels such that
d  MSE  E  X  X    min.
2
ˆ

 

 Solution : Lloyd-Max quantizer
[Lloyd,1957] [Max,1960]
tq   xq 1  xq  q  1, 2,, M -1
1
 M-1 decision thresholds exactly ˆ ˆ
2
half-way between representative tq1
levels.
 M representative levels in the  x f X ( x)dx
centroid of the PDF between two xq  q  0,1, , M -1
tq
ˆ tq 1
successive decision thresholds.
 f X ( x)dx
 Necessary (but not sufficient) tq
conditions
Bernd Girod: EE398A Image and Video Compression Quantization no. 3

4. Iterative Lloyd-Max quantizer design
1. Guess initial set of representative levels xq q  0,1, 2,, M -1
ˆ
2. Calculate decision thresholds
tq   xq 1  xq  q  1, 2,, M -1
1
ˆ ˆ
2
3. Calculate new representative levels
tq1
 x f X ( x)dx
xq  q  0,1,, M -1
tq
ˆ tq1

tq
f X ( x)dx
4. Repeat 2. and 3. until no further distortion reduction
Bernd Girod: EE398A Image and Video Compression Quantization no. 4

5. Example of use of the Lloyd algorithm (I)
 X zero-mean, unit-variance Gaussian r.v.
 Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
 Optimum quantizer, obtained with the Lloyd algorithm
 Decision thresholds -0.98, 0, 0.98
 Representative levels –1.51, -0.45, 0.45, 1.51
 D*=0.12=9.30 dB
Boundary
Reconstruction
Bernd Girod: EE398A Image and Video Compression Quantization no. 5

6. Example of use of the Lloyd algorithm (II)
 Convergence
 Initial quantizer A:  Initial quantizer B:
decision thresholds –3, 0 3 decision thresholds –½, 0, ½
Quantization Function
Quantization Function
6
6
4
4
2 2
0 0
-2 -2
-4 -4
-6
5 10 15 -6
5 10 15
Iteration Number Iteration Number
0.4 0.2
0.2
D
0.15
D
0 0.1
0 5 10 15 0 5
SNROUT final = 9.2978 SNROUT10 = 9.298 15
SNROUT [dB]
SNROUT [dB]
10 10 final
5 8
0 6
0 5 10 15 0 5 10 15
Iteration Number Iteration Number
 After 6 iterations, in both cases, (D-D*)/D*<1%
Bernd Girod: EE398A Image and Video Compression Quantization no. 6

7. Example of use of the Lloyd algorithm (III)
 X zero-mean, unit-variance Laplacian r.v.
 Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
 Optimum quantizer, obtained with the Lloyd algorithm
 Decision thresholds -1.13, 0, 1.13
 Representative levels -1.83, -0.42, 0.42, 1.83
 D*=0.18=7.54 dB
Threshold
Representative
Bernd Girod: EE398A Image and Video Compression Quantization no. 7

8. Example of use of the Lloyd algorithm (IV)
 Convergence
 Initial quantizer A,  Initial quantizer B,
decision thresholds –3, 0 3 decision thresholds –½, 0, ½
Quantization Function
Quantization Function
10 10
5 5
0 0
-5 -5
-10 -10
2 4 6 8 10 12 14 16 2 4 6 8 10 12 14 16
Iteration Number Iteration Number
0.4 0.4
0.2
D
0.2
D
0 0
0 5 10 15 0 5 10 15
8
8
SNR [dB]
SNR [dB]
6 6
SNRfinal = 7.5415 SNRfinal = 7.5415
4 4
0 5 10 15 0 5
Iteration Number Iteration Number10 15
 After 6 iterations, in both cases, (D-D*)/D*<1%
Bernd Girod: EE398A Image and Video Compression Quantization no. 8

9. Lloyd algorithm with training data
1. Guess initial set of representative levels xq ; q  0,1, 2,, M -1
ˆ
2. ˆ
Assign each sample xi in training set T to closest representative xq

Bq  x T : Q x  q  q  0,1,2,, M -1
3. Calculate new representative levels
1
xq 
ˆ
Bq
x
xBq
q  0,1,, M -1
4. Repeat 2. and 3. until no further distortion reduction
Bernd Girod: EE398A Image and Video Compression Quantization no. 9

10. Lloyd-Max quantizer properties
 Zero-mean quantization error
ˆ
E X  X   0
 
 Quantization error and reconstruction decorrelated
 ˆ ˆ
 
E X X X 0
 Variance subtraction property
   E
2
ˆ
X
 X  X 2


2
X
ˆ


 
Bernd Girod: EE398A Image and Video Compression Quantization no. 10

11. High rate approximation
 Approximate solution of the "Max quantization problem,"
assuming high rate and smooth PDF [Panter, Dite, 1951]
1
x( x)  const
3 f X ( x)
Distance between two
Probability density
successive quantizer
function of x
representative levels
 Approximation for the quantization error variance:
3
  1  3 f ( x)dx 

d  E X X 
2
ˆ

  12M 2   X


x 
Number of representative levels
Bernd Girod: EE398A Image and Video Compression Quantization no. 11

12. High rate approximation (cont.)
 High-rate distortion-rate function for scalar Lloyd-Max quantizer
d  R    2 X 22 R
2
3
1  3 
with      f X ( x)dx 
2 2
X
12  x 
Some example values for 
2

uniform 1
Laplacian 9
2  4.5
3
Gaussian  2.721
2
Bernd Girod: EE398A Image and Video Compression Quantization no. 12

13. High rate approximation (cont.)
 Partial distortion theorem: each interval makes an (approximately)
equal contribution to overall mean-squared error
Pr ti  X  ti 1 E 



 X X 2 t  X t 
ˆ
i i 1


 Pr t j  X  t j 1 E


 
 X X 2 t  X t 
ˆ
j j 1


for all i, j
[Panter, Dite, 1951], [Fejes Toth, 1959], [Gersho, 1979]
Bernd Girod: EE398A Image and Video Compression Quantization no. 13

14. Entropy-constrained scalar quantizer
 Lloyd-Max quantizer optimum for fixed-rate encoding, how can we
do better for variable-length encoding of quantizer index?
 Problem : For a signal x with given pdf f X ( x) find a quantizer with
rate M 1
ˆ  
R  H X    pq log 2 pq
q 0
such that

d  MSE  E  X  X    min.
2
ˆ

 

 Solution: Lagrangian cost function

J  d  R  E  X  X     H  X   min.
2
ˆ ˆ

 

Bernd Girod: EE398A Image and Video Compression Quantization no. 14

15. Iterative entropy-constrained scalar quantizer design
1. Guess initial set of representative levels xq ; q  0,1, 2,, M -1
ˆ
and corresponding probabilities pq
2. Calculate M-1 decision thresholds
xq -1  xq
ˆ ˆ log 2 pq 1  log 2 pq
tq =  q  1, 2,, M -1
2 2  xq -1  xq 
ˆ ˆ
3. Calculate M new representative levels and probabilities pq
tq1
 xf X ( x)dx
xq  q  0,1,, M -1
tq
ˆ tq1

tq
f X ( x)dx
4. Repeat 2. and 3. until no further reduction in Lagrangian cost
Bernd Girod: EE398A Image and Video Compression Quantization no. 15

16. Lloyd algorithm for entropy-constrained
quantizer design based on training set
1. Guess initial set of representative levels xq ; q  0,1, 2,, M -1
ˆ
and corresponding probabilities pq
2. Assign each sample xi in training set T to representative ˆ
xq
J x  q    xi  xq    log 2 pq
2
minimizing Lagrangian cost i
ˆ
Bq  x T : Q  x   q q  0,1, 2,, M -1
3. Calculate new representative levels and probabilities pq
1
xq 
ˆ
Bq
x
xBq
q  0,1,, M -1
4. Repeat 2. and 3. until no further reduction in overall Lagrangian
cost
   i 
2
J J  xi x Q x i   log p 2 q  xi 
xi xi
Bernd Girod: EE398A Image and Video Compression Quantization no. 16

17. Example of the EC Lloyd algorithm (I)
 X zero-mean, unit-variance Gaussian r.v.
 Design entropy-constrained scalar quantizer with rate R≈2 bits, and
minimum distortion D*
 Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm
 11 intervals (in [-6,6]), almost uniform
 D*=0.09=10.53 dB, R=2.0035 bits (compare to fixed-length example)
Threshold
Representative level
Bernd Girod: EE398A Image and Video Compression Quantization no. 17

18. Example of the EC Lloyd algorithm (II)
 Same Lagrangian multiplier λ used in all experiments
 Initial quantizer A, 15 intervals  Initial quantizer B, only 4 intervals
(>11) in [-6,6], with the same length (<11) in [-6,6], with the same length
Quantization Function
6 6
Quantization Function
4 4
2 2
0 0
-2 -2
-4 -4
-6 -6
5 10 15 20 25 30 35 40 5 10 15 20
Iteration Number Iteration Number
12 12
R [bit/symbol] SNR [dB]
R [bit/symbol] SNR [dB]
SNRfinal = 8.9452
10 10
SNRfinal = 10.5363 8
8
6 6
0 5 10 15 20 25 30 35 40 0 5 10 15 20
2.5 2.5
Rfinal = 1.7723
2 2
Rfinal = 2.0044 1.5
1.5
1 1
0 5 10 15 20 25 30 35 40 0 5 10 15 20
Iteration Number Iteration Number
Bernd Girod: EE398A Image and Video Compression Quantization no. 18

19. Example of the EC Lloyd algorithm (III)
 X zero-mean, unit-variance Laplacian r.v.
 Design entropy-constrained scalar quantizer with rate R≈2 bits
and minimum distortion D*
 Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm
 21 intervals (in [-10,10]), almost uniform
 D*= 0.07=11.38 dB, R= 2.0023 bits (compare to fixed-length example)
Decision threshold
Representative level
Bernd Girod: EE398A Image and Video Compression Quantization no. 19

20. Example of the EC Lloyd algorithm (IV)
 Same Lagrangian multiplier λ used in all experiments
 Initial quantizer A, 25 intervals  Initial quantizer B, only 4 intervals
(>21 & odd) in [-10,10], with the (<21) in [-10,10], with the same length
same length
Quantization Function
Quantization Function
10 10
5 5
0 0
-5 -5
-10 -10
5 10 15 20 25 30 35 40 5 10 15 20
Iteration Number Iteration Number
15 15
R [bit/symbol] SNR [dB]
R [bit/symbol] SNR [dB]
10 10
SNRfinal = 11.407
5 5 SNRfinal = 7.4111
0 5 10 15 20 25 30 35 40 0 5 10 15 20
3 3
Rfinal = 2.0063
2 2 Rfinal = 1.6186
1 1
0 5 10 15 20 25 30 35 40 0 5 10 15 20
Iteration Number Iteration Number
 Convergence in cost faster than convergence of decision thresholds
Bernd Girod: EE398A Image and Video Compression Quantization no. 20

21. High-rate results for EC scalar quantizers
 For MSE distortion and high rates, uniform quantizers (followed by
entropy coding) are optimum [Gish, Pierce, 1968]
 Distortion and entropy for smooth PDF and fine quantizer interval 

2
 
H X  h  X   log 2 
ˆ
2
d   d 
2
 12
2
 Distortion-rate function
1 2 h X  2 R
d  R  2 2
12
e
is factor or 1.53 dB from Shannon Lower Bound
6
1 2 h X  2 R
D  R  2 2
2 e
Bernd Girod: EE398A Image and Video Compression Quantization no. 21

22. Comparison of high-rate performance of
scalar quantizers
 High-rate distortion-rate function
d  R    2 X 22 R
2
 Scaling factor 2
Shannon LowBd Lloyd-Max Entropy-coded
6
Uniform  0.703 1 1
e
e 9 e2
Laplacian  0.865  4.5  1.232
 2 6
3 e
Gaussian 1  2.721  1.423
2 6
Bernd Girod: EE398A Image and Video Compression Quantization no. 22

23. Deadzone uniform quantizer
Quantizer
ˆ
output x


Quantizer

input x
Bernd Girod: EE398A Image and Video Compression Quantization no. 23

24. Embedded quantizers
 Motivation: “scalability” – decoding of compressed
bitstreams at different rates (with different qualities)
 Nested quantization intervals
Q0 x
Q1
Q2
 In general, only one quantizer can be optimum
(exception: uniform quantizers)
Bernd Girod: EE398A Image and Video Compression Quantization no. 24

25. Example: Lloyd-Max quantizers for Gaussian PDF
0 1
0 0 1 1
0 1 0 1
 2-bit and 3-bit optimal -.98 .98
quantizers not embeddable
0 0 0 01 1 1 1
 Performance loss for 0 0 1 10 0 1 1
embedded quantizers 0 1 0 10 1 0 1
-1.75 -1.05 -.50 .50 1.05 1.75
Bernd Girod: EE398A Image and Video Compression Quantization no. 25

26. Information theoretic analysis
 “Successive refinement” – Embedded coding at multiple
rates w/o loss relative to R-D function
ˆ
E  d ( X , X 1 )   D1 ˆ
I ( X ; X 1 )  R( D1 )
 
ˆ
E  d ( X , X 2 )   D2 ˆ
I ( X ; X )  R( D )
  2 2
 “Successive refinement” with distortions D1 and D2  D1
can be achieved iff there exists a conditional distribution
fX ,X
ˆ ˆ X
( x1 , x2 , x)  f X
ˆ ˆ ˆ X
ˆ
( x2 , x) f X
ˆ ˆ
X2
ˆ ˆ
( x1 , x2 )
1 2 2 1
 Markov chain condition
ˆ ˆ
X  X 2  X1
[Equitz,Cover, 1991]
Bernd Girod: EE398A Image and Video Compression Quantization no. 26

27. Embedded deadzone uniform quantizers
x=0
8   4
Q0 x
4   2
Q1
Q2
2   
Supported in JPEG-2000 with general  for
quantization of wavelet coefficients.
Bernd Girod: EE398A Image and Video Compression Quantization no. 27

28. Vector quantization
Bernd Girod: EE398A Image and Video Compression Quantization no. 28

29. LBG algorithm
 Lloyd algorithm generalized for VQ [Linde, Buzo, Gray, 1980]
Best representative Best partitioning
vectors of training set
for given partitioning for given
of training set representative
vectors
 Assumption: fixed code word length
 Code book unstructured: full search
Bernd Girod: EE398A Image and Video Compression Quantization no. 29

30. Design of entropy-coded vector quantizers
 Extended LBG algorithm for entropy-coded VQ
[Chou, Lookabaugh, Gray, 1989]
 Lagrangian cost function: solve unconstrained problem rather than
constrained problem
J  d  R  E


ˆ


ˆ  
 X  X 2    H X  min.
 Unstructured code book: full search for
J xi  q   xi  xq   log 2 pq
2
ˆ
The most general coder structure:
Any source coder can be interpreted as VQ with VLC!
Bernd Girod: EE398A Image and Video Compression Quantization no. 30

31. Lattice vector quantization
• • • • • • • pdf
•
• • • • • •
• • •
• • • •
• • • •
Amplitude 2
• • •
• • • • • •
• • • • • • • •
• • • • • • • • •
• • • • • •
cell • representative
vector
Amplitude 1
Bernd Girod: EE398A Image and Video Compression Quantization no. 31

32. 8D VQ of a Gauss-Markov source
18 LBG fixed
CWL
12
SNR [dB]
E8-lattice
LBG variable
CWL
6
scalar
0
r  0.95
0 0.5 1.0
Rate [bit/sample]
Bernd Girod: EE398A Image and Video Compression Quantization no. 32

33. 8D VQ of memoryless Laplacian source
9
E8-lattice
LBG variable
CWL
6 scalar
SNR [dB]
3
LBG fixed
CWL
0
0 0.5 1.0
Rate [bit/sample]
Bernd Girod: EE398A Image and Video Compression Quantization no. 33

34. Reading
 Taubman, Marcellin, Sections 3.2, 3.4
 J. Max, “Quantizing for Minimum Distortion,” IEEE Trans.
Information Theory, vol. 6, no. 1, pp. 7-12, March 1960.
 S. P. Lloyd, “Least Squares Quantization in PCM,” IEEE
Trans. Information Theory, vol. 28, no. 2, pp. 129-137,
March 1982.
 P. A. Chou, T. Lookabaugh, R. M. Gray, “Entropy-
constrained vector quantization,” IEEE Trans. Signal
Processing, vol. 37, no. 1, pp. 31-42, January 1989.
Bernd Girod: EE398A Image and Video Compression Quantization no. 34