This document discusses video compression techniques used in standards like MPEG. It examines temporal and spatial redundancy reduction methods like motion-compensated prediction coding. Motion estimation algorithms including full search and fast search methods are described. The document also covers video coding concepts such as frame types, transmission order and syntax elements in video streams.

1. EIE415MultimediaTechnology
VideoCompression
Dr.Y.H.Chan
2EIE415MultimediaTechnology
LearningObjectives
Examineboththetheoreticandpracticalaspectsofthevideo
compressionprocess.
Describethetemporalredundancythatmaybeexploitedby
videocompressionalgorithm.
Addressmotion-compensatedpredictioncodinginthevideo
compression.
DescribeMPEGcompressionstandards.
References:
“MultimediaCommunications:Applications,Networks,Protocols
andStandards”,FredHalsall,Addison-Wesley,Chapter4
3EIE415MultimediaTechnology
Contents
Fundamentalsofvideo
■Pictureformats■Framerates■Temporalredundancy
■spatialredundancy
Motioncompensationprediction
■Motionestimation■Motioncompensation■Motionvector
■Fullsearchalgorithm■3-stepsearchalgorithm
■Complexityvsperformance
Codec
■Typesofframes■Transmissionorder■Codingofdifferentframes
Videostreamsyntax
■Sequence■GOP■Picture■Slice■Macroblock■Block
4EIE415MultimediaTechnology
Contents(Cont.)
Standards:Background&History
■MPEG-1/-2/-4/H-264/AVS
■MPEG-2:■ProfilesandLevels
■Scalability-variousschemesandtheiruse
■MPEG-4:■Goals
■codingofvideoobjects
■H.264:■Newfeatures
Inter-mode:■Sub-macroblocksofvarioussizes
■Multiplereferenceframes
■B-framepredictionweighting
■(1/8)-pixelmotionvectorprecision
Intra-mode:■Spatialprediction
■Integertransform

2. EIE415MultimediaTechnology
FundamentalsofVideo
6EIE415MultimediaTechnology
Introduction
Thebitraterequirementsresultingfromalldigital
videoaresubstantiallylargerthanthebitratesofthe
transmissionchannels.
Compressionisnecessary
NOTjustasinglestandardassociatedwithvideo
H.261,H.263,…
MPEG-1,MPEG-2andMPEG-4
7EIE415MultimediaTechnology
PictureFormat(HighResolution)
CCIR-6014:2:2format
Y:720×480
UV:360×480
Application:OriginaldigitizationformatusedinCCIR-601
(TelevisionStudios)
CCIR-6014:2:0format
Y:720×480
UV:360×240(Absenceofchrominancesamplesinalterativelines)
Application:Digitalvideobroadcastapplications
8EIE415MultimediaTechnology
PictureFormat(LowResolution)
CIF(CommonIntermediateFormat)
4:2:0format
Y:360×288
UV:180×144
Applications:Apicturequalitycomparablewiththatobtainedwith
VCRsqualityorvideoconferencing
QCIF
4:2:0format
Y:180×144
UV:90×72
Applications:Mobilemultimedia

3. 9EIE415MultimediaTechnology
DigitalVideo
Video(movingpictures):asequenceofdigitizedpicture.
10EIE415MultimediaTechnology
DigitalVideo(Framerates)
framerates:
full-motionvideo:24-30frames/sor
even60frames/sforHDTV
animation:15-19frames/s
videotelephony:5-10frames/s
videoconferencing&interactivemediaapplications:
15-30frames/s
11EIE415MultimediaTechnology
VideoCompressionPrinciples
Video(movingpictures):asequenceofdigitizedpicture.
MovingJPEG(10:1to24:1):spatialredundancy
Consider:
apersonorvehiclemovingacrossthescreeninamovie
Lastforatleast3s
Assume:framerate60framepersecond⇒180frames
Bysendingonlyinformationrelatingtothosesegmentsofeachframe
thathavemovementassociatedwiththem
Considerablesavingsinbandwidth
Temporalredundancy
12EIE415MultimediaTechnology
FrameDifference
Coding
Video-TemporalRedundancy
Similaritiesexist
betweensuccessive
frames
MotionEstimation
andCompensation
Processingfor
reducingtemporal
redundancies
framet-1framet
Processingforreducingspatial
redundancies

4. EIE415MultimediaTechnology
Motion-CompensationPrediction
14EIE415MultimediaTechnology
Motion-Compensation
Moving
Object
(u,v)
Displaced
Object
timet
Previousframe
Currentframe
Stationary
background
Motioncompensation
•theprocessofcompensatingforthe
displacementofmovingobjectsfrom
oneframetoanother
•Predictionforluminancesignal
I(x,y,t)withthemovingobject:
)1,,(),,(ˆ−−−=tvyuxItyxI
15EIE415MultimediaTechnology
HybridCodingSchemewithMCPrediction
Motion
Estimation
Motion
Compensation
2D-DCT
CodingI(x,y,t)
I(x,y,t-1)
motionvector
(u,v)
e(x,y,t)
2D-DCT
Decoding
Motion
Compensation
Framememory
e(x,y,t)I(x,y,t)
I(x+u,y+v,t-1)
Encoder
Decoder
Motionestimation-theprocessoffinding
correspondingpixelsamongframes,i.e.
themotionvector(u,v).
Motionvector-therelativedisplacementof
aobjectfromoneframetoanother
Predictionerror:
Thepredictionerrorisencodedby
theDCTcodingmethod
Motioncompensationisperformedinboththe
encoderandthedecoder,butmotionestimation
isneededonlyinencoder
⇒asymmetricproperty
)1,,(),,(
),,(ˆ),,(),,(
−−−−
−=
tvyuxItyxI
tyxItyxItyxe
16EIE415MultimediaTechnology
BlockMatchingMotionEstimation
N
N
N
N
Block
searchwindow(-D,D)
D
D
D
D
framet
framet-1
tormotionvecfinalthe:),(
tormotionveccandidatetheofposition:),(
framereferenceintheblock:),(
framecurrentintheblock:),(
),(minarg),(
|),(),(|),(
1
l)(k,
1
0
1
0
1
vu
lk
I
I
lkSADvu
ljkiIjiIlkSAD
t
t
N
i
N
j
tt
⋅⋅
⋅⋅
=
++−=
−
−
=
−
=
−∑∑
MatchingCriterion:
SumAbsoluteDifference
Theblockthatgivesthebest
matchforagivenmatching
criterionischosenasthe
motionvector.
Fullsearching
mostaccurate
heavycomputation
Fastsearching
reducedcomputationcomplexity
deteriorationinpicturequality

5. 17EIE415MultimediaTechnology
CurrentframeReferenceframe
Searchingwindow
Thelocalcoordinatesystemusedinthe
searchingwindowareusedtodetermine
themotionvectorofthemacroblock.
Macroblock
ofinterest
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
Candidate
macroblockFullSearch
18EIE415MultimediaTechnology
Searchingwindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
1
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
MAE
Macroblock
ofinterest
MAEplaneinthewindow
CurrentframeReferenceframe
FullSearch
19EIE415MultimediaTechnology
12365978842
Searchingwindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
MAE
Macroblock
ofinterest
MAEplaneinthewindow
CurrentframeReferenceframe
FullSearch
20EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
MAE
Macroblock
ofinterest
MAE
Macroblock
ofinterest
CurrentframeReferenceframe
FullSearch

6. 21EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
CurrentframeReferenceframe
FullSearch
1
•minMAE=0happensat(-5,-2)
⇒themotionvectoroftheblockis(-5,-2).
•Realizationcomplexityishigh.
Macroblock
ofinterest
22EIE415MultimediaTechnology
n-StepHierachicalSearch(n-SHS)
(Tutorial)
23EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
CurrentframeReferenceframe
Macroblock
ofinterest
39
52
3-stepsearch
Step1
6
37
3
3
24EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
CurrentframeReferenceframe
Macroblock
ofinterest
39
526
37
3
3
51
635
25
9
6
3-stepsearch
Step2

7. 25EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
CurrentframeReferenceframe
Macroblock
ofinterest
39
526
37
3
35
8102
3
1
751
635
25
9
6
3
3-stepsearch
Step3
26EIE415MultimediaTechnology
7387513483731
4365165242832
5179375493273
8592466675739
1105111028323847
4549317253786
7387513483735
9521478547771
1497825584322
8413373529463
17116861486351
62631183575654
12365978842
Searchingwindow
MAEplaneinthewindow
(0,-6)
(-6,0)(6,0)
(0,6)
(0,0)
11
-6-5-4-3-2-10123456x=
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6y=
CurrentframeReferenceframe
Macroblock
ofinterest
39
526
37
3
35
8102
3
1
751
635
25
9
6
31
1
•TwoinallcheckedcandidateblocksprovidestheminMAE.
⇒themotionvectorfortheconcernedblockis(-1,0)or(-1,-1).
•Lowcomplexitybutmaynotprovidethebestmotionvector.
3-stepsearch
Step3
27EIE415MultimediaTechnology
GlobalandLocalMinima
3-stepsearchcanbetrappedinthelocalminima
Localminimaofthedistortionfunctiongiving
predictionimageswithhighererrorthanfullsearch
So,manyresearchworkshavebeencarriedoutto
findalternativeapproachesforblockmatching
motionestimation.
EIE415MultimediaTechnology
Codec

8. 29EIE415MultimediaTechnology
Frame
Memory
2D-DCTQuantizer
Dequantizer
Inverse
2D-DCT
Frame
Memory
Motion
Compensation
Motion
Estimation
VLC
coder
Buffer
Regulator
011010...111
Compressed
bitstream
Motionvectors
Predictiveframe
Source
video
+
++
-
VideoEncoder
30EIE415MultimediaTechnology
VideoDecoder
BufferDequantizer
Inverse
2D-DCT
VLC
Decoder
Frame
Memory
Motion
Compensation
+
+
Qunatizer
stepsize
Motionvectors
Recontructedvideo0110..11
Compressed
bitstream
31EIE415MultimediaTechnology
PictureTypes
IIPPPPP
Forwardprediction
PP
Intra(I)picture
withoutreferencetoanyotherpictures
provideaccesspointstothecodedsequencewheredecodingbegin
codedwithonlymoderatecompression(10:1)
Predicted(P)picture
referencetotheI-picturesorP-pictures
generallyusedasareferenceforfurtherprediction
Compressionratioisabout20:1
32EIE415MultimediaTechnology
IIBBBBP
Forwardprediction
Bidirectionalprediction
BB
WithB-picture
Bidirectional(B)picture
usingmotioncompensatedpredictionfromapastandafutureIorP
picture
neverusedasareferenceorfurtherprediction
highestcompressionratio(50:1)

9. 33EIE415MultimediaTechnology
TransmissionandDisplayOrder
IBBBPBBBIBBBPBBBI
1234567891011121314151617
DisplayOrder
IPBBBIBBBPBBBIBBB
1523496781310111217141516
TransmissionOrder
Rule:BeforeyousendaBframe,makesureallitsreferenceframeshavebeensent.
Sendthereferenceframefirstifnot.
34EIE415MultimediaTechnology
Example
AnMPEG-1systemusestheframesequenceshownas
below.Deriveasuitablereorderedsequencethatensures
firstly,onlytwoframesmustbestoredinthedecoder,and
secondly,therequiredIand/orP-framesareavailableto
decodeeachP-andB-frameastheyarereceived.
35EIE415MultimediaTechnology
Solution
Asuitablereorderedframesequencethat
meetsthedefinedrequirementsis:
I,P,B,B,P,B,B,P,B,B,I,B,B,...
1,4,2,3,7,5,6,10,8,9,13,11,12
36EIE415MultimediaTechnology
CodingofI-Picture
2-D
FDCT
QVLCI-Frame
Coded
I-Frame
Intraframecoding
8×8DCT
AnyweightingmatrixforDCTcoefficientspossible
DifferentialcodingofDCcoefficient
Uniformqunatization
Runlengthcodingofzeroswithzig-zagscan
Entropycoding

10. 37EIE415MultimediaTechnology
CodingofP-Pictures
MotioncompensatedpredictionfromanI-orP-pictures
Onemotionvectorpermacroblock
Codingofpredictionerrorwith8×8DCT,uniformqunatization,zig-zag
scan(likeI-picture)
38EIE415MultimediaTechnology
CodingofMotionVectors
Highlycorrelatedamongneighboringmacroblocks
Inmanystandards,macroblockistheunitfor
motionestimation.Itwillbefinetunedtotheblock
levelifnecessary
Codethedifferenceusingvariablelengthcode
(VLC)
39EIE415MultimediaTechnology
CodingofB-Pictures
2-D
FDCT
QVLC
refframe1
Coded
residual
signal
+
Motion
Estimator
Interpolation
Encodethe
motion
vectors
Coded
Motion
Vectors
for
B-frame
refframe2
Prediction
Error
ForwardMotionVectors
Motion
Estimator
+
BackwardMotionVectors
currentframe
40EIE415MultimediaTechnology
MotioncompensatedpredictionfromtwoconsecutiveP-orI-pictures
either
onlyforwardprediction(1vector/macroblock)
or
onlybackwardprediction(1vector/macroblock)
or
averageofforwardandbackwardprediction=interpolation(2
vectors/macroblock)
Codingofpredictionerrorwith8×8DCT,uniformqunatization,zig-zag
scan(likeI-picture)

11. EIE415MultimediaTechnology
VideoStreamSyntax
42EIE415MultimediaTechnology
VideoStreamSyntax–SixLayers
SequenceLayer
GOPLayer
PictureLayer
SliceLayer
MacroblockLayer
BlockLayer
43EIE415MultimediaTechnology
VideoStreamSyntax
Groupof
Picture
Picture
YY
YY
CbCr
Macroblock
Block
IBBPBBIBBPBBIB...
GroupofPicture
44EIE415MultimediaTechnology
SupplementarynotesonVideo
StreamSyntax
Y1Y2
Y3Y4
CbCr
16×16
Macroblock
8×8Block
BlocklayerY1Y2Y3Y4CrCb
DCAC…..EOB
DCTUnitDCTUnit
Macrolock
layer
Y1Y2Y3Y4CrCb
Address
Quant.
value
Motion
vector
…….
MotionEstimationUnitMotionEstimationUnit
MBHeader

12. 45EIE415MultimediaTechnology
MBHeader
Macrolocklayer
Y1Y2Y3Y4CrCb
MB1
SlicelayerSliceHeaderMB1MB3MB4…..MBNMB2
Slice
Startcode
Slice
position
Quant.
value
Slice1
Slice4
Slice2
Slice3
Slice5
Slice6
Picturelayer
Picture
Header
Slice
1
Slice
2
Slice
3
Slice
4
Slice
5
Slice
6
46EIE415MultimediaTechnology
Groupof
Picture
IBBPBBIBBPBBIB...
GroupofPicture
Slice
1
Slice
2
Slice
3
Slice
4
Slice
5
Slice
6
PicturelayerPicture
Header
P2P3P4P5P6
Picture
startcode
Picture
type
….
P1
GOP2GOP3GOP4GOP5GOP6
Sequence
Header
Sequence
layer
GOP1
Sequence
startcode
Video
parameters
Quantization
value
GOPHeaderGOPlayer
GOPHeader
47EIE415MultimediaTechnology
GOPstartcode:indicatesthestartofaGOP
Timestamp:usedforsynchronizationpurposes
Parameters:definestheparticularsequenceof
frametypesthatareusedineachGOP(e.g.
IPPBPP)
48EIE415MultimediaTechnology
Picturelayer
Slice
1
Slice
2
Slice
3
Slice
4
Slice
5
Slice
6
Picture
Header
Sequence
layerGOP2GOP3GOP4GOP5GOP6
Sequence
Header
GOP1
P2P3P4P5P6P1GOPHeaderGOPlayer
SlicelayerSliceHeaderMB1MB3MB4…..MBNMB2
MBHeaderY1Y2Y3Y4CrCb
Y1Y2Y3Y4CrCb
Macroblocklayer
Blocklayer

13. 49EIE415MultimediaTechnology
SliceLayer
Eachsliceisacontiguous
sequenceofmacroblocksinraster
scanorder,startingatspecific
addressorpositioninthepicture
specifiedinthesliceorder
Eachelementinthesliceisa
macroblock
ReasonsforSliceconcepts:
(i)toallowflexibilityinsignalchanges
insomeofthecodingparameters
(ii)tooptimizequalityforagivenbit
rate
(iii)tocontrolthebitrate
EIE415MultimediaTechnology
Standards:Background&History
51EIE415MultimediaTechnology
WhatisMPEG?
MPEGmeansMovingPictureExpertsGroup
aworkinggroupofISO/IEC
Developmentofinternationalstandardsforcompression,
decompression,processing,andcodedrepresentationof
movingpictures,audioandtheircombination.
52EIE415MultimediaTechnology
ChronologicalTableofVideoCodingStandards
H.261
(1990)
MPEG-1
(1993)
H.263
(1995/96)H.263+
(1997/98)
H.263++
(2000)
H.264
(MPEG-4
Part10)
(2002)
MPEG-4v1
(1998/99)
MPEG-4v2
(1999/00)
MPEG-4v3
(2001)
19901992199419961998200020022003
MPEG-2
(H.262)
(1994/95)
ISO/IEC
MPEG
ITU-T
VCEG

14. 53EIE415MultimediaTechnology
MPEGAchievements
BuiltMPEG-1,MPEG-2
Widelyadoptedinaudiovisualindustry
DigitalTV;VCD,DVD,Video-On-Demand,archiving,Musiconthe
Internet
MPEG-4isstillon-going
Firstversionreadyatendof1998
Secondversionwasapprovedattheendof1999
MPEG-7“MultimediaContentDescriptionInterface”
Workingdraftwasappearedattheendofyear1999
IShadbeenapprovedatJuly2001
EIE415MultimediaTechnology
MPEG-1
55EIE415MultimediaTechnology
History:MPEG-1
Audio/VideoonDigitalStorageMedia(1.5Mbit/s,
CIF)
StartLate1988
CommitteeDraft9/90(BitstreamSyntaxand
Decoding)
ISO/IEC11172-1…-5(Systems,Video,Audio,
Compliance,Software)
56EIE415MultimediaTechnology
MPEG-1
Progressivescanningwitharefreshrateof30Hz(forNTSC)
and25Hz(forPAL).
SupportI-,P-andB-frames
I-framesmustbeusedforthevariousrandom-access
functionsassociatedwithVCRs.
Anewlayercalledsliceisaddedinthestructureofthe
streamsuchthatthedecodercanresynchronizemorequickly
incaseoferror.
NTSC:Y=352×240Cb=Cr=176×120
PAL:Y=352×288Cb=Cr=176×144

15. 57EIE415MultimediaTechnology
Example
AdigitizedvideoistobecompressedusingtheMPEG-
1standard.Assumingaframesequenceof:
IBBPBBPBBPBBI...
andaveragecompressionratiosof10:1(I),20:1(P)
and50:1(B),derivetheaveragebitratethatis
generatedbytheencoderforboththeNTSCandPAL
digitizationformats.
58EIE415MultimediaTechnology
Solution
Framesequence=IBBPBBPBBPBBI...
Hence:I-frame:1/12
P-frame:3/12
B-frame:8/12
AverageCR=(1×0.1+3×0.05+8×0.02)/12
=0.0342or29.24:1
59EIE415MultimediaTechnology
Solution(Cont.)
NTSCframesize:
Withoutcompression
=352×240⋅8+2(176×120×8)
=1.013760Mbitsperframe
Withcompression=1.01376×1/29.24
=34.670kbitsperframe
Hencebitrategeneratedat30fps=1.040Mbps
60EIE415MultimediaTechnology
PALframesize:
Withoutcompression
=352×288×8+2(176×144×8)
=1.216512Mbitsperframe
Withcompression=1.216512⋅1/29.24
=41.604kbitsperframe
Hencebitrategeneratedat25fps=1.040Mbps
Normally,allowingforpacketizationandmultiplexing
overheads,abandwidthof1.2Mbpsisallocatedforthe
video.Hence,assumingamaximumbitrateof1.5Mbps,
thisleaves300kbpsforthecompressedaudiostream.
Solution(Cont.)

16. EIE415MultimediaTechnology
MPEG-2
62EIE415MultimediaTechnology
History:MPEG-2
Audio/Videoforbroadcast(DigitalTV,HDTV,
Cable,Satellite)
4-20Mbit/sforDigitalTV
CommitteeDraft11/93(BitstreamSyntaxand
Decoding)
Profiles@Levels,supportinginterlacing
ISO/IEC13818-1…-6(Systems,Video,Audio,
Compliance,Software,DSM-CC)
63EIE415MultimediaTechnology
MPEG-2standardisintentedtobegenericandto
serveawiderangeofapplications.Theconceptsof
profileandlevelhavebeenintroducedtomakethe
practicalimplementationsofthespecificationeasier.
Whatisaprofile?
Aprofileisa"definedsubsetofthesyntaxofthe
specification".
Imposessomeboundstothefullsyntax
Defineswhichtoolsorfunctionalitiesmaybeusedtoproducea
bitstreamandhow.
Whatisalevel?
Alevelisa"definedsetofconstraintsonthevalueswhich
maybetakenbytheparametersofthespecificationwithin
aparticularprofile".
64EIE415MultimediaTechnology
Levels
Alevelspecifiestherangeoftheparametersthatare
supportedbytheimplementation
imagesize
framerate
bitrate
FourlevelsaredefinedinMPEG-2:eachtargetedata
particularapplicationdomain.
LevelSamples/linesLines/frameFrames/sec
High(H)1920115260
High-1440(H-14)1440115260
Main(ML)72057630
Low(L)35228830

17. 65EIE415MultimediaTechnology
Thereare5profilesassociatedwitheachlevel:
Simple
Main
Spatialresolution
Quantizationaccuracy
High.
Thedifferentcombinationsoflevelsandprofilesforma
frameworkforallstandardsactivitiesassociatedwithMPEG-
2.
OneofthemostpopularsettingistheMP@MLstandard
whichisfordigitaltelevisionbroadcasting.
advancedtelevision(ATV)inNorthAmerica,
digitalvideobroadcast(DVB)inEurope,and
multiplesub-Nyquistsamplingencoding(MUSE)inJapan.
66EIE415MultimediaTechnology
CommonMPEG-2ProfileandLevelsinSimplified
Form
Profile
Level
Spatial
resolution
layer
Simple
Pictures:I,P
Chroma:
4:2:0
Main
Pictures:
I,P,B
Chroma:
4:2:0
SNR
Pictures:
I,P,B
Chroma:
4:2:0
Spatial
Pictures:
I,P,B
Chroma:
4:2:2
High
Pictures:
I,P,B
Chroma:
4:2:2
HighEnhancementSamples/line
Lines/frame
Frames/sec
1920
1152
60
1920
1152
60
LowerSamples/line
Lines/frame
Frames/sec
960
576
30
High-
1440
EnhancementSamples/line
Lines/frame
Frames/sec
1440
1152
60
1440
1152
60
1440
1152
60
LowerSamples/line
Lines/frame
Frames/sec
720
576
30
720
576
30
MainEnhancementSamples/line
Lines/frame
Frames/sec
720
576
30
720
576
30
720
576
30
720
576
30
LowerSamples/line
Lines/frame
Frames/sec
352
288
30
LowEnhancementSamples/line
Lines/frame
Frames/sec
352
288
30
352
288
30
LowerSamples/line
Lines/frame
Frames/sec

67EIE415MultimediaTechnology
MPEG-2ScalableCodingConcepts
Reasons:
toprovideinteroperabilitybetweendifferentservices
togiveflexiblesupporttoreceiverswithdifferent
displaycapabilities
toprovidealayeredvideobitstreamthatisamenable
forprioritizedtransmission.
tomadeitpossiblefortransmissioninthepresenceof
channelerrors,suchasinATM
AnewfeatureofMPEG-2isbitstreamscalability,which
allowsforalayeredrepresentationofthecodedbitstream
68EIE415MultimediaTechnology
Exampleofuses
ScalingbetweenHDTVandSDTV(Standard
DefinitionTelevision)
importantforHDTVtobecompatiblewithSDTV
transmissionusingHDTVquality
butusingscalablecoding:
lowresolution-baselayer
highresolution-baselayer+enhancementlayer

18. 69EIE415MultimediaTechnology
ScalableEncoderandDecoder
twolayersprovided:
BaseLayer:
lower-resolutionvideowithreducedbitrate(downsamplingspatially
ortemporally)
upscaledreconstructedbase-layervideo(upsampledspatiallyor
temporally)tobeusedasapredictionfortherecoveringofthe
originalinputvideosignal
EnhancementLayer
tocodethepredictionerrors
Reconstruction:
LowqualityReconstruction:
tomakesuseofBaseLayeronly
HighqualityReconstruction:
tomakesuseofbothBaseLayerandEnhancementLayer
70EIE415MultimediaTechnology
ScalableCodingSchemes
DataPartitioning
whentwochannelsareavailable,criticalbitstreamsforbetterchannel
andlesscriticalforpoorerchannel
SNR(Quality)Scalability
multiple-qualityvideoservices
SpatialScalability
multiple-dimensiondisplays
TemporalScalability
Multipleframerates
71EIE415MultimediaTechnology
Datapartitioning
Single-layer
encoder
Data
partitioner
Multiplexer
Video
Enh.Layer
Bitstream
BaseLayer
Bitstream
Output
Bitstream
Priority
breakpoint
baselayer:includeallcritical
information(suchasheader,motion
vectors,…
enhlayer:theremainingbitstream
whentwochannelsareavailable,criticalbitstreamsfor
betterchannelandlesscriticalforpoorerchannel
1
72EIE415MultimediaTechnology
SNRScalability
possiblewayfordegradationofthevideoqualityin
prioritizedtransmissionmedia
reducedqualityvideo:usingbaselayer,protected
fromtransmissionerror
itisbasedonafrequency(DCT-domain)scalability
technique
toobtainhigherqualityvideosignal:usingboth
lowerandhigherlayerbitstreams
2

19. 73EIE415MultimediaTechnology
Enhancement
Encoder
BaseLayer
Encoder
Enhancement
Decoder
BaseLayer
Decoder
Enh.Layer
Bitstream
BaseLayer
Bitstream
Highquality
video
Lowquality
video
encodesandtransmitsthedifferencebetweenthenonquantized
DCTcoefficientsandthequantizedcoefficientsfromthebaselayer
withfinerquantizationstepsize
AtbaseLayer:DCTcoefficientsarecoarsely
quantizedandtransmitted(⇒reducebitrate)
Video
BaseLayer
Decoder
+
-
+
+
74EIE415MultimediaTechnology
SpatialScalability
Itsupportsdisplayswithdifferentspatialresolutionsat
thereceiver.
BaseLayer:
theinputvideoisspatiallydecimatedtolowerspatialresolution
signalandencoded(tosendmoreimportantdata,errorfreeif
possible).
lowerspatialresolutionvideo:tobeconstructedfromthebase
layer.
EnhancementLayer:
thelowerlayerdecoderspatiallyinterpolatesthesignaltofull
spatialresolutionforpredictionintheenhancementlayer.
forhighresolutionvideo:bitstreamsfrombothlayersare
decoded.
3
75EIE415MultimediaTechnology
Enhancement
Encoder
BaseLayer
Encoder
Enhancement
Decoder
Upsampling
Spatial
Downsampling
Spatial
Upsampling
Spatial
BaseLayer
Decoder
Enh.Layer
Bitstream
BaseLayer
Bitstream
Highresolution
video
Lowresolution
video
tocodethedifferencesignal
Lowerresolutionvideowithreducedbitrate
Video
theinputvideois
spatiallydecimated
tolowerspatial
resolutionsignal
thelowerlayerdecoderspatiallyinterpolatesthe
signaltofullspatialresolutionforpredictionin
theenhancementlayer
76EIE415MultimediaTechnology
SpatialScalability-Decoder
Asingle-layerdecoderdecodesonlythebaselayer
toproduceareduced-resolutionoutputsequence.
Atwo-layerdecodercanreconstructafull-
resolutionsequenceasfollows:
1.Decodethebaselayerandup-sampletotheoriginal
resolution.
2.Decodetheenhancementlayer.
3.Addthedecodedresidualfromtheenhancementlayerto
thedecodedbaselayertoformtheoutputframe.

20. 77EIE415MultimediaTechnology
TemporalScalability
Itsharessimilarconceptwiththespatialscalability
BaseLayer:
toprovidethebasictemporalrate(tosendmore
importantdata,errorfreeifpossible).
forlowqualitydisplay-onlythelowerlayerisdecoded.
EnhancementLayer:
tobecodedwithtemporalpredictionwithrespecttothe
lowerlayer.
forhighqualitydisplay-bitstreamsfrombothlayersare
decodedandrecombinedfordisplay.
4
78EIE415MultimediaTechnology
Enhancement
Encoder
BaseLayer
Encoder
Enhancement
Decoder
Upsampling
Temporal
Downsampling
Temporal
Upsampling
Temporal
BaseLayer
Decoder
Enh.Layer
Bitstream
BaseLayer
Bitstream
Fullframe-rate
video
Lowframe-rate
video
tocodethedifferencesignal
Lowerframe-ratevideowithreducedbitrate
Video
theinputvideois
temporallydecimated
tolowertemporal
resolutionsignal
thelowerlayerdecodertemporallyinterpolates
thesignaltofulltemporalresolutionfor
predictionintheenhancementlayer
temporal
79EIE415MultimediaTechnology
Advantages
Layeredcodingfacilitates
Unequalerrorprotection
Efficientuseofresources
Customersatisfaction
Multipleservices
possibletocombinedifferentscalabilitytoolsintoahybrid
codingscheme.
i.e.interoperabilitybetweenserviceswithdifferentspatialresolutions
andframeratescanbesupportedbymeansofcombiningthespatial
scalabilityandthetemporalscalabilitytoolsintoahybridlayered
codingscheme.
80EIE415MultimediaTechnology
ComparisonBetweenMPEG-1andMPEG-2
CodingParameterMPEG-1MPEG-2
Standard19921994
MainapplicationDigitalVideoDigitalTV(andHDTV)
onCD-ROM
Spatialresolution~288x360pixels576x720pixels
(1152x1440pixels)
Temporalresolution25-30frames/s50-60fields/s
(100-120fields/s)
Bitrate1.5Mb/s~4Mb/s
(~20Mb/s)
QualityComparabletoComparableto
VHSNTSC/PALforTV
Compressionratio~20-30~30-40
overPCM(~30-40)

21. 81EIE415MultimediaTechnology
History:MPEG-3
MPEG-3existedonceuponatime,butitsgoal,enabling
HDTV,couldbeaccomplishedusingthetoolsofMPEG-2,and
hencetheworkitemwasabandoned
EIE415MultimediaTechnology
MPEG-4
83EIE415MultimediaTechnology
MPEG-4:theDevelopments
OriginalTarget:AdvancedCodingatRatesLowerthan64
kbit/s
Moreinformationisindigitalform,…
Moreinformationison-line,…
Moreinformationisaudio/visual,…
Applications&servicesbecome‘multimedia’,.
Applications&servicesbecomeinteractive,…
Audiovisualinformationnowcoversallbitratesandall
networks
Inshort:multimediaishappening
84EIE415MultimediaTechnology
IssuesthatupcomingMPEG-4
standardsaddress
InJuly1994,thefocuswasshiftedfromcompression
newfunctionalities+compression
compressionnolongertheonlyreasonfordoingcoding
“lowbitrate’,althoughstillimportant,isnotMPEG-4’sonlyfocus!
HavingAVinformationavailablewhereveryouare
Reusingtheaudiovisualcontent,combiningelementsof
contentinnewways
Morefreedomtointeractwithwhatiswithinthecontent

22. 85EIE415MultimediaTechnology
MPEG-4
(CodingofAudio-VisualObjects)
GenericStandardforCodingofMultimediaContents
1993-1998FirstCalltoInternationalStandard
ISO/IECProject#14496
86EIE415MultimediaTechnology
MPEG-4:NewDirection(Goals)
Content-Based
Interactivity
Universal
Access
Improved
Coding
Efficiency
Videoe-mail
Multimediaauthoring
SyntheticnaturalAV
1.Interactivecomputerapplications
2.Wirelesscommunications
3.IntegrationofAVdataintoanumberofapplications
Wirelessvideo
Internetvideo
Videodatabase
Interactivehomeshopping
Videogames
87EIE415MultimediaTechnology
MPEG-4:CodingofAVObjects
Audiovisualsceneiscomposedof“Objects”.
“Compositor”putsobjectsinscene.
Objectscanbedifferentinnature.
naturalorsyntheticAV,text&graphics,animatedfaces,
arbitraryshapeorrectangular
Codingschemecandifferforindividualobjects.
Principleisindependentofbitrate.
fromlowbitratesto(virtually)losslessquality.
88EIE415MultimediaTechnology
MPEG-4AudiovisualScene
scene
2D
background
personfurniture
AV
presentation
voicevideoglobedesk

23. 89EIE415MultimediaTechnology
DecodedandManipulatedOriginalDecoded
MoreaboutthegoalsofMPEG-4
Interactivity
Contentbasedmanipulationandbitstreamediting
Improvedtemporalrandomaccess
90EIE415MultimediaTechnology
MoreaboutthegoalsofMPEG-4
Integrationofnaturaland
syntheticmaterial
Mixingsyntheticandnatural
objectstogetherinthesamescene
Virtualenvironments
91EIE415MultimediaTechnology
OverviewofMPEG-4System
multiplexer
demultiplexer
(ConventionalSystem)
encoder
decoder 92EIE415MultimediaTechnology
VLBCCoreandtheGenericMPEG-4Coder
Motion
(MV)
Texture
(DCT)
MPEG-4VLBVCoreCoder
(similartoH.263/MPEG-1)
Motion
(MV)
Texture
(DCT)
GenericMPEG-4Coder
Shape

24. 93EIE415MultimediaTechnology
BasicBlockDiagramofMPEG-4VideoCoder
+TextureCoder
ShapeCoder
Processor
andBuffer
ArbitraryShape
+
Motion
Estimator
Motion
Compensator
MVPredictor
andCoder
Previous/Next
Reconstructed
VOPsStore
SystemMultiplexer
VideoObject
partialdata
MOTIONCODER
-
94EIE415MultimediaTechnology
OverviewofMPEG-4System
Scenesegmentation
anddepthlayering
O2
O3
O1
Layered
encoding
contour
motion
texture
contour
motion
texture
contour
motion
texture
bitstream
layer1
bitstream
layer2
bitstream
layer3
multiplexer
demultiplexer
Separate
decoding
compositor
AV-objects
95EIE415MultimediaTechnology
ProcessingStepsinMPEG-4Compositor
MPEG-4terminal
NetworkLayer
Elementary
streams
Demux
Syntactic
decoded
streams
Syntactic
decoding
Decompression
Primitive
AVObject
Upstreamdata(userevents,classrequestetc.)
Composition
information
Scenedescription
(scriptorclasses)
BIFS(BinaryFormatforScenes)
•Setofnodestorepresenttheprimitivescene
objectstobecomposed,thescenegraph
constructs,thebehaviorandinteractivity
•Updatestream:modifythesceneintime
•AnimStream:animatethesceneintime
EIE415MultimediaTechnology
H.264

25. 97EIE415MultimediaTechnology
TheH.264VideoCodingStandard
Multi-mode,multi-referenceMC
Motionvectorcanpointoutofimageborder
1/4-,1/8-pixelmotionvectorprecision
B-framepredictionweighting
Integertransform
Multi-modeintra-prediction
In-loopde-blockingfilter
UVLC(UniformVariableLengthCoding)
NAL(NetworkAbstractionLayer)
SP-slices
98EIE415MultimediaTechnology
ANoteonTerminologyofH.264
Thefollowingtermsareusedinterchangeably:
H.26L
TheWorkoftheJVTor“JVTCODEC”
JM2.x,JM3.x,JM4.x
TheThingBeyondH.26L
The“AVC”orAdvancedVideoCODE
ProperTerminologygoingforward:
MPEG-4Part10(OfficialMPEGTerm)
ISO/IEC14496-10AVC
H.264(OfficialITUTerm)
99EIE415MultimediaTechnology
PositionofH.264
100EIE415MultimediaTechnology
BasicMarcoblockCodingStructure
Entropy
Coding
Scaling&Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf.coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.-
Input
Video
Signal
Splitinto
Macroblocks
of16x16pixels
each
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal

26. 101EIE415MultimediaTechnology
•MotionCompensation
Entropy
Coding
Scaling&Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf.coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.-
Input
Video
Signal
Splitinto
Macroblocks
of16x16pixels
each
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
Variousblocksizesandshapes
8x8
0
4x8
01
01
23
4x48x4
1
08x8
Types
0
16x16
01
8x16
MB
Types
8x8
01
23
16x8
1
0
102EIE415MultimediaTechnology
•MultipleReferenceFrames
Entropy
Coding
Scaling&Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf.coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.-
Input
Video
Signal
Splitinto
Macroblocks
of16x16pixels
each
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
MultipleReferenceFramesfor
MotionCompensation
103EIE415MultimediaTechnology
•B-framePredictionWeighting
Playbackorder:I0B1B2B3P4B5B6……...
Bitstreamorder:I0P4B1B3B2P8B5……...
I0B1B2B3P4B5B6
Time
104EIE415MultimediaTechnology
•IntegerTransform
Entropy
Coding
Scaling&Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf.coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.-
Input
Video
Signal
Splitinto
Macroblocks
16x16pixels
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
8x8DCTisreplacedby4x4
and2x2IntegerTransforms.
e.g.Y=HXH
T
Advantagesofinteger
transform:
Nomultiplcation(only
shift-add)
Noroundingerror
1111
2112
1111
1221

−−=
−−

−−
H

27. 105EIE415MultimediaTechnology
Example:Codinga16x16residualmacroblock
All4x4lumaandchromablocksareorderedasfollowsandtransformed
witha4x4integertransform.
DCcoefficientsofblocks0,1…15areextractedtoformblock-1,whichis
furthertransformedwithanother4x4integertransform
DCcoefficientsofblocks18-21(22-25)areextractedtoformblock16
(17),whichisfurthertransformedwitha2x2integertransform
Blocksareencodedandsentaccordingtothesequentialorder.
NoteallDCcoefficientsinblocks0-15,18-21and22-25areremovedas
theyareencodedinblock-1,16&17.
106EIE415MultimediaTechnology
•Intra-predictionModes
Entropy
Coding
Scaling&Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf.coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.-
Input
Video
Signal
Splitinto
Macroblocks
16x16pixels
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
Inolderstandards,blocksare
encodedindependentlyinintramode
withoutmakinguseoftheinterblock
correlation⇒inefficient
H.264supportsDirectionalspatial
predictioninintramode
(9typesfor4x4luma,1for4x4
chroma,4typesfor16x16luma)
•4x4lumacase:
e.g.,Mode1(horizontal):
a=b=c=d=I,e=f=g=h=J
i=j=k=l=K,m=n=o=p=L
QABCDEFGH
Iabcd
Jefgh
Kijkl
Lmnop
1
0
34
56
7
8
2-DC
107EIE415MultimediaTechnology
Ifablockisencodedinintramode,apredictionis
madebasedonpreviouslyencodedblocksandthe
predictionresultissubtractedfromthecurrentblock
priortoencoding.
Modesofprediction:
9optionalpredictionmodestopredicta4x4lumablock
4optionalmodestopredicta16x16lumablock
1modetoeachpredicta4x4chromablock.
108EIE415MultimediaTechnology

28. 109EIE415MultimediaTechnology110EIE415MultimediaTechnology
Summary(H.264)
Videocodingisbasedonhybridvideocodingandsimilarin
spirittootherstandardsbutwithimportantdifferences
Newkeyfeaturesare:
Enhancedmotioncompensation
Smallblocksfortransformcoding
Enhancedentropycoding
Substantialbit-ratesavings(upto50%)relativetoother
standardsforthesamequality
Enhancementonperceptivequalityseemsbetterthanthaton
PSNR
Thecomplexityoftheencodertriplesthatofthepriorones
Thecomplexityofthedecoderdoublesthatofthepriorones
111EIE415MultimediaTechnology
Comparison
112EIE415MultimediaTechnology

29. EIE415MultimediaTechnology
AVS
114EIE415MultimediaTechnology
AVS(AudioVideocodingStandard)
AVSactivityinChina
Makeasimpleandlow-costindustrystandard
UseasmoreaswecanthepartsfromMPEG,
avoid/arounddisagreementforIPRpolicy
Co-designonbothtechnologyandIP
One-stop-shoppinglicense
Integrationofblockingtechnologyfrompartners
115EIE415MultimediaTechnology
OverviewofAVS1.0Video
High-efficiencycoding
Focusedapplication–initiallyHDTV
Lowercomplexity,lowercost
Completesolution:
Video,audio,systems,DRM
Simple,comprehensivepatentlicensing
116EIE415MultimediaTechnology
OpenstandardImpacttoChina
OrganizationofstandardinChina
SAC(StandardizationadministrationofChina)
Informationstandardizationcommittees
–AVSWorkinggroup
–…

30. 117EIE415MultimediaTechnology
AVS-VideoTools
MajorTools
Transform–16bit-implemented8x8transform(different
kernelascomparedwithH.264)
Quantizationandscaling–scalingonlyinencoder
Intraprediction–5modes
Motioncompensation–16x16/16x8/8x16/8x8modes
Quarter-pelinterpolation–4-tapsinterpolationfilter
Deblocking
Entropycoding
118EIE415MultimediaTechnology
Whatyouhavelearnt?
Videocodingexploitsthetemporalredundancybetween
successiveframesinvideo.
Motion-compensatedprediction:theprocessofcompensatingfor
thedisplacementofmovingobjectsfromoneframetoanother.
Videocompressionstandards:H261,H263,MPEG1,2,4and
H.264.
Motion-compensatedprediction,DCT,differentialcoding,run-
lengthcodingandhuffmancoding
Scalability
EIE415MultimediaTechnology
END