The document discusses error resiliency and concealment techniques in H.264 video coding, aimed at minimizing the impact of channel errors on reconstructed image quality. It highlights various error resilience methods, such as Flexible Macroblock Ordering (FMO), Arbitrary Slice Ordering (ASO), and Data Partitioning (DP), which balance compression and error resistance in heterogeneous environments. Additionally, it explains how these techniques manage error propagation and enhance the robustness of video transmission through networks.

1. Error Resiliency & Concealment in H.264 Code from jm9.8/x264 http:// getonebyone .com/ H264.html

2. What is Error Resiliency? “ To enable the compressed bit-stream to better resist channel errors” OR “ Error resilience techniques enable the compressed bit-stream to resist channel errors so that the impact on the reconstructed image quality is minimal”

3. But… some part of the information is inevitably lost and the goal of error concealment, at the receiving side, is to estimate the losses and conceal them in the displayed video.

4. Computation Consumption Error resilience takes nearly 20% of the consumption

5. Because, generally, the error resiliency schemes introduce some redundancy in the data. On the other hand, compression schemes aim to remove various redundancies from the data In H.264 we can find several parameters that can be tuned so that a trade-off between compression rate and error resilience can be made targeting different type of problems found in heterogeneous environments. Tradeoff Error Resiliency Compression 1

6. H.264 Eco System Heterogeneous Environments Transport

7. Video Coding Layer (VCL), which efficiently represents the video content Network Abstraction Layer (NAL), which formats the VCL representation of the video and provides header information in a manner appropriate for conveyance by particular transport layers (such as Real Time Transport Protocol) or storage media. H.264/AVC has adopted a two-layer structure design VCL NAL Data Partitioning Coded MBs Coded Slice/Partition Control Data H.264/AVC two-layered Structure Design

8. Entropy Coding Data Partition NAL Packets (nal-units or slices) NAL Entropy Decoding Decode Prediction Info, Quantized Transform Coeffs…. VLC or AC VLC CodedData Coded MBs Coded Slice/Partition Packets (nal-units or slices) Encode Abstract Level Flow Diagram internet

9. Resilience & Concealment in Heterogeneous Environment

10. Error Propagation A transmission error such as a bit error or packet loss may cause a video decoder to lose synchronization with the sequence of decoded VLCs. This can cause the decoder to decode incorrectly some or all of the information after the occurrence of the error and this means that part or all of the decoded VOP will be distorted or completely lost (i.e. the effect of the error spreads spatially through the VOP, ‘spatial error propagation’). If subsequent VOPs are predicted from the damaged VOP, the distorted area may be used as a prediction reference, leading to temporal error propagation in subsequent VOPs. Spatial and Temporal Error Propagation Error Position Damaged area Forward Prediction

11. Error Resilience Employed w.r.t Profiles Baseline profile includes some enhanced error resilience tools Flexible Macroblock Ordering (FMO), Arbitrary Slice Ordering (ASO), and Redundant Slices (RS) Extended profile adds further error resilience support in the form of data partitioning (DP). Main profile does not include enhanced error resilience tools like FMO, ASO, RS, DP, SP or SI Slices.

12. Error Resiliency Schemes Flexible Macroblock Ordering (FMO) Arbitrary Slice Ordering (ASO) Data Partitioning (DP) Redundant Slices (RS) SP/SI frame for bitstream switching Reference Frame Selection Intra-block refreshing by R-D control. Random Macroblock Intra Refresh

13. Flexible Macroblock Ordering (FMO) Why FMO? Because it aims at avoiding error propagation from a corrupted packet to subsequent packets. Objective? The objective behind FMO is to scatter possible errors to the whole frame as equally as possible, to avoid error accumulation in a limited region. How it works? It randomizes the data prior to transmission, so that if a segment of data is lost (e.g. a packet or several packets), the errors are distributed more randomly over the video pictures, rather than causing corruption of a complete regions, making it more likely that relevant neighboring data is available for concealment of lost content.

14. FMO cont.. Randomizing Data To achieve this, each MB is statically assigned to a slice group using Macroblock Allocation Map ( MBAmap). How does a MBAmap look like? The MBA_Map is a data structure that maps the spatial address of a MB to a slice group, in a raster-scan order.

15. FMO cont.. MBA_Map in H.264 supports 7 arbitrary shapes: Interleaved Dispersed Foreground & Background Box-out Raster scan Wipe Explicit The parameters that control the mapping functions are included in a Picture Parameter Set (PPS) which is sent to the receiver.

16. FMO cont.. FMOInit FmoGenerateMapUnitToSliceGroupMap FmoGenerateMBAmap FmoGenerateType0MapUnitMap FmoGenerateType1MapUnitMap FmoGenerateType2MapUnitMap FmoGenerateType3MapUnitMap FmoGenerateType4MapUnitMap FmoGenerateType5MapUnitMap FmoGenerateType6MapUnitMap Main Encode_one_frame() Frame_picture() Field_picture () FMOInit

17. Arbitrary Slice Ordering Why ASO? Since each slice of a coded picture can be decoded independently of the other slices in the picture, the H.264 design enables sending and receiving the slices of the picture in any order relative to each other. OR Slices in a coded frame may follow any decoding order. sender receiver Slice # 0 Slice # 1 Slice # 2 0 20 40

18. Slice Size Criteria Encoder intelligently groups MBs into a slice whose size is less than (or equal to) the size of MTU (Maximum Transportation Unit). MTU represents the largest size of a packet that can be transported through networks without being split. Arbitrary Slice Ordering

19. Data partitioning enables unequal error protection (UEP) according to syntax elements’ importance, since some syntax elements in the bitstream are more important than others. The coded data that makes up a slice is placed in three separate Data Partitions (A, B and C), each containing a subset of the coded slice. Each Partition can be placed in a separate NAL unit and may therefore be transported separately. Information regarding the Data Partitioning is put into the slice header. Data Partition (DP)

20. Data Partition (DP) DP A: Contains header information (MB types, quantization params, and motion vectors). With the loss of data of DP A, data of the other two partitions becomes useless . DP B: Contains Intra CBPs and transform coefficients of I-blocks. The loss of this part will severely impair the recovery of successive frames due to error propagation, because intra frame and intra-MBs are used as references. DP C: Contains Inter CBPs and coefficients of P-blocks. Compared to the DPs A and B, the data contained in DP C is less important. However, it is the biggest partition of a coded slice because a large number of frames are coded as p-frames.

21. Concealment in Data Partitioning Error concealment mechanism implemented by the decoder *A = header info *B = Intra CBPs and I-block trans-coeffs *C = Inter CBPs and P-block trans-coeffs

22. Error Concealment API Encoder & Decoder Both Have the Same DataPartition Struct //! DataPartition typedef struct datapartition { Bitstream *bitstream; EncodingEnvironment ee_cabac; int (* writeSyntaxElement )(SyntaxElement *, struct datapartition *); /*!< virtual function;actual method depends on chosen data partition and entropy coding method */ } DataPartition ; //! Slice typedef struct { DataPartition *partArr; … .. } Slice

23. Data Partition (DP) Code_a_picturre Encode_one_Slice() Start_slice() Generates Slice (Partiton) Header Main Encode_one_frame() Frame_picture() Field_picture () SliceGroupId, Picture *, TotalCodedMBs Picture * Picture * Picture * Picture * #define MAXSLICEPERPICTURE 100 typedef struct { int no_slices; Slice *slices[MAXSLICEPERPICTURE]; …… } Picture;

24. Redundant Slices Why? In order to enhance robustness in the event of loss of slices due to packet drops in network during transmission Usage In normal operation, the decoder reconstructs the frame from ‘primary’ pictures and discards any redundant pictures. However, if a primary coded picture is damaged, the decoder may replace the damaged area with decoded data from a redundant picture if available. A redundant slice is another representation of one or more MBs in the same bitstream. Encoder put redundant slice information in PPS

25. SP- /SI- Synchronization/ Switching Frame SP-/SI mechanism is designed for the purpose of video bitstream switching It is regarded as an important error resiliency feature in network environments with feedback channel This scheme assumes that more than one video bitstreams are being sent through network. In the case of synchronization-lost or presence of error in bitstream, the decoder can signal the presence of error in the frame(s) via feedback channel and the SP frame can be generated for error recovery.

26. SP/SI-Switching H.264/AVC uses SP- frames that enable random switching and its info is put in ImageParameters struct Same value Encoder Decoder Feedback channel

27. Reference Frame Selection The reference frame selection mode can be applied both with and without feedback channel (channel which gives lost-packet report) The reference frame selection with feedback channel uses the reference frame based on the feedback acknowledgement

28. Reference Frame Selection There are four possible modes defined in reference frame mode Neither: no back-channel data is returned from the decoder to the encoder. ACK : the decoder returns only acknowledgment messages. In this mode the encoder uses only an acknowledged segment as a reference for inter-frame encoding. NACK : the decoder returns only non-acknowledgment messages. Whenever NACK is received for a frame, the new reference frame is selected from amongst the frames coded before the NACK frame. ACK+NACK : the decoder returns both acknowledgment and non-acknowledgment messages. In the ACK+NACK mode, the encoder switches between two kinds of modes according to the upstream messages from the decoder.

29. Rate Distortion Optimization The error propagation and drift due to predictive coding can be eliminated by periodically inserting intra-coded MBs in the bitstream. Adaptive Intra Refresh (AIR): H.264 uses intelligent intra-block refreshing by R-D (rate-distortion) control, such that an appropriate block coding option o* will be selected that minimizes the Lagrangian cost function , unlike the periodic or random intra refresh in earlier standards (e.g., MPEG-4, H.263)

30. Intra Block Refreshing by R-D Control Here, o = block coding mode (inter/intra mode and block size) D(o) = distortion introduced by encoding with mode o and is computed by the SAD (sum of absolute difference) in low complexity mode and by SSD (sum of squared difference) in high complexity mode R(o) = corresponding coding rate, whereas for inter-block mode it represents the block residual and corresponding motion vector(s). λ = Lagrange parameter for appropriate weighting of rate and distortion.

31. double RDCost_for_8x8blocks ( int* cnt_nonz, // --> number of nonzero coefficients int64* cbp_blk, // --> cbp blk double lambda, // <-- lagrange multiplier int block, // <-- 8x8 block number int mode, // <-- partitioning mode short pdir, // <-- prediction direction short ref, // <-- reference frame short bwd_ref // <-- abp type ) Error Concealment API Desc: Calculates Cost function to find the appropriate block coding option. Return: Get Rate-Distortion Cost Intra Block Refreshing by R-D Control

32. Random Intra MB Refresh In order to enhance robustness to data loss, the H.264/AVC design contains a new ability to allow an encoder to send redundant representations of regions of pictures. void RandomIntraInit ( int xsize, int ysize, int refresh ) Desc: Initializes Random Intra module. xsize, ysize = size of picture in MBs refresh = refresh rate in MBs per picture void RandomIntraNewPicture ( ) Desc: Selects new set of MBs for forced Intra based on the RefreshPattern initialized in RandomIntraInit()

33. Random Intra MB Refresh Main Init_img() RandomIntraInit() int xsize, int ysize, int refresh Code_a_picturre Main Encode_one_frame() Frame_picture() Field_picture () Picture * Picture * Picture * Picture * RandomIntraNewPicture () Called, only after initialization (or changes) of the picture size or the random intra refresh value, immediately after parsing config file Allocates forced INTRA MBs

34. Parameter Sets (SPS/PPS) Why? To achieve synchronization between encoder and decoder in terms of packets as well as syntax Purpose The parameter set mechanism enables an encoder to signal important, infrequently changing sequence and picture parameters separately from the coded slices themselves. The H.264 encoder and decoder keep identical parameter tables in their memory that store various possible combinations of these parameters, known as ParameterSets , including picture size, entropy coding method, motion vector resolution, and so on that the decoder needs to know.

35. Encoder Side Error Resilience Error Concealment ( ) Encode ALL Frames( ) (main) Encode_One_Frame ( ) Field/Frame Picture ( ) (Picture *)

36. Error Concealment Decoder Error Concealment Decode ALL Frames (main) Docode_One_Frame ( Flag, DecParams, DecOutputs ) Read_New_Slice ( DecParams, DecInputs, DecOutputs ) Flag is the state variable for error detector and concealor ( DecParams, DecInputs, DecOutputs )

37. The End http:// getonebyone .com/ H264.html