• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Chapter10
 

Chapter10

on

  • 1,934 views

 

Statistics

Views

Total Views
1,934
Views on SlideShare
1,934
Embed Views
0

Actions

Likes
1
Downloads
113
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Chapter10 Chapter10 Presentation Transcript

    • Chapter 10 Video Multimedia Systems
    • Key Points
      • The display of moving pictures depends on persistence of vision .
      • Uncompressed video requires 26MBytes per sec (NTSC) or 31MBytes per sec (PAL).
      • Digitization may be performed in the camera (e.g. DV ) or using a capture card attached to a computer.
      • NTSC , PAL and SECAM are analogue video standards. All three use interlaced fields .
    • Key Points
      • CCIR 601 is a standard for digital video. It uses Y'CBCR colour with 4:2:2 chrominance sub-sampling . The data rate is 166Mbits per sec.
      • Video compression can make use of spatial (intra-frame) and temporal (inter-frame) compression. Spatial compression is still-image compression applied to individual frames. Temporal compression is based on frame differences and key frames .
      • Motion JPEG applies JPEG compression to each frame. It is usually performed in hardware.
      • Cinepak , Intel Indeo and Sorenson are popular software codecs used in multimedia. They are based on vector quantization .
    • Key Points
      • MPEG video is an elaborate codec that combines DCT-based compression of key frames ( I-pictures ) with forward and backward prediction of intermediate frames ( P-pictures and B-pictures ) using motion compensation .
      • QuickTime is a component-based multimedia architecture providing cross-platform support for video, and incorporating many codecs. It has its own file format that is widely used for distributing video in multimedia.
      • Digital video editing is non-linear (like film editing).
    • Key Points
      • Most digital post-production tasks are applications of image manipulation operations to the individual frames of a clip.
      • For delivery using current technology, it may be necessary to sacrifice frame size, frame rate, colour depth, and image quality.
      • Streamed video is played as soon as it arrives without being stored on disk, so it allows for live transmission and `video on demand'.
    • Moving Pictures
      • All current moving pictures depend on the following phenomena
        • Persistence of vision
          • A lag in the eye's response results 'after-images'
        • Fusion frequency
          • If a sequence of still images is presented above this frequency, we will experience a continuous visual sensation
          • Depend on brightness of image relative to viewing environment
          • Below this frequency will perceived flickering effect
    • Generate Moving Pictures
      • Video
        • Use video camera to capture a sequence of frames
      • Animation
        • Generate each frame individually either by computer or by other means
    • Digital Video
      • A video sequence consists of a number of frames
      • Each frame is a single image produced by digitizing time-varying signal generated by video camera
    • Digital Video
      • Think about the size of the uncompressed digital video
        • NTSC video format
          • Bitmapped images for video frame
            • 640  480 pixels with 24-bit color = 0.9 MB/frame
          • 30 frames per second
            • 900 kb/frame  30 frames/sec = 26 MB/sec
          • 60 seconds per minute
            • 26 MB/sec  60 secs/minute = 1,600 MB/minute
        • Strains on current processing, storage and data transmission !
    • Create Digital Video
      • Get analog/digital video signal from
        • video camera
        • video tape recorder (VTR)
        • broadcast signal
      • Digitize analog video & compress it
    • Digitizing Analog Video
      • In computer
        • Video capture card
          • Convert analog to digital & compress
          • Can also decompress & convert digital to analog
        • Compress through
          • Video capture card (hardware codec)
          • Software (software codec)
    • Digitizing Analog Video
      • In camera
        • Digitize and compress using circuitry inside camera
        • Transfer digitized signal from camera to computer through
          • IEEE 1394 interface ( FireWire ): 400 Mb/sec
          • USB : 12Mb/sec(version 1.1) ~ 480 Mb/sec(version 2.0)
    • Digitize in Computer v.s. Camera
      • Digitize in camera
        • Advantage
          • Digital signals are resistant to corruption when transmitted down cables and stored on tape
        • Disadvantage
          • User has no control between picture quality and data rate (file size)
    • Analog Video Standard 15.652 kHZ / 50 HZ 15.652 kHZ / 50 HZ 15.734 kHZ / 60 HZ Horizontal/Vertical frequency 4 : 3 4 : 3 4 : 3 Aspect Ratio Worldwide Standard Scan Lines Frame/Field Rate (per sec) France, … 625 (576) 25 / 50 SECAM Western Europe, … 625 (576) 25 / 50 PAL US, Japan, Taiwan, … 525 (480) 29.97 / 59.94 NTSC
    • Display Video on TV Cross-section of CRT Delta-delta shadow-mask CRT (Scan From “Computer Graphics: Principles and Practice”)
    • Field and Interlace
      • Transmitting many entire pictures in a second requires a lot of bandwidth
      • Field
        • Divide each frame into two fields
          • One consisting of the odd-numbered lines of each frame, the other of the even lines
      • Interlace
        • Each frame is built up by interlacing the fields
          • PAL
            • 50 fields/sec => 25 frames/sec
          • NTSC
            • 59.94 fields/sec => 29.97 frames/sec
    • Display Video on Computer
      • Progressive scanning
        • Write all lines of each frame to frame buffer
        • Refresh whole screen from frame buffer at high rate
    • Display Video TV Computer Monitor … … frame i frame i+1 frame i+2 frame i+3 …
    • Field & Interlace Artifacts Odd lines of frame i Even lines of frame i+1 Combine previous two analog video for progressive display A video clip of flash light on the water surface
    • Field & Interlace Artifacts
    • Prevent Interlace Artifacts
      • Average two field to construct a single frame
      • Discard half fields and interpolate remained fields to construct a full frame
      • Convert each field into a single frame (reduce frame rate but much better !)
    • Types of Analog Video
      • Component video
        • Three components: Y (luminance), U and V (color)
        • Often use in production and post-production
      • Composite video
        • Combine three components into a signal
        • Color component (U and V) is allocated half bandwidth as the luminance (Y)
        • Often use in transmission
      • S-video
        • Separates the luminance from the two color (total two signals)
    • Digital Video Standards
      • CCIR 601 ( Rec. ITU-R BT.601 )
        • specifies the image format, and coding for digital television signals
      Parameter Value YUV encoding 4:2:2 Sampling frequency for Y (MHz) 13.5 Sampling frequency for U and V (MHz) 6.75 No of samples per line 720 No of levels for Y component 220 No of levels for U,V components 225
    • Perplexing NTSC System 1 2 3 … 480 1 2 3 … 640 Analog to digital Pixels are square 1 2 3 … 480 1 2 3 … 720 CCIR 601 standard Pixels are not square
    • CCIR 601 Sampling Y samples C B and C R samples 4 : 2 : 2 sampling (co-site) 4 : 2 : 0 sampling (not co-site) 4 : 1 : 1 sampling (co-site)
    • Compression & Data Stream Standards
      • Sampling produces a digital representation of a video signal
      • This must be compressed and then formed into a data stream for transmission
      • Further standards are needed to specify the compression algorithm and the format of the data stream
    • Compression & Data Stream Standards
      • DV standard
        • For semi-professional & news-gathering
      • MPEG-2 standard
        • For family use
        • Organized into different profiles and levels
          • The most combination is Main Profile at Main Level ( [email_address] )
          • Used for digital television broadcasts & DVD video
    • Introduction to Video Compression
      • Adapted to consumers’ hardware, video data needs to be compressed twice
        • First during capture
        • Then again when it is prepared for distribution
    • Video Compression
      • Digital video compression algorithms operate on a sequence of bit-mapped images
        • Spatial compression ( intra-frame )
          • Compress each individual image in isolation
        • Temporal compression ( inter-frame )
          • Store the differences between sub-sequences of frames
    • Spatial Compression
      • Compress method is similar to image compression
        • Lossless
          • No information loss
          • Compression ratios is lower
        • Lossy
          • Some information loss
          • Compression ratios is higher
      • Why recompressing video is unavoidable
        • The compressor used for capture are not suitable for multimedia delivery
        • For post-production
    • Temporal Compression
      • Key frames
        • Certain frames in a sequence are designated as key frames
      • Difference frame
        • Each of the frames between the key frames is replaced by a difference frame
        • Records only the differences between the frames
    • Time Required for Compression & Decompression
      • Symmetrical
        • Compression & decompression of a piece of video take the same time
      • Asymmetrical
        • Compression & decompression of a piece of video not take the same time
        • Generally Compression takes longer time
    • Motion JPEG (MJPEG)
      • A popular approach to compressing video during capture
      • Applying JPEG compression to each frame (No temporal compression)
      • Therefore it is called “Motion JPEG”
    • DV
      • Compression based on DCT transform
      • Perform temporal compression (motion compensation) between two fields of each frame
      • Quality is varied dynamically to maintain constant data rate
    • MJPEG v.s DV 5:1 (4:1:1 sample) DCT based compression Motion compensation 25 Mb/sec DV 7:1 (mid-range capture card) JPEG compression 24 Mb/sec MJPEG Compression ratio Compression method Date Rates
    • Software Codecs for Multimedia
      • Popular software codecs
        • MPEG-1
        • Cinepak
        • Intel Indeo
        • Sorenson
    • Vector Quantization Source Output Group into vectors encoder decoder Unblock Reconstruction Find closest code-vector code book index code book index Table lookup
    • MPEG
      • Stand for M otion P icture E xperts G roup ( Joint of the ISO and the IEC)
      • Works on standards for the coding of moving pictures and associated audio
    • MPEG Family
      • MPEG – 1
        • Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mb/s
      • MPEG – 2
        • Generic coding of moving pictures and associated audio
        • For broadcasting & studio work
      • MPEG – 3
        • no longer exists (has been merged into MPEG-2)
      • MPEG – 4
        • Very low bit rate audio-visual (integrated multimedia) coding
    • MPEG Family
      • MPEG – 7
        • Multimedia content description interface
      • MPEG – 21
        • Vision statement
          • To enable transparent & augmented use of multimedia resources across a wide range of networks and devices
        • Objectives
          • To understand how the elements fit together
          • To identify new standards which are required if gaps in the infrastructure exist
          • To accomplish the integration of different standards
    • MPEG–1 Standard
      • Defines a data stream syntax and a decompressor, allowing manufacturers to develop different compressors
      • MPEG-1 compression
        • Temporal compression based on motion compensation
        • Spatial compression based on quantization & coding of frequency coefficients produced by a DCT of the data
    • MPEG –1 Objective
      • Medium quality video (VHS-like)
      • Bit rate < 1.5 Mb/s
        • 1.15 Mb/s for video
        • 350 kb/s for audio & additional data
      • Asymmetrical application
        • Store video & audio on CD-ROM
      • Picture format : SIF (Source Input Format)
        • 4:2:0 sub-sampled
        • Frame size @ frequency rate
          • 352  288 @ 25 HZ
          • 352  240 @ 30 HZ
    • An object moving between frames Area of potential change
    • Motion Compensation
      • Divide each frame into macroblocks of 16  16 pixels
      • Predict where the corresponding macroblock in next frame
        • Try all possible displacements within a limited range
        • Choose the best match
      • Construct difference frame by subtracting each macroblock from its predicted counterpart
        • Keep the motion vectors describing the predicted displacement of macroblocks between frames
    • Picture Type
      • I (intra) pictures
        • Code without reference to other pictures
        • Low compression rate
      • P (predicted) pictures
        • Code using motion compensated prediction from a past I or P picture
        • Higher compression rate than I picture
      • B (bidirectional-predicted) pictures
        • Code bidirectional interpolation between the I or P picture which preceded & followed them
        • Highest compression rate
      All are compressed using the MPEG version of JPEG compression
    • I 01 I B P I P B B B B B B B 02 03 04 05 06 11 12 13 14 15 16 21 Group of Pictures (GOP) An MPEG sequence in display order 01 I B P I P I B B B B B B B 04 02 03 11 05 06 14 12 13 21 15 16 An MPEG sequence in bitstream order (decode order)
    • MPEG-1 視頻壓縮技術
      • 運動補償 (Motion Compensation)
      • 頻率變換 (Frequency Transform)
      • 可變長度編碼 (Variable Length Coding)
      • 彩色信號 subsampling
      • 量化 (Quantization)
      • 預測編碼
      • 圖像插值
    • QuickTime
      • Apple, 1991
      • Time base, non-linear editing
      • Component-based architecture
        • Compressor components
          • Cinepak, Intel Indeo codec
        • Sequence grabber components
        • Movie control component
        • Transcoder
          • Translate data between different formats
        • Video digitizer component
      • Support MPEG-1, DV, OMF, AVI, OpenDML
    • Digital Video Editing & Post-production
      • Editing
      • Compositing
      • Reverse shot
        • Conversation between two people
    • Film & Video Editing
      • Traditional
      • In point and out point
      • Timecode
        • SMPTE timecode
        • Hours, minutes, seconds, frames
      • VHS
        • Two copying operations is to produce serious loss of quality
        • Constructed linearly
    • Digital Video Editing
      • Random access
      • Non-destructive
      • Premiere
        • Three main windows
          • Project, timeline, monitor
          • Figs. 10.12-14
        • Timelines
          • Have several video tracks
        • Transitions, Fig. 10.15
        • Cuts and Transitions
          • In a cut, two clips are butted
          • In transitions, two clips overlap
            • Image processing is required to construct transitional frames
    •  
    • Digital Video Post-production
      • Over- or under-exposed, out of focus, color cast, digital artifacts
        • Provide image manipulation programs
          • Adjust level, sharpen, blur
        • The same correction may be needed for every frame, so the levels can be set for the first frame and the adjustment will be applied to as many frames as user specifies.
        • If light fades during a sequence, it will necessary to increase the brightness gradually to compensate.
          • Apply a suitable correction to each frame and allow their values at intermediate frames to be interpolated
          • Varying parameter values over time
    • Keying
      • Selecting transparent areas
      • Blue screening
        • Chroma keying: any color
        • Alpha channel
        • Luma keying: a brightness threshold is used to determine which areas are transparent
      • Select explicitly
        • Create mask
        • In film and video, mask is called matte
        • Matte out: removing unwanted elements
        • Split-screen effects
      • Alpha channel created in other application
    • Track matte
      • Chroma keying and luma keying
        • Color and brightness changes between frames
        • Use a sequence of masks as matte
          • Separate video track: track matte
        • Track matte
          • Painstaking by hand
          • Generated from a single still image applying simple geometrical transformations over time to create a varying sequence of mattes
    • Adobe After Effects
      • Apply a filter to a clip and vary it over time
      • A wide range of controls for the filter’s parameters
      • Premiere: parameter values are interpolated linearly between key frames
      • After effect: interpolation can use Bezier curves
    • Preparing Video for Multimedia Delivery
      • Frame size, frame rate, color depth, image quality
      • People sit close to monitors, so a large picture is not necessary
      • Higher frame rates are needed to eliminate flicker only if display is refreshed at the same rate.
        • Computer monitors are refreshed at a much higher rate from VRAM.
      • Limiting colors
        • Not all codecs support
    • Streamed Video & Video Conference
      • Streamed video
        • Delivering video data stream from a remote server, to be displayed as it arrives
        • As against downloading an entire files to disk & playing it from there
        • Opens up the possibility of delivering live video on computers
    • Streamed Video & Video Conference
      • Video conference
        • Streamed video doesn't restricted to a single transmitter broadcasting to many consumers: Any suitably equipped computer can act both as receiver & transmitter
        • Users on several machines can communicate visually, taking part in what is usually called a video conference
    • Single transmitter Multiple receiver All computer are receiver & transmitter
    • Obstacle to Streamed Video
      • Bandwidth
        • SIF MPEG-1 video require a bandwidth of 1.86 Mb/sec
        • Decent quality streamed video is restricted to LAN, T1 lines, ADSL & cable modems for now
      • Delivering time over network
        • Deliver data with the minimum of delay
          • Delay may cause independently delivered video & audio stream to lose synchronization
    • Conventional Delivery of WWW Video
      • Embedded video
        • Transfer movie files from a server to the user’s machine
        • Playback from disk once the entire file has arrived
      • Progressive download ( HTTP streaming )
        • Transfer movie files to user’s disk
        • Start playing as soon as enough of it has arrived
        • The file usually remains on the user’s disk after playback is completed
      start playing Cannot be used for live video ! 100 0 download play time
    • True Streaming
      • Each stream frame is played as soon as it arrives over the network
      • Video files is never stored on the user’s disk
        • Length of streamed movie is limited only by the storage size at the server, not by the user’s machine
      • Suit for live video & video on demand (VOD)
        • The network must be able to deliver the data stream fast enough for playback
        • Movie’s data rate & quality is restricted to what the network can deliver
    • State of the Art
      • Leading technique over the Internet
        • RealVideo (Real Networks)
        • Streaming QuickTime (Apple)
        • Media Player (Microsoft)
      • Architecture
        • RTSP (Real Time Streaming Protocol)
          • Control the playback of video streams
        • Providing several versions of a movie compressed
          • Server chooses the appropriate one to fit the speed of the user’s connection
    • Codecs for Video Conferencing
      • H.261
        • Designed for two-way telecommunication applications over ISDN
        • A precursor of MPEG-1
          • DCT-based compression with motion compensation
          • It does not use B-pictures
      • H.263
        • Very low bit rate video
      • H.263+
        • An extension of H.263
    • H.261
      • Real time constrain
        • Video conference cannot tolerate longer delays without becoming disjointed
        • Maximum delay: 150 ms (about 7 frames/sec)
      • Bit rate: p  64 kbps ( p = 1 ~ 30)
      • Picture format
        • CIF (Common Intermediate Format)
          • Component(size): Y(352  288), C b & C r (176  144)
          • Picture rate: 29.97 frames/sec
        • QCIF (Quarter CIF)
          • Component(size): Y(176  144), C b & C r (88  72)
    • H.263
      • Very low bit rate video (< 64 kbps)
      • Primary target rate is about 27 kbps (V.34 modem)
      • Compression techniques
        • Chroma sub-sampling 4:2:0
        • DCT compression with quantization
        • Run-length and variable-length encoding of coefficients
        • Motion compensation with forward & backward prediction
        • Compress a QCIF picture as low as 3.5 frames/sec