Dalí   A Multimedia Software Library Wei Tsang Ooi Brian Smith, Hsi Haye Chan, Matthew Chiu, Sugata Mukhopadhyay, Dan Rubi...
Motivations
Current Software Solution Black Box C Source
MPEG to RealVideo Black Box Real Video  Encoder RGB MPEG Real OpenFile(f) GetNextFrame()
MPEG to RealVideo RGB Decoder YUV to RGB Encoder RGB to YUV Real MPEG YUV compressed data OpenFile(f) GetNextFrame() YUV
MPEG to RealVideo YUV Decoder YUV to RGB Encoder RGB to YUV Real MPEG compressed data
Decode I-Frame only Black Box RGB MPEG ? OpenFile(f) GetNextFrame()
Decode I-Frame only RGB MPEG ? Decoder YUV to RGB compressed data YUV OpenFile(f) GetNextFrame()
Decode I-Frame only RGB MPEG ? Decoder YUV to RGB I Frame filter compressed data I-frames
Others Motivations.. <ul><li>Read from network ? </li></ul><ul><li>Tune I/O ? </li></ul><ul><li>Run on handheld ? </li></u...
Current Software Solution Black Box C Source
Dalí is in between.. Black Box C Source Dalí
Example
Example : <ul><li>Process a subset of the gray scale version of each I-frame </li></ul>
MPEG structures GOP hdr sequence hdr picture picture pic hdr pic data
What we want to do <ul><li>parse sequence header </li></ul>
What we want to do <ul><li>parse sequence header </li></ul><ul><li>while  not eof  do - find & parse picture header </li><...
What we want to do <ul><li>parse sequence header </li></ul><ul><li>while  not eof  do - find & parse picture header -  if ...
What we want to do <ul><li>parse sequence header </li></ul><ul><li>while  not eof  do - find & parse picture header -  if ...
Abstractions
Abstractions <ul><li>For input/output </li></ul><ul><li>For representing elements in MPEG </li></ul><ul><li>For storing im...
BitStream & BitParser BitStream File BitParser BitParser
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>MPEG Headers <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump <...
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>Find <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul...
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>Skip <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul...
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>Dump <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul...
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>Parse <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></u...
<ul><li>Parse </li></ul><ul><li>Encode </li></ul>Encode <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></...
ScImage ScBlock :- {   short dc;   char index[63];   int ac[63]; } represents 8x8 pixels
ByteImage unsigned char 0..255 represents a pixel
The Code
Example MpegSeqHdr *seqHdr = MpegSeqHdrNew (); MpegPicHdr *picHdr = MpegPicHdrNew (); BitStream *bs = BitStreamNew (65536)...
Example BitParser *bp = BitParserNew (); BitParserAttach (bp, bs); BitStream BitParser
Example BitStreamReadFromFile (bs, f, 0); BitStream BitParser File MpegSeqHdr MpegGopHdr MpegPicHdr
Example MpegSeqHdrFind (bp); BitParser MpegSeqHdr MpegGopHdr MpegPicHdr
Example MpegSeqHdrParse (bp, seqHdr); w = (seqHdr->width + 7)/8; h = (seqHdr->height + 7)/8; BitParser MpegSeqHdr MpegGopH...
Example ScImage *y = ScNew (w, h); ScImage *u = ScNew (w/2, h/2); ScImage *v = ScNew (w/2, h/2); y u v
Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); status = MpegPicHdrFind ...
Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FR...
Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FR...
Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FR...
Refill the BitStream BitParser
Refill the BitStream BitParser
Refill the BitStream BitParser File
Memory Mapped BitStream bs = BitStreamMMapReadNew(“tennis.mpg”); File
Operate on a subset
Physical ByteImage width: 8 height: 6 x: 0 y: 0 parentWidth: 8 isVirtual: 0 firstByte *
Virtual ByteImage width: 5 height: 4 x: 1 y: 1 parentWidth: 8 isVirtual: 1 firstByte *
Example ScImage *virt = ScClip (y,6,2,20,20); 6 20 blocks
Example ScImage *virt = ScClip (y,6,2,20,20); ByteImage *byte = ByteNew (160, 160); ScToByte (virt, byte); 6 20 blocks
Implementation
Implementation <ul><li>C library </li></ul><ul><li>Tcl bindings available </li></ul>
Organization <ul><li>Code are organized into packages </li></ul><ul><li>Basic  </li></ul><ul><ul><li>ByteImage, ScImage, e...
Other Packages <ul><li>AVI </li></ul><ul><li>WAVE </li></ul><ul><li>JPEG </li></ul><ul><li>GIF </li></ul>
Availability <ul><li>http://www.cs.cornell.edu/dali </li></ul><ul><li>open source </li></ul><ul><li>binaries for linux/win...
The End
Upcoming SlideShare
Loading in …5
×

Dali: A Multimedia Software Library

635 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
635
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • add animation
  • add animation
  • add animation
  • add animation
  • add animation
  • add animation
  • 8 x 8
  • 8 x 8
  • Dali: A Multimedia Software Library

    1. 1. Dalí A Multimedia Software Library Wei Tsang Ooi Brian Smith, Hsi Haye Chan, Matthew Chiu, Sugata Mukhopadhyay, Dan Rubinovitz, Jiesang Song, Steve Weiss.
    2. 2. Motivations
    3. 3. Current Software Solution Black Box C Source
    4. 4. MPEG to RealVideo Black Box Real Video Encoder RGB MPEG Real OpenFile(f) GetNextFrame()
    5. 5. MPEG to RealVideo RGB Decoder YUV to RGB Encoder RGB to YUV Real MPEG YUV compressed data OpenFile(f) GetNextFrame() YUV
    6. 6. MPEG to RealVideo YUV Decoder YUV to RGB Encoder RGB to YUV Real MPEG compressed data
    7. 7. Decode I-Frame only Black Box RGB MPEG ? OpenFile(f) GetNextFrame()
    8. 8. Decode I-Frame only RGB MPEG ? Decoder YUV to RGB compressed data YUV OpenFile(f) GetNextFrame()
    9. 9. Decode I-Frame only RGB MPEG ? Decoder YUV to RGB I Frame filter compressed data I-frames
    10. 10. Others Motivations.. <ul><li>Read from network ? </li></ul><ul><li>Tune I/O ? </li></ul><ul><li>Run on handheld ? </li></ul>Black Box
    11. 11. Current Software Solution Black Box C Source
    12. 12. Dalí is in between.. Black Box C Source Dalí
    13. 13. Example
    14. 14. Example : <ul><li>Process a subset of the gray scale version of each I-frame </li></ul>
    15. 15. MPEG structures GOP hdr sequence hdr picture picture pic hdr pic data
    16. 16. What we want to do <ul><li>parse sequence header </li></ul>
    17. 17. What we want to do <ul><li>parse sequence header </li></ul><ul><li>while not eof do - find & parse picture header </li></ul>
    18. 18. What we want to do <ul><li>parse sequence header </li></ul><ul><li>while not eof do - find & parse picture header - if it is an I-frame then - parse the picture into DCT domain - IDCT into spatial domain </li></ul>
    19. 19. What we want to do <ul><li>parse sequence header </li></ul><ul><li>while not eof do - find & parse picture header - if it is an I-frame then - parse the picture into DCT domain - IDCT into spatial domain - take the gray scale version of the frame and process it </li></ul>
    20. 20. Abstractions
    21. 21. Abstractions <ul><li>For input/output </li></ul><ul><li>For representing elements in MPEG </li></ul><ul><li>For storing images in DCT domain </li></ul><ul><li>For storing images in spatial domain </li></ul>
    22. 22. BitStream & BitParser BitStream File BitParser BitParser
    23. 23. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>MPEG Headers <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegGopHdr MpegSeqHdr MpegPicHdr Picture Data
    24. 24. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>Find <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegGopHdr MpegSeqHdr MpegPicHdr Picture Data
    25. 25. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>Skip <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegGopHdr MpegSeqHdr MpegPicHdr Picture Data
    26. 26. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>Dump <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegGopHdr MpegSeqHdr MpegPicHdr Picture Data MpegGopHdr
    27. 27. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>Parse <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegGopHdr MpegSeqHdr MpegPicHdr Picture Data width: 320 height: 240 pic rate: 30 fps etc
    28. 28. <ul><li>Parse </li></ul><ul><li>Encode </li></ul>Encode <ul><li>Find </li></ul><ul><li>Skip </li></ul><ul><li>Dump </li></ul>MpegSeqHdr width: 320 height: 240 pic rate: 30 fps etc
    29. 29. ScImage ScBlock :- { short dc; char index[63]; int ac[63]; } represents 8x8 pixels
    30. 30. ByteImage unsigned char 0..255 represents a pixel
    31. 31. The Code
    32. 32. Example MpegSeqHdr *seqHdr = MpegSeqHdrNew (); MpegPicHdr *picHdr = MpegPicHdrNew (); BitStream *bs = BitStreamNew (65536); BitStream
    33. 33. Example BitParser *bp = BitParserNew (); BitParserAttach (bp, bs); BitStream BitParser
    34. 34. Example BitStreamReadFromFile (bs, f, 0); BitStream BitParser File MpegSeqHdr MpegGopHdr MpegPicHdr
    35. 35. Example MpegSeqHdrFind (bp); BitParser MpegSeqHdr MpegGopHdr MpegPicHdr
    36. 36. Example MpegSeqHdrParse (bp, seqHdr); w = (seqHdr->width + 7)/8; h = (seqHdr->height + 7)/8; BitParser MpegSeqHdr MpegGopHdr MpegPicHdr
    37. 37. Example ScImage *y = ScNew (w, h); ScImage *u = ScNew (w/2, h/2); ScImage *v = ScNew (w/2, h/2); y u v
    38. 38. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); status = MpegPicHdrFind (bp); }
    39. 39. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { } status = MpegPicHdrFind (bp); }
    40. 40. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } status = MpegPicHdrFind (bp); }
    41. 41. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } BitStreamFill(bp, f); status = MpegPicHdrFind (bp); }
    42. 42. Refill the BitStream BitParser
    43. 43. Refill the BitStream BitParser
    44. 44. Refill the BitStream BitParser File
    45. 45. Memory Mapped BitStream bs = BitStreamMMapReadNew(“tennis.mpg”); File
    46. 46. Operate on a subset
    47. 47. Physical ByteImage width: 8 height: 6 x: 0 y: 0 parentWidth: 8 isVirtual: 0 firstByte *
    48. 48. Virtual ByteImage width: 5 height: 4 x: 1 y: 1 parentWidth: 8 isVirtual: 1 firstByte *
    49. 49. Example ScImage *virt = ScClip (y,6,2,20,20); 6 20 blocks
    50. 50. Example ScImage *virt = ScClip (y,6,2,20,20); ByteImage *byte = ByteNew (160, 160); ScToByte (virt, byte); 6 20 blocks
    51. 51. Implementation
    52. 52. Implementation <ul><li>C library </li></ul><ul><li>Tcl bindings available </li></ul>
    53. 53. Organization <ul><li>Code are organized into packages </li></ul><ul><li>Basic </li></ul><ul><ul><li>ByteImage, ScImage, etc. </li></ul></ul><ul><li>MPEG </li></ul><ul><ul><li>MPEG-1 Video/Audio/System </li></ul></ul><ul><li>Vision </li></ul><ul><ul><li>Useful vision routines </li></ul></ul>
    54. 54. Other Packages <ul><li>AVI </li></ul><ul><li>WAVE </li></ul><ul><li>JPEG </li></ul><ul><li>GIF </li></ul>
    55. 55. Availability <ul><li>http://www.cs.cornell.edu/dali </li></ul><ul><li>open source </li></ul><ul><li>binaries for linux/win32/sunos </li></ul><ul><li>examples </li></ul><ul><li>documentations </li></ul>
    56. 56. The End

    ×