Your SlideShare is downloading. ×
Copyright © 2001 Intel Corporation. On Multimedia Technology ...
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Copyright © 2001 Intel Corporation. On Multimedia Technology ...

477
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
477
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • 1.6 Mbit MPEG 1 video streamed over 2 Mbit (raw) 802.11 network using the TCP and our Hybrid Automatic Repeat Request method (with Reed-Solomon coding)
  • Transcript

    • 1. On Multimedia Technology & Infrastructure for Emerging Applications: from Desktop to Wireless to Peer-to-Peer Minerva M. Yeung, Ph.D. <minerva.yeung@intel.com> Media Technology Research Microprocessor Research Labs Intel Corporation. Feb 20, 2001. Stanford Seminar Copyright © 2001 Intel Corporation.
    • 2. Intel Media Technology Acknowledgements* MRL/TRL  Yen-Kuang Chen, Ph.D.  Matthew Holliman  Rainer Lienhart, Ph.D.  Skip Macy, Ph.D.  Igor Kozintsev, Ph.D  Andre Zaccarin, Ph.D.  Valery Kuriakin and the iNNL-MPL team (Russia)  And many others * Key collaborators/contributors to the work mentioned in the talk. Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 2
    • 3. Intel Ubiquitous Multimedia Media Technology MRL/TRL Wired and Wireless Interactive, High Performance, Enriched Media Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 3
    • 4. Intel Digital Media: from desktop, to Internet, to Media Technology MRL/TRL hand-helds, to wireless, and to Peer-to-Peer Data Farms / Storage Edge server E-Commerce Server Media Server Web Server Apps/DB Edge server Server Wireless Comm Server Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 4
    • 5. Intel  Implications: Media Technology MRL/TRL – Blurring of Core vs. Edge – Migration from client-server to peer-to-peer framework – Where should data come from? – Where should computations be done? – Optimal partitioning of computations Inequality: compute, bandwidth, storage and display Platform CPU Memory Storage B/W Screen Server Multiple 2GB 100G to 1+ Gbps N/A 1GHz+ Terabytes PC Single 256M 40G 100 Mbps 1600 x 1200 1GHz+ Laptop 600 MHz 128M 10G 100 Mbps 1024 x 768 Media 200MHz 16M 8M 19.2 320 x 240 PDA kbps PDA 70MHz 8M 8M 19.2 kbps 160 x 160 Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 5
    • 6. Intel Media Technology Challenges MRL/TRL  What is the optimal multimedia content distribution mechanism?  What is the optimal multimedia processing methodology?  What are the technology building blocks? Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 6
    • 7. Intel Media Technology Multimedia data MRL/TRL  Same content can exist in different resolutions, bit rate and quality  Can be gracefully degraded  Can be transformed to multiple representations Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 7
    • 8. Intel Media Technology P2P for Multimedia MRL/TRL  Maintain key features and extend – Supports traditional (existing) file sharing systems – Facilitates sharing of resources like CPU cycles, storage, computation process or arbitrary functionalities that can be abstracted by the user as resources.  Facilitates exchange of data with real time requirements, exemplified by video or audio transmission. – Choice and flexibility of transport mechanisms – Meeting diverse application requirements  Efficient multimedia services – Integrates media transformation/transcoding  Flexible query interfaces to facilitate multimedia search Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 8
    • 9. Intel MAPS: Media Accelerating Peer Services Media Technology MRL/TRL  Technology, platform architecture, and software infrastructure – facilitates efficient, transparent and high- performance media content delivery & processing – can take advantage of, integrate into, and bring out the benefits of the P2P computing infrastructure MAPS MAPS MAPS Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 9
    • 10. Intel System Architecture Media Technology MRL/TRL Application MAPS API Script language (Multimedia) Support modules of other Media Multi-media application Accelerating types of applications (Document editing, distributed Support modules Peer processing, etc.) (Transcoding, compression, security/watermarking, etc.) Service Layer Platform Traditional Optimization application MAPS API Peer-to-Peer Service Layer (general security, Directory service, data replication, flow control, etc.) Network Transport Services (TCP socket, CORBA, DCOM, SOAP) operating system Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 10
    • 11. Intel System Components Media Technology MRL/TRL Platform Optimization (Media Acceleration) – Detect and analyze the optimal platform configuration and operation at a particular time instant – Dynamic optimization via cost-function analysis across a global or local set of peer nodes: the result determines a best operational mode for a particular media delivery or processing operation – Optimization is built upon (and calling) a set of support modules, like streaming, transcoding, etc. for multimedia delivery operations, and other support modules like distributed image processing primitives for multimedia processing operations. – Invisible and seamless to the applications – Overall results cascaded into an enhanced service experience – fast delivery, higher satisfaction, lower abort rates, and better viewing/listening/interactive experiences, … Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 11
    • 12. Intel Minimal Cost to Deliver a Media Technology MRL/TRL Resource to a Peer Node  To find the minimal of – cost of getting a copy locally and transcoding if necessary, – cost of getting a copy from a peer node + cost of transmission Minimal cost Minimal cost Minimal cost i j k Minimal cost Minimal cost m n Minimal cost Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 12
    • 13. Intel Media Technology Cost Analysis MRL/TRL  Request to evaluate cost of R Cost of  If the file is not cached retrieval, (local), then find the transcoding, & minimal cost from the set transmission of neighbors (can extend from local to global) Compare the  Cost of transcoding  minimal cost computation  Cost of transmission  Repeat for the set of Cache the available network neighbors result bandwidth  Other cost factors can be formulated Return Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 13
    • 14. Intel Media Technology Minimal Cost to Perform MRL/TRL a computation  For a computation task on data (resource R), optimize with respect to several variables such as delay, power consumption, temp storage, etc. (constraint parameters P)  To find the minimal of – cost of performing the computation locally, – cost of transmitting the data to a peer (or multiple peers), perform the computation there and transmitting the result back – cost of finding the same data on a peer, perform the computation there and transmitting the result back Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 14
    • 15. Intel Media Technology MRL/TRL Cost Computation & Estimation  What is the cost? –Cost should be dynamic – Resource availability at “server” – Resource availability at “client” – Network availability  How to measure?  How to estimate?  Methodology?  Tools? Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 15
    • 16. Intel Methodology for Estimating End-to-End Media Technology MRL/TRL Computing Performance and Analyzing Network Computing Models  In a networked & distributed computing environment, end-to-end performance of applications depend on capability of components in the system and their interaction.  Need methodology to estimate end-to-end system performance.  Results of performance estimation used to make computing decisions. – Best distribution of computational tasks. – Best algorithm choice. – Most effective hardware improvement. – Enhanced security Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 16
    • 17. Intel Overview Methodology Media Technology MRL/TRL inputs performance operations models results metrics queuing best processing rates grayscale response time network distribution maximum lowest data size scale size calculations throughput cost best network bandwidth compression cost algorithm cost weights halftone quality … encryption security digital signatures Sample Operations Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 17
    • 18. Intel Sample Results: Computation Distribution Media Technology MRL/TRL Color convert 512x512 to gray, scale to 128x128, halftone to 1-bit/pixel  Assumptions: 0.4 – server processing rate >> client 0.35 processing rate response time (sec) 0.3 server process – Simplest model; other server/client 0.25 server network disparities not included 0.2 client network Figure 1 0.15 client process 0.1  Server: grayscale, scale, halftone total 0.05  Client: none 0  Results: 0 4 8 12 arrival rate (requests/sec) – Processing rate limited. – Lowest response time. 1.4 Figure 2 (Left) 3 1.2 response time (sec) response time (sec)  Server: grayscale, scale 2.5 1  Client: halftone 2 0.8 1.5 0.6 1 0.4 0.2 Figure 3 (Right) 0.5 0  Server: grayscale, 0 0 2 4 6 8 10 scale, compress 0 3 6 9 12 15 arrival rate (requests/sec)  Client: decompress, arrival rate (requests/sec) halftone Results: Results:  Bandwidth rate limited. – Highest request throughput Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 18 – client process raises time
    • 19. Intel Sample Results: Quality as a Performance Metric Media Technology MRL/TRL Compare JPEG2000, SPIHT and JPEG – Order of processing rate: JPEG, SPIHT, JPEG2000 – Order of compression ratio for given PSNR: JPEG2000, SPIHT, JPEG  Server and Client can have equal capability (measured on PIII 600 MHz systems for the following) 50 Compression Decompression 40 Method PSNR (Mb/s) (Mb/s) compression 30 JPEG2000 JPEG 20 SPIHT JPEG2000 2 3 30.9 10 0 JPEG l6 15 27.8 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 PSNR SPIHT 12 12 29.0 Compression vs PSNR 128x128 24-bit Lena Comparison of processing rates 8 Delay increases 3 maximum throughput with PSNR 2.5 6 delay (sec) JPEG2000 2 JPEG2000 4 JPEG 1.5 JPEG SPIHT 1 SPIHT 2 @1.5 Mbps. 0.5 0 0 Network limits 26 28 30 32 34 36 38 JPEG2000 closer to 40 throughput 26 28 30 32 34 36 38 40 JPEG and SPIHT Processing deceases with PSNR PSNR for slower network limited PSNR Delay for low arrival rate against PSNR Maximum throughput for 1.5 Mbps Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 19
    • 20. Intel Multimedia Support Media Technology MRL/TRL  “Malleability”/“flexibility” of (bulky) Multimedia – Graceful degradation – Various representations – Complexity  b/w and compute intensive! – More flexibility  more opportunities  Special technology components in MAPS – Universal across multiple platforms – Media Adaptive Transcoding – e.g. MPEG2 (6Mbps)  MPEG4 (300kps) – Audio/video Streaming – Robust, error resilient coding and transmission – Enhanced Multimedia Processing – Technology modules for media security & tracking (encryption/watermarking) – Technology that facilitates multimedia browsing/search/management Intel® MPL can serve as a basic media software infrastructure that provides some core media technology components while facilitates the building of the other ones Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 20
    • 21. Intel Intel MPL - Media Processing Library Media Technology MRL/TRL (presented at IDF Fall’99, Fall’00)  General purpose, high performance software infrastructure with API’s for MPEG encode/decode and processing  BUILDING BLOCK enabling NEW rich-media apps: targeted for applications beyond standard encode/decode/display, e.g., ease-of-use of video, allow interactivity with digital video  Universal & Ubiquitous: – Common API’s across multiple OS: Windows, Linux – Optimized for multiple architecture: IA, XScale™ (in progress) – Can extend to network applications (Internet, wireless, etc.)  Scalable solution – Low bit rate to high bit rate – Low resolution to high resolution – 1 processor to multiple processors – Small devices to high end servers Rich features, high performance, ease-of-development Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 21
    • 22. Intel MPL – PC and Internet Multimedia S/W Media Technology MRL/TRL infrastructure for Diverse Platforms Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 22
    • 23. Intel Reference: MPL features Media Technology MRL/TRL Support MPEG-1 and MPEG-2 up to HDTV, MPEG4 Optimized with MMX™ technology and SIMD Streaming Extensions, Pentium™ 4 Advanced Features - the highlights: – Random access to any frames with near constant-time – Fast extraction of reduced frames – Simultaneous decode of multiple MPEG sequences. – Access via callbacks (API) to non-frame level information in the MPEG bitstream, such as raw bits, blocks, macroblocks, GOP and slice, etc. [structured access to compressed video/audio] – Flexible input plugins: MPL - DTV, MPL – Network – Audio support (AC3, audio level 1, 2 and 3 – MP3, AAC) – SMP support; multithreaded design – Transcoding capabilities Ease-of-development and deployment – Windows*, Linux* – MPL as Component: COM objects (for VB, VC++, DirectShow*) – IPP support for audio & video (2001) – cross-platform optimization: Pentium™ processor families, Xscale™ core architecture based processors (in planning) * Other names and brands are property of their respective owners Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 23
    • 24. Comparison of MPL decoding (MPEG-2 704x480) Intel 136,2 Inside the MPL pipeline Media Technology Decoding rate 150 126 128,2 MRL/TRL (FPS) 100 50 0 Tbird (1GHz) PIII (1GHz) P4 (1.1 GHz) Various input streams High performance from various devices performance Media Infrastructure Media input Video Video output processing MPEG encoder control MPEG decoder Decoder user callback function Motion Compensation Encoder user callback function Control algorithm Preconditioning Motion vector search algorithm Motion estimation Inverse DCT Video encoding DCT control Dequantization Bitrate control Quantization Huffman decoding Users may easily Huffman coding plug in their own algorithms to Headers generation Headers processing encoder’s pipeline and, also, fully Mixer Splitter control encoding and decoding processes. Output system Input system Copyright © 2001 Intel Corporation. Encoded stream Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 24
    • 25. Intel Core Bitstream MPL Next Steps Media Technology MRL/TRL Enhancement Bitstream 1 Enhancement Bitstream 2 MPEG-4 (simple to advanced profiles) Demultiplex Low bitrate, acceptable quality Scene Graph Decoder A Elementary sreams Scenes Person 2D background Furniture A/V object 1 A/V object 2 N decoding E Voice Sprite Globe Desk T BIFS W O Objects R decoding K 2D-Mesh 3D-Mesh Face&Body animation Natural video Synthetic video Decoder B Average bitrate, good quality Decoder C High bitrate, excellent quality Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 25
    • 26. Intel MPL Next Steps Media Technology MRL/TRL  Extend to embrace & integrate more technology building blocks – wireless – P2P – embrace technology catering to and take advantage of the “inequalities” inherent in the computing platforms – low power considerations – Merging of video & graphics  Strive to – Cater to diversity (universal) – Be among the best in performance – Be flexible, and maximize utility/usability – Be a key building block infrastructure for emerging applications Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 26
    • 27. Intel Media Technology MRL/TRL Facilitating Multimedia Search  Automatic media content analysis algorithms AND meta-data can improve search efficiency.  New browsing tools and methodologies can be deployed for distributed media sharing, storage and retrieval, and collaboration.  Flexible query interfaces to facilitate multimedia search  Search can be based on BOTH the file names and content descriptions. – e.g. MPEG7 or XML Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 27
    • 28. Intel Media Technology Content analysis & Search MRL/TRL  Example: Video Search Matched Clips – Very compact representations – Fast Matching on Compact Signatures representation: 4.26 real numbers per frame of video* 250 min of video searched within 7 sec on a PII 400 against 50 frame query * achieved using window size 30 and polynomial degree 1; histograms of Y, U, V channels quantized into 16, 8 and 8 bins Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 28
    • 29. Intel Media Technology On Compact Representations & Fast Search MRL/TRL (Naphade, Yeo & Yeung, earlier work)  Extract compressed low-resolution temporal signature – Compute spatial low resolution signature - for MPEG video, extract/estimate DC images for I/P/B frames, on compressed data – Fit polynomial of N degrees over M samples in time – Describe temporal trace using the N degree polynomial for each M samples  Achieved matching rate of 6000+ frames/sec on a low-end PC  Reduce complex MM search/matching to a fast string-matching  Same signature & compact size regardless of resolution of video DC Sequence Histogram Sequence Incoming Video Database of Polynomial Coefficients Retrieved Clips Similarity Metric Approximated Histogram Computation Sequence Histogram Sequence DC Sequence Incoming Query Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 29
    • 30. Intel Media Technology MRL/TRL Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 30
    • 31. Intel Building MAPS into a P2P Media Technology MRL/TRL computing platform  The enhanced multimedia peer service layer can be integrated into existing/emerging P2P infrastructure – To make P2P media-aware and optimized for media delivery and media applications…  APIs can be defined on P2P platform to facilitate multimedia applications development  Prototype & experimentation at Intel MRL Internet Service “edge” service Peer-to-Peer Service Media Accelerating Peer Service MAPS Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 31
    • 32. Intel Media Technology Prototype system MRL/TRL (Intel MRL 2001) Other Peer Peer  Daemon Other – Low level Peer P2P primitives Daemon – High level controllers Internet Other Client  Client APIs API Peer Peer – Application Applications access to P2P P2P Daemon network The Global Network Peer -to- peer “super” daemon Universe Application -to - Daemon -to -daemon Client daemon APIs communication API Media transcoding Cost evaluation Applications module module User defined modules Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 32
    • 33. Intel Seamless Multimedia for Dynamic and Media Technology MRL/TRL Heterogeneous Environment • Transcoding for bandwidth management • Robust Wireless Streaming/transmission • Multi-platform Encode/Decode • Resource Optimization 10/100baseT HUB 10/100baseT Tablets PDA’s & devices Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 33
    • 34. Intel Media Technology Peer Media Explorer MRL/TRL Snapshot Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 34
    • 35. Intel Media Technology Extending Multimedia Apps and MRL/TRL Technology from Desktop to P2P Emerging media-centric applications:  Content Transaction, Tracking and Monitoring  Media content indexing, browsing & organization  Interactive video, and merged video and graphics  Enhanced virtual reality  Enhanced video conferencing  Networked multimedia games Goal: to enable multimedia apps development and deployment on P2P with enhanced user experience. Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 35
    • 36. Intel Streaming, sharing and Media Technology MRL/TRL collaborating on digital media Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 36
    • 37. Intel Building Blocks Media Technology MRL/TRL Ease-of-use Interactive Video Media Delivery as a natural data type Manipulation/processing “Rendering” of video, Scalable streaming fast, intuitive, graphics (& audio) FGS on un/compressed media, “Word for Video” Optimization of Robust/resilient delivery interactive streaming theory and practice Automatic content analysis & abstraction MPEG4 Technology Transcoding and Apps Utilization B/W-compute adaptability Search, browse, and Smarter delivery schemes retrieve Watermarking Object recognition & Encryption, Security MPL Call-back/hooks segmentation Next-gen CODEC’s Desktop  Networked  Wired to wireless ubiquity  P2P Performance Power Resource Inequality Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 37
    • 38. Intel Wireless Multimedia: Media Technology MRL/TRL Robust Video Delivery over Wireless Networks (IP, Bluetooth, and beyond) TCP MPEG Generate Packetize FEC file File Selection Wireless Got ACK? Access Point ACK Packets Decode Buffer Decode MPEG FEC New Results Buffer Wireless Can Access Point Decode? Send Packets ACK Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 38
    • 39. Intel Looking forward… Media Technology MRL/TRL  What is the optimal multimedia content distribution mechanism?  What is the optimal multimedia processing methodology?  What are the technology building blocks?  Many more questions… – What are the computing models of the future? – What are the best algorithms? – What are the performance criteria?  Many good problems for research  New framework, new formulation, … Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 39
    • 40. More on Intel MRL Media Research University collaborations Summer internships Copyright © 2001 Intel Corporation.
    • 41. Intel Augmenting Research via University Media Technology MRL/TRL Collaboration & Partnerships  Collaborations (2000-01) –UC Berkeley –Polytech University (NY) –others  Partnerships –Internships –Technical exchanges, hosting visits –Courses, resource utilization (e.g. s/w h/w research infrastructure) Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 41
    • 42. Intel Media Technology Summer Internships @ Intel MRL MRL/TRL  May/June to August/September 2001 (12-13 weeks, can be renewed to 9 months)  Looking for outstanding candidates (MS, Ph.D., and very exceptional BS students)  Areas: 1. Algorithms research and development (general, in various fields) – video, audio 2. Low power research 3. P2P System research 4. Wireless multimedia, communications – Email: matthew.holliman@intel.com and Matt will forward to appropriate researchers Copyright © 2001 Intel Corporation. Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 42