• Like
Course notes (.ppt)
Upcoming SlideShare
Loading in...5
×
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,886
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
42
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Chapter 4: Software That Enables Multimedia
    • Operating systems
    • Development software
    • Delivery software
  • 2. Operating System Software
    • Device drivers
    • Multi-tasking, multi-threading
    • Memory management
    • Disk space management
    • Processor (CPU) management & scheduling
    • Window management
    • Servers (e.g., printing, email, web, telnet, ssh)
    • User level software
      • Includes programs to access and deal with lower levels of operating system (e.g., format a disk, partition a disk, change the priority of a task or process)
      • And user level applications to get work done
        • E.g., Macromedia Director, Power Point, Web browser
  • 3. Device Drivers
    • E.g., Display adapter driver, Firewire card driver, Network card driver, Video Camera Driver, Microphone driver
    • Devices have specific hardware
      • Drivers are specific to the hardware
      • Enable the operating system to use that specific hardware
    • Device drivers depend
      • on operating system
      • on the specific device
    • Device drivers enable programs
      • To be less dependent on hardware; more generic
  • 4. Codec’s
    • Compression and Decompression software for video and audio
      • “ drivers” for the particular format used for the files
      • Depend on format, not on hardware
    • Within a particular general format (e.g., AVI or Quicktime)
      • E.g., with AVI, a Cinepak or Indeo codec can be used
      • Sorensen is available with Quicktime
      • Many codec’s can be used
      • Codec must be on development system
      • And on the destination system
  • 5. Networking - 1
    • Networking functions are incorporated into operating systems
      • Remote disk mounting
      • Network device (e.g., printer) access
      • Lower levels of network data transmission (e.g., physical, data link layers)
    • Multimedia software often depends on networks
      • Dynamic loading of codec & other components
      • Access to web-based presentations
  • 6. Networking - 2
    • Communication protocols
      • Physical layer protocols (ISO layer 1)
        • E.g., Ethernet, token ring, ATM
      • Higher protocol layers
        • TCP/IP (layers 2 & 3)
        • SMTP – simple mail transfer protocol (layer 6)
        • HTTP – hypertext transfer protocol (layer 6)
    • Performance objectives
      • Accuracy, transmission speed, error correction, user-perceived delays, overall throughput
  • 7. Client/Server Architecture - 1
    • Server
      • A process that provides access to particular hardware or software resources
        • Process: A program that runs independently on a computer
    • Often used in the context of networks
      • Used within software design generally, as well
    • “ Server” may designate a software process
      • Or may designate a computer (hardware)
      • Depends on the kind of service being provided
  • 8. Client/Server Architecture - 2
    • E.g., database server
      • Computer will have database software
      • May also have additional hardware resources (e.g., more disk space)
    • E.g., a software license server (“metering”)
      • Enables some maximum number of people to be using a software resource at a given time.
    • Server receives requests
      • Remote requests from a client process on another computer
      • Local requests from a client process on same computer
      • Requests are sometimes for data sometimes for programs
  • 9. Operating System & Multimedia Requirements
    • Development of multimedia
    • Running multimedia
      • Some operating systems may be more suited for running multimedia
        • E.g., BeOS has been touted as a multimedia operating system
    • Issues include:
      • Memory (RAM) required
      • Operating system availability
      • Application availability
      • Speed
      • Networking
  • 10. Multimedia Development Software
    • Software needed to develop multimedia
      • And software needed to deliver
      • E.g., Shockwave playback in a browser has different requirements than running Macromedia Director
    • Development
      • Preproduction, production, postproduction
    • Types of development software
      • 2D image capture, scanning & processing; Digital video capture & processing
      • Animation; 3D modeling, rendering; Sound capture & processing; Text; Web development; Authoring
  • 11. Vector vs. Bitmap Graphics - 1
    • Representation of objects in images
      • E.g., a rectangle with a black outline and a gray interior
    • Bitmap (raster) graphics
      • Specify value for each pixel
      • Application data file (e.g., Photoshop) stores collection of pixels making up rectangle
        • E.g., 300 x 600 = 180,000 pixels of rectangle
        • Requires 176KB to 527KB of data (uncompressed)
    • Vector graphics
      • Use rectangle drawing tool & color-fill
      • Application program generates a data file that represents rectangle as a command and parameters
      • E.g.,
        • Rectangle, 0, 0, 300, 600, black, gray (32)
      • Requires about 100 bytes of information
  • 12. Vector vs. Bitmap Graphics - 2
    • Space Tradeoff
      • Bitmap graphics (uncompressed) take more space
      • Vector graphics take less space
    • Speed Tradeoff
      • Bitmap graphics (uncompressed) are drawn faster
      • Vector graphics need to be decoded and then the bitmap (pixels) generated
        • Will be slower than bitmap representation
        • An issue on slower computers or if a very large number of objects need to be displayed
    • Expressiveness
      • Some objects can’t be easily drawn in vector fashion
    • Contemporary drawing programs may provide both vector and bitmap graphics
  • 13. Video - 1
    • Video is captured in discrete units called frames
      • Frame: Similar to a still picture
    • Temporal and spatial compression
      • Spatial: within a frame
      • Temporal: across frames
    • Key-frame concept in temporal compression
      • Some compression methods use this form of coding
      • Store information on one frame at particular intervals
        • This is a “key frame”
      • Then store changes from this point
      • Uses frame differencing
      • Makes use of common background often shared between multiple frames
  • 14. Video - 2
    • Time-dependent medium
      • Illusion of motion depends on frame rate
        • Don’t perceive motion below about 15 fps
      • Also: animation and sound
    • Time-dependent media place high computational demands on a computer
      • E.g., with 1024 x 768 image resolution, and 24 bit color depth @ 30 fps, what is the (apparent) data transfer rate in KB/second?
  • 15. Video - 3
    • 1024 * 768 = 786, 432 pixels/frame
    • 786, 432 pixels/frame* 3 bytes/pixel =
    • 2,359,296 bytes/frame
    • 2,359,296 bytes/frame * 30 frame/sec =
    • 70,778,880 bytes/sec
    • (70,778,880 bytes/sec) / (1024 bytes/KB) =
    • 69,120 KB/sec
    • (Apparent: because this is without any data compression)
  • 16. Animation
    • Similar to video
    • Each frame created via software
      • not from a camera source
    • History
      • traditionally done by hand, frame by frame
      • E.g., cel-based, or clay-mation
    • Model-based animation
      • Online (real-time) versus offline
      • Model-based: software representation of world
      • Online: Frames are generated on-the-fly, in response to dynamic changes of the model
      • Offline: Frames generated in response to model changes
        • No requirement for real-time algorithms
  • 17. Related Techniques
    • Biological animation
      • Animal gaits, bipedal motion
    • Morphing
      • Gradual changes in one image to transform it to a second image
      • Applying changes gradually
        • E.g., taking a few seconds @ 30 fps
        • Can be used for animation purposes
    • VRML
      • Virtual reality modeling language
      • "an open standard for 3D multimedia and shared virtual worlds on the Internet."
          • http://home.hiwaay.net/~crispen/vrmlworks/faq/faq1.html#q1
      • “ Briefly, VRML 1.0 worlds are static. VRML 2.0 worlds can move and interact with the visitor to those worlds.”
  • 18. MPEG, Quicktime, AVI
    • Standard formats for video and audio information
      • Includes techniques/standards for compression, decompression
    • MPEG
      • http://mpeg.telecomitalialab.com/mpeg_general.htm
      • http://mpeg.telecomitalialab.com/
      • MPEG Audio
        • For a paper on this, see http://citeseer.nj.nec.com/421409.html
      • ISO/IEC standardization
      • Open standard
        • Specification is available to everyone (for a fee)
        • No single company “owns” the standard
  • 19. Quicktime, AVI
    • Quicktime
      • Primarily Macintosh/Apple
      • http://www.quicktime.com
    • AVI
      • Primarily Windows/Microsoft
      • AVI: “Audio Video Interleave ”
      • E.g., see http://www.jmcgowan.com/avi.html
  • 20. MPEG Standards
    • MPEG-1:
      • Parts: Systems, Video, Audio
      • Applications: MP3, Video CD
    • MPEG-2
      • Parts: Systems, Video, Audio, Conformance, Reference Software Digital Storage Media Command and Control (DSM-CC), Advanced Audio Coding (AAC), Real Time Interface
      • MP3 , Digital Television set top boxes, Digital Versatile Discs (DVD).
    • MPEG-3 (See MPEG-2)
    • MPEG-4
      • Proposed fixed & mobile web standard
    • MPEG-7: Multimedia Content Description
    • MPEG-21: Multimedia Framework
  • 21. MPEG: Video
    • MPEG-1
      • Roughly VHS quality at 1,15 Mbit/s
    • MPEG-2
      • “ MPEG-2 Video was at least good, and in many cases even better than standards or specifications developed for high bitrate or studio applications”
      • http://mpeg.telecomitalialab.com/standards/mpeg-2/mpeg-2.htm
  • 22. MPEG: Audio
    • MPEG standards have layers
      • Layer: Operating modes with increasing complexity and performance
    • MPEG-1 provides
      • Mono and stereo coding at 32, 44.1, and 48 kHz sampling rate
      • Layer I, II, III: varying bit rates from 32 to 448 kbit/s
      • Optimized for 128 kbits/s stero
    • MPEG-2 BC provides
      • Backwards compatible multichannel extension to MPEG-1
      • Up to 5 channels plus a 'low frequent enhancement' channel
      • Bit rate range is extended up to about 1 Mbit/s;
      • an extension of MPEG-1 towards lower sampling rates 16, 22.05, and 24 kHz for bitrates from 32 to 256 kbit/s (Layer I) and from 8 to 160 kbit/s (Layer II & Layer III).
    • http://mpeg.telecomitalialab.com/faq/audio.htm#5
  • 23. MPEG-2 & MPEG-4: Audio
    • MPEG-2 Advanced Audio Coding (AAC)
      • High-quality audio coding standard for 1 to 48 channels
      • Sampling rates of 8 to 96 kHz
      • Multichannel, multilingual, and multiprogram capabilities
      • Bit rates from 8 kbit/s to more thanf 160 kbit/s/channel
      • Multiple encode/decode cycles
      • Three profiles (layers) of AAC provide varying levels of complexity and scalability.
    • MPEG-4 will provide
      • coding and composition of natural and synthetic audio objects at a very wide range of bit rates.
      • Emphasis on new functionalities
  • 24. MPEG-4
    • Goal: Higher levels of interaction
    • 1. represent units of aural, visual or audiovisual content, called "media objects". These media objects can be of natural or synthetic origin; this means they could be recorded with a camera or microphone, or generated with a computer;
    • 2. describe the composition of these objects to create compound media objects that form audiovisual scenes;
    • 3. multiplex and synchronize the data associated with media objects, so that they can be transported over network channels providing a QoS appropriate for the nature of the specific media objects; and
    • 4. interact with the audiovisual scene generated at the receiver’s end.
    • http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm
  • 25. MP3
    • MPEG-1/2 Audio Layer-3
      • MPEG-1 Audio Layer-3
      • MPEG-2 Audio Layer-3 (successor)
    • MP3 and AAC Explained
      • http://citeseer.nj.nec.com/421409.html
    • MP3  MPEG-3!
  • 26. Sound Representation
    • On a sample basis
      • E.g., about 44 kHz for CD quality audio
      • Each sample is a number
        • gives the amplitude of the analog sound wave at that point
    • On a feature basis
      • E.g., frequency, duration, musical note
      • Also possible: for speech, phonemes
        • Phoneme is a unit of sound in a particular language
        • Phonemes may sound differently dependent on context
        • E.g. /p/ ( p ersonal, p aternal, com p uter, ra p id), /b/, and /s/
    • Difference is analogous to the difference between bitmapped (raster) & vector graphics
    • Tradeoffs– Similar to those between bitmap & vector graphics
  • 27. Example
    • 1) With a sampling rate of 44.1 kHz, and 16 bit sound resolution, how many KB does it take to represent 3 minutes of sound? (Without compression)
    • 2) If we are representing speech phonemes, and we want to represent individual words, how much data does it take to encode the English word “Hello”?
      • Assume that English has 39 phonemes
      • Assume that we have one phoneme per character
      • Assume that our coding scheme needs to accommodate arbitrary English words
  • 28. Example- Solution (1)
    • 44,100 samples/sec * 2 bytes/sample = 88, 200 bytes/sec
    • 88, 200 bytes/sec * 60 sec/minute =
    • 5,292,000 bytes/min
    • 5, 292, 000 bytes/min * 3 min =
    • 15,876,000 bytes
    • 15,876,000 bytes/(1024 bytes/KB) =
    • 15, 503.90625 KB
    • (Or about 15 MB)
  • 29. Example- Solution (2)
    • Problem is to represent sequences of phonemes
      • Each phoneme can take on 39 possible values
      • Need at least 6 bits of information to represent 64 different values
        • So, need at least 6 bits per phoneme
    • Word “Hello” has 5 characters
      • Assuming also 5 phonemes
      • 5 * 6 = 30 bits
      • 30 bits/(8 bits/byte) = 3.75 bytes
      • Need 4 bytes
  • 30. Sound Filtering
    • A sound can be filtered to remove a certain set of frequencies
    • Some types of filtering
      • Low pass: Allows only low frequencies through
      • High pass: Allows only high through
      • Band pass: Allows only a band range of frequencies through
    • Other types of filtering or processing
      • Trimming, splicing, volume, brightness
  • 31. Java - 1
    • Sun Microsystems
    • Goal: Platform independent programs
    • Java code compiled into “byte-code”
      • To be executed on a Java virtual machine
      • Really a byte-code interpreter
      • Executes individual byte-code commands
      • These are the files that are executed by browsers
    • Some Java features
      • No pointers
      • Garbage collection
  • 32. Java - 2
    • Efficiency
      • Just-in-time compilers
        • Compile byte-code to native machine code when used
      • Java processing hardware
    • Security
      • Java code downloaded from the Internet is “untrusted”
      • Applet restrictions
      • Byte-code verification
        • Byte-codes checked before running to ensure they are valid
        • Goal: ensure no unauthorized references to discs etc.
  • 33. Web browsers
    • Process various file types
      • HTML
      • Media formats: images, sounds, movies
    • Some types of data are processed directly
      • Built into the browser
    • Other types
      • Require additional programs or “plugin-ins”
      • E.g., a plugin to display PDF or Adobe Portable Document format files or a plugin to display Shockwave files
      • Some plugins are called “players” or “viewers”
        • E.g., a viewer to display MS Word documents
  • 34. Kappner & Steinmetz (1995): Multimedia platform requirements:
    • Multimedia platform should
      • Support
        • the development of a wide variety of multimedia applications
        • applications in a distributed system
      • Be able to
        • establish and control streams of continuous media
        • to choose the quality of service for a data stream
        • to express relations between several media streams
  • 35. Multimedia Delivery Software
    • Two general means of delivery
      • Compiled or Interpreted
    • Compiled
      • Machine code generated for a computer CPU (e.g., Intel-based) and operating system (O/S)
      • Faster; limited to computer type & O/S; stand-alone
      • E.g., Macromedia Director “Projector”
    • Interpreted
      • Instructions created that can be interpreted by a program; E.g., a plugin
      • Slower; more portable; need interpreter (e.g., player) software
      • E.g., Macromedia Shockwave
    • In both cases, media files are embedded and/or retained as separate files