Graphics in Mobile Terminals: Principles and Applications Tomi Aarnio Nokia Research Center Visual Communications Lab Sept...
What Is Computer Graphics? <ul><li>Computer graphics concerns visual synthesis of real or imaginary objects from their com...
Graphics in Mobile Terminals <ul><li>2D Vector Graphics </li></ul><ul><ul><li>What is it? </li></ul></ul><ul><ul><li>Appli...
2D Vector Graphics
2D Vector Graphics <ul><li>Raster  images are composed of individual pixels (e.g., 320x240) </li></ul><ul><ul><li>The colo...
Rasterization <ul><li>Conversion of vector images into pixel arrays </li></ul><ul><ul><li>Vector images have infinite reso...
Benefits of 2D Vector Graphics <ul><li>Zooming.  Uncovers new details instead of magnifying pixels. </li></ul><ul><li>Disp...
2D Applications: Interactive Maps <ul><li>Very little network traffic when downloading </li></ul><ul><li>No network traffi...
2D Applications: Cartoons, Micromovies <ul><li>Vector graphics often a better format than video </li></ul><ul><li>TV-quali...
2D Applications: Multimedia Messaging <ul><li>Animation effects combined with photos, audio, text </li></ul><ul><li>Messag...
2D Applications: Graphical User Interfaces <ul><li>Lots of freedom in UI design, compared to raster graphics </li></ul><ul...
2D Applications: Graphics Overlays for Video <ul><li>Video can be overlaid with text, buttons, logos, etc. </li></ul><ul><...
2D Applications: Presentations, e-Books <ul><li>Flexible page layout: capabilities far beyond HTML </li></ul><ul><li>Effic...
More Applications for 2D Vector Graphics <ul><li>Personalization </li></ul><ul><ul><li>Compact animated UI elements </li><...
2D Vector Graphics Infrastructure <ul><li>Vector graphics data can come from </li></ul><ul><ul><li>Web sites, WAP sites (m...
2D Vector Graphics Formats: Flash <ul><li>Proprietary format from Macromedia </li></ul><ul><ul><li>Flash Player installed ...
2D Vector Graphics Formats: SVG <ul><li>Scalable Vector Graphics by W3C </li></ul><ul><ul><li>Open standard from the World...
2D Vector Graphics Formats: VML <ul><li>Vector Markup Language by Microsoft </li></ul><ul><ul><li>Office2000 native format...
2D Vector Graphics Formats: MPEG-4 <ul><li>ISO/IEC standard for &quot;Coding of Audio-Visual Objects&quot; </li></ul><ul><...
Flash vs. SVG <ul><li>Flash is geared for multimedia presentations </li></ul><ul><ul><li>Best choice for non-interactive c...
3D Graphics
Modeling Objects in 3D <ul><li>Geometric descriptions of objects in 3D space </li></ul><ul><li>Usually only the surface of...
Lighting <ul><li>Lighting makes objects look three-dimensional </li></ul><ul><li>Approximates the behavior of light in the...
Texture Mapping <ul><li>Photograph or some other raster image projected on a surface </li></ul><ul><li>Practical alternati...
Environment Mapping <ul><li>Object textured with an image of its environment </li></ul><ul><li>Relatively cheap method to ...
Scene Graph <ul><li>Hierarchic data structure of objects in a visual scene and their relationships </li></ul><ul><li>Relat...
3D Applications: Personal Navigation <ul><li>Location sensitive map viewed from an upwards angle </li></ul><ul><li>&quot;U...
3D Applications: Cartoons, Micromovies <ul><li>The 3rd dimension gives a lot of freedom to authors </li></ul><ul><li>More ...
3D Applications: Multimedia Messaging <ul><li>3D animation gives more possibilities than 2D animation </li></ul><ul><li>An...
3D Applications: Games <ul><li>Visual element will be important also in mobile gaming </li></ul><ul><li>Real-time graphics...
3D Applications: E-Commerce <ul><li>Interactive product visualization </li></ul><ul><li>3D models used instead of photogra...
3D Applications: Augmented Reality <ul><li>Virtual Reality Overlaid on Reality </li></ul><ul><li>Virtual objects placed in...
3D Graphics Formats: VRML / X3D <ul><li>Virtual Reality Modeling Language </li></ul><ul><ul><li>An ISO standard from mid 1...
3D Graphics Formats: MetaStream <ul><li>Proprietary technology from MetaCreations </li></ul><ul><ul><li>Web browser plug-i...
3D Graphics Formats: Pulse3D <ul><li>Proprietary technology from Pulse Entertainment </li></ul><ul><ul><li>Web browser plu...
Graphics Software Architecture
The Role of a Graphics API <ul><li>&quot;Application Programming Interface&quot; </li></ul><ul><li>Provides a defined meth...
Components of a Graphics System Graphics Hardware ? Hardware Specific API Low-Level Generic API High-Level API / Applicati...
Existing Graphics Systems: Case 2D Windows GDI, Epoc GDI, Unix Xlib 2D vector graphics rendering library, 2D Game Engine F...
Existing Graphics Systems: Case 3D OpenGL, Direct3D IRIS Performer, 3D Game Engine VRML, MetaStream, Pulse3D (format + vie...
Graphics in Mobile Terminals: Principles and Applications Tomi Aarnio Nokia Research Center Visual Communications Lab Sept...
Upcoming SlideShare
Loading in …5
×

Graphics in mobile terminals

1,435 views
1,377 views

Published on

Slides from a lecture I gave about 2D/3D graphics in mobile devices, pitching the case for open standard formats and APIs, such as OpenGL and SVG.

Dated September 15, 2000.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,435
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • - Hierarchic, integrated data structure - Parent - children / root - subtree - Describes object locations, relationships to each other, … - Also audio, collision detection, ...
  • - historically &amp;quot;application programming interface&amp;quot; - in practice, a library of graphics functions, implemented in hardware and/or software - Makes life easier!
  • Graphics in mobile terminals

    1. 1. Graphics in Mobile Terminals: Principles and Applications Tomi Aarnio Nokia Research Center Visual Communications Lab September 15, 2000
    2. 2. What Is Computer Graphics? <ul><li>Computer graphics concerns visual synthesis of real or imaginary objects from their computer-based models </li></ul><ul><li>Includes creation, storage and manipulation of models and images of objects, often in an interactive way </li></ul><ul><li>Closely related to image processing, which treats analysis and manipulation of existing images </li></ul>
    3. 3. Graphics in Mobile Terminals <ul><li>2D Vector Graphics </li></ul><ul><ul><li>What is it? </li></ul></ul><ul><ul><li>Applications </li></ul></ul><ul><ul><li>Technologies </li></ul></ul><ul><li>3D Graphics </li></ul><ul><ul><li>What is it? </li></ul></ul><ul><ul><li>Applications </li></ul></ul><ul><ul><li>Technologies </li></ul></ul><ul><li>Graphics Software Architecture </li></ul>
    4. 4. 2D Vector Graphics
    5. 5. 2D Vector Graphics <ul><li>Raster images are composed of individual pixels (e.g., 320x240) </li></ul><ul><ul><li>The color of each pixel is stored separately </li></ul></ul><ul><ul><li>Good for natural images (photographs and video) </li></ul></ul><ul><li>Vector images are composed of shapes: lines, curves, arcs, etc. </li></ul><ul><ul><li>Example: A circle is defined by its center and radius </li></ul></ul><ul><ul><li>Good for hand-drawn and computer-generated images </li></ul></ul><ul><li>Vector graphics usually means mixed vector/raster graphics </li></ul><ul><ul><li>Shapes may be filled or blended with raster images </li></ul></ul>
    6. 6. Rasterization <ul><li>Conversion of vector images into pixel arrays </li></ul><ul><ul><li>Vector images have infinite resolution, current displays do not </li></ul></ul><ul><ul><li>Images are sampled at some resolution (e.g., 320x240, 64k colors) </li></ul></ul><ul><li>Rasterization is done only when the image needs to be displayed </li></ul><ul><ul><li>Terminal decides the resolution and number of colors </li></ul></ul><ul><ul><li>Gives optimal quality for any display </li></ul></ul>
    7. 7. Benefits of 2D Vector Graphics <ul><li>Zooming. Uncovers new details instead of magnifying pixels. </li></ul><ul><li>Display independence. Optimal quality on any screen or printer. </li></ul><ul><li>Bandwidth efficiency. Orders of magnitude savings in some cases. </li></ul><ul><li>Interactivity. Allows terminal-side manipulation of individual shapes. </li></ul>Zoom
    8. 8. 2D Applications: Interactive Maps <ul><li>Very little network traffic when downloading </li></ul><ul><li>No network traffic when zooming or panning </li></ul><ul><li>Can be overlaid with arrows, paths, text, etc. </li></ul><ul><li>Can be printed out with perfect quality </li></ul>
    9. 9. 2D Applications: Cartoons, Micromovies <ul><li>Vector graphics often a better format than video </li></ul><ul><li>TV-quality South Park episodes at ~25 kbits/s </li></ul><ul><li>Scalable to any screen resolution with no loss of quality </li></ul><ul><li>Many popular sites on the web already (cartoons, music videos) </li></ul>Vector graphic image Compressed with JPEG
    10. 10. 2D Applications: Multimedia Messaging <ul><li>Animation effects combined with photos, audio, text </li></ul><ul><li>Messages composed on the terminal from simple elements </li></ul><ul><li>Collections of animation clips and effects on a server </li></ul><ul><li>Building on the success of SMS and first generation MMS </li></ul>
    11. 11. 2D Applications: Graphical User Interfaces <ul><li>Lots of freedom in UI design, compared to raster graphics </li></ul><ul><li>Innovative, more intuitive UIs possible for mobile devices also </li></ul><ul><li>Efficient use of screen space with zooming, panning, antialiasing </li></ul><ul><li>Example: Apple's Aqua is based on real-time vector graphics </li></ul>
    12. 12. 2D Applications: Graphics Overlays for Video <ul><li>Video can be overlaid with text, buttons, logos, etc. </li></ul><ul><li>Encoding overlays as video is a serious waste of bandwidth </li></ul><ul><li>Solution: transmit video and overlays separately </li></ul><ul><li>Use vector graphics for the overlays </li></ul>
    13. 13. 2D Applications: Presentations, e-Books <ul><li>Flexible page layout: capabilities far beyond HTML </li></ul><ul><li>Efficient screen space utilization with vector fonts </li></ul><ul><li>Text can be rotated, scaled, slanted, put on a curve, etc. </li></ul><ul><li>Hyperlinks and other interactive elements can be used </li></ul>
    14. 14. More Applications for 2D Vector Graphics <ul><li>Personalization </li></ul><ul><ul><li>Compact animated UI elements </li></ul></ul><ul><ul><li>Animated &quot;personal assistants&quot; </li></ul></ul><ul><li>Charts, graphs, figures </li></ul><ul><ul><li>Considerable bandwidth savings </li></ul></ul><ul><ul><li>Can be zoomed to uncover details </li></ul></ul><ul><ul><li>Interactive pop-up elements, etc. </li></ul></ul><ul><li>Web site intro animations </li></ul><ul><ul><li>Bandwidth efficiency is key </li></ul></ul><ul><ul><li>Could be used in WAP as well </li></ul></ul><ul><li>Server-based games </li></ul><ul><ul><li>No graphical games can be done in HTML/WML </li></ul></ul><ul><ul><li>Vector graphics & scripting provide a solution </li></ul></ul>
    15. 15. 2D Vector Graphics Infrastructure <ul><li>Vector graphics data can come from </li></ul><ul><ul><li>Web sites, WAP sites (maps, cartoons, games, etc.) </li></ul></ul><ul><ul><li>Other terminals (animation messages, etc.) </li></ul></ul><ul><ul><li>Local storage (logos, fonts, preloaded maps) </li></ul></ul><ul><li>Vector graphics engine can be integrated into </li></ul><ul><ul><li>Web/WAP browser </li></ul></ul><ul><ul><li>Operating system </li></ul></ul><ul><ul><li>Application program </li></ul></ul><ul><li>Two methods to deliver long animations </li></ul><ul><ul><li>Download the animation as one big file, then play it back </li></ul></ul><ul><ul><li>Perform playback while downloading (streaming) </li></ul></ul>
    16. 16. 2D Vector Graphics Formats: Flash <ul><li>Proprietary format from Macromedia </li></ul><ul><ul><li>Flash Player installed in over 90 % of web browsers </li></ul></ul><ul><ul><li>Good authoring tool support (Adobe, Corel, Macromedia) </li></ul></ul><ul><li>Multimedia presentations - not just graphics </li></ul><ul><ul><li>Poor integration with other web content (HTML) </li></ul></ul><ul><ul><li>Frame-based animation with synchronized audio </li></ul></ul><ul><ul><li>Streamable, but only with a reliable transport </li></ul></ul><ul><ul><li>Binary format with very simple compression </li></ul></ul><ul><ul><li>Limited client-side interactivity </li></ul></ul><ul><li>Macromedia aims to get control of the web </li></ul><ul><ul><li>Web sites built with Flash instead of HTML </li></ul></ul><ul><ul><li>Licence fees from everyone but the end users </li></ul></ul>
    17. 17. 2D Vector Graphics Formats: SVG <ul><li>Scalable Vector Graphics by W3C </li></ul><ul><ul><li>Open standard from the World Wide Web Consortium (W3C) </li></ul></ul><ul><ul><li>SVG viewer available as a browser plug-in from Adobe </li></ul></ul><ul><ul><li>Supported in authoring tools from Adobe, Corel, others </li></ul></ul><ul><li>SVG is a part of the &quot;next generation web&quot; </li></ul><ul><ul><li>XML-based text format (no compression) </li></ul></ul><ul><ul><li>Excellent integration with other web standards </li></ul></ul><ul><ul><li>Multimedia presentations built with SVG+SMIL </li></ul></ul><ul><li>Technically more advanced than Flash </li></ul><ul><ul><li>Image post-processing (filter effects) </li></ul></ul><ul><ul><li>Sophisticated animation model </li></ul></ul><ul><ul><li>Full client-side interactivity </li></ul></ul>
    18. 18. 2D Vector Graphics Formats: VML <ul><li>Vector Markup Language by Microsoft </li></ul><ul><ul><li>Office2000 native format for vector drawings </li></ul></ul><ul><ul><li>Only supported in Office2000 and Internet Explorer </li></ul></ul><ul><ul><li>Microsoft's current plans for VML are unknown </li></ul></ul><ul><li>One of the initial proposals that led to SVG </li></ul><ul><ul><li>XML-based text format, no compression </li></ul></ul><ul><ul><li>Good integration with other web content </li></ul></ul><ul><ul><li>Full client-side interactivity </li></ul></ul><ul><li>Technically somewhat challenged </li></ul><ul><ul><li>Geared towards typical office applications </li></ul></ul><ul><ul><li>No animation capabilities - still images only </li></ul></ul><ul><ul><li>No image post-processing capabilities </li></ul></ul>
    19. 19. 2D Vector Graphics Formats: MPEG-4 <ul><li>ISO/IEC standard for &quot;Coding of Audio-Visual Objects&quot; </li></ul><ul><ul><li>No support in authoring tools </li></ul></ul><ul><ul><li>No commercial viewers/players </li></ul></ul><ul><li>A presentation environment similar to Flash, but far larger </li></ul><ul><ul><li>Supports 2D/3D graphics, video, audio, etc. </li></ul></ul><ul><ul><li>The only format that is designed for streaming </li></ul></ul><ul><ul><li>Binary format with advanced compression </li></ul></ul><ul><ul><li>No compatibility with web standards </li></ul></ul><ul><li>2D vector graphics features a bit lacking </li></ul><ul><ul><li>No arbitrary clipping paths </li></ul></ul><ul><ul><li>No image post-processing </li></ul></ul>
    20. 20. Flash vs. SVG <ul><li>Flash is geared for multimedia presentations </li></ul><ul><ul><li>Best choice for non-interactive content </li></ul></ul><ul><ul><li>Published presentations cannot be changed </li></ul></ul><ul><ul><li>Simple to implement, low complexity </li></ul></ul><ul><ul><li>Owned by Macromedia </li></ul></ul><ul><li>SVG is for vector graphics content in general </li></ul><ul><ul><li>Best choice for interactive content </li></ul></ul><ul><ul><li>All content is readable and reusable </li></ul></ul><ul><ul><li>Very advanced graphics capabilities </li></ul></ul><ul><ul><li>Open standard, licence-free </li></ul></ul><ul><ul><li>Excellent web integration </li></ul></ul><ul><li>SVG and Flash will probably coexist for a long time </li></ul>
    21. 21. 3D Graphics
    22. 22. Modeling Objects in 3D <ul><li>Geometric descriptions of objects in 3D space </li></ul><ul><li>Usually only the surface of an object is modeled </li></ul><ul><li>Surfaces are most often represented with triangle meshes </li></ul>
    23. 23. Lighting <ul><li>Lighting makes objects look three-dimensional </li></ul><ul><li>Approximates the behavior of light in the real world </li></ul><ul><li>More complex lighting equations yield better visual quality </li></ul>
    24. 24. Texture Mapping <ul><li>Photograph or some other raster image projected on a surface </li></ul><ul><li>Practical alternative to modeling surfaces with millimeter accuracy </li></ul>
    25. 25. Environment Mapping <ul><li>Object textured with an image of its environment </li></ul><ul><li>Relatively cheap method to simulate reflective surfaces </li></ul>
    26. 26. Scene Graph <ul><li>Hierarchic data structure of objects in a visual scene and their relationships </li></ul><ul><li>Relationships may include transformation, visibility, and many other kinds of attributes </li></ul>House Living Room Kitchen Table TV Set Sofa Teapot
    27. 27. 3D Applications: Personal Navigation <ul><li>Location sensitive map viewed from an upwards angle </li></ul><ul><li>&quot;Up&quot; on the screen is always &quot;forward&quot; in the real world </li></ul><ul><li>Buildings, characters, arrows, etc. can be placed on the map </li></ul><ul><li>Quicker and easier to navigate than a static 2D map </li></ul>
    28. 28. 3D Applications: Cartoons, Micromovies <ul><li>The 3rd dimension gives a lot of freedom to authors </li></ul><ul><li>More life-like animation, more movie-like camera angles </li></ul><ul><li>Modest (and adjustable) bandwidth requirements </li></ul>
    29. 29. 3D Applications: Multimedia Messaging <ul><li>3D animation gives more possibilities than 2D animation </li></ul><ul><li>Animation sequences and 3D models stored on a server </li></ul><ul><li>Example: Virtual Jay Leno at http://www.nbc.com </li></ul>
    30. 30. 3D Applications: Games <ul><li>Visual element will be important also in mobile gaming </li></ul><ul><li>Real-time graphics is needed to go beyond Minesweeper and Solitaire </li></ul><ul><li>Multi-player networked games more interesting than solo games </li></ul>
    31. 31. 3D Applications: E-Commerce <ul><li>Interactive product visualization </li></ul><ul><li>3D models used instead of photographs </li></ul><ul><li>Customers get a better view of the product </li></ul>
    32. 32. 3D Applications: Augmented Reality <ul><li>Virtual Reality Overlaid on Reality </li></ul><ul><li>Virtual objects placed in your physical surroundings </li></ul><ul><li>Viewed through a Head-Mounted Display (HMD) </li></ul>
    33. 33. 3D Graphics Formats: VRML / X3D <ul><li>Virtual Reality Modeling Language </li></ul><ul><ul><li>An ISO standard from mid 1990's </li></ul></ul><ul><ul><li>Implemented as web browser plug-ins </li></ul></ul><ul><li>VRML never became popular </li></ul><ul><ul><li>Aimed to be a generic platform for all applications </li></ul></ul><ul><ul><li>Had several flaws in design and was too complex </li></ul></ul><ul><ul><li>No complete and correct implementations exist </li></ul></ul><ul><ul><li>Poor integration with other web content </li></ul></ul><ul><li>Efforts continued under the name X3D (Extensible 3D) </li></ul><ul><ul><li>A reformulation of VRML97 in XML </li></ul></ul><ul><ul><li>Carries the legacy of VRML to maintain compatibility </li></ul></ul><ul><ul><li>Will probably not succeed - should have started from scratch </li></ul></ul>
    34. 34. 3D Graphics Formats: MetaStream <ul><li>Proprietary technology from MetaCreations </li></ul><ul><ul><li>Web browser plug-in required (~100 kbytes) </li></ul></ul><ul><ul><li>Geared for product visualization in e-commerce </li></ul></ul><ul><li>Technologically impressive, but not perfect </li></ul><ul><ul><li>Nearly photorealistic rendering </li></ul></ul><ul><ul><li>Integrates well with other web content </li></ul></ul><ul><ul><li>XML-based scene descriptions </li></ul></ul><ul><ul><li>Compressed object definitions </li></ul></ul><ul><ul><li>Progressive downloading </li></ul></ul><ul><ul><li>Limited animation capabilities </li></ul></ul><ul><li>For more information and demos, see http://www.metastream.com </li></ul>
    35. 35. 3D Graphics Formats: Pulse3D <ul><li>Proprietary technology from Pulse Entertainment </li></ul><ul><ul><li>Web browser plug-in required (~100 kbytes) </li></ul></ul><ul><ul><li>Aimed for entertainment: cartoons, music videos, etc. </li></ul></ul><ul><ul><li>A lot of content already available on the web </li></ul></ul><ul><li>Technology not as advanced as MetaStream's </li></ul><ul><ul><li>Rendering quality not as impressive </li></ul></ul><ul><ul><li>Scalability and streaming features lacking </li></ul></ul><ul><ul><li>Limited integration with other web content </li></ul></ul><ul><ul><li>Good animation capabilities </li></ul></ul><ul><li>For more information and demos, see http://www.pulse3d.com </li></ul>
    36. 36. Graphics Software Architecture
    37. 37. The Role of a Graphics API <ul><li>&quot;Application Programming Interface&quot; </li></ul><ul><li>Provides a defined method of accessing graphics functions </li></ul><ul><li>Abstracts underlying hardware and software implementations </li></ul><ul><li>Allows easy development of graphics applications for different platforms </li></ul>?
    38. 38. Components of a Graphics System Graphics Hardware ? Hardware Specific API Low-Level Generic API High-Level API / Application Engine Presentation Platform Applications
    39. 39. Existing Graphics Systems: Case 2D Windows GDI, Epoc GDI, Unix Xlib 2D vector graphics rendering library, 2D Game Engine Flash, SVG, VML, PDF (format + viewer) Hardware Specific API Graphics Hardware Cartoons, MMS, e-Books, GUIs, Games, Web/WAP services
    40. 40. Existing Graphics Systems: Case 3D OpenGL, Direct3D IRIS Performer, 3D Game Engine VRML, MetaStream, Pulse3D (format + viewer) Hardware Specific API Graphics Hardware Games, Maps, Cartoons, GUIs, etc.
    41. 41. Graphics in Mobile Terminals: Principles and Applications Tomi Aarnio Nokia Research Center Visual Communications Lab September 15, 2000

    ×