Introduction to 2D/3D Graphics


Published on

Takes the reader through the various components of windowing systems, and how to develop and benchmark various Graphics applications using OpenGL and other toolsets. Also includes a Cheatsheet that covers various terminologies used in the Graphics world.

Published in: Technology, Art & Photos
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Problem Statement and why is it important ( applications, issues etc)
  • PIX COP – does packing/unpacking of data after rasterisation Texture cop – Generates texture address for fetch into the multi-level cache Tiling cop – Generates tiles from vertex shader (USSE) output and Rejects them if needed based on culling/clipping before giving to pixel shader USSE – vertex,pixel shading done, 4kb cached program Multilevel cache-L0=Textureonly,8pixels,L1=genpurpose 1KB,L2=genpurpose 1kB 32 linesx256bit, 4way set associative
  • Figure, description, blow-up of part of the diagram if relevant Analogies if any
  • Introduction to 2D/3D Graphics

    1. 1. Engaging the consumer with visual effects – Architectures and implementations for graphics Prabindh Sundareson ( [email_address] ) Texas Instruments India TIITC2009 : Customer Centric Strategies DSPS
    2. 2. Outline <ul><li>2D Graphics </li></ul><ul><li>3D Graphics </li></ul><ul><li>The OpenGL API </li></ul><ul><li>Internals of SGX, benchmarking </li></ul><ul><li>Future roadmap </li></ul><ul><li>Conclusion </li></ul><ul><li>[Graphics Cheatsheets] </li></ul>TIITC2009 : Customer Centric Strategies
    3. 3. What can Graphics do for me ? TIITC2009 : Customer Centric Strategies
    4. 4. Graphics Frameworks Cheatsheet TIITC2009 : Customer Centric Strategies FB, V4l2 Cairo Clutter, QT, Opera DirectFB Surfaces X Window System/Mgrs Application Frameworks Buffers / blits Note: Cheatsheet table in appendix GDI DDr a w Win32/ Windowing System Win32, .NET, Silverlight
    5. 5. 2D Graphics <ul><li>Almost all existing Consumer Graphical User Interfaces are still built around 2D, and not around OpenGL </li></ul><ul><ul><li>Exception of course, iPhone, other Mobile phones, Games… </li></ul></ul><ul><ul><li>Specific operations can still be accelerated with 3D accelerators. Like blits of large sizes </li></ul></ul><ul><li>What are the typical 2D applications ? </li></ul><ul><ul><li>All Windowing Systems (X, Win32, …) </li></ul></ul><ul><ul><li>Application Frameworks like QT, Flash </li></ul></ul><ul><ul><li>Browsers </li></ul></ul><ul><ul><li>All 3D operations ultimately end in 2D blitting (ex, EGL uses PVR2D) </li></ul></ul><ul><li>An example, a Browser application… </li></ul>TIITC2009 : Customer Centric Strategies
    6. 6. Case study: Webkit’s Call Sequence TIITC2009 : Customer Centric Strategies FillRectangles () BatchBlit() Blit(), mem_copy() Clear() Blit() TileBlit() FILL RECTANGLE SHOW GLYPHS SET CLIP REGION ACQUIRE FONT CACHE ALLOCATE FONT CACHE SURFACE COMPOSITE PREPARE/FINISH COMPOSITE
    7. 7. Summary of 2D – Browser Rendering Processes <ul><li>To render one page of Youtube, for example </li></ul><ul><ul><li>A large blit of size = page width x block size ~ typical (500 x 100 pix) </li></ul></ul><ul><ul><li>Followed by multiple blits of a glyph cache line of one line of text (page width x text height ) ~ ex (50 x 8), this renders blocks line by line </li></ul></ul><ul><ul><li>Followed by multiple compositing operations for each block to render the whole page ~ ex (16 x 16 to page size) </li></ul></ul>TIITC2009 : Customer Centric Strategies
    8. 8. 2D Features - Anti-aliasing TIITC2009 : Customer Centric Strategies Coverage proportional to (1/x) Many advances – Supersampling is common now in all GPUs x
    9. 9. 2D Features - Compositing TIITC2009 : Customer Centric Strategies SRC 1 -a src DST 1-a dst a dst SRC & DST SRC & NO-DST NO-DST & NO-SRC Compositing
    10. 10. Porter-Duff Operations <ul><li>Composition modes were originally proposed by Thomas Porter and Tom Duff in the article Compositing Digital Images in 1984 </li></ul><ul><ul><li>Describes a model for combining the pixels in a source image with the pixels in a destination, there are 12 basic composition modes </li></ul></ul><ul><li>Formula for combining source and destination pixels </li></ul><ul><ul><ul><li>c dst and c src represent color components of destination and source </li></ul></ul></ul><ul><ul><ul><li>a dst and a src represent alpha components of destination and source </li></ul></ul></ul><ul><ul><ul><li>C dst and C src are the color components pre-multiplied by the alpha components </li></ul></ul></ul><ul><ul><ul><li>The parameters X , Y and Z are either 0 or 1. </li></ul></ul></ul><ul><li>Permutations of X, Y, Z, and f produce various composition modes </li></ul><ul><ul><li>Most Common = SRC, SRC_OVER </li></ul></ul>TIITC2009 : Customer Centric Strategies
    11. 11. A typical 2D Graphics HW Feature Set <ul><li>Bit blit feature supporting all P-D combinations and ROPs and blending </li></ul><ul><li>Data expansion (Gray scale to full colour) </li></ul><ul><li>Line/curve drawing based on paths </li></ul><ul><li>Area/pattern filling </li></ul><ul><li>Anti-aliased drawing with enable/disable </li></ul><ul><li>Alpha </li></ul><ul><li>Color-keys </li></ul><ul><li>Hardware cursor support </li></ul><ul><li>SW implementations also have to take care of Glyphs/cache </li></ul>TIITC2009 : Customer Centric Strategies
    12. 12. 3D vs 2D <ul><li>Windowing systems already comprehend z-order </li></ul><ul><li>3D != 2D with depth </li></ul><ul><li>So what is 3D ? </li></ul>TIITC2009 : Customer Centric Strategies
    13. 13. Pixology – Video & Graphics <ul><li>Video Compression </li></ul><ul><ul><li>Analytic approach to describe a scene </li></ul></ul><ul><ul><ul><li>Describes a scene using its pixel values. Quality is dependent on the source content/ encoder. </li></ul></ul></ul><ul><ul><ul><li>Encoder can be highly complex, decoder is constrained by spec </li></ul></ul></ul><ul><ul><ul><li>Residual compressor </li></ul></ul></ul><ul><li>OpenGL </li></ul><ul><ul><li>Object based </li></ul></ul><ul><ul><ul><li>Describes a scene using its components and properties. Output quality is dependent on the renderer, not source </li></ul></ul></ul><ul><ul><ul><li>GL authoring tools are complex, but quality depends on renderer </li></ul></ul></ul><ul><ul><ul><ul><li>GL Spec is intentionally non-pixel compliant ! </li></ul></ul></ul></ul><ul><ul><ul><li>Each frame rendering is independent of any other </li></ul></ul></ul><ul><ul><li>Complexity now dependent on shaders !! </li></ul></ul>TIITC2009 : Customer Centric Strategies Basics
    14. 14. 3D Concepts TIITC2009 : Customer Centric Strategies Light Source Shadows, Realism 1 2 3 4 This is hidden by another object ! Not visible in the current Viewport !
    15. 15. 3D Graphics <ul><li>3D Graphics APIs allow recreating a natural scene, by using </li></ul><ul><ul><li>One or more light sources </li></ul></ul><ul><ul><li>Replicating natural phenomena like reflections, shadows, surface textures, and cloth surface properties like bending or water properties like waves </li></ul></ul><ul><ul><li>A specified viewport from where viewer is looking </li></ul></ul><ul><li>This emphasis on recreating realistic “Physics”, has resulted in driving computational complexity higher and higher, in GPUs </li></ul><ul><li>In order to accelerate this in HW, several standard blocks are defined </li></ul><ul><li>What is OpenGL anyway ? </li></ul>TIITC2009 : Customer Centric Strategies
    16. 16. Key OpenGL Terminologies <ul><li>OpenGL </li></ul><ul><ul><li>An API from Khronos (from SGI), for constructing a 3D object, doing operations on the object, and displaying it </li></ul></ul><ul><li>Primitives </li></ul><ul><ul><li>Triangles, Lines, Points, that can be specified through vertices to define an arbitrary shape </li></ul></ul><ul><li>Texture </li></ul><ul><ul><li>Small (!) bitmap to make objects more realistic </li></ul></ul><ul><li>EGL </li></ul><ul><ul><li>The EGL API defines a portable mechanism for creating GL contexts and windows for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer </li></ul></ul><ul><li>Shader Language </li></ul><ul><ul><li>A C like language, for writing vertex and fragment shaders. For GL ES2.0. </li></ul></ul>TIITC2009 : Customer Centric Strategies
    17. 17. The OpenGL Bazaar TIITC2009 : Customer Centric Strategies OPENGL Full version ES version Common Common-Lite GLSL companion GLSL-ES companion What we miss in ES compared to desktop version: Polygons, Display lists, Accumulation buffers,… Currently in 3.0 Currently in 2.0 Currently in 1.0.16 Currently in 1.20 EGL Currently in 1.3 Core GL Spec Vertex/Frag shaders Platform interface and Configuration setup EGL Currently in 1.3 2.0 vs 1.1
    18. 18. ES2.0,ES1.1 and Full Function <ul><li>ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1 </li></ul><ul><ul><li>Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write code to generate these effects. </li></ul></ul><ul><ul><li>More flexibility, little more work, but lot of available code </li></ul></ul><ul><ul><li>Android uses ES 1.1 </li></ul></ul><ul><li>ES2.0 NOT backward compatible with ES1.1 </li></ul><ul><li>See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification </li></ul><ul><li>Ok, so what exactly is the OpenGL pipeline ? </li></ul>TIITC2009 : Customer Centric Strategies
    19. 19. ES2.0 Pipeline TIITC2009 : Customer Centric Strategies
    20. 20. Sample OpenGL ES 2.0 program TIITC2009 : Customer Centric Strategies Handle = get_platform_specific_window_handle(); eglGetDisplay (handle); eglInitialize (); eglBindAPI ( EGL_OPENGL_ES_API ); eglChooseConfig (); eglCreateWindowSurface (); eglCreateContext (); eglMakeCurrent (); //Compile and Load the shaders before this step … float afVertices [] = {G,H,F, F,H,E, E,A,F, F,A,B, …}; glEnableVertexAttribArray (0); glVertexAttribPointer ( VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) ; eglSwapBuffers(eglDisplay, eglSurface); Setup Actors Show Keep showing the actors, in a loop, change view/position/texture .. How does the shader code look like ?
    21. 21. Simple Shader <ul><li>char* pszFragShader = &quot; </li></ul><ul><li>uniform sampler2D sTexture; </li></ul><ul><li>varying mediump vec2 TexCoord; </li></ul><ul><li>void main (void) </li></ul><ul><li>{ </li></ul><ul><li>gl_FragColor = texture2D(sTexture, TexCoord); </li></ul><ul><li>}&quot;; </li></ul><ul><li>char* pszVertShader = &quot; </li></ul><ul><li>attribute highp vec4 inVertex; </li></ul><ul><li>attribute mediump vec2 inTexCoord; </li></ul><ul><li>uniform mediump mat4 MVPMatrix; </li></ul><ul><li>varying mediump vec2 TexCoord; </li></ul><ul><li>void main() </li></ul><ul><li>{ </li></ul><ul><li>gl_Position = MVPMatrix * inVertex; </li></ul><ul><li>TexCoord = inTexCoord; </li></ul><ul><li>}&quot;; </li></ul>TIITC2009 : Customer Centric Strategies GL ES 2.0 permits shader code to be precompiled and loaded, or dynamically compiled and loaded
    22. 22. Alpha Blending <ul><li>Many options for blending – </li></ul><ul><ul><li>Multi-textures, Blending with Framebuffer, Can be looped… </li></ul></ul><ul><li>Blending with framebuffer </li></ul><ul><ul><li>glEnable(GL_BLEND); //disabled by default </li></ul></ul><ul><ul><li>glBlendEquation(GL_FUNC_ADD); //many other equations available </li></ul></ul><ul><ul><li>glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other modes available </li></ul></ul>TIITC2009 : Customer Centric Strategies
    23. 23. Texturing using OpenGL <ul><li>Pass Texture coordinates </li></ul><ul><ul><li>glEnableVertexAttribArray (); </li></ul></ul><ul><ul><li>glVertexAttribPointer (); </li></ul></ul><ul><li>Use glTexImage2D () to load image buffer for texture </li></ul><ul><li>Apply texture in the fragment shader program </li></ul><ul><ul><li>gl_FragColor = texture2D( ); </li></ul></ul><ul><li>OMAP3 supports ARGB 8888 textures </li></ul><ul><ul><li>Also YUV textures using extensions (IMG_STREAM) </li></ul></ul>TIITC2009 : Customer Centric Strategies When we do texturing, scaling is automatic
    24. 24. Rotation with OpenGL <ul><li>Rotation is accomplished by modifying the viewing angle </li></ul><ul><li>Update the Model View Projection Matrix before drawing </li></ul><ul><li>Update the vertex position in the vertex shader </li></ul><ul><ul><li>gl_Position = MVPMatrix * inVertex; // </li></ul></ul><ul><li>Along with texturing, can be used to create “wet floor” effects </li></ul><ul><ul><li>Apply texture in reverse way </li></ul></ul>TIITC2009 : Customer Centric Strategies
    25. 25. Texturing with SGX - Example TIITC2009 : Customer Centric Strategies An example Window manager using texturing and SGX only Efficient Rendering
    26. 26. Efficient Rendering Paths TIITC2009 : Customer Centric Strategies Discussion: How many ways can we implement this ? ..SGX Core Display Draw
    27. 27. Overview of the SGX core TIITC2009 : Customer Centric Strategies SGX uses “deferred” rendering REF: POWERVR SGX.OpenGL ES 2.0 Application Development Recommendations.1.8f.External.pdf (from AnandTech)
    28. 28. Measuring Performance <ul><li>Performance of Graphics is not a single number, but a collection of parameters </li></ul><ul><li>Almost all operations other than texturing, take about 10-15% of 500 MHz of Cortex-A8. 2D Texturing can take 100% CPU depending on size of texture </li></ul><ul><li>Optimised 2D Texturing methods using SGXSink bring CPU utilisation to minimum </li></ul>TIITC2009 : Customer Centric Strategies DDR Bandwidth Raw Throughput Quality Raw Throughput Quality CPU Utilisation
    29. 29. Benchmarking Tools <ul><li>Throughput measured under 2 cases, </li></ul><ul><ul><li>Vertex Throughput </li></ul></ul><ul><ul><li>Pixel Throughput </li></ul></ul><ul><li>Vertex throughput is very important for gaming scenerios (Millions of vertices) </li></ul><ul><li>UI/ Blit use cases have more dependency on Pixel throughput </li></ul><ul><li>SGXPERF – tool allows benchmarking of various texturing methods (2D Pixel throughput) </li></ul><ul><ul><li> </li></ul></ul><ul><li>How do we compare against ARM-NEON, and all APIs available on SGX ? </li></ul>TIITC2009 : Customer Centric Strategies
    30. 30. SGX Debugging Tools <ul><li>For debugging, PDUMP tools are provided that is similar to a stack trace – useful for debugging crash scenarios during power management transitions </li></ul><ul><li>In addition, Imagination Tech provides PVRTUNE for tuning the performance </li></ul><ul><ul><li>Shows loading of each shader, number of tasks </li></ul></ul>TIITC2009 : Customer Centric Strategies GPU futures…
    31. 31. Texture as data <ul><li>Fragment Shader is primarily SIMD, processing all pixels as a parallel stream </li></ul><ul><li>Each computation is independent of the other </li></ul><ul><li>Vertex Shader can do MIMD processing, but General Purpose processing is done with the more efficient Fragment Shader </li></ul><ul><li>Right way to represent data arrays is as a Texture Stream </li></ul>TIITC2009 : Customer Centric Strategies
    32. 32. The OpenCL standard <ul><li>OpenCL/DirectCompute take the concept of multi-core programming to a common model across the industry ! </li></ul><ul><li>Each core is considered a “Compute Unit” </li></ul><ul><li>The host can “submit” “kernels” (like FFT or DCT or any C function) to any Compute Unit </li></ul><ul><li>Event interactions are defined, so that results of computation can be retrieved </li></ul><ul><li>Textures as Arrays </li></ul><ul><ul><li>N elements of 1D array, as AxB 2D Texture, where AxB = N </li></ul></ul><ul><ul><li>1:1 mapping of buffers to be used </li></ul></ul><ul><li>GPU Computing != GP.GPU which is shader based </li></ul><ul><li>Refer OpenCL Cheatsheet </li></ul>TIITC2009 : Customer Centric Strategies Possibilities
    33. 33. Possibilities of GPU moving forward <ul><li>General Purpose programming - C/C++ (+) OpenGL (OpenCL) </li></ul><ul><ul><li>More complicated because, Games just have to look good, Computations have to be accurate ! </li></ul></ul><ul><ul><li>Will make more processing power available to general applications ! </li></ul></ul><ul><li>Integrated Web with OpenGL (WebGL) </li></ul><ul><li>Medical Imaging applications </li></ul><ul><li>Usage of higher computing power - Raytracing, Physics processing </li></ul>Conclusion
    34. 34. Conclusion <ul><li>Core clocks are not going to get much faster, but multi-core is here to stay </li></ul><ul><li>Requirements for 2D Graphics vastly different from 3D Graphics </li></ul><ul><li>Continued focus needed on enabling UI frameworks like QT, Browsers using 2D accelerators </li></ul><ul><li>Full usage of 3D in commercial UI is soon approaching - Video texturing, Coverflow are examples that show that it is possible </li></ul><ul><li>Video+Graphics+display integration is key to having a high texturing performance </li></ul>TIITC2009 : Customer Centric Strategies
    35. 35. References <ul><li>Porter & Duff Operations </li></ul><ul><li>SGXPERF </li></ul><ul><li>The SGX architecture used in IPhone </li></ul><ul><ul><li>http:// =3579 </li></ul></ul><ul><li>The OpenGL Redbook </li></ul><ul><ul><li> </li></ul></ul><ul><li>OpenGL ES2.0 programming handbook </li></ul><ul><ul><li>OpenGL® ES 2.0 Programming Guide, Aaftab Munshi ; Dan Ginsburg; Dave Shreiner </li></ul></ul><ul><li>Khronos organisation (OpenGL, OpenMAX, OpenVG ..) </li></ul><ul><ul><li>http:// </li></ul></ul>TIITC2009 : Customer Centric Strategies
    36. 36. Which Graphics ? TIITC2009 : Customer Centric Strategies
    37. 37. Graphics Cheatsheet TIITC2009 : Customer Centric Strategies Name Author Description Cairo Cairo-graphics API that can take care of interfacing to multiple backends like X, DirectFB etc Flash Adobe Application framework for games, animation, video and others Clutter Intel UI development framework, based completely on OpenGL, with different backends Gst-Gl Gstreamer Plugin for allowing video to be played as OpenGL ES textures XGL,WGL,EGL Multiple Interfaces for native platform support for interfacing to the windowing system OpenGL/ES Khronos An API for 3D Graphics programming OpenCL Khronos An API for multi core programming QT Nokia/Trolltech An application framework to develop native applications Silverlight Microsoft An API primarily for Web Application programming, but also for native .NET Microsoft An API for application programming, with C# language Surfaceflinger Android/Google Framework for surface creation and managing drawing DirectFB DirectFB Framework for surface creation and managing drawing, mostly using HW support X11 X org A windowing system/Manager very common on Linux system DirectX Microsoft A collective name for interfaces like DirectDraw etc. Currently in DirectX11 version. Handles 3D and 2D GDI Microsoft Native API for Win32 programming Opera Opera Optimised Browser for embedded platforms Mesa3D/Gallium Mesa An Opensource implementation of OpenGL API on Linux, can use HW if present. Moving to Gallium3D Vincent3D Vincent An Opensource pure software implementation of OpenGL API on WinCE/Mo SDL SDL A UI API that is cross platform, primarily meant for Games but also used as backend for rendering GLEW - Linux library that makes it easier for an OpenGL application to use GL extensions GLUT - Cross platform library, having window system and GL utility funcs on different platforms