The document discusses various architectures and implementations for graphics, focusing on both 2D and 3D graphics, the OpenGL API, and benchmarking methods. It highlights consumer-centric strategies, graphics frameworks, and different rendering processes, particularly in web applications, as well as the complexities of implementing shaders and textures for efficient performance. Additionally, it addresses the evolution of GPU computing and future directions for graphics technology, emphasizing the integration of video and graphics for enhanced performance.

1. Engaging the consumer with visual effects – Architectures and implementations for graphics Prabindh Sundareson ( [email_address] ) Texas Instruments India TIITC2009 : Customer Centric Strategies DSPS

2. Outline 2D Graphics 3D Graphics The OpenGL API Internals of SGX, benchmarking Future roadmap Conclusion [Graphics Cheatsheets] TIITC2009 : Customer Centric Strategies

3. What can Graphics do for me ? TIITC2009 : Customer Centric Strategies http://cg.cs.tsinghua.edu.cn:8080/cmm/?page_id=155

4. Graphics Frameworks Cheatsheet TIITC2009 : Customer Centric Strategies FB, V4l2 Cairo Clutter, QT, Opera DirectFB Surfaces X Window System/Mgrs Application Frameworks Buffers / blits Note: Cheatsheet table in appendix GDI DDr a w Win32/ Windowing System Win32, .NET, Silverlight

5. 2D Graphics Almost all existing Consumer Graphical User Interfaces are still built around 2D, and not around OpenGL Exception of course, iPhone, other Mobile phones, Games… Specific operations can still be accelerated with 3D accelerators. Like blits of large sizes What are the typical 2D applications ? All Windowing Systems (X, Win32, …) Application Frameworks like QT, Flash Browsers All 3D operations ultimately end in 2D blitting (ex, EGL uses PVR2D) An example, a Browser application… TIITC2009 : Customer Centric Strategies

6. Case study: Webkit’s Call Sequence TIITC2009 : Customer Centric Strategies FillRectangles () BatchBlit() Blit(), mem_copy() Clear() Blit() TileBlit() FILL RECTANGLE SHOW GLYPHS SET CLIP REGION ACQUIRE FONT CACHE ALLOCATE FONT CACHE SURFACE COMPOSITE PREPARE/FINISH COMPOSITE

7. Summary of 2D – Browser Rendering Processes To render one page of Youtube, for example A large blit of size = page width x block size ~ typical (500 x 100 pix) Followed by multiple blits of a glyph cache line of one line of text (page width x text height ) ~ ex (50 x 8), this renders blocks line by line Followed by multiple compositing operations for each block to render the whole page ~ ex (16 x 16 to page size) TIITC2009 : Customer Centric Strategies

8. 2D Features - Anti-aliasing TIITC2009 : Customer Centric Strategies Coverage proportional to (1/x) Many advances – Supersampling is common now in all GPUs x

9. 2D Features - Compositing TIITC2009 : Customer Centric Strategies SRC 1 -a src DST 1-a dst a dst SRC & DST SRC & NO-DST NO-DST & NO-SRC Compositing

10. Porter-Duff Operations Composition modes were originally proposed by Thomas Porter and Tom Duff in the article Compositing Digital Images in 1984 Describes a model for combining the pixels in a source image with the pixels in a destination, there are 12 basic composition modes Formula for combining source and destination pixels c dst and c src represent color components of destination and source a dst and a src represent alpha components of destination and source C dst and C src are the color components pre-multiplied by the alpha components The parameters X , Y and Z are either 0 or 1. Permutations of X, Y, Z, and f produce various composition modes Most Common = SRC, SRC_OVER TIITC2009 : Customer Centric Strategies

11. A typical 2D Graphics HW Feature Set Bit blit feature supporting all P-D combinations and ROPs and blending Data expansion (Gray scale to full colour) Line/curve drawing based on paths Area/pattern filling Anti-aliased drawing with enable/disable Alpha Color-keys Hardware cursor support SW implementations also have to take care of Glyphs/cache TIITC2009 : Customer Centric Strategies

12. 3D vs 2D Windowing systems already comprehend z-order 3D != 2D with depth So what is 3D ? TIITC2009 : Customer Centric Strategies

13. Pixology – Video & Graphics Video Compression Analytic approach to describe a scene Describes a scene using its pixel values. Quality is dependent on the source content/ encoder. Encoder can be highly complex, decoder is constrained by spec Residual compressor OpenGL Object based Describes a scene using its components and properties. Output quality is dependent on the renderer, not source GL authoring tools are complex, but quality depends on renderer GL Spec is intentionally non-pixel compliant ! Each frame rendering is independent of any other Complexity now dependent on shaders !! TIITC2009 : Customer Centric Strategies Basics

14. 3D Concepts TIITC2009 : Customer Centric Strategies Light Source Shadows, Realism 1 2 3 4 This is hidden by another object ! Not visible in the current Viewport !

15. 3D Graphics 3D Graphics APIs allow recreating a natural scene, by using One or more light sources Replicating natural phenomena like reflections, shadows, surface textures, and cloth surface properties like bending or water properties like waves A specified viewport from where viewer is looking This emphasis on recreating realistic “Physics”, has resulted in driving computational complexity higher and higher, in GPUs In order to accelerate this in HW, several standard blocks are defined What is OpenGL anyway ? TIITC2009 : Customer Centric Strategies

16. Key OpenGL Terminologies OpenGL An API from Khronos (from SGI), for constructing a 3D object, doing operations on the object, and displaying it Primitives Triangles, Lines, Points, that can be specified through vertices to define an arbitrary shape Texture Small (!) bitmap to make objects more realistic EGL The EGL API defines a portable mechanism for creating GL contexts and windows for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer Shader Language A C like language, for writing vertex and fragment shaders. For GL ES2.0. TIITC2009 : Customer Centric Strategies

17. The OpenGL Bazaar TIITC2009 : Customer Centric Strategies OPENGL Full version ES version Common Common-Lite GLSL companion GLSL-ES companion What we miss in ES compared to desktop version: Polygons, Display lists, Accumulation buffers,… Currently in 3.0 Currently in 2.0 Currently in 1.0.16 Currently in 1.20 EGL Currently in 1.3 Core GL Spec Vertex/Frag shaders Platform interface and Configuration setup EGL Currently in 1.3 2.0 vs 1.1

18. ES2.0,ES1.1 and Full Function ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1 Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write code to generate these effects. More flexibility, little more work, but lot of available code Android uses ES 1.1 ES2.0 NOT backward compatible with ES1.1 See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification Ok, so what exactly is the OpenGL pipeline ? TIITC2009 : Customer Centric Strategies

19. ES2.0 Pipeline TIITC2009 : Customer Centric Strategies

20. Sample OpenGL ES 2.0 program TIITC2009 : Customer Centric Strategies Handle = get_platform_specific_window_handle(); eglGetDisplay (handle); eglInitialize (); eglBindAPI ( EGL_OPENGL_ES_API ); eglChooseConfig (); eglCreateWindowSurface (); eglCreateContext (); eglMakeCurrent (); //Compile and Load the shaders before this step … float afVertices [] = {G,H,F, F,H,E, E,A,F, F,A,B, …}; glEnableVertexAttribArray (0); glVertexAttribPointer ( VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) ; eglSwapBuffers(eglDisplay, eglSurface); Setup Actors Show Keep showing the actors, in a loop, change view/position/texture .. How does the shader code look like ?

21. Simple Shader char* pszFragShader = "\ uniform sampler2D sTexture; \ varying mediump vec2 TexCoord; \ void main (void)\ {\ gl_FragColor = texture2D(sTexture, TexCoord);\ }"; char* pszVertShader = "\ attribute highp vec4 inVertex;\ attribute mediump vec2 inTexCoord;\ uniform mediump mat4 MVPMatrix;\ varying mediump vec2 TexCoord;\ void main()\ {\ gl_Position = MVPMatrix * inVertex;\ TexCoord = inTexCoord;\ }"; TIITC2009 : Customer Centric Strategies GL ES 2.0 permits shader code to be precompiled and loaded, or dynamically compiled and loaded

22. Alpha Blending Many options for blending – Multi-textures, Blending with Framebuffer, Can be looped… Blending with framebuffer glEnable(GL_BLEND); //disabled by default glBlendEquation(GL_FUNC_ADD); //many other equations available glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other modes available TIITC2009 : Customer Centric Strategies

23. Texturing using OpenGL Pass Texture coordinates glEnableVertexAttribArray (); glVertexAttribPointer (); Use glTexImage2D () to load image buffer for texture Apply texture in the fragment shader program gl_FragColor = texture2D( ); OMAP3 supports ARGB 8888 textures Also YUV textures using extensions (IMG_STREAM) TIITC2009 : Customer Centric Strategies When we do texturing, scaling is automatic

24. Rotation with OpenGL Rotation is accomplished by modifying the viewing angle Update the Model View Projection Matrix before drawing Update the vertex position in the vertex shader gl_Position = MVPMatrix * inVertex; // Along with texturing, can be used to create “wet floor” effects Apply texture in reverse way TIITC2009 : Customer Centric Strategies

25. Texturing with SGX - Example TIITC2009 : Customer Centric Strategies An example Window manager using texturing and SGX only Efficient Rendering

26. Efficient Rendering Paths TIITC2009 : Customer Centric Strategies Discussion: How many ways can we implement this ? ..SGX Core Display Draw

27. Overview of the SGX core TIITC2009 : Customer Centric Strategies SGX uses “deferred” rendering REF: POWERVR SGX.OpenGL ES 2.0 Application Development Recommendations.1.8f.External.pdf (from AnandTech)

28. Measuring Performance Performance of Graphics is not a single number, but a collection of parameters Almost all operations other than texturing, take about 10-15% of 500 MHz of Cortex-A8. 2D Texturing can take 100% CPU depending on size of texture Optimised 2D Texturing methods using SGXSink bring CPU utilisation to minimum TIITC2009 : Customer Centric Strategies DDR Bandwidth Raw Throughput Quality Raw Throughput Quality CPU Utilisation

29. Benchmarking Tools Throughput measured under 2 cases, Vertex Throughput Pixel Throughput Vertex throughput is very important for gaming scenerios (Millions of vertices) UI/ Blit use cases have more dependency on Pixel throughput SGXPERF – tool allows benchmarking of various texturing methods (2D Pixel throughput) https://gforge.ti.com/gf/project/gleslayer/ How do we compare against ARM-NEON, and all APIs available on SGX ? TIITC2009 : Customer Centric Strategies

30. SGX Debugging Tools For debugging, PDUMP tools are provided that is similar to a stack trace – useful for debugging crash scenarios during power management transitions In addition, Imagination Tech provides PVRTUNE for tuning the performance Shows loading of each shader, number of tasks TIITC2009 : Customer Centric Strategies GPU futures…

31. Texture as data Fragment Shader is primarily SIMD, processing all pixels as a parallel stream Each computation is independent of the other Vertex Shader can do MIMD processing, but General Purpose processing is done with the more efficient Fragment Shader Right way to represent data arrays is as a Texture Stream TIITC2009 : Customer Centric Strategies

32. The OpenCL standard OpenCL/DirectCompute take the concept of multi-core programming to a common model across the industry ! Each core is considered a “Compute Unit” The host can “submit” “kernels” (like FFT or DCT or any C function) to any Compute Unit Event interactions are defined, so that results of computation can be retrieved Textures as Arrays N elements of 1D array, as AxB 2D Texture, where AxB = N 1:1 mapping of buffers to be used GPU Computing != GP.GPU which is shader based Refer OpenCL Cheatsheet TIITC2009 : Customer Centric Strategies Possibilities

33. Possibilities of GPU moving forward General Purpose programming - C/C++ (+) OpenGL (OpenCL) More complicated because, Games just have to look good, Computations have to be accurate ! Will make more processing power available to general applications ! Integrated Web with OpenGL (WebGL) Medical Imaging applications Usage of higher computing power - Raytracing, Physics processing Conclusion

34. Conclusion Core clocks are not going to get much faster, but multi-core is here to stay Requirements for 2D Graphics vastly different from 3D Graphics Continued focus needed on enabling UI frameworks like QT, Browsers using 2D accelerators Full usage of 3D in commercial UI is soon approaching - Video texturing, Coverflow are examples that show that it is possible Video+Graphics+display integration is key to having a high texturing performance TIITC2009 : Customer Centric Strategies

35. References Porter & Duff Operations SGXPERF The SGX architecture used in IPhone http:// www.anandtech.com/printarticle.aspx?i =3579 The OpenGL Redbook http://www.glprogramming.com/red/chapter01.html OpenGL ES2.0 programming handbook OpenGL® ES 2.0 Programming Guide, Aaftab Munshi ; Dan Ginsburg; Dave Shreiner Khronos organisation (OpenGL, OpenMAX, OpenVG ..) http:// Khronos.org TIITC2009 : Customer Centric Strategies

36. Which Graphics ? TIITC2009 : Customer Centric Strategies

37. Graphics Cheatsheet TIITC2009 : Customer Centric Strategies Name Author Description Cairo Cairo-graphics API that can take care of interfacing to multiple backends like X, DirectFB etc Flash Adobe Application framework for games, animation, video and others Clutter Intel UI development framework, based completely on OpenGL, with different backends Gst-Gl Gstreamer Plugin for allowing video to be played as OpenGL ES textures XGL,WGL,EGL Multiple Interfaces for native platform support for interfacing to the windowing system OpenGL/ES Khronos An API for 3D Graphics programming OpenCL Khronos An API for multi core programming QT Nokia/Trolltech An application framework to develop native applications Silverlight Microsoft An API primarily for Web Application programming, but also for native .NET Microsoft An API for application programming, with C# language Surfaceflinger Android/Google Framework for surface creation and managing drawing DirectFB DirectFB Framework for surface creation and managing drawing, mostly using HW support X11 X org A windowing system/Manager very common on Linux system DirectX Microsoft A collective name for interfaces like DirectDraw etc. Currently in DirectX11 version. Handles 3D and 2D GDI Microsoft Native API for Win32 programming Opera Opera Optimised Browser for embedded platforms Mesa3D/Gallium Mesa An Opensource implementation of OpenGL API on Linux, can use HW if present. Moving to Gallium3D Vincent3D Vincent An Opensource pure software implementation of OpenGL API on WinCE/Mo SDL SDL A UI API that is cross platform, primarily meant for Games but also used as backend for rendering GLEW - Linux library that makes it easier for an OpenGL application to use GL extensions GLUT - Cross platform library, having window system and GL utility funcs on different platforms