OpenGL ES based UI Development on TI Platforms


Published on

This presentation made at TI Developer Conference 2008, introduces the options available for developers to create User Interfaces on TI SGX based platforms.

Published in: Technology, Art & Photos
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OpenGL ES based UI Development on TI Platforms

  1. 1. Building advanced user interfaces using SGX Graphics core on OMAP3 Prabindh Sundareson ( [email_address] ) DSPS Texas Instruments
  2. 2. Outline <ul><li>Overview of OpenGL </li></ul><ul><li>Developing an UI with 3D components </li></ul><ul><li>Advanced 3D features – texturing, rotation, materials </li></ul><ul><li>Integrating the whole system </li></ul><ul><li>Performance and Power </li></ul><ul><li>Call For Action </li></ul><ul><li>Q & A </li></ul>
  3. 3. Video vs Graphics It’s a state machine - Setup the stage, lighting, actors… Then draw it ! Based on motion between frames OpenGL based Graphics Video
  4. 4. The OpenGL ES 2.0 Picture OPENGL Full version ES version Common Common-Lite GLSL companion GLSL-ES companion What we miss in ES compared to desktop version: Polygons, Display lists, Accumulation buffers,… Currently in 3.0 Currently in 2.0 Currently in 1.0.16 Currently in 1.20 EGL Currently in 1.3 Core GL Spec Vertex/Frag shaders Platform interface and Configuration setup EGL Currently in 1.3
  5. 5. ES2.0,ES1.1 and Full Function <ul><li>ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1 </li></ul><ul><ul><li>Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write code to generate these effects. </li></ul></ul><ul><ul><li>More flexibility, little more work, but lot of available code </li></ul></ul><ul><ul><li>Android uses ES 1.1 </li></ul></ul><ul><li>ES2.0 NOT backward compatible with ES1.1 </li></ul><ul><ul><li>See Companion paper for definitive comparison of available functions in ES1.1 and ES2.0 </li></ul></ul><ul><li>See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification </li></ul><ul><li>Ok, so what exactly is openGL ? </li></ul>
  6. 6. Primitives, Vertices … D B C A *Note the orientation
  7. 7. The pipeline… Vertex shader outputs position of vertex, calculates orientation Fragment shader outputs colour
  8. 8. Sample OpenGL ES 2.0 program Handle = get_platform_specific_window_handle(); eglGetDisplay (handle); eglInitialize (); eglBindAPI ( EGL_OPENGL_ES_API ); eglChooseConfig (); eglCreateWindowSurface (); eglCreateContext (); eglMakeCurrent (); //Compile and Load the shaders before this step … float afVertices [] = {G,H,F, F,H,E, E,A,F, F,A,B, …}; glEnableVertexAttribArray (0); glVertexAttribPointer ( VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) ; eglSwapBuffers(eglDisplay, eglSurface); Setup Actors Show Keep showing the actors, in a loop, change view/position/texture ..
  9. 9. Simple Shader <ul><li>char* pszFragShader = &quot; </li></ul><ul><li>uniform sampler2D sTexture; </li></ul><ul><li>varying mediump vec2 TexCoord; </li></ul><ul><li>void main (void) </li></ul><ul><li>{ </li></ul><ul><li>gl_FragColor = texture2D(sTexture, TexCoord); </li></ul><ul><li>}&quot;; </li></ul><ul><li>char* pszVertShader = &quot; </li></ul><ul><li>attribute highp vec4 inVertex; </li></ul><ul><li>attribute mediump vec2 inTexCoord; </li></ul><ul><li>uniform mediump mat4 MVPMatrix; </li></ul><ul><li>varying mediump vec2 TexCoord; </li></ul><ul><li>void main() </li></ul><ul><li>{ </li></ul><ul><li>gl_Position = MVPMatrix * inVertex; </li></ul><ul><li>TexCoord = inTexCoord; </li></ul><ul><li>}&quot;; </li></ul>Note: Read the GL ES Shading Language specification for details
  10. 10. OMAP3 + SGX530 <ul><li>The OMAP3 Graphics Engine supports </li></ul><ul><ul><li>OpenGL ES1.1 + EGL 1.3  Fixed function pipeline </li></ul></ul><ul><ul><li>OpenGL ES2.0 + EGL 1.3  Programmable function pipeline (shaders) </li></ul></ul><ul><ul><li>OpenVG 1.0.1 </li></ul></ul><ul><ul><li>Arbitrary texture sizes </li></ul></ul><ul><li>Several useful application libraries provided for </li></ul><ul><ul><li>Font support (Fixed fonts) </li></ul></ul><ul><ul><li>Loading textures </li></ul></ul><ul><ul><li>Loading compressed textures and with mip-maps </li></ul></ul><ul><ul><li>Loading effects files </li></ul></ul><ul><li>See the Graphics SDK for details </li></ul>
  11. 11. OMAP3 from a UI point of view <ul><li>High performance ARM Cortex-A8 </li></ul><ul><li>VFP Lite with NEON </li></ul><ul><ul><li>Floating point acceleration </li></ul></ul><ul><ul><li>Use the below GCC flags on Linux when compiling, to get best performance on Graphics applications </li></ul></ul><ul><ul><li>“ -mcpu=cortex-a8 -mfpu=neon -mfloat-abi=softfp” </li></ul></ul><ul><li>SGX core </li></ul><ul><li>Display subsystem that supports </li></ul><ul><ul><li>Output resolution upto 1280 x 720 (24 bit Digital) </li></ul></ul><ul><ul><li>Graphics/Video planes </li></ul></ul>
  12. 12. OMAP3 from a UI point of view <ul><li>Linux/WinCE </li></ul><ul><ul><li>QT/GTK based toolkits </li></ul></ul><ul><ul><li>Other 3Ps </li></ul></ul><ul><li>Browser as an OS independent User Interface – Opera, Firefox, other Webkit based browsers </li></ul><ul><ul><li>Many of these can use Cairo (See companion paper) as backend for 2D rendering </li></ul></ul><ul><li>Flash, Flash-lite </li></ul><ul><li>Java – M3G </li></ul><ul><li>How to integrate 3D graphics efficiently into these ? </li></ul>
  13. 13. Typical UI development with OpenGL ES <ul><li>Shapes  </li></ul><ul><ul><li>Convert to meshes based on primitives (ex, using 3DStudio plugins) </li></ul></ul><ul><li>Artwork  </li></ul><ul><ul><li>Texture, Mip Maps, filtering modes, lighting (in compressed or uncompressed formats) </li></ul></ul><ul><li>Animation  </li></ul><ul><ul><li>Path description, timing, dynamic changes, rotation (Write shader logic) </li></ul></ul><ul><li>Transparency  </li></ul><ul><ul><li>Alpha settings </li></ul></ul><ul><li>Programming the display path  </li></ul><ul><ul><li>Display driver interfaces and HW </li></ul></ul><ul><li>Performance tuning – </li></ul><ul><ul><li>Pipelining </li></ul></ul><ul><ul><li>Compressed textures </li></ul></ul>
  14. 14. Alpha Blending with SGX <ul><li>glEnable(GL_BLEND); //disabled by default </li></ul><ul><li>glBlendEquation(GL_FUNC_ADD); //many other equations available </li></ul><ul><li>glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other modes available </li></ul>
  15. 15. Texturing <ul><li>Gives more realistic views to otherwise “dull” objects </li></ul><ul><ul><li>Ex, Lighting effects </li></ul></ul><ul><li>Textures can be calculated programmatically, or can be taken from images specified by application at runtime (ex, live video thumbnails) </li></ul><ul><li>Can be applied to lines, or surfaces </li></ul><ul><li>Coordinates mapped in Vertex shaders </li></ul><ul><li>“ Applying the texture” done in Fragment shaders </li></ul><ul><li>Can have multiple textures that can be combined </li></ul><ul><ul><li>One for lighting effects, one for material properties (ex. Gloss vs Matt …) </li></ul></ul>
  16. 16. Adding Texturing using OpenGL <ul><li>Pass Texture coordinates </li></ul><ul><ul><li>glEnableVertexAttribArray (); </li></ul></ul><ul><ul><li>glVertexAttribPointer (); </li></ul></ul><ul><li>Use glTexImage2D () to load image buffer for texture </li></ul><ul><li>Apply texture in the fragment shader program </li></ul><ul><ul><li>gl_FragColor = texture2D( ); </li></ul></ul><ul><li>OMAP3 supports ARGB 8888 textures </li></ul>
  17. 17. Texturing with SGX - Example An example Window manager using texturing and SGX only
  18. 18. Rotation with OpenGL <ul><li>Rotation is accomplished by modifying the viewing angle </li></ul><ul><li>Update the Model View Projection Matrix before drawing </li></ul><ul><li>Update the vertex position in the vertex shader </li></ul><ul><ul><li>gl_Position = MVPMatrix * inVertex; // Recall the shader code </li></ul></ul><ul><li>Along with texturing, can be used to create “wet floor” effects </li></ul><ul><ul><li>Apply texture </li></ul></ul><ul><ul><li>Rotate object to make back side visible, or apply texture in reverse way (remember the orientation of primitives ?) </li></ul></ul><ul><ul><li>Need to disable CULLing to make back side visible if first method is used </li></ul></ul>
  19. 19. Key Decision Points <ul><li>Decide - Compositing also on SGX, or only 3D on SGX ? </li></ul><ul><ul><li>Decision should be based on ARM loading for the target UI </li></ul></ul><ul><ul><li>Benchmark before deciding – What other applications run on ARM while running the UI ? </li></ul></ul><ul><li>Use indexed vertices instead of strips for large number of vertices </li></ul><ul><ul><li>How many > 500 vertex objects are being submitted for rendering ? </li></ul></ul><ul><li>Avoid dependency on CPU for SGX, and SGX for CPU </li></ul><ul><li>Mip-Maps increase the size of textures </li></ul><ul><ul><li>But increase the quality discernibly, especially for near-far cases </li></ul></ul>
  20. 20. Getting the final image to the display <ul><li>Case 1 (Partner mode – work with other applications): </li></ul><ul><ul><li>OpenGL allows reading back the composited image from the gl frame buffer </li></ul></ul><ul><ul><ul><li>glReadPixels(xpos, ypos, xwidth, yheight, format, type, *userdataptr); </li></ul></ul></ul><ul><ul><ul><li>Caveat: This flushes the pipeline. Benchmark for your target application </li></ul></ul></ul><ul><ul><ul><li>Now the data is in the external application/driver </li></ul></ul></ul><ul><ul><ul><li>Regular alpha blending/compositing operations can be performed on this data </li></ul></ul></ul><ul><ul><li>OMAP3 Display System HW supports 3 display planes </li></ul></ul><ul><ul><ul><li>2 video planes with up/down scaling, 1 Graphics plane </li></ul></ul></ul><ul><ul><li>Any of these planes can be used for blending with outputs from other applications </li></ul></ul><ul><li>Case 2 (Full ownership mode) </li></ul><ul><ul><li>The final image can be directly written to the framebuffer in full screen mode by EGL itself </li></ul></ul>
  21. 21. Application Integration <ul><li>Multiple options available </li></ul><ul><li>Write separate application, directly use EGL for display to frame buffer </li></ul><ul><ul><li>Use SGX for all compositing and window operations </li></ul></ul><ul><ul><li>No other window management required </li></ul></ul><ul><li>Use application framework like Clutter to build your applications. Configure Clutter to use OpenGL-ES as backend </li></ul><ul><ul><li>More application support becomes available with this option, ex. Browsers .. </li></ul></ul><ul><ul><li>Clutter is still under active development </li></ul></ul><ul><li>Use GTK to manage the Graphics output created by SGX application </li></ul>
  22. 22. Performance and Power <ul><li>The SGX core is connected to the L3 interface (Refer SPRU66B) </li></ul><ul><ul><li>Internal DMA, MMU for operating without interfering ARM </li></ul></ul><ul><li>Tops at 111 MHz, other variants in the future </li></ul><ul><li>Power management handled internally in the SGX HW/driver </li></ul><ul><li>Power numbers comparable to ARM at equivalent MHz (contact TI support for details) </li></ul>
  23. 23. Performance and Power <ul><li>Target upto 30 fps with 512x512 (window like) textures in 2D </li></ul><ul><li>Frame rate goes down as number/size of textures, and complexity of objects and operations goes up </li></ul><ul><ul><li>Use compressed PVRTC mode textures for static textures </li></ul></ul><ul><li>Rendering full screen with OpenGL allows ARM to do other stuff </li></ul><ul><ul><li>Network stack, Browsers, Multimedia, … </li></ul></ul>
  24. 24. Call For Action <ul><li>Download the OMAP3 Graphics SDK </li></ul><ul><li>Use Beagle or EVM or other suitable OMAP3 platform for development </li></ul><ul><li>Use the examples provided to start, download examples from net </li></ul><ul><li>Use mailing list for support </li></ul><ul><li>Engage at the TI open-source symposium (7/11/08) </li></ul>
  25. 25. Support <ul><li>For support on Graphics related queries on OMAP3, write to </li></ul><ul><li>[email_address] </li></ul><ul><li>We will be glad to support you to create that great UI !! </li></ul>
  26. 26. References <ul><li>TRM for SGX core </li></ul><ul><ul><li> /lit/ug/spruff6b/spruff6b.pdf </li></ul></ul><ul><li>PC version of PowerVR SDK </li></ul><ul><ul><li> </li></ul></ul><ul><li>TI DVEVM software (Needs EVM serial #, and account for registration) </li></ul><ul><ul><li> </li></ul></ul><ul><li>The OpenGL Redbook </li></ul><ul><ul><li> </li></ul></ul><ul><li>OpenGL ES2.0 programming handbook </li></ul><ul><ul><li>OpenGL® ES 2.0 Programming Guide, Aaftab Munshi ; Dan Ginsburg; Dave Shreiner </li></ul></ul><ul><li>Khronos organisation </li></ul><ul><ul><li> </li></ul></ul><ul><li>Brief description of OMAP3 Graphics usage (Needs acc) </li></ul><ul><ul><li> </li></ul></ul>