Advanced Graphics Workshop - GFX2011

Advanced Graphics Workshop - Prabindh Sundareson, Texas Instruments GFX2011 Dec 3 rd 2011 Bangalore Note: This slide set is a public version of the actual slides presented at the workshop

GFX2011 8.30 AM [Registration and Introduction, Equipment setup] 9.00 AM Why Graphics ? Present and Future – Prof. Vijay Natarajan, Assistant Professor, Department of Computer Science and Automation, IISc, Bangalore 9.45 AM Introduction to the OpenGL/ES Rendering pipeline, and algorithms Detailed walkthrough of the OpenGL ES2.0 spec and APIs – Part 1 1.00 PM [Lunch] Detailed walkthrough of the OpenGL ES2.0 spec and APIs – Part 2 - Break - Framework and platform integration - EGL, Android (SurfaceFlinger) Tools for performance benchmarking, and Graphics Development Q&A, Certificate presentation to participants – Networking

Detailed Agenda Inaugural Talk – Dr. Vijay Natarajan – “Graphics – Present and Future” - Break GPU HW Architectures, and the GL API The CFF APIs Lab 1 Texturing objects Lab 94 – Rectangle wall Vertices and Transformations Lab 913 – Eyeing the eyes Lunch Break Real-life modeling and loading 3D models Lab 9 – Review .obj file loading method Shaders – Vertex and Fragment Lab 96 – Squishing the slices - Break Rendering targets Lab 910 – 3D in a 2D world Creating special effects EGL – and Platform Integration Overview of GLES2.0 usage in Android/ iOS *This PPT is to be used in conjunction with the labs at http://www.gpupowered.org

GPU HW Architectures CPUs are programmed with sequential code Typical C program – linear code Well defined Pre-fetch architectures, cache mechanisms Problem ? Limited by how fast “a” processor can execute, read, write GPUs are parallel Small & same code, multiple data Don’t care - control dependencies If used, drastically reduces throughput “ Output” is a result of a matrix operation (n x n) Graphics output – color pixels Computational output – matrix values

GPU integrated SOCs The A5 chipset CPU size ~= GPU size

Integrated GPU architectures (Samsung SRP) From – Khronos presentation

Vivante GPU Architecture GPUs vary in Unified vs separate shader HW architecture internal cache size Bus size rendering blocks Separated 2D and 3D blocks From - http://www.socip.org/socip/speech/pdf/2-Vivante-SoCIP%202011%20Presentation.pdf Spec evolution

OpenGL specification evolution Reading the spec

A note on reading the GLES specifications It must be clear that GLES specification is a derivative of the GL specification It is recommended to read the OpenGL 2.0 specification first Then read the OpenGL ES2.0 specification Similarly, for shading language It is recommended to read the OpenGL SL specification first Then read the OpenGL ES SL specification Extensions

What are Extensions ? Extension types OES – Conformance tested by Khronos EXT – Extension supported by >1 IP vendor Proprietary (vendor_ prefix) – Extension from 1 IP vendor How to check for extensions ? getSupportedExtensions (WebGL), getExtension() glGetString (openGL ES) Number of extensions OpenGL  400 + OpenGL ES  100+ Dependencies

OpenGL Dependencies OpenGL depends on a number of external systems to run A Windowing system (abstracted by EGL/ WGL/ XGL …) External inputs – texture files, 3D modelling tools, shaders, sounds, … OpenGL is directly used by OS/ Driver developers (ofcourse!) HW IP designers Game studios (optimisation) Researchers (Modelling, Realism, ) … Tools developers Application developers do not generally program on OpenGL, but rather do it on an Android API binding, or Java binding GL vs ES

OpenGL 2.0 vs OpenGL ES2 All Fixed function functionality is removed Specific drawing calls like Fog etc Matrix setup Replaced with programmable entities, GLES SL is ~= GL SL Compatibility issues GL  GLES Shaders GLSL does not enable fixed function state to be available ex, gl_TexCoord To enable compatibility, Architecture Review Board (ARB) extension – “ GL_ARB_ES2_compatibility” http://www.opengl.org/registry/specs/ARB/ES2_compatibility.txt Good reference http://developer.amd.com/gpu_assets/GDC06-GLES_Tutorial_Day-Munshi-OpenGLES_Overview.pdf GLES API

The OpenGL ES API From the Khronos OpenGL ES Reference Card “ OpenGL ® ES is a software interface to graphics hardware. The interface consists of a set of procedures and functions that allow a programmer to specify the objects and operations involved in producing high-quality graphical images, specifically color images of three-dimensional objects” Keep this document handy for API reference http://www.khronos.org/opengles/sdk/docs/reference_cards/OpenGL-ES-2_0-Reference-card.pdf Client server

The GL Client – Server Model Client (application on host), server (OpenGL on GPU) Server can have multiple rendering contexts, but has a global state Client will connect to one of the contexts at any point of time Client can set the states of the server by sending commands Further API calls will thus be affected by the previous states set by the client Server expects not-to-be interrupted by the client during operation Inherent nature of the parallel processor GL,SL, EGL spec versions

OpenGL Specifications OPENGL Full version ES version Common Common-Lite GLSL companion GLSL-ES companion What we miss in ES compared to desktop version: Polygons, Display lists, Accumulation buffers,… Currently in 4.0+ Currently in 2.0 Currently in 1.0.16 Currently in 1.20 EGL Currently in 1.3 Core GL Spec Shader Spec Platform Integration EGL Currently in 1.3 Programming Flow

Programming in OpenGL / ES Step1: Initialise EGL for rendering – context, surface, window Step2: Describe the scene (VBOs, Texture coordinates) – objects with Triangles, lighting Step3: Load the textures needed for the objects in the scene Step4: Compile the Vertex and Fragment Shaders Step 5: Select the output target (FBO, Fb, Pixmap …) Step5: Draw the scene Step 6 Run this in a loop Frames/vertex/basics

Preliminaries Pixel Throughput Memory bandwidth for a 1080P display @ 60 fps Did we forget Overdraw ? Vertex Throughput Vertex throughput for a 100k triangle scene Tearing Frame switch (Uniform) Driver frame draw (non-uniform) Real frame switch happens here Triangles

Why Triangles ? Connectivity, Simplicity, Cost + + ??? pipeline

ES2.0 Pipeline What do the APIs look like ? …

The GLES API – Overall view Global Platform Management Vertex Operations Texture Operations Shader Operations Rendering Operations VBOs Attaching attributes Attaching textures Loading texture data Mipmaps Loading, compiling, Linking to program Binary shaders Rendering to Framebuffer Rendering to FBO RTT State Management CFF Front/Back facing Enable/Disable (culling, ..) Get/ Set uniforms egl, wgl, glx .. Antialiasing, Configuration Context Management Surface - window Threading models Context sharing EGL GL

Flush, and Finish Several types of rendering methods adopted by GPUs Immediate rendering Deferred rendering Tiled rendering (ex, QCOM_tiled_rendering) Immediate rendering – everyone is happy Except the memory bus! Deferred – the GPU applies its “intelligence” to do/not do certain draw calls/ portions of draw calls Used most commonly in embedded GPUs Flush() – The call ensures pending operations are kicked off, returns Finish() – The call ensures pending operations are kicked off, “waits for completion” , returns

A note on WebGL In native code, need to handle Surface, context For example, refer to this native code eglGetDisplay eglInitialize – for this display eglBindAPI – bind to GLES/ VG context eglChooseConfig – configure surface type, GLES1/2/VG, Depth/Stencil buffer size eglCreateWindowSurface / eglCreatePixmapSurface eglCreateContext eglMakeCurrent Platform initialisation in WebGL is handled by browser Only configurations are – stencil, depth, AA, Alpha, preserve No EGL calls in JS application code No multi-threading issues (not yet, but “workers” are coming) The hands on labs will focus on GL, not EGL Note : - GL context gets lost, when user account is locked/screen saver mode etc – Restart browser as required.

Programming ClearColor Clear – clears with color specified earlier Flush/ Finish “ setup”, “vertex”, “fragment”, “animate” functions in class Will be called by framework Clear web cache first time Login Open “Default Lab #1” - Copy Open “My labs” #1 – Paste code Change code in setupFunc() Save, Run again Lab #1 Below link contains introductory video for starting the labs: http://www.youtube.com/watch?v=TM6t2ev9MHk

A note on Binding, Buffer Objects What is “Binding” ? Binding a server to a client – ex, VBO to a texture All objects are associated with a context state Binding an object is ~ copying the object state  context Removes client  server movement everytime “ Xfer-once-to-server, keep the token, Use-multipletimes-later” Good practice to “unbind” after operations– set binding to 0/null to avoid rogue programs changing state of bound object Buffer-Objects Allows data to be stored on the “server” ie, the GPU memory, rather than client memory (via pointer) GPU can decide where to place it for the fastest performance

Correct approach of data transfers Generate Object (ex, glGenBuffers, glGenTextures ) Bind Object to an ID “xyz” ( glBindBuffer(xyz), .. ) Transfer data to Object ( glBufferData, glTexImage2D ) Unbind ( glBindBuffer(0) )  After this point, the data remains bound to “xyz” and is managed by GPU. Can be accessed later by referencing “xyz”  Applies to VBOs, Textures, … Note the implicit “no atomicity” – needs locking

Texturing basics Texture Formats available RGB* formats, Luminance only formats Relevance of YUV Texture Filtering Maps texture coordinates to object coordinates – think of wrapping cloth over object Mipmaps Local optimisation – use pre-determined “reduced” size images, if object is far away from viewer – as compared to filtering full image Objective is to reduce bandwidth, not necessarily higher quality Application can generate and pass through TexImage2D() GPU can generate using GenerateMipMap() Occupies more memory Uv mapping

Texturing 3D objects Mapping from a bitmap to a 3D object involves matching the texture coordinates to the object surface Texture coordinates are calculated along with the vertex coordinates 3D tools output Texture coordinates along with vertex information, for the scene Lab 12 (Sphere), Lab 13 compression

Texture Compression types GLES spec supports RGBA textures, Luminance … To reduce memory bandwidth, compression used Texture Compression major types PVRTC ETC1 Others Android primarily supports ETC1 iOS supports PVRTC (and no other) Extension support queryable using GL API queries How to store this information in an uniform manner ? Texture file formats PVRTC (using Textool converter from IMG) commonly used KTX file format KTX

Khronos KTX file format To render a texture, steps to be used today: Application needs apriori knowledge of texture type, format, storage type, mipmap levels, and filename or the buffer data itself Then load into server using TexImage2D() Proprietary formats exist to separate this application+texture dependency – ex, PVRT from IMG KTX file format from Khronos is a standard way to store texture information, and the texture itself See next slide for structure of the file

Texture coordinates to GPU Texture coordinates are passed to GPU as “Attributes” along with the vertices Gen-bind-bufferdata, then bindAttrib WebGL/Textures

Note on WebGL and Textures Because of the way file loads work on browsers (asynchronous), texture loading may not happen before the actual draw Expect black screen for a very short-while till the Texture image loads from the website On native applications, due to the synchronous nature of loading the texture this issue will not be present Programming

Programming with Textures bindTexture pixelStorei (webGL only) UNPACK_FLIP_Y_WEBGL texImage2D texParameteri TEXTURE_MAG_FILTER TEXTURE_MIN_FILTER Note: WebGL “null” binding instead of “0” Lab #2 (why not square ?) Lab #94 – The wall

Note on the lab hints Each session ends with a lab. The lab sessions online intentionally have errors that the reader has to debug to show the rendered object on screen. Keys are provided for each such lab at the end of the section in this PPT

Lab 94 – Texturing (Keys to remove errors) var indexArray = new Uint16Array([0, 1, 2, 2 , 1, 3]); var texCoordArray = new Float32Array([0,0, 10 ,0, 0, 10, 10,10]); context.enableVertexAttribArray( 1 ); context.vertexAttribPointer(1, 2 , context.FLOAT, context.FALSE, 0, 0);

What are vertices ? Vertices – Points defined in a specific coordinate axes, to represent 3D geometry Atleast 3 vertices are used to define a Triangle – one of the primitives supported by GLES

Vertex operations Where do vertices come from ? Output of Modelling tools Mesh rendering / transforms – optimisations For 2D operations (ex Window systems), just 2 triangles Attributes

Vertex Attributes A vertex is characterised by its position {x,y,z} {x,y,z} are floating point values Additionally, normals are required for directional lighting calculations in shader 3D Tools output the normal map also along with vertex information Additionally, texture coordinates are required Again, 3D tools output the texture coordinates Each HW implementation must support a minimum number of vertex attributes Maximum number can be queried using MAX_VERTEX_ATTRIBS CPU to GPU xfer

Vertices – CPU to GPU Optimising Vertex operations A 3D object will have a lot of “common” vertices Ex – Cube has 6*2 triangles, (6*2)*3 vertices, but only 8 “points” So rather than passing vertices, pass 8 vertices, and 36 indices to the vertices to reduce Bandwidth Indices can be 16bit, so reduce BW by ~50% GL_ELEMENT_ARRAY_BUFFER and GL_ARRAY_BUFFER STATIC_DRAW, DYNAMIC_DRAW Not uploading again and again but re-use What are Vertex Buffer Objects ? genBuffers (createBuffer in WebGL), binding, bufferData/offset and usage Usage of Index Buffers (ELEMENT_ARRAY_BUFFER) Cartesian

Translation matrix1.translate(X,0.0,0); X = 0 X = 0.4  Translation applied to all objects (effect is not dependent on depth of object)

Rotation x y z Rotation  Observe effect of x offset! Refresh M,V,P after every rotate -0 Lookat

Getting the eye to see the object “ Model” Matrix made the object “look” right Now make the object visible to the “eye” – The “View” Eye is always at the origin {0,0,0} So using matrices, move the current object to the eye “ LookAt” is implemented in many standard toolkits The LookAt transformation is defined by Viewpoint - from where the view ray starts (eye) A Reference point (where the view ray ends) – in middle of scene (center) A look-”up” direction (up) ex – gluLookAt Utility function Significant contributor of grey-hair Viewport

Viewport Transformation Convert from the rendering to the final screen size ie physical screen Define the viewport using glViewport() Viewport can be an area anywhere within the physical screen This takes care of aspect ratio Ex, square becomes rectangle in laptop After the transformation, successful triangles get to the rasterisation HW, and then to the Fragment shader HW optimisations

Summary - The Transformation Sequence Translation example mathematical step - w

HW Optimisations Not all triangles are visible HW can reject based on depth coverage Front-facing or back-facing (Culling) Culling is disabled by default per specification However, most HW do this optimisation by default to save on bandwidth/ later pixel processing Programming

Programming ! Recall the Bandwidth needs for the vertex transfers / frame Passing Vertices Create Buffer Object bindBuffer bufferData Indices are passed as type ELEMENT_ARRAY Passing Attributes bindAttribLocation enableVertexAttribArray vertexAttribPointer matrix.getAsArray() Lab #913 – Eyeing the eyes

Lab 913 – Keys for “Eyeing the eyes” var totalArcs = 36 ; //shadervertexsetup_tunnel texCoords.push( 10 *numArcs / totalArcs); texCoords.push( 10 *zslice / numZSlices); matrix1.scale(1.0, 1.0, 1.0 ); //not 15.0

Real-life modelling of objects 3D models are stored in a combination of Vertices Indices / Faces * Normals Texture coordinates Ex, .OBJ, 3DS, STL, FBX … f, v, v//norm, v/t, o Export of vertices => scaling to 1.0-1.0 Vertex normals vs face normals Materials (mtl), animations Problem of multiple indices not allowed in openGL Tools and Models Blender, Maya, … http://assimp.sourceforge.net/ - tool for importing multiple types http://www.blendswap.com/ - Blender models Tessellation of meshes can be aided by HW in GPUs

Programming Loading 3D models is an application functionality No new APIs from OpenGLES are needed A parser is required to parse the model files, and extract the vertex, attribute, normal, texture coordinate information Look through objdata.js in Lab #9

Vertices, Fragments - Revisited Vertices – Points defined in a specific coordinate axes, to represent 3D geometry Atleast 3 vertices are used to define a Triangle – one of the primitives supported by GLES Fragments The primitives are “rasterised” to convert the “area” under the primitive to a set of color pixels that are then placed in the output buffer Shader characteristics

Shader characteristics Uniforms – uniform for all shader passes Can be updated at run time from application Attributes – changes per shader pass Varying – Passed between vertex and fragment shaders Ex, written by Vertex shader, and used by Fragment shader gl_Position Programs Why do we need multiple programs in an application for offscreen animation, different effects MAX VARYING VECTORS – enum Inputs to shader

Inputs to the Shaders Vertex Shader Vertices, attributes, Uniforms Fragment Shader Rasterised fragments (ie, after rasteriser fixed function HW) Varyings from vertex shader Uniforms Shader types

Fragment Shaders A fragment is – a pixel belonging to an area of the target render screen (on-screen or off-screen) Primitives are rasterised, after clipping Fragment shader is responsible for the output colour, just before the post-processing operations A Fragment shader can operate on “1” fragment at a time Minimum number of “TEXTURE UNITS” is 8 Calculation of colors Colors are interpolated across vertices automatically (Ref Lab 6 in the hands-on session) – ie, “varyings” are interpolated in Fragment shaders during rendering Colors can be generated from a texture “sampler” Each HW has a specific number of “Texture Units” that need to be activated, and textures assigned to it for operation in the shader Additional information from vertex shader through “varyings” Outputs gl_FragColor Sample Frag shader

Program Each program consists of 1 fragment shader, and 1 vertex shader Within a program, all uniforms share a single global space Precision

Advanced Shaders Animation Environment Mapping Per-Pixel Lighting (As opposed to textured lighting) Bump Mapping Ray Tracers Procedural Textures CSS – shaders (HTML5 – coming up)

Programming with Shaders Pass in shader strings Compile, link, Use Set uniforms Do calculations Lab #96

Lab 96 – Keys for “Squishing the slice” uniform mediump float skyline; vec4 tempPos; tempPos = MVPMatrix * inVertex; tempPos.y=min(skyline, tempPos.y); //or, try below – one of the 2 tempPos.y=min(sin(inVertex.x*5.0)+cos(inVertex.y*2.0), tempPos.y); gl_Position = tempPos; var skylineLoc = context.getUniformLocation(sprogram,"skyline"); context.uniform1f(skylineLoc, -0.1); context.drawArrays(context.TRIANGLES, 0, vertexparams[1 ] /3 );

Rendering Targets A rendering context is required before drawing a scene. And a correponding Framebuffer Recall bindFramebuffer() It can be Window system Framebuffer (Fb) Offscreen buffer (Implemented in a Frame Buffer Object) FBO is not a memory area – it is information about the actual color buffer in memory, depth/ stencil buffers By default, rendering happens to the Window system framebuffer (ID ‘0’) Need

Need for offscreen rendering Special effects Refer the fire effect specified earlier (Multiple passes) Interfacing to “non-display” use-cases Ex, passing video through GPU, perform 3D effects, then re-encode back to compressed format Edge detection/ computation – output is sent to a memory buffer for use by other (non-GL) engines FBO

FrameBuffer Object A Frame Buffer Object Can be just a color buffer (ex, a buffer of size 1920x1080x 4) Typically also has depth/ stencil buffer By default – FBO – ID “0” is never assigned to new FBO It is assigned to Window system provided Frame Buffer (onscreen) Renderbuffers and Textures can be “attached” to FBO For RB – application has to allocate storage For FBO, the GL server will allocate the storage rtt

Render-To-Texture By binding a Texture to a FBO, the FBO can be used as Stage 1 – target of a rendering operation Stage 2 – used as a texture to another draw This is “Render-To-Texture” (RTT) This allows the flexibility of “discreetly” using the server to do 3D operations (not visible onscreen), then use this output as texture input to a visible object If not for RTT, we have to render to regular Framebuffer then do CopyTexImage2D() or readPixels() which are inefficient Offscreen rendering is needed for dynamic-reflections APIs

Post-processing operations Blending with Framebuffer - enables nice effects (Ref Lab #6) Standard Alpha-Blending glEnable ( GL_BLEND ); glBlendFunc ( GL_SRC_ALPHA, GL_ONE ); Is a “bad” way of creating effects Reads back previous framebuffer contents, then blend Makes application memory bound, specially at larger resolutions Stalls parallel operations within the GPU Recommended way is to perform Render-To-Texture, and blending where necessary in the shader But needed for medical image viewing – ex Ultrasound images, > 128 slices blending programming

Programming FBO and back to Fb glGenFramebuffers glBindFramebuffer Makes this FBO used glFramebufferTexture2D(id) Indicate ‘id’ is to be used for rendering to TEXTURE, so storage is different glDeleteFramebuffers Then, create separate object to texture with TEXTURE ‘id’ Then, use previous textureID id as input to texImage2D next Switching to FB Change binding to screen FB Load different set of vertices as needed, different program as needed Set texture binding to FBO texture drawn previously DrawElements call FBOs are used to do post-processing effects

Programming Draw a textured rectangle to a FBO Using this FBO as texture, render another rectangle on-screen CheckFramebufferStatus very important Lab #910

Lab 910 – Keys for “Render to Texture” lab Location, location, location ! Also note that readpixels doesn’t show anything! // context.clearColor(1.0, 0.0, 0.0, 1.0); // context.clear(context.COLOR_BUFFER_BIT | context.DEPTH_BUFFER_BIT); // context.flush();

Setting up the platform - EGL Context, Window, Surface OpenGL ES – EGL_SWAP_BEHAVIOR == “EGL_BUFFER_PRESERVED” Reduces performance Anti-aliasing configurations EGL_SAMPLES (4 to 16 typically, 4 on embedded platforms) WebGL - preserveDrawingBuffer – attribute Optimisations done if it is known that app is clearing the buffer – no dirty region check and whole scene is drawn efficiently Dirty region check made in some systems Android

Android Integration Details Android composition uses GLES2.0 mostly as a pixel processor, not a vertex processor Uninteresting rectangular windows, treated as a texture 6 vertices Blending of translucent screens/ buttons/ text 3D (GLES2.0) is natively integrated 3D Live wallpaper backgrounds Video morphing during conferencing (?) Use the NDK Surfaceflinger

Android SurfaceFlinger architecture Introduction to OpenGL interface on Android http://code.google.com/p/gdc2011-android-opengl/wiki/TalkTranscript HW acceleration on Android 3.0 / 4.0 http://android-developers.blogspot.com/2011/11/android-40-graphics-and-animations.html composition

Optimising OpenGL / ES applications Graphics performance is closely tied to a specific HW Size of interface to memory, cache lines HW shared with CPU – ex, dedicated memory banks Power vs Raw performance Intelligent Discarding of vertices/ objects (!) Performance is typically limited by Memory throughput GPU pixel operations per GPU clock CPU throughput for operations involving vertices Load balancing of units – within the GPU GPUs that are integrated into SOCs are more closely tied to the CPU for operations, than separate GPUs Ex, GPU drivers offload some operations to CPU debugging

Debugging OpenGL Vanishing vertices, Holes Improper lighting Missing objects in complex scenes Windows Tools Perfkit/ GLExpert / gDEBugger Intel GPA Linux Tools PVRTune (IMG) GDebugger Standard kernel tools Intel GPA Pixel vs Vertex throughput, CPU loading, FPS, Memory limited – tuning knobs

References Specs - http://khronos.org/opengles CanvasMatrix.js https://github.com/toji/gl-matrix Tools - http://www.iquilezles.org/apps/shadertoy/ http://www.inka3d.com/ (from Maya) http://assimp.sourceforge.net/ - Asset importer ARM – Mali – Architecture Recommendations http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0363d/CJAFCCDE.html Optimising games – simple tips http://glenncorpes.blogspot.com/2011/09/topia-optimising-for-opengles20.html

Appendix: Video and Graphics Graphics is computed creation Video is recorded as-is Graphics is object – based Video (today) is not Graphics is computed every frame fully Video is mostly delta sequences Motion-detection, construction, compensation But extensions like swap_region (Nokia) exist

Q & A, Feedback Feedback http://ewh.ieee.org/r10/bangalore/ces/gfx2011.html

Advanced Graphics Workshop - GFX2011

More Related Content

What's hot

Viewers also liked

Similar to Advanced Graphics Workshop - GFX2011

More from Prabindh Sundareson

Recently uploaded

Advanced Graphics Workshop - GFX2011