Downloaded 146 times
![DIFFUSE SHADER
uniform sampler2D my_color_texture;
void main()
{
// Defining The Material Colors
const vec4 DiffuseColor = vec4(1.0, 0.0, 0.0, 1.0);
// Scaling The Input Vector To Length 1
vec3 normalized_normal = normalize(normal);
vec3 normalized_vertex_to_light_vector = normalize(vertex_to_light_vector);
// Calculating The Diffuse Term And Clamping It To [0;1]
float DiffuseTerm = clamp(dot(normal, vertex_to_light_vector), 0.0, 1.0);
// Calculating The Final Color
gl_FragColor = AmbientColor + DiffuseColor * DiffuseTerm;
}
Shader compiler
Input Fragment <diffuseShader>: Output Shaded Fragment
Sample r0,v4, t0, s0
mul r3, v0, cb0[0]
madd r3, v1, cb0[1], r3
madd r3, v2, cb0[2], r3
clmp r3, r3, 1(0.0), 1(1.0)
mul o0,r0,r3
mul o1,r1,r3
mul o2,r2,r3
mov o0, 1(1.0)](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-14-320.jpg)
![THE DRIVER ARCHITECTURE
Shader Compiler
User Space
OpenGL ES EGL
L2-Stack
Kernel Space
GPU Driver FB Driver
Hardware [ GPU + Display Controller ]](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-15-320.jpg)
![GRAPHICS PC EMULATOR
Shader Compiler
User Space
OpenGL ES EGL
MESA GL /
Proprietary OGL GLSL
Kernel Space
GPU Driver FB Driver
Hardware [ GPU + Display Controller ]](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-18-320.jpg)
![DRIVER ARCHITECTURE
Shader Compiler
User Space
OpenGL ES EGL
L2-Stack
Kernel Space
GPU Driver FB Driver Open Source
Hardware [ GPU + Display Controller ]](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-29-320.jpg)
![REALITY CHECK
OpenCL
• OpenCL compilation process on Compute Program
SnowLeopard (OSX 10.6) OpenCL
front-end [ clang ]
• Step1: Compile OCL to LLVM IR
LLVM IR
(Intermediate Representation)
• Step2: Compile to target device X86 LLVM
NVIDIA Back-End
• NVIDIA GPU device compiles the LLVM X86
Binary
IR in two steps: PTX IR
• LLVM IR to PTX (CUDA IR) – PTX to target
GPU
G80 G92 G200
Binary Binary Binary
• CPU device uses LLVM x86 BE to compile
directly to x86 Binary code.
http://www.haifux.org/lectures/212/OpenCL_for_Halifux_new.pdf](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-34-320.jpg)
![MISC REFERENCES IN SLIDES
• [Slide-1] http://www.nobodysplace.nl/pacman.jpg
• ATI RUBY : http://www.pcgameshardware.com/
&menu=browser&image_id=999114&article_id=680582&page=1&show=original
• http://cdn.erictric.com/wp-content/uploads/2010/06/samsung-galaxy-s-stock-shot.jpg
• http://media.photobucket.com/image/iPhone4/defskilz14/blog%20pics/torch_iphone4.jpg
• http://s08.idav.ucdavis.edu/fatahalian-gpu-architecture.pdf
• http://s08.idav.ucdavis.edu/fatahalian-gpu-architecture.pdf](https://image.slidesharecdn.com/graphicsdriversfossin-110215045012-phpapp02/85/Mobile-Graphics-The-Need-for-Open-Source-Drivers-36-320.jpg)
Uploaded byHarsha Padmanabha
Mobile Graphics, The Need for Open Source Drivers
The document discusses the importance and complexities of mobile graphics, emphasizing the need for open-source drivers and compatibility across various operating systems and hardware. It covers the architecture of mobile graphics such as OpenGL ES, the challenges in development due to fragmented ecosystems, and various case studies on mobile systems like Android and Meego. Additionally, it highlights the evolution of graphics APIs and standards necessary for efficient graphics rendering on mobile devices.
More Related Content
Similar to Mobile Graphics, The Need for Open Source Drivers
Mobile Graphics, The Need for Open Source Drivers
- 1. MOBILE GRAPHICS The need for Open Source Drivers & Stack Harsha Padmanabha http://soulbuzz.net
- 2. TOPICS • Mobile Graphics : An Introduction • Application Processors : Missing datasheets • Mobile graphics pipeline & cores •A developers perspective on Graphics & Games • Case Study : Developing a Android based product
- 3. TYPICAL MOBILE SOC Android,Linux, MeeGo, Symbian, ARM Windows Mobile Bus Interconnect Graphics Communication Audio Display Video OpenGL ES 1.1 OpenGL ES 2.0 Post Processing LTE, 3G, Wifi MP3, AAC, 3GPP DeInterlacing Bluetooth MPEG-1/2/4, H. ....... 264, VC-1
- 4. IPHONE : A4PROCESSOR • A4 Processor ARM A8 !"#$%&'( • Main processor for the Apple iPad • 1GHz ARM Cortex A8 45nm core • NEON SIMD • Cache size: VXD • L1I$=32KB • L1D$=32KB SGX 535 • L2=512KB • Graphics engine: • PowerVR SGX 3D engine from Imagination Technologies !""#$%!&%'()*$++)( ,-.%, .>;"%,;?$%@%/2%66A .> • Video Engine 2 ! !""#$%!&%3+4%,56+789%,/'.001! .)8:;<$8=;5# • PowerVR VXD / VXE : MultiProfile video encode/decode • 53mm² on 45nm LP www.ubmtechinsights.com/.../Apple%20A4%20vs%20SEC%20S5PC110A01.pdf
- 5. NVIDIA TEGRA 250 •8 separate cores Dual ARM A9 ARM 9 • Dual ARM Cortex A9 • ARM9 300MHz HD Decode •1 HD Encode, 1 Decode HD Encode • OGLES 2.0 Graphics core • Image Processor, for HD Camera • Lots of internal memories 2D/3D GPU http://www.anandtech.com/show/2911
- 6. MOBILE OS MARKETSHARE 2%2% 18% 41% 17% 5% 14% http://www.gartner.com/it/page.jsp?id=1421013 Symbian iOS Windows Mobile Android RIM Linux Others
- 7. TOP APPLICATIONS RANK CATEGORIES 1 Games 2 Social Networking 3 Mail & Messaging 4 Music 5 Entertainment 6 Weather 7 Sports 8 Education & Employment 9 News 10 Health & Fitness http://www.millennialmedia.com/wp-content/images/mobilemix/MM-MobileMix-Oct2.pdf
- 8. TODAY’S GRAPHICS PERFORMANCE Processor GPU Mhz (M) Tri/Sec (M) Pixels/Sec Texas Instruments PowerVR 130 28 500* OMAP SGX 535 ARM ST Micro 275 30 400* MALI 400 Marvell Vivante 250 25 375* ARMADA GC 800 References http://arm.com/products/multimedia/mali-graphics-hardware/mali-400-mp.php http://www.vivantecorp.com/p_mvr.html#GC800 * Estimates as per marketing data
- 9. VISUAL COMPUTING Authoring & Accessibility 3D Digital Asset Plugin-free 3D Mobile OS resource Exchange format Web Content abstraction High-level Steaming Media High-level Multimedia Frameworks Recording and Playback Enhanced Audio Inter-API Embedded 3D Interoperability Hub Acceleration Heterogeneous Parallel Programming Streaming Media and Image Processing Vector 2D Codec Creation Window System Acceleration System Integration All logos & trademarks are copyright www.khronors.org
- 10. MOBILE GRAPHICS API • OpenGL ES 1.1 • Fixed Function API derived from OpenGL 1.3/1.4 • OpenGL ES 2.0 • Complete programmable API, GLSL support derived from OpenGL 2.0 • OpenGL ES 3.0* • Future, planned for 2011/12, cross compatibility with OpenGL 4.0 • EGL 1.4 • Visual API interoperability hub • OpenCL 1.1 • Open compute, supports GPGPU on multicore architecture
- 11. OPENGL ES VERSIONS • OpenGL ES 1.1 – fixed-function pipeline • Based on OpenGL 1.5 • Vertex Arrays / Buffer Objects • Transform & Lighting • Multi-texturing (min 2 units) • Fixed-point & Floating-point profiles Images Copyright Rightware • OpenGL ES 2.0 – programmable pipeline • Based on OpenGL 2.0 • Adds vertex and fragment shader programming • Removes fixed function pipeline • Super-compact, efficient API • High level language (GLSL ES) Images & text Copyright Khronos.org • On-line or off-line compilation
- 12. OPENGL ES 2.0PIPELINE Triangles/Lines/Points Primitive Vertex Primitive Primitive Processing Shader Processing Processing API Vertex Buffer Objects Fragment Shader Colour Alpha Depth Buffer Dither Framebuffer Test Test Blend
- 13. PROGRAMMABLE HARDWARE Input Assembly Texture Shader Shader Cache Core Core Rasterizer Fetch Alpha / Depth Test Output Blend Texture Shader Shader Cache Core Core Fetch Scheduler
- 14. DIFFUSE SHADER uniform sampler2Dmy_color_texture; void main() { // Defining The Material Colors const vec4 DiffuseColor = vec4(1.0, 0.0, 0.0, 1.0); // Scaling The Input Vector To Length 1 vec3 normalized_normal = normalize(normal); vec3 normalized_vertex_to_light_vector = normalize(vertex_to_light_vector); // Calculating The Diffuse Term And Clamping It To [0;1] float DiffuseTerm = clamp(dot(normal, vertex_to_light_vector), 0.0, 1.0); // Calculating The Final Color gl_FragColor = AmbientColor + DiffuseColor * DiffuseTerm; } Shader compiler Input Fragment <diffuseShader>: Output Shaded Fragment Sample r0,v4, t0, s0 mul r3, v0, cb0[0] madd r3, v1, cb0[1], r3 madd r3, v2, cb0[2], r3 clmp r3, r3, 1(0.0), 1(1.0) mul o0,r0,r3 mul o1,r1,r3 mul o2,r2,r3 mov o0, 1(1.0)
- 15. THE DRIVER ARCHITECTURE Shader Compiler User Space OpenGL ES EGL L2-Stack Kernel Space GPU Driver FB Driver Hardware [ GPU + Display Controller ]
- 16. KERNEL DRIVERS • Device State & Information • Memory Management • Register Allocation • IRQ handling • Performance Counters • Framebuffer Access
- 17. USER SPACE DRIVER •Based on OpenGL ES state trackers • Consider OpenGL ES as a big state machine • Some time contains essential algorithms GPU manipulation • Use of host CPU for setup • Tessellation for example • Emulate calls not supported by GPU • Just-In-Time Shader compiler
- 18. GRAPHICS PC EMULATOR Shader Compiler User Space OpenGL ES EGL MESA GL / Proprietary OGL GLSL Kernel Space GPU Driver FB Driver Hardware [ GPU + Display Controller ]
- 19. MEEGO QEMU EMULATOR Kernel Application X Server Virtual I/O Driver LibGL Client Client OS QEMU HOST Virtual I/O Device Process Frame Buffer Management LibGL Server Stub Offscreen Buffer http://conference2010.meego.com/sites/all/files/sessions/ opengl_acceleration_in_meegosimulatorandemulator.pdf
- 20. THE EMULATOR DRAWBACKS • Does not emulate all OpenGL ES functions • OpenGL ES is usually mapped to OS OpenGL calls • Shader compilers are not optimized • Too many SW layers, consider QMEU as well • Rendering is not pixel accurate • Most emulators don’t support texture compression formats • Texture bandwidth & Memory is not accurately depicted • On Intel integrated GPUs rendering is mostly SW • Only recently have Nvidia & AMD started supporting OGLES 2 • Performance counters ?, PBuffers, Depth Size variations........
- 21. ANDROID GRAPHICS ARCHITECTURE Android JSR Surface Flinger JNI Wrapper JNI Wrapper OpenGL ES SKIA EGL copyBLIT libagl SW renderer libSkiaHW User Space Kernel Space HAL ARM + NeoN Gralloc GPU KDrv FBDev
- 22. MEEGO GRAPHICS ARCHITECTURE QML + API QT Paint Engine QT OpenGL Wrapper QScreen X Server (optional) OpenGL ES 2.0 OpenGL ES 1.1 EGL User Space Kernel Space GPU KDrv FBDev
- 23. A PLATFORM SCENARIO • Consider a SoC with ARM Cortex A9 + SGX535 GPU • Intel Atom Z6xx with x86 + SGX535 • As a system integrator you want to “sell products” • Simpler to maintain a common set of drivers • But GPU drivers are an issue • Android userspace drivers are compiled against bionic libc • MeeGo for example uses glibc and generic linux stack • DirectX support only on Intel SGX, not on TI OMAP • Symbol & linker errors • PC development uses an emulation layer with PC Graphics • Performance on the same GPU varies
- 24. MOBILE GRAPHICS BIGPICTURE PC Development Device PowerVR SGX QEMU ARM Emulation Android Qualcomm AMD OGLES Based on AMD Nvidia Tegra Simulator, x86 code PowerVR MBX iPhone OpenGL ->OGLES PowerVR SGX QEMU ARM Emulation PowerVR SGX MeeGo X11, OGLES Passthrough ARM MALI
- 25. GPU IP Emulators Emulator Features API Support Win32 - OpenGL Imagination Tech No PVRTC support OpenGL ES 2.0 Linux - Mesa/OpenGL Win32 - OpenGL ARM MALI OpenGL ES 2.0 Linux - Mesa/OpenGL No Antialiasing Nvidia Tegra Win32 - OpenGL ES OpenGL ES 2.0 No ETC1 Vivante Win32 - OpenGL Basic OpenGL ES 2.0 Win32 - OpenGL Supports performance Qualcomm* OpenGL ES 2.0 Android - OpenGL counters
- 26. THE DEVELOPER • OpenGL ES is a standard, conformance guarantees the implementation & functionality • Fragmentation, a lot of players • Khronos does not specify • Texture Compression formats, ETC, ETC2, S3 • Non power of 2 textures, wrapping & mip-mapping • Shader Application Binary Interface • Depth size is not constant , varies 24bit PVR, 16Bit Tegra • Font rendering , Tessellation of Geometry • Although FSAA has become a standard • There is no API to turn off AA feature
- 27. WHAT HIDDEN INSW SoC OMAP 3730 Screen Resolution 800x480 SoC Process 45nm CPU ARM Cortex A8 OS Android 2.2 DSP C64x TI DSP Mobile DDR2 256MBytes GPU SGX 530 ARM 600 MHz ARM 800MHz GPU 200MHz Draw Image 16.0 21.0 Utah TeaPot 30.33 39.19 OGLES Fog 110.71 129.78 OGLES Blending 105.43 118.92 http://code.google.com/p/0xbench/wiki/Benchmarks
- 28. SUMMARY OF THEISSUES • Many API ( OpenGL ES, OpenVG, OpenCL, OpenMAX ) • Many OS ( Linux, Android, MeeGo, Symbian, WinPhone7 ) • Many GPU IP ( SGX, MALI, TEGRA, Vivante, DMP ) • Binary portability of Shader programs • Extensions differentiate GPUs, also fragments • Does not provide for easy transition among vendors IPs • SW/HW partition is not transparent • Performance of emulators ( Linux, Windows, OS X )
- 29. DRIVER ARCHITECTURE Shader Compiler User Space OpenGL ES EGL L2-Stack Kernel Space GPU Driver FB Driver Open Source Hardware [ GPU + Display Controller ]
- 30. USER SPACE DRIVER OpenGL ES 2.0 SGX LLVM Backend OpenGL ES 2.0 Gallium PC Emulation MALI LLVM Backend LLVM OpenGL ES 1.1 PC GPU Emulation LLVM Backend OpenCL 1.1 Source : http://llvm.org , 10-lattner-OpenGL.pdf
- 31. HOW DOES ITWORK • OpenGL ES, OpenCL, OpenMAX are all standard APIs • Gallium outputs LLVM intermediate representation • LLVM optimizes this IR • Each GPU driver has a LLVM backend description • LLVM Backend describes • Instruction Set • Registers • Constraints
- 32. WHAT DOES THISOFFER • Portable infrastructure supports all GPUs & CPUs • Reuse of optimization paths • PowerPC ( Altivec ) • ARM ( Neon ) • x86 ( SSE ) • Smaller driver, simpler to maintain • Interface to inject proprietary code as LLVM backend • Support a common set of graphics functions across platforms • Support a number of different APIs and bring cohesion
- 33. REALITY CHECK C Code Bytecode Opcode- Llvm-gcc Disk functions OpenGL OpenGL to GLSL LLVM Optimizer LLVM JIT Parser LLVM OpenGL AST LLVM IR LLVM IR • OpenGL support on x86 without explicit Graphics card Leopard (OS X 10.5) • Step1: gcc front end parses OpenGL C code • Step2: GLSL or shader is compiled using clang/LLVM • Step3 : LLVM IR is produced to be further optimized by LLVM JIT
- 34. REALITY CHECK OpenCL • OpenCL compilation process on Compute Program SnowLeopard (OSX 10.6) OpenCL front-end [ clang ] • Step1: Compile OCL to LLVM IR LLVM IR (Intermediate Representation) • Step2: Compile to target device X86 LLVM NVIDIA Back-End • NVIDIA GPU device compiles the LLVM X86 Binary IR in two steps: PTX IR • LLVM IR to PTX (CUDA IR) – PTX to target GPU G80 G92 G200 Binary Binary Binary • CPU device uses LLVM x86 BE to compile directly to x86 Binary code. http://www.haifux.org/lectures/212/OpenCL_for_Halifux_new.pdf
- 35. CAN IT BECOMEA REALITY • Fabless companies should force this on IP vendors • Projects like Android & MeeGo can help push SoC providers • Ultimately its the product companies, Nokia, Motorola etc • Developers • Maybe a commercial FOSS company
- 36. MISC REFERENCES INSLIDES • [Slide-1] http://www.nobodysplace.nl/pacman.jpg • ATI RUBY : http://www.pcgameshardware.com/ &menu=browser&image_id=999114&article_id=680582&page=1&show=original • http://cdn.erictric.com/wp-content/uploads/2010/06/samsung-galaxy-s-stock-shot.jpg • http://media.photobucket.com/image/iPhone4/defskilz14/blog%20pics/torch_iphone4.jpg • http://s08.idav.ucdavis.edu/fatahalian-gpu-architecture.pdf • http://s08.idav.ucdavis.edu/fatahalian-gpu-architecture.pdf
- 37. WinXP GDI eglpla+orm.h libEGL.lib eglext.h egl.h OpenGL32.dll k lin incl ude libEGL.dll libGLESv2.dll include khrpla+orm.h ApplicaCon de c lu libGLESv2.so GPU in libEGL.so link gl2pla+orm.h gl2ext.h xxx_DRI.so gl2.h libGLESv2.lib Linux X11 Components of OpenGL ES 2.0 simulator