Download as PDF, PPTX
Uploaded byPatricia Aas
Isolating GPU Access in its Own Process (Foss-North 2018)
The document discusses the architecture and benefits of isolating GPU access in web browsers, particularly within the Chromium framework. Key reasons for using a separate GPU process include enhancing security, robustness, and dependency separation, which help prevent crashes and resource leaks. It also details the communication architecture for processing video frames and provides insights on implementation design in browser rendering processes.
More Related Content
Similar to Isolating GPU Access in its Own Process (Foss-North 2018)
Isolating GPU Access in its Own Process (Foss-North 2018)
- 1.
- 2. Isolating GPU Access inits own process @pati_gallardo Patricia Aas Foss-North, Gothenburg, Sweden, 2018
- 3. Patricia Aas - VivaldiBrowser Programmer - mainly in C++ Currently : Vivaldi Technologies Previously : Cisco Systems, Knowit, Opera Software Master in Computer Science - main language Java Twitter : @pati_gallardo
- 4.
- 5. @pati_gallardo Disclaimer: This TalkIs a bit Linux Focused ¯_(ツ)_/¯
- 6. - What isChromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can I Use? @pati_gallardo
- 7.
- 8. Konqueror Safari Chrome Brave Vivaldi Opera KHTMLWebkit Blink KDE Apple Google
- 9. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygoteInit Renderer Renderer Process Relationships Tabs
- 10. Composition The Browser Windowis composed of many views produced by many cooperating processes @pati_gallardo
- 11. - What isChromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can I Use? @pati_gallardo
- 12. @pati_gallardo Why Not DoGPU Composition in The Browser Process?
- 13. Well, Actually… OnAndroid It Does… But I Digress… @pati_gallardo
- 14. 1. Security 2. Robustness 3.Dependency Separation @pati_gallardo
- 15.
- 16. Example Texture memory beingleaked across processes - From Other Programs on the Users Machine - From Other Tabs - From the Browser @pati_gallardo
- 17.
- 19.
- 20. Graphics Drivers Crashingthe Browser - Prevent bugs in GPU drivers from crashing the browser - Make sure graphics code in WebGL can’t crash the browser - Compensate for Graphics Driver Bugs/Inconsistencies @pati_gallardo
- 21.
- 22. Keep GPU ProcessDependencies Out of the Renderer process - Minimize the renderer sandbox - Can Have Different Dependencies @pati_gallardo
- 23.
- 24. Before We DiveIn - Deep Breath :D@pati_gallardo
- 25. “We can solveany problem by introducing an extra level of indirection. …except for the problem of too many levels of indirection” Fundamental theorem of software engineering Andrew Koenig/Butler Lampson/David J. Wheeler @pati_gallardo
- 26. - What isChromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can I Use? @pati_gallardo
- 27. IPC, Commands &Synchronization @pati_gallardo
- 28. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygoteInit Renderer Renderer Process Relationships Tabs
- 29. Communication Architecture Renderer Gpu Process BrowserRenderer IPC Channels Shared Memory Gpu Memory BuffersCommand Buffers
- 30. High Level Design Client- Server Architecture Emulates OpenGl ES2.0 Actual Graphics Implementation is Platform Specific Composition in GPU Process Page Composition Controlled From Renderer @pati_gallardo
- 31. OpenGL ES2 callsare serialized into “Commands” That are placed in a Ring Buffer in Shared Memory called a “Command Buffer”. The GPU process de-serializes them, does validation, maybe some workarounds, and executes them using the Platform Graphics System. @pati_gallardo Implementation Design
- 32. Client Renderer / Browser -Write commands to shared memory - Update ‘put’ pointer - Signal GPU process @pati_gallardo
- 33. Server GPU Process - Readcommands from shared memory - Validate command and arguments - Make actual call@pati_gallardo
- 34.
- 35. Gpu Process Command Buffer IPC Channel Command Stream Ordering Barrier Unverified SyncToken Wait Sync Token Wait Sync TokenCommandCommand Command Command Command Command CommandVerified Sync TokenBrowser Renderer Renderer Command Command CommandCommand CommandCommand
- 36. Sync Token -Inserts a synchronization fence into the command stream - Can be attached to a resource (texture) that cannot be used before all previous commands have been processed @pati_gallardo
- 37. Mailbox Holder 1. Mailbox- unique name 2. SyncToken - fence 3. Texture Target Type (if texture backed) @pati_gallardo
- 38. - What isChromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can this work for you? @pati_gallardo
- 39.
- 40. Compositing in Chromium -Software Compositing is Done In the Browser Process - Hardware/GPU Compositing is Done in the GPU Process (Except on Android where there is a GPU thread in the Browser Process) @pati_gallardo
- 41. Insert Some HandWaving The full architecture is massive We will follow one path A software decoded video frame @pati_gallardo
- 42. Getting a VideoFrame into the Page @pati_gallardo
- 43. Software Decoded VideoFrame - Decoded Frame in Memory in RENDERER PROCESS - GPU Composition is done in the GPU PROCESS - The Frame needs to be uploaded to the GPU as a Texture @pati_gallardo
- 44. “At a highenough level of abstraction, everything looks the same.” Law of PowerPoint Architecture Patricia Aas, 2018 @pati_gallardo
- 45.
- 46. Decoding Video Browser Process Network stack Renderer Decoder* VideoFrame MemoryBuffer Y Plane U Plane V Plane Internet * Sometimes decodeing is done in the GPU process
- 47.
- 48.
- 49. OES_EGL_image_external Extension that createsEGLImage texture targets from EGLImages “Each TEXTURE_EXTERNAL_OES texture object may require up to 3 texture image units for each texture unit to which it is bound.” @pati_gallardo
- 50. VideoFrame VideoFrame Memory Buffer VPlane Y Plane U Plane Shared Memory Gpu ProcessRenderer Transform the Video Frame into a GPU Resource Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken MailboxMailbox MailboxHolder
- 51.
- 52. VideoFrame Shared Memory Gpu ProcessRenderer YPlane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken Mailbox Mailbox MailboxHolder Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 Move into a Transferrable Resource
- 53.
- 54. YUVVideoDrawQuad Gpu ProcessRenderer Y PlaneTexture UV Plane Texture MailboxHolder SyncToken Mailbox MailboxHolder Mailbox RenderPass Resources Id: 0 Id: 1 Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 LayerTreeResourceProvider
- 55.
- 56. YUVVideoDrawQuad Resources Id: 0 Id:1 Render The Frame! GLRenderer::DrawYUVVideoQuad clip_region
- 57.
- 58. Examples : ChromiumGLES2 Extensions ● CHROMIUM_image ● CHROMIUM_texture_mailbox ● CHROMIUM_sync_point @pati_gallardo
- 59. - What isChromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can I Use? @pati_gallardo
- 60. Not Exactly CutAnd Paste @pati_gallardo
- 61. Three APIs arein use in the renderer 1. Opengl ES2 2. Chromium GL ES2 Extensions 3. Chromium APIs @pati_gallardo
- 62. “All non-trivial abstractions,to some degree, are leaky.” Law of Leaky Abstractions Joel Spolsky, 2002 @pati_gallardo
- 63. - Ok, but…Can I Use? - Hm, don’t know… Maybe? ¯_(ツ)_/¯ @pati_gallardo
- 64.
- 65. Make Everyone FeelSafe To Be Themselves @pati_gallardo
- 66. Vivaldi Swag Patricia Aas,Vivaldi Technologies @pati_gallardo Photos from pixabay.com
- 67.
- 68. Copy Video FrameTo GPU Memory Buffer - CopyVideoFrameToGpuMemoryBuffers - OutputFormat::NV12_SINGLE_GMB - CopyRowsToNV12Buffer - libyuv::I420ToNV12 - GpuMemoryBufferImplSharedMemory @pati_gallardo
- 69. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resourcelifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3
- 70.
- 71. VideoFrameProvider Client VideoFrameController Client InputHandler Client LayerTreeHostImpl VideoFrameCompositor VideoRendererSink OnBeginFrame DidDrawFrame UpdateCurrentFrame GetCurrentFrame PutCurrentFrame VideoRendererImpl Render OnFrameDropped VideoFrameProviderClientImplVideoFrameProviderClientImplVideoFrameProviderClientImpl Video FramePainting VideoFrame current_frame_ VideoLayerImpl active_video_layer_ DecodersVideoResourceUpdater
- 72. Useful files toread gpu_memory_buffer_video_frame_pool.cc video_resource_updater.cc gl_renderer.cc (GLRenderer::DrawYUVVideoQuad) program_binding.cc (ProgramKey::YUVVideo) @pati_gallardo
- 73.
- 74. Share Group - CommandBuffers in the same share group must be in the same Command Stream - gl::GLFence - eglFenceSyncKHR (EGL_KHR_fence_sync) - eglWaitSyncKHR (EGL_KHR_wait_sync) @pati_gallardo
- 75.