Download as PDF, PPTX
Uploaded byPatricia Aas
Isolating GPU Access in its Own Process
The document discusses the architecture of Chromium's GPU process, including its communication structure and the reasons for isolating GPU access. It covers the processing of video frames, the synchronization architecture necessary for rendering, and the benefits of GPU composition in terms of security, robustness, and performance. Additionally, it addresses potential usage and factors to consider when implementing GPU features in applications.
More Related Content
Similar to Isolating GPU Access in its Own Process
![[2018 GDC] Real-Time Ray-Tracing Techniques for Integration into Existing Ren...](https://cdn.slidesharecdn.com/ss_thumbnails/gdc2018takahiroharada-180330041526-thumbnail.jpg?width=640&height=640&fit=bounds)
![Lect 1 Number systems and base conversions. [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/lect1numbersystemsandbaseconversions-260111134109-67c2d865-thumbnail.jpg?width=640&height=640&fit=bounds)
Isolating GPU Access in its Own Process
- 1.
- 2. Isolating GPU Access inits own process Patricia Aas, T S NDC TechTown 2018 T S @pati_gallardo
- 3. Patricia Aas - Consultant C++Programmer, Application Security Currently : T S Previously : Vivaldi, Cisco Systems, Knowit, Opera Software Master in Computer Science - main language Java Pronouns: she/her T S @pati_gallardo
- 4. - What isChromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
- 5. Some Browser Trivia@pati_gallardo
- 6. Konqueror Safari Chrome Brave Vivaldi Opera KHTMLWebkit Blink KDE Apple Google
- 7. Composition The Browser Windowis composed of many views produced by many cooperating processes @pati_gallardo
- 8. Demo of Composition ...Imight have made a browser... ...I might have decided to have a demo last night... @pati_gallardo T S
- 9. Renderer Process Webkit Browser Process Software Composition OriginalChromium Software Composition Architecture GUI
- 10. Renderer Process Webkit Browser Process GUI GpuProcess Hardware Composition Moving Composition to the GPU Process
- 11. - What isChromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
- 12.
- 13. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygoteInit Renderer Renderer Process Relationships Tabs IPC & Commands
- 14. Components of Communication Renderer GpuProcess Browser Renderer IPC Channels Shared Memory Gpu Memory Buffers Command Buffers (Ring buffer) Gpu Memory Buffers Gpu Memory BuffersCommand Command Command
- 15. Faking OpenGL ES2 (for fun and profit?) Command CommandCommand Command Render/Browser Process Gpu Process Client Encoder/Proxy ServerDecoder/Validator Shared Memory OpenGL ES 2 Interface
- 16. - Write Commandsto Command Buffer in Shared Memory - Update ‘put’ pointer - Signal GPU process @pati_gallardo Client Renderer / Browser
- 17. - Read Commandsfrom Command Buffer in Shared Memory - Validate Command and arguments - Make actual call@pati_gallardo Server GPU Process
- 18. Server (Gpu Process) IPC Channel Command Client (Renderer / Browser process) CommandCommandCommandCommand Command Stream Command BufferCommand Conceptual Model
- 19.
- 20. - Inserts asynchronization fence into the command stream - Can be attached to a resource (texture) that cannot be used before all previous commands have been processed @pati_gallardo Sync Token
- 21.
- 22. Gpu Process Command Buffer IPC Channel CommandStream Ordering Barrier Unverified Sync Token Wait Sync Token Wait Sync TokenCommandCommand Command Command Command Command CommandVerified Sync TokenBrowser Renderer Renderer Command Command CommandCommand CommandCommand
- 23. - What isChromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
- 24. Getting a VideoFrame into the Page @pati_gallardo
- 25. Software Decoded VideoFrame - Decoded Frame in Memory in RENDERER PROCESS - GPU Composition is done in the GPU PROCESS - The Frame needs to be uploaded to the GPU as a Texture BEFORE it can be composed @pati_gallardo
- 26. Decode Frame into RendererMemory Copy Frame to GPU Memory Buffer Issue Draw Commands to GPU Wait SyncToken Using the SyncToken to Reorder
- 27. Insert Some HandWaving The full architecture is massive We will follow one path A software decoded video frame @pati_gallardo
- 28. “At a highenough level of abstraction, everything looks the same.” Law of PowerPoint Architecture Patricia Aas, 2018 @pati_gallardo
- 29.
- 30. Decoding Video Browser Process Network stack Renderer Decoder* VideoFrame MemoryBuffer Y Plane U Plane V Plane Internet * Sometimes decoding is done in the GPU process
- 31.
- 32. 1. Mailbox -unique name 2. SyncToken - fence 3. Texture Target Type (if texture backed) @pati_gallardo Mailbox Holder
- 33. VideoFrame VideoFrame Memory Buffer VPlane Y Plane U Plane Shared Memory Gpu ProcessRenderer Transform the Video Frame into a GPU Resource Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBuffer UV Planes GpuMemoryBuffer MailboxHolder SyncToken MailboxMailbox MailboxHolder
- 34.
- 35. VideoFrame Shared Memory Gpu ProcessRenderer YPlane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken Mailbox Mailbox MailboxHolder Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 Move into a Transferrable Resource
- 36.
- 37. YUVVideoDrawQuad Gpu ProcessRenderer Y PlaneTexture UV Plane Texture MailboxHolder SyncToken Mailbox MailboxHolder Mailbox RenderPass Resources Id: 0 Id: 1 Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 LayerTreeResourceProvider
- 38.
- 39. YUVVideoDrawQuad Resources Id: 0 Id:1 Render The Frame! GLRenderer::DrawYUVVideoQuad clip_region
- 40. Gpu Process Wait Sync Token Command Command VerifiedSync Token Browser Renderer CommandCommand Command
- 41.
- 42. Examples : ChromiumGLES2 Extensions ● CHROMIUM_image ● CHROMIUM_texture_mailbox ● CHROMIUM_sync_point @pati_gallardo
- 43. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resourcelifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3
- 44. - What isChromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
- 45. @pati_gallardo Why Not DoGPU Composition in The Browser Process?
- 46. Well, Actually… OnAndroid It Does… But I Digress… @pati_gallardo
- 47. 1. Security 2. Robustness 3.Dependency Separation 4. Performance ? @pati_gallardo
- 48.
- 49. Gives Fine GrainedControl Texture memory being leaked across processes - From Other Programs on the Users Machine - From Other Tabs - From the Browser @pati_gallardo
- 50.
- 52.
- 53. Graphics Drivers Crashingthe Browser - Prevent bugs in GPU drivers from crashing the browser - Make sure graphics code in WebGL can’t crash the browser - Compensate for Graphics Driver Bugs/Inconsistencies @pati_gallardo
- 54.
- 55. Keep GPU ProcessDependencies Out of the Renderer process - Minimize the renderer sandbox - Can Have Different Dependencies @pati_gallardo
- 56.
- 57. “We can solveany problem by introducing an extra level of indirection. …except for the problem of too many levels of indirection” Fundamental theorem of software engineering Andrew Koenig/Butler Lampson/David J. Wheeler @pati_gallardo
- 58. - What isChromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
- 59. - Ok, but…Can I Use? - Hm, don’t know… Maybe? ¯_(ツ)_/¯ @pati_gallardo
- 60. Not Exactly CutAnd Paste @pati_gallardo
- 61.
- 62. Three APIs arein use in the renderer 1. Opengl ES2 2. Chromium GL ES2 Extensions 3. Chromium APIs @pati_gallardo
- 63. “All non-trivial abstractions,to some degree, are leaky.” Law of Leaky Abstractions Joel Spolsky, 2002 @pati_gallardo
- 64. - Ok, ok,but… Can I Use? @pati_gallardo
- 65. ...I’d probably adviceagainst it @pati_gallardo
- 66. But knowing thatit can be done has value. It makes giving it a go less crazy. @pati_gallardo
- 67.
- 68. Patricia Aas, Consultant TS C++ and Application Security T S @pati_gallardo
- 69.
- 70.
- 71. Appendix / SomeNotes @pati_gallardo
- 72. High Level Design Client- Server Architecture Emulates OpenGl ES2.0 Actual Graphics Implementation is Platform Specific Composition in GPU Process Page Composition Controlled From Renderer @pati_gallardo
- 73. Copy Video FrameTo GPU Memory Buffer Interesting Code - CopyVideoFrameToGpuMemoryBuffers - OutputFormat::NV12_SINGLE_GMB - CopyRowsToNV12Buffer - libyuv::I420ToNV12 - GpuMemoryBufferImplSharedMemory @pati_gallardo
- 74. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resourcelifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3
- 75.
- 76. OES_EGL_image_external Extension that createsEGLImage texture targets from EGLImages “Each TEXTURE_EXTERNAL_OES texture object may require up to 3 texture image units for each texture unit to which it is bound.” @pati_gallardo
- 77.
- 78. Share Group - CommandBuffers in the same share group must be in the same Command Stream - gl::GLFence - eglFenceSyncKHR (EGL_KHR_fence_sync) - eglWaitSyncKHR (EGL_KHR_wait_sync) @pati_gallardo
- 79. VideoFrameProvider Client VideoFrameController Client InputHandler Client LayerTreeHostImpl VideoFrameCompositor VideoRendererSink OnBeginFrame DidDrawFrame UpdateCurrentFrame GetCurrentFrame PutCurrentFrame VideoRendererImpl Render OnFrameDropped VideoFrameProviderClientImplVideoFrameProviderClientImplVideoFrameProviderClientImpl Video FramePainting VideoFrame current_frame_ VideoLayerImpl active_video_layer_ DecodersVideoResourceUpdater
- 80. Useful files toread gpu_memory_buffer_video_frame_pool.cc video_resource_updater.cc gl_renderer.cc (GLRenderer::DrawYUVVideoQuad) program_binding.cc (ProgramKey::YUVVideo) @pati_gallardo
- 81. P f . PatriciaAas, T S @pati_gallardo T S
- 82.