Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Isolating GPU Access in its Own Process

156 views

Published on

Chromium's process architecture has graphics access restricted to a separate GPU-process. There are several reasons why this could make sense, three common ones are: Security, Robustness and Dependency Separation.

GPU access restricted to a single process requires an efficient framework for communication over IPC from the other processes, and most likely a framework for composition of surfaces. This talk describes both the possible motivations for this kind of architecture and Chromium's solution for the IPC framework. We will demonstrate how a multi-process program can compose into a single window on Linux.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Isolating GPU Access in its Own Process

  1. 1. T S @pati_gallardo
  2. 2. Isolating GPU Access in its own process Patricia Aas, T S NDC TechTown 2018 T S @pati_gallardo
  3. 3. Patricia Aas - Consultant C++ Programmer, Application Security Currently : T S Previously : Vivaldi, Cisco Systems, Knowit, Opera Software Master in Computer Science - main language Java Pronouns: she/her T S @pati_gallardo
  4. 4. - What is Chromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
  5. 5. Some Browser Trivia @pati_gallardo
  6. 6. Konqueror Safari Chrome Brave Vivaldi Opera KHTML Webkit Blink KDE Apple Google
  7. 7. Composition The Browser Window is composed of many views produced by many cooperating processes @pati_gallardo
  8. 8. Demo of Composition ...I might have made a browser... ...I might have decided to have a demo last night... @pati_gallardo T S
  9. 9. Renderer Process Webkit Browser Process Software Composition Original Chromium Software Composition Architecture GUI
  10. 10. Renderer Process Webkit Browser Process GUI Gpu Process Hardware Composition Moving Composition to the GPU Process
  11. 11. - What is Chromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
  12. 12. Communication Architecture @pati_gallardo
  13. 13. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygote Init Renderer Renderer Process Relationships Tabs IPC & Commands
  14. 14. Components of Communication Renderer Gpu Process Browser Renderer IPC Channels Shared Memory Gpu Memory Buffers Command Buffers (Ring buffer) Gpu Memory Buffers Gpu Memory BuffersCommand Command Command
  15. 15. Faking OpenGL ES 2 (for fun and profit?) Command CommandCommand Command Render/Browser Process Gpu Process Client Encoder/Proxy ServerDecoder/Validator Shared Memory OpenGL ES 2 Interface
  16. 16. - Write Commands to Command Buffer in Shared Memory - Update ‘put’ pointer - Signal GPU process @pati_gallardo Client Renderer / Browser
  17. 17. - Read Commands from Command Buffer in Shared Memory - Validate Command and arguments - Make actual call@pati_gallardo Server GPU Process
  18. 18. Server (Gpu Process) IPC Channel Command Client (Renderer / Browser process) CommandCommandCommand Command Command Stream Command BufferCommand Conceptual Model
  19. 19. Synchronization Architecture @pati_gallardo
  20. 20. - Inserts a synchronization fence into the command stream - Can be attached to a resource (texture) that cannot be used before all previous commands have been processed @pati_gallardo Sync Token
  21. 21. SyncToken @pati_gallardo SyncToken CommandBufferNamespace release_count_ CommandBufferId
  22. 22. Gpu Process Command Buffer IPC Channel Command Stream Ordering Barrier Unverified Sync Token Wait Sync Token Wait Sync TokenCommandCommand Command Command Command Command CommandVerified Sync TokenBrowser Renderer Renderer Command Command CommandCommand CommandCommand
  23. 23. - What is Chromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
  24. 24. Getting a Video Frame into the Page @pati_gallardo
  25. 25. Software Decoded Video Frame - Decoded Frame in Memory in RENDERER PROCESS - GPU Composition is done in the GPU PROCESS - The Frame needs to be uploaded to the GPU as a Texture BEFORE it can be composed @pati_gallardo
  26. 26. Decode Frame into Renderer Memory Copy Frame to GPU Memory Buffer Issue Draw Commands to GPU Wait SyncToken Using the SyncToken to Reorder
  27. 27. Insert Some Hand Waving The full architecture is massive We will follow one path A software decoded video frame @pati_gallardo
  28. 28. “At a high enough level of abstraction, everything looks the same.” Law of PowerPoint Architecture Patricia Aas, 2018 @pati_gallardo
  29. 29. Decode @pati_gallardo
  30. 30. Decoding Video Browser Process Network stack Renderer Decoder* VideoFrame Memory Buffer Y Plane U Plane V Plane Internet * Sometimes decoding is done in the GPU process
  31. 31. @pati_gallardo “Texturize”
  32. 32. 1. Mailbox - unique name 2. SyncToken - fence 3. Texture Target Type (if texture backed) @pati_gallardo Mailbox Holder
  33. 33. VideoFrame VideoFrame Memory Buffer V Plane Y Plane U Plane Shared Memory Gpu ProcessRenderer Transform the Video Frame into a GPU Resource Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBuffer UV Planes GpuMemoryBuffer MailboxHolder SyncToken MailboxMailbox MailboxHolder
  34. 34. Prepare @pati_gallardo
  35. 35. VideoFrame Shared Memory Gpu ProcessRenderer Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken Mailbox Mailbox MailboxHolder Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 Move into a Transferrable Resource
  36. 36. Add to Render @pati_gallardo
  37. 37. YUVVideoDrawQuad Gpu ProcessRenderer Y Plane Texture UV Plane Texture MailboxHolder SyncToken Mailbox MailboxHolder Mailbox RenderPass Resources Id: 0 Id: 1 Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 LayerTreeResourceProvider
  38. 38. Render! @pati_gallardo
  39. 39. YUVVideoDrawQuad Resources Id: 0 Id: 1 Render The Frame! GLRenderer::DrawYUVVideoQuad clip_region
  40. 40. Gpu Process Wait Sync Token Command Command Verified Sync Token Browser Renderer CommandCommand Command
  41. 41. GLES2 Extensions @pati_gallardo
  42. 42. Examples : Chromium GLES2 Extensions ● CHROMIUM_image ● CHROMIUM_texture_mailbox ● CHROMIUM_sync_point @pati_gallardo
  43. 43. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resource lifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3
  44. 44. - What is Chromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
  45. 45. @pati_gallardo Why Not Do GPU Composition in The Browser Process?
  46. 46. Well, Actually… On Android It Does… But I Digress… @pati_gallardo
  47. 47. 1. Security 2. Robustness 3. Dependency Separation 4. Performance ? @pati_gallardo
  48. 48. Security @pati_gallardo
  49. 49. Gives Fine Grained Control Texture memory being leaked across processes - From Other Programs on the Users Machine - From Other Tabs - From the Browser @pati_gallardo
  50. 50. User : Lxgr security.stackexchange.com
  51. 51. Robustness @pati_gallardo
  52. 52. Graphics Drivers Crashing the Browser - Prevent bugs in GPU drivers from crashing the browser - Make sure graphics code in WebGL can’t crash the browser - Compensate for Graphics Driver Bugs/Inconsistencies @pati_gallardo
  53. 53. Dependency Separation @pati_gallardo
  54. 54. Keep GPU Process Dependencies Out of the Renderer process - Minimize the renderer sandbox - Can Have Different Dependencies @pati_gallardo
  55. 55. Performance? ¯_(ツ)_/¯ @pati_gallardo
  56. 56. “We can solve any problem by introducing an extra level of indirection. …except for the problem of too many levels of indirection” Fundamental theorem of software engineering Andrew Koenig/Butler Lampson/David J. Wheeler @pati_gallardo
  57. 57. - What is Chromium? - Communication Architecture - Passing A Video Frame - Why have a GPU process? - Can I Use? @pati_gallardo
  58. 58. - Ok, but… Can I Use? - Hm, don’t know… Maybe? ¯_(ツ)_/¯ @pati_gallardo
  59. 59. Not Exactly Cut And Paste @pati_gallardo
  60. 60. Check : <chrome>://gpu @pati_gallardo
  61. 61. Three APIs are in use in the renderer 1. Opengl ES2 2. Chromium GL ES2 Extensions 3. Chromium APIs @pati_gallardo
  62. 62. “All non-trivial abstractions, to some degree, are leaky.” Law of Leaky Abstractions Joel Spolsky, 2002 @pati_gallardo
  63. 63. - Ok, ok, but… Can I Use? @pati_gallardo
  64. 64. ...I’d probably advice against it @pati_gallardo
  65. 65. But knowing that it can be done has value. It makes giving it a go less crazy. @pati_gallardo
  66. 66. So… maybe? ¯_(ツ)_/¯ @pati_gallardo
  67. 67. Patricia Aas, Consultant T S C++ and Application Security T S @pati_gallardo
  68. 68. T SD P
  69. 69. @pati_gallardo T S
  70. 70. Appendix / Some Notes @pati_gallardo
  71. 71. High Level Design Client - Server Architecture Emulates OpenGl ES2.0 Actual Graphics Implementation is Platform Specific Composition in GPU Process Page Composition Controlled From Renderer @pati_gallardo
  72. 72. Copy Video Frame To GPU Memory Buffer Interesting Code - CopyVideoFrameToGpuMemoryBuffers - OutputFormat::NV12_SINGLE_GMB - CopyRowsToNV12Buffer - libyuv::I420ToNV12 - GpuMemoryBufferImplSharedMemory @pati_gallardo
  73. 73. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resource lifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3
  74. 74. texture_target Mac GL_TEXTURE_RECTANGLE_ARB Android/Linux GL_TEXTURE_EXTERNAL_OES Fallback GL_TEXTURE_2D @pati_gallardo
  75. 75. OES_EGL_image_external Extension that creates EGLImage texture targets from EGLImages “Each TEXTURE_EXTERNAL_OES texture object may require up to 3 texture image units for each texture unit to which it is bound.” @pati_gallardo
  76. 76. CHROMIUM_image CreateImageCHROMIUM ReleaseTexImage2DCHROMIUM BindTexImage2DCHROMIUM DestroyImageCHROMIUM @pati_gallardo
  77. 77. Share Group - Command Buffers in the same share group must be in the same Command Stream - gl::GLFence - eglFenceSyncKHR (EGL_KHR_fence_sync) - eglWaitSyncKHR (EGL_KHR_wait_sync) @pati_gallardo
  78. 78. VideoFrameProvider Client VideoFrameController Client InputHandler Client LayerTreeHostImpl VideoFrameCompositor VideoRendererSink OnBeginFrame DidDrawFrame UpdateCurrentFrame GetCurrentFrame PutCurrentFrame VideoRendererImpl Render OnFrameDropped VideoFrameProviderClientImplVideoFrameProviderClientImplVideoFrameProviderClientImpl Video Frame Painting VideoFrame current_frame_ VideoLayerImpl active_video_layer_ DecodersVideoResourceUpdater
  79. 79. Useful files to read gpu_memory_buffer_video_frame_pool.cc video_resource_updater.cc gl_renderer.cc (GLRenderer::DrawYUVVideoQuad) program_binding.cc (ProgramKey::YUVVideo) @pati_gallardo
  80. 80. P f . Patricia Aas, T S @pati_gallardo T S
  81. 81. @pati_gallardo T S

×