Isolating GPU Access in its Own Process (Foss-North 2018)

1. @pati_gallardo

2. Isolating GPU Access in its own process @pati_gallardo Patricia Aas Foss-North, Gothenburg, Sweden, 2018

3. Patricia Aas - Vivaldi Browser Programmer - mainly in C++ Currently : Vivaldi Technologies Previously : Cisco Systems, Knowit, Opera Software Master in Computer Science - main language Java Twitter : @pati_gallardo

4. Building A Browser @pati_gallardo

5. @pati_gallardo Disclaimer: This Talk Is a bit Linux Focused ¯_(ツ)_/¯

6. - What is Chromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can I Use? @pati_gallardo

7. @pati_gallardoSome Browser Trivia

8. Konqueror Safari Chrome Brave Vivaldi Opera KHTML Webkit Blink KDE Apple Google

9. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygote Init Renderer Renderer Process Relationships Tabs

10. Composition The Browser Window is composed of many views produced by many cooperating processes @pati_gallardo

12. @pati_gallardo Why Not Do GPU Composition in The Browser Process?

13. Well, Actually… On Android It Does… But I Digress… @pati_gallardo

14. 1. Security 2. Robustness 3. Dependency Separation @pati_gallardo

15. Security @pati_gallardo

16. Example Texture memory being leaked across processes - From Other Programs on the Users Machine - From Other Tabs - From the Browser @pati_gallardo

17. User : Lxgr security.stackexchange.com

19. Robustness @pati_gallardo

20. Graphics Drivers Crashing the Browser - Prevent bugs in GPU drivers from crashing the browser - Make sure graphics code in WebGL can’t crash the browser - Compensate for Graphics Driver Bugs/Inconsistencies @pati_gallardo

21. Dependency Separation @pati_gallardo

22. Keep GPU Process Dependencies Out of the Renderer process - Minimize the renderer sandbox - Can Have Different Dependencies @pati_gallardo

23. Performance? ¯_(ツ)_/¯ @pati_gallardo

24. Before We Dive In - Deep Breath :D@pati_gallardo

25. “We can solve any problem by introducing an extra level of indirection. …except for the problem of too many levels of indirection” Fundamental theorem of software engineering Andrew Koenig/Butler Lampson/David J. Wheeler @pati_gallardo

27. IPC, Commands & Synchronization @pati_gallardo

28. Process Architecture Browser Gpu BrokerZygote Renderer GpuZygote Init Renderer Renderer Process Relationships Tabs

29. Communication Architecture Renderer Gpu Process Browser Renderer IPC Channels Shared Memory Gpu Memory BuffersCommand Buffers

30. High Level Design Client - Server Architecture Emulates OpenGl ES2.0 Actual Graphics Implementation is Platform Specific Composition in GPU Process Page Composition Controlled From Renderer @pati_gallardo

31. OpenGL ES2 calls are serialized into “Commands” That are placed in a Ring Buffer in Shared Memory called a “Command Buffer”. The GPU process de-serializes them, does validation, maybe some workarounds, and executes them using the Platform Graphics System. @pati_gallardo Implementation Design

32. Client Renderer / Browser - Write commands to shared memory - Update ‘put’ pointer - Signal GPU process @pati_gallardo

33. Server GPU Process - Read commands from shared memory - Validate command and arguments - Make actual call@pati_gallardo

34. Synchronization of Commands @pati_gallardo

35. Gpu Process Command Buffer IPC Channel Command Stream Ordering Barrier Unverified Sync Token Wait Sync Token Wait Sync TokenCommandCommand Command Command Command Command CommandVerified Sync TokenBrowser Renderer Renderer Command Command CommandCommand CommandCommand

36. Sync Token - Inserts a synchronization fence into the command stream - Can be attached to a resource (texture) that cannot be used before all previous commands have been processed @pati_gallardo

37. Mailbox Holder 1. Mailbox - unique name 2. SyncToken - fence 3. Texture Target Type (if texture backed) @pati_gallardo

38. - What is Chromium? - Why have a GPU process? - Communication Architecture - Passing A Video Frame - Can this work for you? @pati_gallardo

39. Exploring Composition @pati_gallardo

40. Compositing in Chromium - Software Compositing is Done In the Browser Process - Hardware/GPU Compositing is Done in the GPU Process (Except on Android where there is a GPU thread in the Browser Process) @pati_gallardo

41. Insert Some Hand Waving The full architecture is massive We will follow one path A software decoded video frame @pati_gallardo

42. Getting a Video Frame into the Page @pati_gallardo

43. Software Decoded Video Frame - Decoded Frame in Memory in RENDERER PROCESS - GPU Composition is done in the GPU PROCESS - The Frame needs to be uploaded to the GPU as a Texture @pati_gallardo

44. “At a high enough level of abstraction, everything looks the same.” Law of PowerPoint Architecture Patricia Aas, 2018 @pati_gallardo

45. @pati_gallardo Decode

46. Decoding Video Browser Process Network stack Renderer Decoder* VideoFrame Memory Buffer Y Plane U Plane V Plane Internet * Sometimes decodeing is done in the GPU process

47. @pati_gallardo Texturize

48. texture_target Mac GL_TEXTURE_RECTANGLE_ARB Android/Linux GL_TEXTURE_EXTERNAL_OES Fallback GL_TEXTURE_2D @pati_gallardo

49. OES_EGL_image_external Extension that creates EGLImage texture targets from EGLImages “Each TEXTURE_EXTERNAL_OES texture object may require up to 3 texture image units for each texture unit to which it is bound.” @pati_gallardo

50. VideoFrame VideoFrame Memory Buffer V Plane Y Plane U Plane Shared Memory Gpu ProcessRenderer Transform the Video Frame into a GPU Resource Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken MailboxMailbox MailboxHolder

51. @pati_gallardo Prepare

52. VideoFrame Shared Memory Gpu ProcessRenderer Y Plane Texture UV Plane Texture Plane Resources Y Plane GpuMemoryBufferUV Planes GpuMemoryBuffer MailboxHolder SyncToken Mailbox Mailbox MailboxHolder Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 Move into a Transferrable Resource

53. @pati_gallardo Add to Render

54. YUVVideoDrawQuad Gpu ProcessRenderer Y Plane Texture UV Plane Texture MailboxHolder SyncToken Mailbox MailboxHolder Mailbox RenderPass Resources Id: 0 Id: 1 Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Transferrable Resource Texture filter GL_LINEAR Texture target GL_TEXTURE_2D Id: 0 Id: 1 LayerTreeResourceProvider

55. @pati_gallardo Render!

56. YUVVideoDrawQuad Resources Id: 0 Id: 1 Render The Frame! GLRenderer::DrawYUVVideoQuad clip_region

57. GLES2 Extensions @pati_gallardo

58. Examples : Chromium GLES2 Extensions ● CHROMIUM_image ● CHROMIUM_texture_mailbox ● CHROMIUM_sync_point @pati_gallardo

60. Not Exactly Cut And Paste @pati_gallardo

61. Three APIs are in use in the renderer 1. Opengl ES2 2. Chromium GL ES2 Extensions 3. Chromium APIs @pati_gallardo

62. “All non-trivial abstractions, to some degree, are leaky.” Law of Leaky Abstractions Joel Spolsky, 2002 @pati_gallardo

63. - Ok, but… Can I Use? - Hm, don’t know… Maybe? ¯_(ツ)_/¯ @pati_gallardo

64. @pati_gallardo

65. Make Everyone Feel Safe To Be Themselves @pati_gallardo

66. Vivaldi Swag Patricia Aas, Vivaldi Technologies @pati_gallardo Photos from pixabay.com

67. Appendix @pati_gallardo

68. Copy Video Frame To GPU Memory Buffer - CopyVideoFrameToGpuMemoryBuffers - OutputFormat::NV12_SINGLE_GMB - CopyRowsToNV12Buffer - libyuv::I420ToNV12 - GpuMemoryBufferImplSharedMemory @pati_gallardo

69. VideoFrame FrameResources gfx::Size PlaneResource Mailbox Unique Name SyncToken MailboxHolder PlaneResource PlaneResource 2. CreateImageCHROMIUM GpuMemoryBuffer GpuMemoryBufferVideoFramePool Resource lifetime ownership MailboxHolder MailboxHolder 3. BindTexImage2DCHROMIUM image_id 1. BindTexture texture_target texture_id 1 to 3 1 to 3

70. CHROMIUM_image CreateImageCHROMIUM ReleaseTexImage2DCHROMIUM BindTexImage2DCHROMIUM DestroyImageCHROMIUM @pati_gallardo

71. VideoFrameProvider Client VideoFrameController Client InputHandler Client LayerTreeHostImpl VideoFrameCompositor VideoRendererSink OnBeginFrame DidDrawFrame UpdateCurrentFrame GetCurrentFrame PutCurrentFrame VideoRendererImpl Render OnFrameDropped VideoFrameProviderClientImplVideoFrameProviderClientImplVideoFrameProviderClientImpl Video Frame Painting VideoFrame current_frame_ VideoLayerImpl active_video_layer_ DecodersVideoResourceUpdater

72. Useful files to read gpu_memory_buffer_video_frame_pool.cc video_resource_updater.cc gl_renderer.cc (GLRenderer::DrawYUVVideoQuad) program_binding.cc (ProgramKey::YUVVideo) @pati_gallardo

73. SyncToken @pati_gallardo SyncToken CommandBufferNamespace release_count_ CommandBufferId

74. Share Group - Command Buffers in the same share group must be in the same Command Stream - gl::GLFence - eglFenceSyncKHR (EGL_KHR_fence_sync) - eglWaitSyncKHR (EGL_KHR_wait_sync) @pati_gallardo

75. @pati_gallardo#oslove

Isolating GPU Access in its Own Process (Foss-North 2018)

Isolating GPU Access in its Own Process (Foss-North 2018)

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Isolating GPU Access in its Own Process (Foss-North 2018)

Similar to Isolating GPU Access in its Own Process (Foss-North 2018)(20)

More from Patricia Aas

More from Patricia Aas(20)

Recently uploaded

Recently uploaded(20)

Isolating GPU Access in its Own Process (Foss-North 2018)