Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[122]책에서는 맛볼 수 없는 HTML5 Canvas 이야기

1,073 views

Published on

[122]책에서는 맛볼 수 없는 HTML5 Canvas 이야기

Published in: Technology

[122]책에서는 맛볼 수 없는 HTML5 Canvas 이야기

  1. 1. 책에서는 맛 볼 수 없는 HTML5 Canvas 이야기 Name : Jinho Bang E-Mail : zino@chromium.org
  2. 2. http://bit.ly/deview_canvas
  3. 3. 목차 (http://bit.ly/deview_canvas) ● Canvas의 역사 ● 브라우저는 어떻게 그림을 그리나? ● Canvas는 어떻게 그림을 그리나? ● Canvas Animation의 문제점 ● 기존의 Canvas Animation 개선안들 ● 새로운 API의 도입 OffscreenCanvas ● How to use Offscreen Canvas ● 사례 연구 및 그 밖의 실험
  4. 4. 1. CANVAS의 역사
  5. 5. WebCore/html/HTMLCanvasElement.h /* * Copyright (C) 2004-2017 Apple Inc. All rights reserved. * Copyright (C) 2007 Alp Toker <alp@atoker.com> * Copyright (C) 2010 Torch Mobile (Beijing) Co. Ltd. All rights reserved. * * ... * */
  6. 6. 이후 <canvas>를 도입한 브라우저들
  7. 7. Ian Hickson’s WHATWG
  8. 8. 2. 브라우저는 어떻게 그림을 그리나?
  9. 9. https://www.naver.com |
  10. 10. Rendering Engine Javascript Engine Graphics Library
  11. 11. <body> <button>Hello</button> <canvas></canvas> </body>
  12. 12. <body> <button>Hello</button> <canvas></canvas> </body> Rendering Engine
  13. 13. <body> <button>Hello</button> <canvas></canvas> </body> <head> <html> <body> . . . <button> <canvas> Parsing DOM (어떤 모양을 그릴지 결정)
  14. 14. <head> <html> <body> . . . <button> <canvas> LayoutBlock LayoutBlock LayoutButton LayoutCanvas Parsing CSS (어떤 모양을 그릴지 결정)
  15. 15. LayoutBlock LayoutBlock LayoutButton LayoutCanvas <body> <button> <canvas> (0, 0, 800, 400) (50, 0, 300, 150) (0, 0, 50, 20) Layouting (위치와 사이즈를 결정)
  16. 16. RedPaintLayer PurplePaintLayerGreenPaintLayer BluePaintLayer z-index: 2 z-index: -1 z-index: 0 z-index: 1 Layerization (그리는 순서를 결정)
  17. 17. Rendering Engine에 의해 그리는 방법 결정 LayoutCanvas Root PaintLayer PaintLayer LayoutBlock LayoutBlock LayoutButton Rendering Engine
  18. 18. Rendering Engine에 의해 그리는 방법 결정 LayoutCanvas Root PaintLayer PaintLayer LayoutBlock LayoutBlock LayoutButton Rendering Engine 중요한 것은 어떤 모양으로, 어느 위치에 어느정도 크기로, 어떤 순서로 그릴지를 결정
  19. 19. Rendering Engine에 의해 그리는 방법 결정 Root PaintLayer LayoutBlock LayoutBlock LayoutButton Graphics Library Hello drawRoundRect(); drawText(); 어떻게 그림을 그릴지에 대한 정보가 담겨있다
  20. 20. 여기서 중요한 것은 Rendering Engine이 <button></button>을 어떻게 그릴지를 알고있다
  21. 21. 3. CANVAS는 어떻게 그림을 그리나?
  22. 22. <canvas></canvas>도 HTML Element이다
  23. 23. DOM을 그릴 때와 동일한 방법으로 그린다 Graphics Library LayoutCanvas PaintLayer 내부를 어떻게 채울지에 대한 정보가 없다 비어있는 Canvas
  24. 24. <canvas></canvas>의 내부를 어떻게 그릴지는 개발자의 손에 달려있다
  25. 25. Canvas Rendering은 Javascript로.. const canvas = document.getElementById('canvas'); const context = canvas.getContext('2d'); context.drawRect(...); context.drawText(...);
  26. 26. Javascript Engine Rendering Engine Graphics Library
  27. 27. Javascript Engine Rendering Engine Graphics Library <script></script>
  28. 28. Javascript Engine Rendering Engine Graphics Library const canvas = document.getElementById('canvas'); const context = canvas.getContext('2d'); context.drawRect(...); context.drawText(...);
  29. 29. Javascript Engine Rendering Engine Graphics Library const canvas = document.getElementById('canvas'); const context = canvas.getContext('2d'); context.drawRect(...); context.drawText(...);
  30. 30. Javascript Engine Rendering Engine Graphics Library const canvas = document.getElementById('canvas'); const context = canvas.getContext('2d'); context.drawRect(...); context.drawText(...); drawRoundRect(); drawText();
  31. 31. <canvas></canvas>의 내부를 어떻게 그릴지는 개발자의 Javascript 코드에 달려있다
  32. 32. 4. CANVAS ANIMATION의 문제점
  33. 33. <canvas></canvas>의 목적은 무엇인가?
  34. 34. 다이나믹하게 변하는 그래픽을 처리
  35. 35. 즉, Animation
  36. 36. 16.7 ms V-Sync timeline 참고1: “브라우저는 VSync를 어떻게 활용하고 있을까?”, 데뷰 2015, 홍영기 - https://deview.kr/2015/schedule#session/87 참고2: “웹 성능 최적화에 필요한 브라우저의 모든 것”, 데뷰 2018, 이형욱 - https://deview.kr/2018/schedule/252
  37. 37. 16.7 ms V-Sync timeline Rendering Engine Javascript Engine Graphics Library
  38. 38. 16.7 ms PaintingDOM V-Sync timeline Javascript Idle
  39. 39. Render Loop function draw() { requestAnimationFrame(draw); // draw something in 16.7ms } requestAnimationFrame(draw);
  40. 40. Canvas Render Loop function draw() { requestAnimationFrame(draw); for (let i = 0; i < 10000000000; i++) context.drawRect(randomX, randomY, randomWidth, randomHeight); } requestAnimationFrame(draw);
  41. 41. 16.7 ms PaintingDOM V-Sync timeline의 현실 Javascript
  42. 42. Main thread가 하는 일이 너무 많아서 16.7ms 동안 그림을 그리기가 힘들다
  43. 43. 5. 기존의 CANVAS ANIMATION 개선안들
  44. 44. 브라우저 관점에서의 개선
  45. 45. 16.7 ms PaintingDOM V-Sync timeline의 현실 Javascript
  46. 46. 여러분이 만약 화가라면..
  47. 47. PaintLayer PaintLayer Recording (Main Thread) drawRect(); drawText(); ... drawRect(); ... 그림 그리는 일을 남에게 미루자
  48. 48. PaintLayer PaintLayer Recording (Main Thread) drawRect(); drawText(); ... drawRect(); ... <canvas> <button> Playback (Raster Threads) drawRect(); drawText(); ... 그림 그리는 일을 남에게 미루자
  49. 49. PaintLayer drawRect(); drawText(); ... <body> <button> <canvas> PaintLayer <canvas> Recording (Main Thread) Playback (Raster Threads) Compositing (Compositor Thread) drawRect(); ... <button> 그림 그리는 일을 남에게 미루자
  50. 50. 참고1: “Accelerated compositing in WebKit: Now and in the future”, 데뷰 2015, 황광윤 - https://deview.kr/2015/schedule#session/75 참고2: “WebKit의 GPU 렌더링”, 데뷰 2012, 황동성 - https://deview.kr/2012/xe/index.php?mid=track&document_srl=374&time_srl=265 참고3: “웹 성능 최적화에 필요한 브라우저의 모든 것”, 데뷰 2018, 이형욱 - https://deview.kr/2018/schedule/252
  51. 51. 16.7 ms16.7 ms Req Raster Canvas JavascriptMain thread Req Raster Canvas Javascript GPU Process Raster Raster Raster Raster Raster Raster GPU 가속 Canvas의 Rendering
  52. 52. 웹 개발자 관점에서의 개선
  53. 53. 개선안1: Fullscreen Canvas로 Rendering 하기
  54. 54. 개선안2: WebGL로 Rendering 하기
  55. 55. 개선안3: Background Canvas
  56. 56. Normal Canvas VS Background Canvas backContext.drawRect(); backContext.drawOval(); backContext.drawTriangle(); context.drawImage(backCanvas);
  57. 57. drawImage()를 활용한 Rendering 개선 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d');
  58. 58. drawImage()를 활용한 Rendering 개선 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = document.createElement('canvas'); const backgroundContext = backgroundCanvas.getContext('2d');
  59. 59. drawImage()를 활용한 Rendering 개선 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = document.createElement('canvas'); const backgroundContext = backgroundCanvas.getContext('2d'); drawComplexObject(backgroundContext); presentationContext.drawImage(backgroundCanvas, ...);
  60. 60. 6. 새로운 API의 도입 OFFSCREEN CANVAS
  61. 61. 많은 개선 방법에도 불구하고.. ● Javascript Engine과 Rendering Engine 사이의 Binding Overhead ● GPU 가속을 사용하더라도 Skia 내부의 Overhead ● DOM Rendering을 처리하기에도 벅찬 Main-thread
  62. 62. 16.7 ms16.7 ms Req Raster Canvas JavascriptMain thread Req Raster Canvas Javascript GPU Process Raster Raster Raster Raster Raster Raster 이상적인 GPU 가속 Canvas의 Rendering
  63. 63. 16.7 ms16.7 ms Main thread Javascript Overhead Skia Overhead DOM Overhead GPU Process Raster Raster Raster Raster Raster Raster 다시 참혹한 현실..
  64. 64. 새로운 방법이 필요하다 ● Javascript의 부하를 줄이기 위해 다른 thread에서 실행할 수 있으면 좋겠다 ● 그리 크진 않지만 Skia의 부하를 줄이기 위해 다른 thread에서 실행하면 좋겠다 ● DOM Rendering과 분리되면 좋겠다
  65. 65. 결국 우리는 다른 thead에서 Canvas Rendering을 하고싶다
  66. 66. WebWorker
  67. 67. <canvas></canvas>는 DOM의 일부이기 때문에 불가
  68. 68. DOM으로부터 분리할 수 있는 방법은 없을까?
  69. 69. OffscreenCanvas!
  70. 70. OffscreenCanvas란? ● Canvas Rendering을 DOM과는 별개로 Worker thread에서 수행할 수 있도록 해주는 새로운 API ● 기존의 Canvas Rendering Logic은 고치지 않고 thread만 옮겨가서 그대로 수행할 수 있도록 고안 ● Chrome 69 Stable에 이미 Shipping
  71. 71. 16.7 ms16.7 ms Main thread Javascript Overhead Skia Overhead DOM Overhead GPU Process Raster Raster Raster Raster Raster Raster 다시 참혹한 현실..
  72. 72. 16.7 ms16.7 ms Main thread Javascript DOM Rendering Worker thread Canvas Javascript Skia Canvas Javascript Skia Skia OffscreenCanvas를 도입하면..
  73. 73. 16.7 ms16.7 ms Main thread Javascript DOM Rendering Worker thread Canvas Javascript Skia Canvas Javascript Skia Skia GPU Process Raster Raster Raster Raster Raster Raster OffscreenCanvas를 도입하면..
  74. 74. 7. HOW TO USE OFFSCREEN CANVAS
  75. 75. Main-thread에서 해야 할 일..
  76. 76. Canvas 객체 얻어오기 const canvas = document.getElementById(‘canvas’);
  77. 77. OffscreenCanvas 객체 얻어오기 const canvas = document.getElementById(‘canvas’); const offscreen = canvas.transferControlToOffscreen();
  78. 78. Worker thread 생성하기 const canvas = document.getElementById(‘canvas’); const offscreen = canvas.transferControlToOffscreen(); const worker = new Worker(‘canvas-worker.js’);
  79. 79. Worker thread로 OffscreenCanvas 전달 const canvas = document.getElementById(‘canvas’); const offscreen = canvas.transferControlToOffscreen(); const worker = new Worker(‘canvas-worker.js’); worker.postMessage({ canvas: offscreen }, [offscreen]);
  80. 80. Worker thread에서 해야 할 일..
  81. 81. Message Handler 설정 self.onmessage = event => { };
  82. 82. OffscreenCanvas 전달받기 self.onmessage = event => { const offscreen = event.data.canvas; };
  83. 83. CanvasRenderingContext 생성 self.onmessage = event => { const offscreen = event.data.canvas; const context = canvas.getContext(‘2d’); };
  84. 84. 일반적인 Canvas와 동일하게 그림을 그린다 self.onmessage = event => { const offscreen = event.data.canvas; const context = canvas.getContext(‘2d’); function render(time) { drawSomething(context); requestAnimationFrame(render); } requestAnimationFrame(render); };
  85. 85. 16.7 ms16.7 ms Main thread Javascript DOM Rendering Worker thread Canvas Javascript Skia Canvas Javascript Skia Skia GPU Process Raster Raster Raster Raster Raster Raster OffscreenCanvas를 도입하면..
  86. 86. DEMO
  87. 87. 8. CASE STUDY
  88. 88. CASE1: Three.js에서 OffscreenCanvas 사용하기
  89. 89. The aim of the project is to create an easy to use, lightweight, 3D library. The library provides Canvas 2D, SVG, CSS3D and WebGL renderers.
  90. 90. 일반적인 Three.js의 Renderer 생성 const renderer = new THREE.WebGLRenderer(); renderer.render(scene, camera);
  91. 91. Three.js에서 OffscreenCanvas 사용 const offscreen = canvas.transferControlToOffscreen(); const renderer = new THREE.WebGLRenderer({ canvas: offscreen }); renderer.render(scene, camera);
  92. 92. 일반적인 Three.js의 Texture 생성 // TextureLoader has a DOM dependency (HTMLImageElement) const t = new THREE.TextureLoader().load('textures/crate.gif');
  93. 93. OffscreenCanvas를 사용할 때 Texture 생성 const loader = new THREE.ImageBitmapLoader(); loader.load( '../../textures/crate.gif', imageBitmap => { const texture = new THREE.CanvasTexture(imageBitmap); }, ...);
  94. 94. 여기까지 하고나면 대부분 잘 동작한다
  95. 95. 한 가지 또 다른 문제점은..
  96. 96. Three.js의 내부 setSize() this.setSize = function (width, height, updateStyle) { ... if ( updateStyle !== false ) { _canvas.style.width = width + 'px'; _canvas.style.height = height + 'px'; } ... };
  97. 97. OffscreenCanvas는 DOM의 일부가 아니므로 style property가 존재하지 않는다
  98. 98. Uncaught TypeError: Cannot read property ‘width’ of undefined
  99. 99. style property에 대한 stub을 만들어 준다 const offscreen = canvas.transferControlToOffscreen(); offscreen.style = { width: 0, height: 0 }; const renderer = new THREE.WebGLRenderer({ canvas: offscreen }); renderer.render(scene, camera);
  100. 100. 여기까지 하고나면 대부분 잘 동작한다
  101. 101. CASE2: Zero-copy Background Rendering
  102. 102. 앞서 살펴 본 “drawImage()를 통한 개선”을 더 효율적인 방법으로 대체할 수 있음
  103. 103. drawImage()를 활용한 Rendering 개선 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = document.createElement('canvas'); const backgroundContext = backgroundCanvas.getContext('2d'); drawComplexObject(backgroundContext); presentationContext.drawImage(backgroundCanvas, ...);
  104. 104. 이 방법으로는 Worker에서 그릴 수 없고, Image 복사가 발생한다
  105. 105. OffscreenCanvas를 활용하여 개선하자!
  106. 106. OffscreenCanvas의 Background Rendering const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = new OffscreenCanvas(width, height); const backgroundContext = backgroundCanvas.getContext('2d'); drawComplexObject(backgroundContext); const image = backgroundCanvas.transferToImageBitmap(); presentationContext.drawImage(image, ...);
  107. 107. TODO: Image zero-copy 그림
  108. 108. CASE3: Multi-view rendering with WebGL
  109. 109. OffscreenCanvas의 Multi-view Rendering const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = new OffscreenCanvas(width, height); const backgroundContext = backgroundCanvas.getContext('webgl'); drawComplexObject(backgroundContext); let snapshot = backgroundCanvas.transferToImageBitmap(); presentationContext.drawImage(snapshot, ...); rotateObject(backgroundContext); snapshot = backgroundCanvas.transferToImageBitmap(); presentationContext.drawImage(snapshot, ...);
  110. 110. Multi-view Rendering 개선하기 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext('2d'); const backgroundCanvas = new OffscreenCanvas(width, height); const backgroundContext = backgroundCanvas.getContext('webgl'); drawComplexObject(backgroundContext); let snapshot = backgroundCanvas.transferToImageBitmap(); presentationContext.drawImage(snapshot, ...);
  111. 111. Multi-view Rendering 개선하기 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext(‘bitmaprenderer’); const backgroundCanvas = new OffscreenCanvas(width, height); const backgroundContext = backgroundCanvas.getContext('webgl'); drawComplexObject(backgroundContext); let snapshot = backgroundCanvas.transferToImageBitmap(); presentationContext.drawImage(snapshot, ...);
  112. 112. Multi-view Rendering 개선하기 const presentationCanvas = document.getElementById('canvas'); const presentationContext = presentationCanvas.getContext(‘bitmaprenderer’); const backgroundCanvas = new OffscreenCanvas(width, height); const backgroundContext = backgroundCanvas.getContext('webgl'); drawComplexObject(backgroundContext); let snapshot = backgroundCanvas.transferToImageBitmap(); presentationContext.transferFromImageBitmap(snapshot);
  113. 113. CASE4: WebXR with OffscreenCanvas
  114. 114. CASE5: Tensorflow.js with OffscreenCanvas
  115. 115. Q & A
  116. 116. Thank you

×