SlideShare a Scribd company logo
WebCodecs in WebKit
WebCodecs in WebKit
With GStreamer!
With GStreamer!
Philippe Normand
Philippe Normand
GStreamer conference 2023
GStreamer conference 2023
1
1 /
/ 8
8
WebCodecs, what is this?
WebCodecs, what is this?
“This specification defines interfaces to codecs for encoding and decoding of
“This specification defines interfaces to codecs for encoding and decoding of
audio, video, and images.”
audio, video, and images.”
Gives JS direct access to encoders and decoders provided by the WebEngine
Gives JS direct access to encoders and decoders provided by the WebEngine
Ability to modify
Ability to modify VideoFrame
VideoFrame s and
s and AudioData
AudioData before encoding or after
before encoding or after
decoding
decoding
No need for 3rd-party WASM-based framework
No need for 3rd-party WASM-based framework
Muxing and demuxing handled at JS layer (MP4Box.js, etc)
Muxing and demuxing handled at JS layer (MP4Box.js, etc)
https://www.w3.org/TR/webcodecs/
https://www.w3.org/TR/webcodecs/
2
2 /
/ 8
8
WebCodecs, use-cases
WebCodecs, use-cases
Video editors
Video editors
WebRTC “Funny hats”
WebRTC “Funny hats”
Integration with WebAudio, WebGL, WebTransport
Integration with WebAudio, WebGL, WebTransport
{en,de}coding can be moved to another thread with WebWorkers
{en,de}coding can be moved to another thread with WebWorkers
3
3 /
/ 8
8
WebCodecs, in WebKit!
WebCodecs, in WebKit!
First approach with
First approach with appsrc
appsrc !
! {en,de}codebin
{en,de}codebin !
! appsink
appsink worked, but was not
worked, but was not
spec compliant
spec compliant
We need “direct” access to the codec element, no additional thread (queue)
We need “direct” access to the codec element, no additional thread (queue)
Push data to sink pad, pull from src pad
Push data to sink pad, pull from src pad
Something a bit similar to GstHarness
Something a bit similar to GstHarness
4
4 /
/ 8
8
WebKit’s GStreamerElementHarness
WebKit’s GStreamerElementHarness
A C++ class, no pipeline per-se
A C++ class, no pipeline per-se
Simple API,
Simple API, pushBuffer()
pushBuffer() ,
, pushEvent()
pushEvent() ,
, flush()
flush() ,…
,…
Internal state: output buffers and events stored in Double Ended Queues (deque)
Internal state: output buffers and events stored in Double Ended Queues (deque)
Simple case: one input pad, one output pad:
Simple case: one input pad, one output pad:
decoder or encoder
decoder or encoder
sink
sink src
src
src
src sink
sink
Push data on src pad
Push data on src pad
Processing ends in … sink pad
Processing ends in … sink pad
C++ closure-based processing triggered with
C++ closure-based processing triggered with ::processOutputBuffers()
::processOutputBuffers()
5
5 /
/ 8
8
Chaining
Chaining
one to many pads
one to many pads
callback-based “plugging”
callback-based “plugging”
“sometimes” pad handling, with possibility of sending buffers to an
“sometimes” pad handling, with possibility of sending buffers to an identity
identity -
-
based harness
based harness
decoder
decoder
parsebin
parsebin
sink
sink src
src
sink
sink src
src sink
sink src
src
src
src sink
sink
6
6 /
/ 8
8
Implementation status in WebKitGTK &
Implementation status in WebKitGTK &
WPEWebKit
WPEWebKit
(H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42
(H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42
(AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding:
(AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding:
landed in main, will ship in 2.44
landed in main, will ship in 2.44
WebPlatform tests coverage is OK
WebPlatform tests coverage is OK
Room for improvement regarding interaction with WebAudio and WebGL
Room for improvement regarding interaction with WebAudio and WebGL
7
7 /
/ 8
8
Demo
Demo
All usable from the GNOME Web Canary flatpak
All usable from the GNOME Web Canary flatpak
Ask me!
Ask me!
8
8 /
/ 8
8
WebCodecs in WebKit With GStreamer!

More Related Content

Similar to WebCodecs in WebKit With GStreamer!

IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
Christopher Diamantopoulos
 
Introduction to WebRTC
Introduction to WebRTCIntroduction to WebRTC
Introduction to WebRTC
Patrick Cason
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVM
John Lee
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eve
chiportal
 
[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래
NAVER D2
 
Shakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud PlatformShakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud Platform
Minku Lee
 
Project ACRN hypervisor introduction
Project ACRN hypervisor introduction Project ACRN hypervisor introduction
Project ACRN hypervisor introduction
Project ACRN
 
How to write clean & testable code without losing your mind
How to write clean & testable code without losing your mindHow to write clean & testable code without losing your mind
How to write clean & testable code without losing your mind
Andreas Czakaj
 
HKG15-300: Art's Quick Compiler: An unofficial overview
HKG15-300: Art's Quick Compiler: An unofficial overviewHKG15-300: Art's Quick Compiler: An unofficial overview
HKG15-300: Art's Quick Compiler: An unofficial overview
Linaro
 
OOPSLA Talk on Preon
OOPSLA Talk on PreonOOPSLA Talk on Preon
OOPSLA Talk on Preon
Wilfred Springer
 
Seaside Portability
Seaside PortabilitySeaside Portability
Seaside Portability
jfitzell
 
Web of Technologies
Web of TechnologiesWeb of Technologies
Web of Technologies
dynamis
 
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdfDo it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
adamsapparelsformen
 
Software Define your Current Storage with Opensource
Software Define your Current Storage with OpensourceSoftware Define your Current Storage with Opensource
Software Define your Current Storage with Opensource
Antonio Romeo
 
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
Akihiro Suda
 
Docker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshitDocker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshit
Yusuf Hadiwinata Sutandar
 
Parsing and Type checking all 2^10000 configurations of the Linux kernel
Parsing and Type checking all 2^10000 configurations of the Linux kernelParsing and Type checking all 2^10000 configurations of the Linux kernel
Parsing and Type checking all 2^10000 configurations of the Linux kernel
chk49
 
Simplify Networking for Containers
Simplify Networking for ContainersSimplify Networking for Containers
Simplify Networking for Containers
LinuxCon ContainerCon CloudOpen China
 
Building a modern SaaS in 2020
Building a modern SaaS in 2020Building a modern SaaS in 2020
Building a modern SaaS in 2020
Nikolay Stoitsev
 
NetDevOps Developer Environments with Vagrant @ SCALE16x
NetDevOps Developer Environments with Vagrant @ SCALE16xNetDevOps Developer Environments with Vagrant @ SCALE16x
NetDevOps Developer Environments with Vagrant @ SCALE16x
Hank Preston
 

Similar to WebCodecs in WebKit With GStreamer! (20)

IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
 
Introduction to WebRTC
Introduction to WebRTCIntroduction to WebRTC
Introduction to WebRTC
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVM
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eve
 
[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래
 
Shakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud PlatformShakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud Platform
 
Project ACRN hypervisor introduction
Project ACRN hypervisor introduction Project ACRN hypervisor introduction
Project ACRN hypervisor introduction
 
How to write clean & testable code without losing your mind
How to write clean & testable code without losing your mindHow to write clean & testable code without losing your mind
How to write clean & testable code without losing your mind
 
HKG15-300: Art's Quick Compiler: An unofficial overview
HKG15-300: Art's Quick Compiler: An unofficial overviewHKG15-300: Art's Quick Compiler: An unofficial overview
HKG15-300: Art's Quick Compiler: An unofficial overview
 
OOPSLA Talk on Preon
OOPSLA Talk on PreonOOPSLA Talk on Preon
OOPSLA Talk on Preon
 
Seaside Portability
Seaside PortabilitySeaside Portability
Seaside Portability
 
Web of Technologies
Web of TechnologiesWeb of Technologies
Web of Technologies
 
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdfDo it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
Do it in Java Please ExamPrep4_Spring2023 Source Packages lo.pdf
 
Software Define your Current Storage with Opensource
Software Define your Current Storage with OpensourceSoftware Define your Current Storage with Opensource
Software Define your Current Storage with Opensource
 
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
[DockerCon 2023] Reproducible builds with BuildKit for software supply chain ...
 
Docker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshitDocker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshit
 
Parsing and Type checking all 2^10000 configurations of the Linux kernel
Parsing and Type checking all 2^10000 configurations of the Linux kernelParsing and Type checking all 2^10000 configurations of the Linux kernel
Parsing and Type checking all 2^10000 configurations of the Linux kernel
 
Simplify Networking for Containers
Simplify Networking for ContainersSimplify Networking for Containers
Simplify Networking for Containers
 
Building a modern SaaS in 2020
Building a modern SaaS in 2020Building a modern SaaS in 2020
Building a modern SaaS in 2020
 
NetDevOps Developer Environments with Vagrant @ SCALE16x
NetDevOps Developer Environments with Vagrant @ SCALE16xNetDevOps Developer Environments with Vagrant @ SCALE16x
NetDevOps Developer Environments with Vagrant @ SCALE16x
 

More from Igalia

The journey towards stabilizing Chromium’s Wayland support
The journey towards stabilizing Chromium’s Wayland supportThe journey towards stabilizing Chromium’s Wayland support
The journey towards stabilizing Chromium’s Wayland support
Igalia
 
Sustainable Futures: Funding the Web Ecosystem
Sustainable Futures: Funding the Web EcosystemSustainable Futures: Funding the Web Ecosystem
Sustainable Futures: Funding the Web Ecosystem
Igalia
 
Status of the Layer-Based SVG Engine in WebKit
Status of the Layer-Based SVG Engine in WebKitStatus of the Layer-Based SVG Engine in WebKit
Status of the Layer-Based SVG Engine in WebKit
Igalia
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
 
Building End-user Applications on Embedded Devices with WPE
Building End-user Applications on Embedded Devices with WPEBuilding End-user Applications on Embedded Devices with WPE
Building End-user Applications on Embedded Devices with WPE
Igalia
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Igalia
 
Automated Testing for Web-based Systems on Embedded Devices
Automated Testing for Web-based Systems on Embedded DevicesAutomated Testing for Web-based Systems on Embedded Devices
Automated Testing for Web-based Systems on Embedded Devices
Igalia
 
Embedding WPE WebKit - from Bring-up to Maintenance
Embedding WPE WebKit - from Bring-up to MaintenanceEmbedding WPE WebKit - from Bring-up to Maintenance
Embedding WPE WebKit - from Bring-up to Maintenance
Igalia
 
Optimizing Scheduler for Linux Gaming.pdf
Optimizing Scheduler for Linux Gaming.pdfOptimizing Scheduler for Linux Gaming.pdf
Optimizing Scheduler for Linux Gaming.pdf
Igalia
 
Running JS via WASM faster with JIT
Running JS via WASM      faster with JITRunning JS via WASM      faster with JIT
Running JS via WASM faster with JIT
Igalia
 
To crash or not to crash: if you do, at least recover fast!
To crash or not to crash: if you do, at least recover fast!To crash or not to crash: if you do, at least recover fast!
To crash or not to crash: if you do, at least recover fast!
Igalia
 
Implementing a Vulkan Video Encoder From Mesa to GStreamer
Implementing a Vulkan Video Encoder From Mesa to GStreamerImplementing a Vulkan Video Encoder From Mesa to GStreamer
Implementing a Vulkan Video Encoder From Mesa to GStreamer
Igalia
 
8 Years of Open Drivers, including the State of Vulkan in Mesa
8 Years of Open Drivers, including the State of Vulkan in Mesa8 Years of Open Drivers, including the State of Vulkan in Mesa
8 Years of Open Drivers, including the State of Vulkan in Mesa
Igalia
 
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por IgaliaIntroducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
Igalia
 
2023 in Chimera Linux
2023 in Chimera                    Linux2023 in Chimera                    Linux
2023 in Chimera Linux
Igalia
 
Building a Linux distro with LLVM
Building a Linux distro        with LLVMBuilding a Linux distro        with LLVM
Building a Linux distro with LLVM
Igalia
 
turnip: Update on Open Source Vulkan Driver for Adreno GPUs
turnip: Update on Open Source Vulkan Driver for Adreno GPUsturnip: Update on Open Source Vulkan Driver for Adreno GPUs
turnip: Update on Open Source Vulkan Driver for Adreno GPUs
Igalia
 
Graphics stack updates for Raspberry Pi devices
Graphics stack updates for Raspberry Pi devicesGraphics stack updates for Raspberry Pi devices
Graphics stack updates for Raspberry Pi devices
Igalia
 
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOSDelegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
Igalia
 
MessageFormat: The future of i18n on the web
MessageFormat: The future of i18n on the webMessageFormat: The future of i18n on the web
MessageFormat: The future of i18n on the web
Igalia
 

More from Igalia (20)

The journey towards stabilizing Chromium’s Wayland support
The journey towards stabilizing Chromium’s Wayland supportThe journey towards stabilizing Chromium’s Wayland support
The journey towards stabilizing Chromium’s Wayland support
 
Sustainable Futures: Funding the Web Ecosystem
Sustainable Futures: Funding the Web EcosystemSustainable Futures: Funding the Web Ecosystem
Sustainable Futures: Funding the Web Ecosystem
 
Status of the Layer-Based SVG Engine in WebKit
Status of the Layer-Based SVG Engine in WebKitStatus of the Layer-Based SVG Engine in WebKit
Status of the Layer-Based SVG Engine in WebKit
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Building End-user Applications on Embedded Devices with WPE
Building End-user Applications on Embedded Devices with WPEBuilding End-user Applications on Embedded Devices with WPE
Building End-user Applications on Embedded Devices with WPE
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Automated Testing for Web-based Systems on Embedded Devices
Automated Testing for Web-based Systems on Embedded DevicesAutomated Testing for Web-based Systems on Embedded Devices
Automated Testing for Web-based Systems on Embedded Devices
 
Embedding WPE WebKit - from Bring-up to Maintenance
Embedding WPE WebKit - from Bring-up to MaintenanceEmbedding WPE WebKit - from Bring-up to Maintenance
Embedding WPE WebKit - from Bring-up to Maintenance
 
Optimizing Scheduler for Linux Gaming.pdf
Optimizing Scheduler for Linux Gaming.pdfOptimizing Scheduler for Linux Gaming.pdf
Optimizing Scheduler for Linux Gaming.pdf
 
Running JS via WASM faster with JIT
Running JS via WASM      faster with JITRunning JS via WASM      faster with JIT
Running JS via WASM faster with JIT
 
To crash or not to crash: if you do, at least recover fast!
To crash or not to crash: if you do, at least recover fast!To crash or not to crash: if you do, at least recover fast!
To crash or not to crash: if you do, at least recover fast!
 
Implementing a Vulkan Video Encoder From Mesa to GStreamer
Implementing a Vulkan Video Encoder From Mesa to GStreamerImplementing a Vulkan Video Encoder From Mesa to GStreamer
Implementing a Vulkan Video Encoder From Mesa to GStreamer
 
8 Years of Open Drivers, including the State of Vulkan in Mesa
8 Years of Open Drivers, including the State of Vulkan in Mesa8 Years of Open Drivers, including the State of Vulkan in Mesa
8 Years of Open Drivers, including the State of Vulkan in Mesa
 
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por IgaliaIntroducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por Igalia
 
2023 in Chimera Linux
2023 in Chimera                    Linux2023 in Chimera                    Linux
2023 in Chimera Linux
 
Building a Linux distro with LLVM
Building a Linux distro        with LLVMBuilding a Linux distro        with LLVM
Building a Linux distro with LLVM
 
turnip: Update on Open Source Vulkan Driver for Adreno GPUs
turnip: Update on Open Source Vulkan Driver for Adreno GPUsturnip: Update on Open Source Vulkan Driver for Adreno GPUs
turnip: Update on Open Source Vulkan Driver for Adreno GPUs
 
Graphics stack updates for Raspberry Pi devices
Graphics stack updates for Raspberry Pi devicesGraphics stack updates for Raspberry Pi devices
Graphics stack updates for Raspberry Pi devices
 
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOSDelegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOS
 
MessageFormat: The future of i18n on the web
MessageFormat: The future of i18n on the webMessageFormat: The future of i18n on the web
MessageFormat: The future of i18n on the web
 

Recently uploaded

Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 

Recently uploaded (20)

Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 

WebCodecs in WebKit With GStreamer!

  • 1. WebCodecs in WebKit WebCodecs in WebKit With GStreamer! With GStreamer! Philippe Normand Philippe Normand GStreamer conference 2023 GStreamer conference 2023 1 1 / / 8 8
  • 2. WebCodecs, what is this? WebCodecs, what is this? “This specification defines interfaces to codecs for encoding and decoding of “This specification defines interfaces to codecs for encoding and decoding of audio, video, and images.” audio, video, and images.” Gives JS direct access to encoders and decoders provided by the WebEngine Gives JS direct access to encoders and decoders provided by the WebEngine Ability to modify Ability to modify VideoFrame VideoFrame s and s and AudioData AudioData before encoding or after before encoding or after decoding decoding No need for 3rd-party WASM-based framework No need for 3rd-party WASM-based framework Muxing and demuxing handled at JS layer (MP4Box.js, etc) Muxing and demuxing handled at JS layer (MP4Box.js, etc) https://www.w3.org/TR/webcodecs/ https://www.w3.org/TR/webcodecs/ 2 2 / / 8 8
  • 3. WebCodecs, use-cases WebCodecs, use-cases Video editors Video editors WebRTC “Funny hats” WebRTC “Funny hats” Integration with WebAudio, WebGL, WebTransport Integration with WebAudio, WebGL, WebTransport {en,de}coding can be moved to another thread with WebWorkers {en,de}coding can be moved to another thread with WebWorkers 3 3 / / 8 8
  • 4. WebCodecs, in WebKit! WebCodecs, in WebKit! First approach with First approach with appsrc appsrc ! ! {en,de}codebin {en,de}codebin ! ! appsink appsink worked, but was not worked, but was not spec compliant spec compliant We need “direct” access to the codec element, no additional thread (queue) We need “direct” access to the codec element, no additional thread (queue) Push data to sink pad, pull from src pad Push data to sink pad, pull from src pad Something a bit similar to GstHarness Something a bit similar to GstHarness 4 4 / / 8 8
  • 5. WebKit’s GStreamerElementHarness WebKit’s GStreamerElementHarness A C++ class, no pipeline per-se A C++ class, no pipeline per-se Simple API, Simple API, pushBuffer() pushBuffer() , , pushEvent() pushEvent() , , flush() flush() ,… ,… Internal state: output buffers and events stored in Double Ended Queues (deque) Internal state: output buffers and events stored in Double Ended Queues (deque) Simple case: one input pad, one output pad: Simple case: one input pad, one output pad: decoder or encoder decoder or encoder sink sink src src src src sink sink Push data on src pad Push data on src pad Processing ends in … sink pad Processing ends in … sink pad C++ closure-based processing triggered with C++ closure-based processing triggered with ::processOutputBuffers() ::processOutputBuffers() 5 5 / / 8 8
  • 6. Chaining Chaining one to many pads one to many pads callback-based “plugging” callback-based “plugging” “sometimes” pad handling, with possibility of sending buffers to an “sometimes” pad handling, with possibility of sending buffers to an identity identity - - based harness based harness decoder decoder parsebin parsebin sink sink src src sink sink src src sink sink src src src src sink sink 6 6 / / 8 8
  • 7. Implementation status in WebKitGTK & Implementation status in WebKitGTK & WPEWebKit WPEWebKit (H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42 (H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42 (AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding: (AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding: landed in main, will ship in 2.44 landed in main, will ship in 2.44 WebPlatform tests coverage is OK WebPlatform tests coverage is OK Room for improvement regarding interaction with WebAudio and WebGL Room for improvement regarding interaction with WebAudio and WebGL 7 7 / / 8 8
  • 8. Demo Demo All usable from the GNOME Web Canary flatpak All usable from the GNOME Web Canary flatpak Ask me! Ask me! 8 8 / / 8 8