Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Embedded Recipes 2019 - Pipewire a new foundation for embedded multimedia

140 views

Published on

PipeWire is an open source project that aims to greatly improve audio and video handling under Linux. Utilising a fresh design, it bridges use cases that have been previously addressed by different tools – or not addressed at all -, providing ground for building complex, yet secure and efficient, multimedia systems.

In this talk, Julien is going to present the PipeWire project and the concepts that make up its design. In addition, he is going to give an update of the current and future work going on around PipeWire, both upstream and in Automotive Grade Linux, an early adopter that Julien is actively working on.
Julian Bouzas

Published in: Software
  • Be the first to comment

  • Be the first to like this

Embedded Recipes 2019 - Pipewire a new foundation for embedded multimedia

  1. 1. Open FirstOpen First PipeWire: a new foundation for the embedded multimedia Julian Bouzas Multimedia Software Engineer
  2. 2. 2 What is PipeWire?
  3. 3. 3 Fresh multimedia framework for linux ● Written by Wim Taymans (creator of PulseAudio) ● Daemon that handles both audio and video devices ● Handles format conversion, mixing, and re-position ● Pipeline Based Design ● Security Permissions ● Like PulseAudio, but for video Video as well
  4. 4. 4 PipeWire ● Initial idea: PulseAudio for video (PulseVideo) ● Now: generic multimedia platform service – Video capture server ● Camera and other video sources (ex. gnome-shell screencast) – Audio server ● PulseAudio and JACK (pro-audio) replacement ● Borrowing ideas also from CoreAudio, AudioFlinger, and others…
  5. 5. 5 The Multimedia Stack
  6. 6. 7 External PipeWire Session Manager ● Setup of devices – Format – Mixers – Effects ● Creation of links and nodes ● Grants security and access control of clients ● Prioritize Devices – Policy (User Specific)
  7. 7. 8 Security ● Fine-grained object access controls per client – Visible, Read data (R) – Write data (W) – Execute methods (X) ● Each client can be made to “see” an entirely different graph ● Session manager applies permissions
  8. 8. 9 Design ● Modules and Plugins ● Plugins based on SPA (Simple Plugin API) – Header-only C library with zero dependencies – Extremely lightweight data structures – “Like Glib, but not so heavy” - Wim Tayman
  9. 9. 10 Performance & Efficiency ● Uses modern linux kernel APIs: – memfd & dmabuf to zero-copy buffers – eventfd & timerfd to wake up the processes ● Low-latency real-time capable – < 1.5 milliseconds possible on desktop ● Much lower CPU usage than PulseAudio
  10. 10. 11 CPU Usage Statistics Playback of a 24bit 96kHz 5.1 channel file, downmixed to 3.1 and resampled to 48kHz 21.33 ms 1.33 ms 2 clients @ 800 MHz 0.7 2.3 2.7 6 % CPU ● Measurements: – 21.33 ms (1024 samples / buffer) – 1.33 ms (64 samples / buffer) – 1.33 ms with 2 clients – 1.33 ms with CPU pinned @ 800 MHz ● Measurements on Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz ● Comparatively, on 1.33 ms, PulseAudio uses 100% CPU and fails (underruns)
  11. 11. 12 Compatibility APIs ● ALSA apps – PipeWire PCM plugin – Built with streams API ● PulseAudio apps – Replacement libpulse.so, libpulse-mainloop-glib.so – Built with streams API ● JACK apps – Replacement libjack.so – Built on top of low-level PipeWire
  12. 12. 13 Who is behind this ● Author: Wim Taymans – Well-known old GStreamer developer & ex-maintainer – Sponsored by: Red Hat ● Embraced by PulseAudio developers – Seen as the next generation of PulseAudio ● Welcomed by ALSA and JACK developers ● License: MIT
  13. 13. 14 Status ● Version 0.2 distributed in fedora – Used for video only ● Version 0.3 to be released soon – Estimated for later this year – A lot of audio work & refactoring done – Bluetooth starting to be fully supported – Used in AGL (Automotive Grade Linux)
  14. 14. 15 PipeWire in AGL
  15. 15. 16 PipeWire in Automotive Grade Linux ● Audio system implemented with PipeWire for Connected Cars – Replaces PulseAudio – Mixer, Media player & Radio player using the native API – ALSA compatibility plugin available ● Session manager: WirePlumber ● Merged in Happy Halibut (8.0.0) RC2 ● In the future also: video
  16. 16. 17 WirePlumber ● First session manager implementation ● Target: reusable session manager for embedded use cases – we’ll see about desktop… ● Modular & extensible, like PipeWire ● Based on GObject – To support writing modules using bindings in other languages
  17. 17. 18 WirePlumber concepts ● Graph abstraction: Endpoint – Closer to PulseAudio’s representation (sources, sinks, source-outputs, sink-inputs) – Described by name & media-class (Audio/Source, Audio/Sink, Stream/Audio/Input, Stream/Audio/Output) ● Extensibility: Modules & Factories like PipeWire ● Pipewire Proxies and Properties – GObject-ify the pipewire API (ease of use + access from bindings)
  18. 18. 19 Software DSP Endpoint
  19. 19. 20 Hardware DSP Endpoint (Unicens)
  20. 20. 21 Policy Management ● Decided what app nodes are linked with what device nodes ● Fully configurable with TOML files ● Default Configuration: – Clients have roles with priorities: ● Multimedia, Navigation, Emergency, ... – Highest priority wins – Equal priority: last one wins – Audio in other apps is automatically paused & restored
  21. 21. 22 Other areas needing work ● Finish Bluetooth audio support (planned for AGL 8.0.2) ● Unicens hardware support (planned for AGL 8.0.3) ● Improve API and make it stable – We are constantly changing it and figuring out what the best API could be ● Better security – Mechanisms exist but currently all clients are granted full permissions ● Documentation
  22. 22. 23 PipeWire Audio Contributions
  23. 23. 24 Audio Stream (pw_stream) ● Nicer than PipeWire low-level API ● Takes input from client (asynchronously) ● Does conversion – Resampling – Channel mixing / volume – Format conversion – Channel splitting into DSP – Decouples server buffer size from client requested latency ● Flush / drain
  24. 24. 25 In GStreamer ● pipewiresrc / pipwiresink – Available upstream – Built mainly for video; have issues with audio at the moment ● pwaudiosrc / pwaudiosink – Available in AGL – Smooth operation with audio – Upstream-Status: Submitted [https://github.com/PipeWire/pipewire/pull/140] ● All built with the streams API
  25. 25. 26 Supported Bluetooth Profiles ● Only A2DP Sink was included – For high quality audio over bluetooth sinks (music - speakers) ● Included A2DP Source node – For high quality audio over bluetooth sources (music - phones) ● Started including HSP/HFP Sink and Source nodes – For comunications (voice – phones) – 2 Setups: Head Unit, Gateway
  26. 26. 27 Showtime
  27. 27. 28 Thank you!
  28. 28. 29 We are hiring col.la/careers

×