[COSCUP 2013] Audio Competing

489 views

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
489
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
5
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

[COSCUP 2013] Audio Competing

  1. 1. Web Developer or OS Developer? Experience from FirefoxOS Alive COSCUP 2013
  2. 2. me @alivedise http://alivedise.github.io http://github.com/alivedise alive@mozilla.com alegnadise@gmail.com ~2.5year Frontend Exp. 聽說COSCUP一定要放
  3. 3. Operating System ? http://www.flickr.com/photos/klaikong/6041270516/
  4. 4. Operating System
  5. 5. FirefoxOS….OS? Storage Management fileSystem(yet) DeviceStorageWatcher Memory Management PageVisibility mozAppType User Interface Multi Window Management Statusbar System Overlay App Overlay Web Activity Keyboard Hardware Resource Management Wi-Fi Radio GPS Bluetooth Power CPU WakeLock Audio/Media Multi-Tasking mozBrowser remote=true Card view
  6. 6. • Multi Channel • Audio Competing • Background Play Audio Management
  7. 7. The beginning of everything.. http://bugzil.la/805333
  8. 8. 805333
  9. 9. Why do we need multi- channel on OS?
  10. 10. Sounds within FxOS - <audio/>
  11. 11. Sounds within FxOS - radio
  12. 12. Sounds within FxOS - alarm
  13. 13. Sounds within FxOS - voice
  14. 14. The world of single channel.. Peace
  15. 15. But.. Too peaceful.
  16. 16. This means: When you mute the device, your alarm also is muted. So..
  17. 17. Overslept ♪♪♪~ Zzzzz..
  18. 18. How does OS avoid that? Different settings for audios on different purposes.
  19. 19. On the other hand.. Competing
  20. 20. On Desktop browser, we don’t need competing.
  21. 21. How about mobile OS?
  22. 22. Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
  23. 23. It’s all about UX. User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
  24. 24. Android solution: audio focus • http://developer.android.com/training/managing -audio/audio-focus.html • App is responsible to request the audio focus on demand and release the focus actively.
  25. 25. Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
  26. 26. Proposal • New attribute for media tags. • Background play and competing needs permission check. • Expose current occupied media info to system.
  27. 27. Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs. • A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
  28. 28. mozAudioChannelType telephony alarm notification content normal(default)
  29. 29. Multi channel volume settings • ringer, desktop-notification audio.volume.notification • music, FMRadio audio.volume.content • alarm audio.volume.alarm • telephony audio.volume.voice • Bluetooth SCO audio.volume.bt_sco
  30. 30. Usage OR manifest.webapp app.html app.js
  31. 31. Interrupted Event
  32. 32. Current Competing Policies •  Channel with higher priority interrupts the lower channel. •  Foreground page is always playable. But interruption occurs or not depends on the channel of the media element. •  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!) •  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’. content normal
  33. 33. Gecko side mozAudioChannelType audio App nsIDOMHTMLAudioElement nsIDOMHTMLMediaElement nsHTMLMediaElement nsIAudioChannelAgentCallback nsIAudioChannelAgent nsBuiltinDecoder nsMediaOmxDecoder nsMediaDecoder DataSource MediaStreamSource MediaResource - SetReadMode() create IOMX OMX MediaExtractorOMXCodec MediaSource - Tell() - Seek() - Read() - GetLength() hw codec sw codec MediaBuffer - CanPlayChanged() - mozinterruptbegin event - mozinterruptend event OmxDecoder nsHTMLAudioElement nsIChannel create nsMediaOmxReader nsBuiltinDecoderReader create create MediaOmxStateMachine nsBuiltinDecoderStateMachine nsDecoderStateMachine create nsIThread nsIThread control create nsDocument - NotifyOwnerDocumentActivityChanged() FileMediaResource ChannelMediaResource nsAudioStream nsNativeAudioStream libsydneyaudio AudioTrack AudioFlinger - Write() - Pause() - Resume() - IsPaused() - Drain() - GetMinWriteSize() - GetPosition() - SetVolume() platform: Firefox OS(gonk) branch: Firefox OS 1.01 (b2g18) will be replaced to libcubeb in v1.2 Audio Streams Diagram https://github.com/sotaroikeda/firefox-diagrams
  34. 34. Since API is implemented.. World peace?
  35. 35. No… Don’t forget the dependency tree.
  36. 36. Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
  37. 37. Policy for Silent mode • Easy for single channel • Multi channels • The default channel for volume rockers is ‘notification’. (Same as Android) • Silent mode interesting channels • notification • content/normal • Alarm is not affected by silent mode.
  38. 38. Policy for video element • video doesn’t need to play at background. • Currently, use ‘visibilitychange’ event. • video by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
  39. 39. Background play issues • Existing web pages: “Hey, I’m a music site!” • grooveshark • youtube • … • How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
  40. 40. CE Certification •  EN 50332-1 •  Output level through earphone should be under 85 dBA. •  Prompt a warning when earphone is plugged and the volume is greater than 85 dBA. •  The cool down time of CE warning popup is continuous 20 hours music listening.
  41. 41. OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
  42. 42. After all..
  43. 43. UX logic in backend and frontend

×