Your SlideShare is downloading. ×
[COSCUP 2013] Audio Competing
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

[COSCUP 2013] Audio Competing


Published on

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Web Developer or OS Developer? Experience from FirefoxOS Alive COSCUP 2013
  • 2. me @alivedise ~2.5year Frontend Exp. 聽說COSCUP一定要放
  • 3. Operating System ?
  • 4. Operating System
  • 5. FirefoxOS….OS? Storage Management fileSystem(yet) DeviceStorageWatcher Memory Management PageVisibility mozAppType User Interface Multi Window Management Statusbar System Overlay App Overlay Web Activity Keyboard Hardware Resource Management Wi-Fi Radio GPS Bluetooth Power CPU WakeLock Audio/Media Multi-Tasking mozBrowser remote=true Card view
  • 6. • Multi Channel • Audio Competing • Background Play Audio Management
  • 7. The beginning of everything..
  • 8. 805333
  • 9. Why do we need multi- channel on OS?
  • 10. Sounds within FxOS - <audio/>
  • 11. Sounds within FxOS - radio
  • 12. Sounds within FxOS - alarm
  • 13. Sounds within FxOS - voice
  • 14. The world of single channel.. Peace
  • 15. But.. Too peaceful.
  • 16. This means: When you mute the device, your alarm also is muted. So..
  • 17. Overslept ♪♪♪~ Zzzzz..
  • 18. How does OS avoid that? Different settings for audios on different purposes.
  • 19. On the other hand.. Competing
  • 20. On Desktop browser, we don’t need competing.
  • 21. How about mobile OS?
  • 22. Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
  • 23. It’s all about UX. User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
  • 24. Android solution: audio focus • -audio/audio-focus.html • App is responsible to request the audio focus on demand and release the focus actively.
  • 25. Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
  • 26. Proposal • New attribute for media tags. • Background play and competing needs permission check. • Expose current occupied media info to system.
  • 27. Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs. • A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
  • 28. mozAudioChannelType telephony alarm notification content normal(default)
  • 29. Multi channel volume settings • ringer, desktop-notification audio.volume.notification • music, FMRadio audio.volume.content • alarm audio.volume.alarm • telephony audio.volume.voice • Bluetooth SCO audio.volume.bt_sco
  • 30. Usage OR manifest.webapp app.html app.js
  • 31. Interrupted Event
  • 32. Current Competing Policies •  Channel with higher priority interrupts the lower channel. •  Foreground page is always playable. But interruption occurs or not depends on the channel of the media element. •  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!) •  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’. content normal
  • 33. Gecko side mozAudioChannelType audio App nsIDOMHTMLAudioElement nsIDOMHTMLMediaElement nsHTMLMediaElement nsIAudioChannelAgentCallback nsIAudioChannelAgent nsBuiltinDecoder nsMediaOmxDecoder nsMediaDecoder DataSource MediaStreamSource MediaResource - SetReadMode() create IOMX OMX MediaExtractorOMXCodec MediaSource - Tell() - Seek() - Read() - GetLength() hw codec sw codec MediaBuffer - CanPlayChanged() - mozinterruptbegin event - mozinterruptend event OmxDecoder nsHTMLAudioElement nsIChannel create nsMediaOmxReader nsBuiltinDecoderReader create create MediaOmxStateMachine nsBuiltinDecoderStateMachine nsDecoderStateMachine create nsIThread nsIThread control create nsDocument - NotifyOwnerDocumentActivityChanged() FileMediaResource ChannelMediaResource nsAudioStream nsNativeAudioStream libsydneyaudio AudioTrack AudioFlinger - Write() - Pause() - Resume() - IsPaused() - Drain() - GetMinWriteSize() - GetPosition() - SetVolume() platform: Firefox OS(gonk) branch: Firefox OS 1.01 (b2g18) will be replaced to libcubeb in v1.2 Audio Streams Diagram
  • 34. Since API is implemented.. World peace?
  • 35. No… Don’t forget the dependency tree.
  • 36. Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
  • 37. Policy for Silent mode • Easy for single channel • Multi channels • The default channel for volume rockers is ‘notification’. (Same as Android) • Silent mode interesting channels • notification • content/normal • Alarm is not affected by silent mode.
  • 38. Policy for video element • video doesn’t need to play at background. • Currently, use ‘visibilitychange’ event. • video by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
  • 39. Background play issues • Existing web pages: “Hey, I’m a music site!” • grooveshark • youtube • … • How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
  • 40. CE Certification •  EN 50332-1 •  Output level through earphone should be under 85 dBA. •  Prompt a warning when earphone is plugged and the volume is greater than 85 dBA. •  The cool down time of CE warning popup is continuous 20 hours music listening.
  • 41. OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
  • 42. After all..
  • 43. UX logic in backend and frontend