Your SlideShare is downloading. ×
[COSCUP 2013] Audio Competing
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

[COSCUP 2013] Audio Competing


Published on

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Web Developer or OS Developer? Experience from FirefoxOS Alive COSCUP 2013
  • 2. me @alivedise ~2.5year Frontend Exp. 聽說COSCUP一定要放
  • 3. Operating System ?
  • 4. Operating System
  • 5. FirefoxOS….OS? Storage Management fileSystem(yet) DeviceStorageWatcher Memory Management PageVisibility mozAppType User Interface Multi Window Management Statusbar System Overlay App Overlay Web Activity Keyboard Hardware Resource Management Wi-Fi Radio GPS Bluetooth Power CPU WakeLock Audio/Media Multi-Tasking mozBrowser remote=true Card view
  • 6. • Multi Channel • Audio Competing • Background Play Audio Management
  • 7. The beginning of everything..
  • 8. 805333
  • 9. Why do we need multi- channel on OS?
  • 10. Sounds within FxOS - <audio/>
  • 11. Sounds within FxOS - radio
  • 12. Sounds within FxOS - alarm
  • 13. Sounds within FxOS - voice
  • 14. The world of single channel.. Peace
  • 15. But.. Too peaceful.
  • 16. This means: When you mute the device, your alarm also is muted. So..
  • 17. Overslept ♪♪♪~ Zzzzz..
  • 18. How does OS avoid that? Different settings for audios on different purposes.
  • 19. On the other hand.. Competing
  • 20. On Desktop browser, we don’t need competing.
  • 21. How about mobile OS?
  • 22. Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
  • 23. It’s all about UX. User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
  • 24. Android solution: audio focus • -audio/audio-focus.html • App is responsible to request the audio focus on demand and release the focus actively.
  • 25. Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
  • 26. Proposal • New attribute for media tags. • Background play and competing needs permission check. • Expose current occupied media info to system.
  • 27. Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs. • A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
  • 28. mozAudioChannelType telephony alarm notification content normal(default)
  • 29. Multi channel volume settings • ringer, desktop-notification audio.volume.notification • music, FMRadio audio.volume.content • alarm audio.volume.alarm • telephony audio.volume.voice • Bluetooth SCO audio.volume.bt_sco
  • 30. Usage OR manifest.webapp app.html app.js
  • 31. Interrupted Event
  • 32. Current Competing Policies •  Channel with higher priority interrupts the lower channel. •  Foreground page is always playable. But interruption occurs or not depends on the channel of the media element. •  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!) •  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’. content normal
  • 33. Gecko side mozAudioChannelType audio App nsIDOMHTMLAudioElement nsIDOMHTMLMediaElement nsHTMLMediaElement nsIAudioChannelAgentCallback nsIAudioChannelAgent nsBuiltinDecoder nsMediaOmxDecoder nsMediaDecoder DataSource MediaStreamSource MediaResource - SetReadMode() create IOMX OMX MediaExtractorOMXCodec MediaSource - Tell() - Seek() - Read() - GetLength() hw codec sw codec MediaBuffer - CanPlayChanged() - "mozinterruptbegin" event - "mozinterruptend" event OmxDecoder nsHTMLAudioElement nsIChannel create nsMediaOmxReader nsBuiltinDecoderReader create create MediaOmxStateMachine nsBuiltinDecoderStateMachine nsDecoderStateMachine create nsIThread nsIThread control create nsDocument - NotifyOwnerDocumentActivityChanged() FileMediaResource ChannelMediaResource nsAudioStream nsNativeAudioStream libsydneyaudio AudioTrack AudioFlinger - Write() - Pause() - Resume() - IsPaused() - Drain() - GetMinWriteSize() - GetPosition() - SetVolume() platform: Firefox OS(gonk) branch: Firefox OS 1.01 (b2g18) will be replaced to libcubeb in v1.2 Audio Streams Diagram
  • 34. Since API is implemented.. World peace?
  • 35. No… Don’t forget the dependency tree.
  • 36. Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
  • 37. Policy for Silent mode • Easy for single channel • Multi channels • The default channel for volume rockers is ‘notification’. (Same as Android) • Silent mode interesting channels • notification • content/normal • Alarm is not affected by silent mode.
  • 38. Policy for video element • <video> doesn’t need to play at background. • Currently, use ‘visibilitychange’ event. • <video> by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
  • 39. Background play issues • Existing web pages: “Hey, I’m a music site!” • grooveshark • youtube • … • How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
  • 40. CE Certification •  EN 50332-1 •  Output level through earphone should be under 85 dBA. •  Prompt a warning when earphone is plugged and the volume is greater than 85 dBA. •  The cool down time of CE warning popup is continuous 20 hours music listening.
  • 41. OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
  • 42. After all..
  • 43. UX logic in backend and frontend backend frontend
  • 44. Find out solutions for existing web sites without new API. • As a platform, it’s difficult to make decisions. • New API takes time to be mature.
  • 45. The scope is undefined before you look into it. 
  • 46. To be honest, I wrote few code in these issues. But lots of online/offline discussions.
  • 47. The position of Frontend
  • 48. • It doesn’t matter you are making a web site, a web app or an operating system. • Frontend is the one connecting UX and Backend, and struggle for ideal design and real requirement.
  • 49. Thanks! Not the end, to be continued..