Web Developer or OS
Developer?
Experience from FirefoxOS
Alive
COSCUP 2013
me
@alivedise	

	

http://alivedise.github.io	

http://github.com/alivedise	

	

alive@mozilla.com	

alegnadise@gmail.com	...
Operating System
?
http://www.flickr.com/photos/klaikong/6041270516/
Operating System
FirefoxOS….OS?
Storage
Management	

fileSystem(yet)	

DeviceStorageWatcher	

Memory
Management	

PageVisibility	

mozAppTyp...
• Multi Channel	

• Audio Competing	

• Background Play	

Audio Management
The beginning of everything..
http://bugzil.la/805333
805333
Why do we need multi-
channel on OS?
Sounds within FxOS - <audio/>
Sounds within FxOS - radio
Sounds within FxOS - alarm
Sounds within FxOS - voice
The world of single channel..
Peace
But..
Too peaceful.
This means:
When you mute the device, your
alarm also is muted. So..
Overslept ♪♪♪~
Zzzzz..
How does OS avoid that?
Different settings for audios on
different purposes.
On the other hand..
Competing
On Desktop browser, we don’t need
competing.
How about mobile OS?
Cases
• one <audio/> in one app.	

• two <audio/> in one app.	

• two <audio/> in two apps.	

• one <audio/> and one not-a...
It’s all about UX.
User wants to listen music in background..	

User has difficulty to mute the background app..	

User nee...
Android solution: audio focus
• http://developer.android.com/training/managing
-audio/audio-focus.html	

• App is responsi...
Problems
• How to request audio focus in web?	

• Volume control of different type of audio source.	

• Telephony is not u...
Proposal
• New attribute for media tags.	

• Background play and competing needs permission
check.	

• Expose current occu...
Page Visibility
• HTML5 Page Visibility	

• The page visibility state is simple on desktop
browsers by toggling tabs.	

• ...
mozAudioChannelType
telephony	

alarm	

notification	

content	

normal(default)
Multi channel volume settings
• ringer, desktop-notification	

audio.volume.notification	

• music, FMRadio	

audio.volume.c...
Usage
OR
manifest.webapp
app.html
app.js
Interrupted Event
Current Competing Policies
•  Channel with higher priority interrupts the
lower channel.	

•  Foreground page is always pl...
Gecko side
mozAudioChannelType
audio
App
nsIDOMHTMLAudioElement
nsIDOMHTMLMediaElement
nsHTMLMediaElement
nsIAudioChannelA...
Since API is implemented..
World peace?
No…
Don’t forget the dependency tree.
Incoming Issues
• Silent mode under multi channels	

• Background play ability of existing web pages	

• Background play a...
Policy for Silent mode
• Easy for single channel	

• Multi channels	

• The default channel for volume rockers is
‘notifica...
Policy for video element
• video doesn’t need to play at background.	

• Currently, use ‘visibilitychange’ event.	

• vide...
Background play issues
• Existing web pages: “Hey, I’m a music site!”	

• grooveshark	

• youtube	

• …	

• How to play wh...
CE Certification
•  EN 50332-1	

•  Output level through earphone
should be under 85 dBA.	

•  Prompt a warning when earpho...
OS level policy for CE
• Only apply to content/normal channel.	

• Output power is hardware-dependent. Exactly,
we don’t k...
After all..
UX logic in backend and frontend
[COSCUP 2013] Audio Competing
[COSCUP 2013] Audio Competing
[COSCUP 2013] Audio Competing
[COSCUP 2013] Audio Competing
[COSCUP 2013] Audio Competing
[COSCUP 2013] Audio Competing
Upcoming SlideShare
Loading in...5
×

[COSCUP 2013] Audio Competing

227

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
227
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
4
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

[COSCUP 2013] Audio Competing

  1. 1. Web Developer or OS Developer? Experience from FirefoxOS Alive COSCUP 2013
  2. 2. me @alivedise http://alivedise.github.io http://github.com/alivedise alive@mozilla.com alegnadise@gmail.com ~2.5year Frontend Exp. 聽說COSCUP一定要放
  3. 3. Operating System ? http://www.flickr.com/photos/klaikong/6041270516/
  4. 4. Operating System
  5. 5. FirefoxOS….OS? Storage Management fileSystem(yet) DeviceStorageWatcher Memory Management PageVisibility mozAppType User Interface Multi Window Management Statusbar System Overlay App Overlay Web Activity Keyboard Hardware Resource Management Wi-Fi Radio GPS Bluetooth Power CPU WakeLock Audio/Media Multi-Tasking mozBrowser remote=true Card view
  6. 6. • Multi Channel • Audio Competing • Background Play Audio Management
  7. 7. The beginning of everything.. http://bugzil.la/805333
  8. 8. 805333
  9. 9. Why do we need multi- channel on OS?
  10. 10. Sounds within FxOS - <audio/>
  11. 11. Sounds within FxOS - radio
  12. 12. Sounds within FxOS - alarm
  13. 13. Sounds within FxOS - voice
  14. 14. The world of single channel.. Peace
  15. 15. But.. Too peaceful.
  16. 16. This means: When you mute the device, your alarm also is muted. So..
  17. 17. Overslept ♪♪♪~ Zzzzz..
  18. 18. How does OS avoid that? Different settings for audios on different purposes.
  19. 19. On the other hand.. Competing
  20. 20. On Desktop browser, we don’t need competing.
  21. 21. How about mobile OS?
  22. 22. Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
  23. 23. It’s all about UX. User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
  24. 24. Android solution: audio focus • http://developer.android.com/training/managing -audio/audio-focus.html • App is responsible to request the audio focus on demand and release the focus actively.
  25. 25. Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
  26. 26. Proposal • New attribute for media tags. • Background play and competing needs permission check. • Expose current occupied media info to system.
  27. 27. Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs. • A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
  28. 28. mozAudioChannelType telephony alarm notification content normal(default)
  29. 29. Multi channel volume settings • ringer, desktop-notification audio.volume.notification • music, FMRadio audio.volume.content • alarm audio.volume.alarm • telephony audio.volume.voice • Bluetooth SCO audio.volume.bt_sco
  30. 30. Usage OR manifest.webapp app.html app.js
  31. 31. Interrupted Event
  32. 32. Current Competing Policies •  Channel with higher priority interrupts the lower channel. •  Foreground page is always playable. But interruption occurs or not depends on the channel of the media element. •  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!) •  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’. content normal
  33. 33. Gecko side mozAudioChannelType audio App nsIDOMHTMLAudioElement nsIDOMHTMLMediaElement nsHTMLMediaElement nsIAudioChannelAgentCallback nsIAudioChannelAgent nsBuiltinDecoder nsMediaOmxDecoder nsMediaDecoder DataSource MediaStreamSource MediaResource - SetReadMode() create IOMX OMX MediaExtractorOMXCodec MediaSource - Tell() - Seek() - Read() - GetLength() hw codec sw codec MediaBuffer - CanPlayChanged() - mozinterruptbegin event - mozinterruptend event OmxDecoder nsHTMLAudioElement nsIChannel create nsMediaOmxReader nsBuiltinDecoderReader create create MediaOmxStateMachine nsBuiltinDecoderStateMachine nsDecoderStateMachine create nsIThread nsIThread control create nsDocument - NotifyOwnerDocumentActivityChanged() FileMediaResource ChannelMediaResource nsAudioStream nsNativeAudioStream libsydneyaudio AudioTrack AudioFlinger - Write() - Pause() - Resume() - IsPaused() - Drain() - GetMinWriteSize() - GetPosition() - SetVolume() platform: Firefox OS(gonk) branch: Firefox OS 1.01 (b2g18) will be replaced to libcubeb in v1.2 Audio Streams Diagram https://github.com/sotaroikeda/firefox-diagrams
  34. 34. Since API is implemented.. World peace?
  35. 35. No… Don’t forget the dependency tree.
  36. 36. Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
  37. 37. Policy for Silent mode • Easy for single channel • Multi channels • The default channel for volume rockers is ‘notification’. (Same as Android) • Silent mode interesting channels • notification • content/normal • Alarm is not affected by silent mode.
  38. 38. Policy for video element • video doesn’t need to play at background. • Currently, use ‘visibilitychange’ event. • video by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
  39. 39. Background play issues • Existing web pages: “Hey, I’m a music site!” • grooveshark • youtube • … • How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
  40. 40. CE Certification •  EN 50332-1 •  Output level through earphone should be under 85 dBA. •  Prompt a warning when earphone is plugged and the volume is greater than 85 dBA. •  The cool down time of CE warning popup is continuous 20 hours music listening.
  41. 41. OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
  42. 42. After all..
  43. 43. UX logic in backend and frontend
  1. ¿Le ha llamado la atención una diapositiva en particular?

    Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

×