Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Nerd sniping myself into a rabbit hole...
Streaming online audio to a Sonos
speaker
Maarten Balliauw
@maartenballiauw
Disclaimer
I will share bits of source code where they matter, but will
not be sharing the full application.
I have built ...
Living room speakers
January 2020
“Let’s replace our old speakers with new and shiny!”
Requirements:
“Smart” speakers that can stream from the ...
Now what…
Searched online for solutions…
…all I found was excuses.
Legal, patents, … Don’t really care as a consumer!
https://support.sonos.com/s/article/79?language=en_US
Nerd Sniping
1. The act of presenting someone, often
a mathematician/physicist with a time consuming problem or
challenge ...
Research
What would I need?
1. Connect to speakers
2. Get MP4 URL of online video
3. Send URL to speakers
4. Enjoy music!
https://python-soco.com/
Connect to speakers
Connect to one speaker
Play an MP3 from webserver
This seems very, very promising!
#!/usr/bin/env pyth...
Get MP4 URL of online video
Video metadata endpoint, used by web player
Returns urlencoded data about video (with JSON spr...
Get MP4 URL of online video
{
"responseContext":{ },
"playabilityStatus":{ },
"streamingData":{
"expiresInSeconds":"21540"...
Get MP4 URL of online video
More reading by Alexey Golub
https://tyrrrz.me/blog/reverse-engineering-youtube
Signed videos
...
Send URL to speakers
#!/usr/bin/env python
from soco import SoCo
if __name__ == '__main__’:
sonos = SoCo('192.168.1.102’)
...
Side track: speaker webserver
Anything useful to find?
http://192.168.1.123:1400/status
http://192.168.1.123:1400/support/...
Troubleshooting
💡 Maybe it’s that SoCo library!
“Because maybe 65 contributors have it wrong!”
The official application can send a stream ...
Sniffing the network
WireShark https://www.wireshark.org/
Sniff traffic that passes your computer’s network adapter
Traffi...
Sniffing the network
🤓 On my Windows box, connected to wired network
Run Ubuntu
SSH into access point and run tcpdump
Pipe...
Your fancy wifi speaker uses SOAP
SOAP payload
<?xml version="1.0" encoding="UTF-8"?>
<s:envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:enco...
Replaying SOAP payload
Tried MP3 URLs and MP4 URLs
MP3 worked, MP4 did not
The SoCo library did not have any issues...
Sea...
💡 Maybe it’s the MP4 format!
https://support.sonos.com/s/article/79?language=en_US
Our (potential) options...
Download MP4, push it to speaker as a local file
or
Proxy MP4 and do on-the-fly transcoding to ...
Containers
MP4
MPEG-4 Part 14 or MP4 is a digital multimedia container
format most commonly used to store video and audio, but
it can...
FFMpeg to the rescue!
“A complete, cross-platform solution to record, convert and stream
audio and video.”
Swiss army knif...
deadf00d
“How I hacked Sonos and YouTube in the same day.”
https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-th...
MP4 to AAC with ADTS
ffmpeg -i original.mp4 -acodec copy 
-f adts -vn output-adts.aac
Extracts the audio track from MP4 co...
Making it an app
Let the app building start!
1. Connect to speakers ✅
2. Get MP4 URL of online video ✅
3. (new) Extract MP4 audio track to ...
Architecture
YouTube app
Share
https://youtube.com/watch?
v=-zJoP2qPgTg
YouTube.com
Sonos
speaker
Main
MP4 to ADTS
MP4 aud...
Technology choices
Native app?
Xamarin?
Flutter?
Jetpack Compose?
🤓♂️ Installed Android Studio as it’s similar to JetBrain...
Getting started…
Template
Start with empty and add things?
Start with <any> and remove things?
Language
Java?
Kotlin?
Research…
Android-specific
Activity (main screen to handle everything)
Intent (ACTION_SEND to receive data from others)
Li...
It all starts with a manifest!
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/...
And checking the intent…
class MainActivity : CoroutineScope, AppCompatActivity() {
override fun onCreate(savedInstanceSta...
SSDP Simple Device Discovery Protocol
Used to discover printers, routers, Sonos, ...
Send UDP datagram as multicast/broadc...
Android – Running webserver
Prevent server stop when application is closed or device goes to sleep
<service
android:name="...
MP4 to ADTS – FFMpeg?
MP4, ADTS, Atoms, Boxes
https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-the-same-day.html
MP4, ADTS, Atoms, Boxes
MP4 with AAC Audio (simplified)
ADTS with AAC Audio (simplified)
moof
“Hey, 25 samples coming.
128...
MP4, ADTS, Atoms, Boxes
Had to manually write conversion logic...
Open MP4 stream (https://github.com/sannies/mp4parser)
F...
Exploring the code
demo
In summary…
In summary…
Enjoy music with an app ✅
Connect to speakers ✅
Get MP4 URL of online video ✅
Extract MP4 audio track to ADTS ...
“Do you use it often?”
In summary…
Learned a lot of random things along the way ✅
There is so much knowledge out there!
Talk to people (thanks, d...
Thank you!
https://blog.maartenballiauw.be
@maartenballiauw
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker

Download to read offline

After buying a set of Sonos-compatible speakers at IKEA, I was disappointed there's no support for playing audio from a popular video streaming service. They stream Internet radio, podcasts and what not. Well, not that service I want it to play!
Determined - and not knowing how deep the rabbit hole would be - I ventured on a trip that included network sniffing on my access point, learning about UPnP and running a web server on my phone (without knowing how to write anything Android), learning how MP4 audio is packaged (and has to be re-packaged). This ultimately resulted in an Android app for personal use, which does what I initially wanted: play audio from that popular video streaming service on Sonos.
Join me for this story about an adventure that has no practical use, probably violates Terms of Service, but was fun to build!

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker

  1. 1. Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos speaker Maarten Balliauw @maartenballiauw
  2. 2. Disclaimer I will share bits of source code where they matter, but will not be sharing the full application. I have built this application for personal and learning use, and I do not intend to share it. Don’t ask, the answer is no.
  3. 3. Living room speakers
  4. 4. January 2020 “Let’s replace our old speakers with new and shiny!” Requirements: “Smart” speakers that can stream from the Internet 2 for living room, 1-2 for home office
  5. 5. Now what… Searched online for solutions… …all I found was excuses. Legal, patents, … Don’t really care as a consumer!
  6. 6. https://support.sonos.com/s/article/79?language=en_US
  7. 7. Nerd Sniping 1. The act of presenting someone, often a mathematician/physicist with a time consuming problem or challenge (often impossible to solve or complete) in the hopes of it appealing to a person's obsessive tendencies. Urban Dictionary And https://xkcd.com/356
  8. 8. Research
  9. 9. What would I need? 1. Connect to speakers 2. Get MP4 URL of online video 3. Send URL to speakers 4. Enjoy music!
  10. 10. https://python-soco.com/
  11. 11. Connect to speakers Connect to one speaker Play an MP3 from webserver This seems very, very promising! #!/usr/bin/env python from soco import SoCo if __name__ == '__main__’: sonos = SoCo('192.168.1.102’) sonos.play_uri('http://host/file.mp3') https://python-soco.com/
  12. 12. Get MP4 URL of online video Video metadata endpoint, used by web player Returns urlencoded data about video (with JSON sprinkled in) https://www.youtube.com/watch?v=-zJoP2qPgTg  https://www.youtube.com/get_video_info?video_id=-zJoP2qPgTg
  13. 13. Get MP4 URL of online video { "responseContext":{ }, "playabilityStatus":{ }, "streamingData":{ "expiresInSeconds":"21540", "formats":[ { "itag":18, "url":"https://r3---sn-uxaxoxu-cg0k.googlevideo.com/videoplayback?expire=1599574779u0026ei=mz5XX6-fKdOIgQeBp6PgDwu0026 "mimeType":"video/mp4;+codecs="avc1.42001E,+mp4a.40.2"", "bitrate":234221, "width":640, "height":360, "lastModified":"1586173729658545", "contentLength":"92057863", "quality":"medium",
  14. 14. Get MP4 URL of online video More reading by Alexey Golub https://tyrrrz.me/blog/reverse-engineering-youtube Signed videos Video player JS code contains decryption routine as JavaScript Need to evaluate that to be able to access video (or Regex the cipher) Too much hassle to write manually! There exist scripts & libraries in many programing languages In summary: we have the MP4 URL now.
  15. 15. Send URL to speakers #!/usr/bin/env python from soco import SoCo if __name__ == '__main__’: sonos = SoCo('192.168.1.102’) sonos.play_uri('http://host/file.mp4') https://python-soco.com/ Expected: Actual:
  16. 16. Side track: speaker webserver Anything useful to find? http://192.168.1.123:1400/status http://192.168.1.123:1400/support/review http://192.168.1.123:1400/tools.htm https://bsteiner.info/articles/hidden-sonos-interface
  17. 17. Troubleshooting
  18. 18. 💡 Maybe it’s that SoCo library! “Because maybe 65 contributors have it wrong!” The official application can send a stream to the speakers... ...can I listen on the network and see what the request looks like?
  19. 19. Sniffing the network WireShark https://www.wireshark.org/ Sniff traffic that passes your computer’s network adapter Traffic does not pass my computer :-/ Phone on wifi, speaker on wifi, computer on wifi – huh? Turns out access point does just send all traffic to all devices 💡💡 Unifi access point is *nix tcpdump there?
  20. 20. Sniffing the network 🤓 On my Windows box, connected to wired network Run Ubuntu SSH into access point and run tcpdump Pipe data back to Windows Access point IP Capture ethernet side From/to my phone IP
  21. 21. Your fancy wifi speaker uses SOAP
  22. 22. SOAP payload <?xml version="1.0" encoding="UTF-8"?> <s:envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingstyle="http://schema <s:body> <u:setavtransporturi xmlns:u="urn:schemas-upnp-org:service:AVTransport:1"> <instanceid>0</instanceid> <currenturi>x-rincon-mp3radio://host/media.mp3</currenturi> <currenturimetadata> <DIDL-Lite xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/" xmlns:dc="http <item id="R:0/0/6" parentID="R:0/0" restricted="true"> <dc:title>Title here</dc:title> <upnp:class>object.item.audioItem.audioBroadcast</upnp:class> <upnp:albumArtURI>https://host/art.jpg</upnp:albumArtURI> <r:description>Description here</r:description> <desc id="cdudn" nameSpace="urn:schemas-rinconnetworks-com:metadata-1- </item> </DIDL-Lite> </currenturimetadata> </u:setavtransporturi> </s:body> </s:envelope>
  23. 23. Replaying SOAP payload Tried MP3 URLs and MP4 URLs MP3 worked, MP4 did not The SoCo library did not have any issues... Searched around for DIDL-Lite in payload Seems speakers use good old UPnP http://www.upnp.org/schemas/av/didl-lite-v2.xsd
  24. 24. 💡 Maybe it’s the MP4 format! https://support.sonos.com/s/article/79?language=en_US
  25. 25. Our (potential) options... Download MP4, push it to speaker as a local file or Proxy MP4 and do on-the-fly transcoding to MP3 Send MP3 URL as “Internet Radio” or Investigate MP4 and see if they indeed use AAC Send AAC URL as “Internet Radio”
  26. 26. Containers
  27. 27. MP4 MPEG-4 Part 14 or MP4 is a digital multimedia container format most commonly used to store video and audio, but it can also be used to store other data such as subtitles and still images. (…) allows streaming (…) Wikipedia Can we extract this to a separate file? MP4 file Header Video 1 Video N Audio 1 Audio N Subtitles MP4 file (optimized for streaming) Header Video 1 (short) Audio 1 (short) Video N (short) Audio N (short)
  28. 28. FFMpeg to the rescue! “A complete, cross-platform solution to record, convert and stream audio and video.” Swiss army knife for video/audio formats. ffmpeg -i original.mp4 -c:a copy output-aac.m4a Extracts the audio track from MP4 container Use SoCo to send file to speakers. https://ffmpeg.org/ Expected: Actual:
  29. 29. deadf00d “How I hacked Sonos and YouTube in the same day.” https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-the-same-day.html @deadf0od - https://twitter.com/deadf0od “HEY, kAn 1 Dm J00? w0rK1n' 0N 51M1Lar 7h1n' AnD wE M19h7 8E a8le 70 HElP EaCH 07heR.” ... “It’s AAC, but in ADTS format. Each atom needs a header in every frame!”
  30. 30. MP4 to AAC with ADTS ffmpeg -i original.mp4 -acodec copy -f adts -vn output-adts.aac Extracts the audio track from MP4 container Adds ADTS headers Use SoCo to send file to speakers. Expected: Actual:
  31. 31. Making it an app
  32. 32. Let the app building start! 1. Connect to speakers ✅ 2. Get MP4 URL of online video ✅ 3. (new) Extract MP4 audio track to ADTS ✅ 4. Send URL to speakers ✅ 5. Enjoy music!
  33. 33. Architecture YouTube app Share https://youtube.com/watch? v=-zJoP2qPgTg YouTube.com Sonos speaker Main MP4 to ADTS MP4 audio Reverse proxy https://<ip>/audio.adts
  34. 34. Technology choices Native app? Xamarin? Flutter? Jetpack Compose? 🤓♂️ Installed Android Studio as it’s similar to JetBrains Rider, IntelliJ IDEA, ...
  35. 35. Getting started… Template Start with empty and add things? Start with <any> and remove things? Language Java? Kotlin?
  36. 36. Research… Android-specific Activity (main screen to handle everything) Intent (ACTION_SEND to receive data from others) Libraries (or code) YouTube metadata extractor (get audio URL) Sonos communication library (discover speakers, send URL to speakers) Webserver (reverse proxy) Something to do the MP4 to AAC (ADTS) conversion
  37. 37. It all starts with a manifest! <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="be.maartenballiauw.android.sonostube"> <application android:allowBackup="true" android:icon="@mipmap/ic_launcher" android:label="@string/app_name" android:roundIcon="@mipmap/ic_launcher_round" android:usesCleartextTraffic="true" android:theme="@style/AppTheme"> <activity android:name=".MainActivity"> <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> <intent-filter> <action android:name="android.intent.action.SEND" /> <category android:name="android.intent.category.DEFAULT" /> <data android:scheme="https" android:host="youtu.be" android:mimeType="text/*"/> <data android:scheme="https" android:host="youtube.com" android:mimeType="text/*"/> <data android:scheme="https" android:host="www.youtube.com" android:mimeType="text/*"/> </intent-filter> </activity> </application> </manifest>
  38. 38. And checking the intent… class MainActivity : CoroutineScope, AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) val url = intent?.extras?.getString(Intent.EXTRA_TEXT)
  39. 39. SSDP Simple Device Discovery Protocol Used to discover printers, routers, Sonos, ... Send UDP datagram as multicast/broadcast M-SEARCH * HTTP/1.1 HOST: 239.255.255.250:1900 MAN: "ssdp:discover" MX: 1 ST: urn:schemas-upnp-org:device:ZonePlayer:1 Devices send back HTTP response as UDP Supported services, endpoint URL, ... https://en.wikipedia.org/wiki/Simple_Service_Discovery_Protocol https://github.com/vmichalak/ssdp-client
  40. 40. Android – Running webserver Prevent server stop when application is closed or device goes to sleep <service android:name=".RunEmbeddedWebServerService" android:enabled="true" android:exported="true" /> class RunEmbeddedWebServerService : CoroutineScope, Service() { private val server = embeddedServer(Netty, 36362) { routing { get("/{videoId}.mp4") { /* ... */ } } } override fun onStartCommand(...): Int { server.start(wait = false) return START_NOT_STICKY } override fun onDestroy() { server.stop(0, 1000) super.onDestroy() } https://developer.android.com/guide/components/services#Types-of-services
  41. 41. MP4 to ADTS – FFMpeg?
  42. 42. MP4, ADTS, Atoms, Boxes https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-the-same-day.html
  43. 43. MP4, ADTS, Atoms, Boxes MP4 with AAC Audio (simplified) ADTS with AAC Audio (simplified) moof “Hey, 25 samples coming. 128kbps, 2 channel audio!” mdat Raw, binary data for 25 samples. moof “Hey, 12 samples coming. 128kbps, 2 channel audio!” mdat Raw, binary data for 12 samples. Header “40 bytes coming, 2 ch” + 40 bytes for 1 sample Header “36 bytes coming, 2 ch” + 36 bytes for 1 sample Header “42 bytes coming, 2 ch” + 42 bytes for 1 sample Header “40 bytes coming, 2 ch” + 40 bytes for 1 sample
  44. 44. MP4, ADTS, Atoms, Boxes Had to manually write conversion logic... Open MP4 stream (https://github.com/sannies/mp4parser) Foreach moof, read metadata Foreach sample, write ADTS header, write sample
  45. 45. Exploring the code demo
  46. 46. In summary…
  47. 47. In summary… Enjoy music with an app ✅ Connect to speakers ✅ Get MP4 URL of online video ✅ Extract MP4 audio track to ADTS ✅ Send URL to speakers ✅
  48. 48. “Do you use it often?”
  49. 49. In summary… Learned a lot of random things along the way ✅ There is so much knowledge out there! Talk to people (thanks, deadF00d!) You can build anything! Even if it seems impossible at first!
  50. 50. Thank you! https://blog.maartenballiauw.be @maartenballiauw

After buying a set of Sonos-compatible speakers at IKEA, I was disappointed there's no support for playing audio from a popular video streaming service. They stream Internet radio, podcasts and what not. Well, not that service I want it to play! Determined - and not knowing how deep the rabbit hole would be - I ventured on a trip that included network sniffing on my access point, learning about UPnP and running a web server on my phone (without knowing how to write anything Android), learning how MP4 audio is packaged (and has to be re-packaged). This ultimately resulted in an Android app for personal use, which does what I initially wanted: play audio from that popular video streaming service on Sonos. Join me for this story about an adventure that has no practical use, probably violates Terms of Service, but was fun to build!

Views

Total views

209

On Slideshare

0

From embeds

0

Number of embeds

14

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×