1.
Nerd sniping myself into a rabbit hole...
Streaming online audio to a Sonos
speaker
Maarten Balliauw
@maartenballiauw

2.
Disclaimer
I will share bits of source code where they matter, but will
not be sharing the full application.
I have built this application for personal and learning use,
and I do not intend to share it.
Don’t ask, the answer is no.

3.
Living room speakers

4.
January 2020
“Let’s replace our old speakers with new and shiny!”
Requirements:
“Smart” speakers that can stream from the Internet
2 for living room, 1-2 for home office

5.
Now what…
Searched online for solutions…
…all I found was excuses.
Legal, patents, … Don’t really care as a consumer!

6.
https://support.sonos.com/s/article/79?language=en_US

7.
Nerd Sniping
1. The act of presenting someone, often
a mathematician/physicist with a time consuming problem or
challenge (often impossible to solve or complete) in the hopes
of it appealing to a person's obsessive tendencies.
Urban Dictionary
And https://xkcd.com/356

8.
Research

9.
What would I need?
1. Connect to speakers
2. Get MP4 URL of online video
3. Send URL to speakers
4. Enjoy music!

10.
https://python-soco.com/

11.
Connect to speakers
Connect to one speaker
Play an MP3 from webserver
This seems very, very promising!
#!/usr/bin/env python
from soco import SoCo
if __name__ == '__main__’:
sonos = SoCo('192.168.1.102’)
sonos.play_uri('http://host/file.mp3')
https://python-soco.com/

12.
Get MP4 URL of online video
Video metadata endpoint, used by web player
Returns urlencoded data about video (with JSON sprinkled in)
https://www.youtube.com/watch?v=-zJoP2qPgTg
 https://www.youtube.com/get_video_info?video_id=-zJoP2qPgTg

13.
Get MP4 URL of online video
{
"responseContext":{ },
"playabilityStatus":{ },
"streamingData":{
"expiresInSeconds":"21540",
"formats":[
{
"itag":18,
"url":"https://r3---sn-uxaxoxu-cg0k.googlevideo.com/videoplayback?expire=1599574779u0026ei=mz5XX6-fKdOIgQeBp6PgDwu0026
"mimeType":"video/mp4;+codecs="avc1.42001E,+mp4a.40.2"",
"bitrate":234221,
"width":640,
"height":360,
"lastModified":"1586173729658545",
"contentLength":"92057863",
"quality":"medium",

14.
Get MP4 URL of online video
More reading by Alexey Golub
https://tyrrrz.me/blog/reverse-engineering-youtube
Signed videos
Video player JS code contains decryption routine as JavaScript
Need to evaluate that to be able to access video (or Regex the cipher)
Too much hassle to write manually!
There exist scripts & libraries in many programing languages
In summary: we have the MP4 URL now.

15.
Send URL to speakers
#!/usr/bin/env python
from soco import SoCo
if __name__ == '__main__’:
sonos = SoCo('192.168.1.102’)
sonos.play_uri('http://host/file.mp4')
https://python-soco.com/
Expected:
Actual:

16.
Side track: speaker webserver
Anything useful to find?
http://192.168.1.123:1400/status
http://192.168.1.123:1400/support/review
http://192.168.1.123:1400/tools.htm
https://bsteiner.info/articles/hidden-sonos-interface

17.
Troubleshooting

18.
💡 Maybe it’s that SoCo library!
“Because maybe 65 contributors have it wrong!”
The official application can send a stream to the speakers...
...can I listen on the network and see what the request looks like?

19.
Sniffing the network
WireShark https://www.wireshark.org/
Sniff traffic that passes your computer’s network adapter
Traffic does not pass my computer :-/
Phone on wifi, speaker on wifi, computer on wifi – huh?
Turns out access point does just send all traffic to all devices
💡💡 Unifi access point is *nix
tcpdump there?

20.
Sniffing the network
🤓 On my Windows box, connected to wired network
Run Ubuntu
SSH into access point and run tcpdump
Pipe data back to Windows
Access point IP Capture ethernet side From/to my phone IP

21.
Your fancy wifi speaker uses SOAP

22.
SOAP payload
<?xml version="1.0" encoding="UTF-8"?>
<s:envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingstyle="http://schema
<s:body>
<u:setavtransporturi xmlns:u="urn:schemas-upnp-org:service:AVTransport:1">
<instanceid>0</instanceid>
<currenturi>x-rincon-mp3radio://host/media.mp3</currenturi>
<currenturimetadata>
<DIDL-Lite xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/" xmlns:dc="http
<item id="R:0/0/6" parentID="R:0/0" restricted="true">
<dc:title>Title here</dc:title>
<upnp:class>object.item.audioItem.audioBroadcast</upnp:class>
<upnp:albumArtURI>https://host/art.jpg</upnp:albumArtURI>
<r:description>Description here</r:description>
<desc id="cdudn" nameSpace="urn:schemas-rinconnetworks-com:metadata-1-
</item>
</DIDL-Lite>
</currenturimetadata>
</u:setavtransporturi>
</s:body>
</s:envelope>

23.
Replaying SOAP payload
Tried MP3 URLs and MP4 URLs
MP3 worked, MP4 did not
The SoCo library did not have any issues...
Searched around for DIDL-Lite in payload
Seems speakers use good old UPnP
http://www.upnp.org/schemas/av/didl-lite-v2.xsd

24.
💡 Maybe it’s the MP4 format!
https://support.sonos.com/s/article/79?language=en_US

25.
Our (potential) options...
Download MP4, push it to speaker as a local file
or
Proxy MP4 and do on-the-fly transcoding to MP3
Send MP3 URL as “Internet Radio”
or
Investigate MP4 and see if they indeed use AAC
Send AAC URL as “Internet Radio”

26.
Containers

27.
MP4
MPEG-4 Part 14 or MP4 is a digital multimedia container
format most commonly used to store video and audio, but
it can also be used to store other data such
as subtitles and still images. (…) allows streaming (…)
Wikipedia
Can we extract this to a separate file?
MP4 file
Header
Video 1
Video N
Audio 1
Audio N
Subtitles
MP4 file (optimized for streaming)
Header
Video 1 (short)
Audio 1 (short)
Video N (short)
Audio N (short)

28.
FFMpeg to the rescue!
“A complete, cross-platform solution to record, convert and stream
audio and video.”
Swiss army knife for video/audio formats.
ffmpeg -i original.mp4 -c:a copy output-aac.m4a
Extracts the audio track from MP4 container
Use SoCo to send file to speakers.
https://ffmpeg.org/
Expected: Actual:

29.
deadf00d
“How I hacked Sonos and YouTube in the same day.”
https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-the-same-day.html
@deadf0od - https://twitter.com/deadf0od
“HEY, kAn 1 Dm J00?
w0rK1n' 0N 51M1Lar 7h1n' AnD wE M19h7 8E a8le 70 HElP EaCH 07heR.”
...
“It’s AAC, but in ADTS format. Each atom needs a header in every frame!”

30.
MP4 to AAC with ADTS
ffmpeg -i original.mp4 -acodec copy
-f adts -vn output-adts.aac
Extracts the audio track from MP4 container
Adds ADTS headers
Use SoCo to send file to speakers.
Expected: Actual:

31.
Making it an app

32.
Let the app building start!
1. Connect to speakers ✅
2. Get MP4 URL of online video ✅
3. (new) Extract MP4 audio track to ADTS ✅
4. Send URL to speakers ✅
5. Enjoy music!

33.
Architecture
YouTube app
Share
https://youtube.com/watch?
v=-zJoP2qPgTg
YouTube.com
Sonos
speaker
Main
MP4 to ADTS
MP4 audio
Reverse proxy
https://<ip>/audio.adts

34.
Technology choices
Native app?
Xamarin?
Flutter?
Jetpack Compose?
🤓♂️ Installed Android Studio as it’s similar to JetBrains Rider, IntelliJ IDEA, ...

35.
Getting started…
Template
Start with empty and add things?
Start with <any> and remove things?
Language
Java?
Kotlin?

36.
Research…
Android-specific
Activity (main screen to handle everything)
Intent (ACTION_SEND to receive data from others)
Libraries (or code)
YouTube metadata extractor (get audio URL)
Sonos communication library (discover speakers, send URL to speakers)
Webserver (reverse proxy)
Something to do the MP4 to AAC (ADTS) conversion

37.
It all starts with a manifest!
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="be.maartenballiauw.android.sonostube">
<application android:allowBackup="true" android:icon="@mipmap/ic_launcher" android:label="@string/app_name"
android:roundIcon="@mipmap/ic_launcher_round" android:usesCleartextTraffic="true" android:theme="@style/AppTheme">
<activity android:name=".MainActivity">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
<intent-filter>
<action android:name="android.intent.action.SEND" />
<category android:name="android.intent.category.DEFAULT" />
<data android:scheme="https" android:host="youtu.be" android:mimeType="text/*"/>
<data android:scheme="https" android:host="youtube.com" android:mimeType="text/*"/>
<data android:scheme="https" android:host="www.youtube.com" android:mimeType="text/*"/>
</intent-filter>
</activity>
</application>
</manifest>

38.
And checking the intent…
class MainActivity : CoroutineScope, AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val url = intent?.extras?.getString(Intent.EXTRA_TEXT)

39.
SSDP Simple Device Discovery Protocol
Used to discover printers, routers, Sonos, ...
Send UDP datagram as multicast/broadcast
M-SEARCH * HTTP/1.1
HOST: 239.255.255.250:1900
MAN: "ssdp:discover"
MX: 1
ST: urn:schemas-upnp-org:device:ZonePlayer:1
Devices send back HTTP response as UDP
Supported services, endpoint URL, ...
https://en.wikipedia.org/wiki/Simple_Service_Discovery_Protocol
https://github.com/vmichalak/ssdp-client

40.
Android – Running webserver
Prevent server stop when application is closed or device goes to sleep
<service
android:name=".RunEmbeddedWebServerService"
android:enabled="true"
android:exported="true" /> class RunEmbeddedWebServerService : CoroutineScope, Service() {
private val server = embeddedServer(Netty, 36362) {
routing {
get("/{videoId}.mp4") { /* ... */ }
}
}
override fun onStartCommand(...): Int {
server.start(wait = false)
return START_NOT_STICKY
}
override fun onDestroy() {
server.stop(0, 1000)
super.onDestroy()
}
https://developer.android.com/guide/components/services#Types-of-services

41.
MP4 to ADTS – FFMpeg?

42.
MP4, ADTS, Atoms, Boxes
https://www.deadf00d.com/post/how-i-hacked-sonos-and-youtube-the-same-day.html

43.
MP4, ADTS, Atoms, Boxes
MP4 with AAC Audio (simplified)
ADTS with AAC Audio (simplified)
moof
“Hey, 25 samples coming.
128kbps, 2 channel audio!”
mdat
Raw, binary data for 25
samples.
moof
“Hey, 12 samples coming.
128kbps, 2 channel audio!”
mdat
Raw, binary data for 12
samples.
Header
“40 bytes coming, 2 ch”
+ 40 bytes for 1 sample
Header
“36 bytes coming, 2 ch”
+ 36 bytes for 1 sample
Header
“42 bytes coming, 2 ch”
+ 42 bytes for 1 sample
Header
“40 bytes coming, 2 ch”
+ 40 bytes for 1 sample

44.
MP4, ADTS, Atoms, Boxes
Had to manually write conversion logic...
Open MP4 stream (https://github.com/sannies/mp4parser)
Foreach moof, read metadata
Foreach sample, write ADTS header, write sample

45.
Exploring the code
demo

46.
In summary…

47.
In summary…
Enjoy music with an app ✅
Connect to speakers ✅
Get MP4 URL of online video ✅
Extract MP4 audio track to ADTS ✅
Send URL to speakers ✅

48.
“Do you use it often?”

49.
In summary…
Learned a lot of random things along the way ✅
There is so much knowledge out there!
Talk to people (thanks, deadF00d!)
You can build anything!
Even if it seems impossible at first!

50.
Thank you!
https://blog.maartenballiauw.be
@maartenballiauw