Second screen iot_day_stockholm_2014


Published on

Second screen, multi screen using XMPP. What is second screen, why is this popular both for the consumers and for the streaming companies? Why should you use XMPP for doing the second screen experience and why not DIAL or other protocols.

Published in: Software, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Second screen iot_day_stockholm_2014

  1. 1. SECOND SCREEN, XMPP Steffen Larsen twitter: @zooldk IoT Day, April 2014, Stockholm
  2. 2. <STREAM> ABOUT ME • Developing XMPP since 2007, MSc. Copenhagen Uni. - developed pidginVOIP/Video plugin, XMPP for signalling • Member of the XMPP Standard Foundation (XSF) since 2009 - now also on the editorial team • Developed software since 1998 for various companies: telcos, streaming companies etc. • Have my own consulting company (BrainTrust) - Developing software and doing some startups!
  3. 3. RELATED PROJECTS • I have been working with: • Open source:Tigase server, components, Strophe.js plugins etc. • Various chat applications, for video playback commentator, support tools etc. • XMPP over satellites - standardised maritime satellite project (MSDS) for maritime safety service (advanced group chat) • XMPP in gaming and betting industry • XMPP forTV presence for various telcos and cable operators in scandinavia and Europe (first one in 2008!) I’ve done
  4. 4. AGENDA • Intro to Second Screen • Brief intro to XMPP • Second Screen in a XMPP environment (no demo - sorry!) • Beyond Second Screen
  5. 5. SECOND SCREEN? Second screen, sometimes also referred to as “companion device” (or “companion apps” when referring to software applications), is a term that refers to an additional electronic device (e.g. tablet, smartphone) that allows a content consumer to interact with the content they are consuming, such as TV shows, movies, music, or video games. Extra data is displayed on a portable device synchronised with the content being viewed on television. -wikipedia
  6. 6. EVES C-CAST control multiple view and cameras Do replays etc. Do betting
  8. 8. AIRPLAY • Proprietary protocol (formerly known as AirTunes) • Allows wireless streaming of audio, video and photos • Troublesome license..Audio part can be licensed • Screen casting • Discover Airplay devices on same network • Request device control such as trick play (play, pause, ff, rev, stop etc.) • Streams can only be decrypted by using apple keys (no other DRM systems)
  10. 10. CHROMECAST • An implementation of DIAL (netflix) • DIscover And Launch - Simple! - Multicast for discover, HTTP/ REST to open and launch apps (urls). • Not “locked” to any device, but can discovered on the same network • Gives you playback of Netflix,Youtube etc. on a simple cheap stick • Supports screen casting from chrome etc.
  11. 11. CHROMECAST • Libraries for IOS and Android only (no windows) • Browser to browser support, JS (Chrome only) • Supports Play Ready and Encrypted Media Extensions (DRM) • Uses Smooth Streaming and MPEG-Dash - No HLS? WTF! • Uses Google DNS.. :-( • White listings of AppIds
  12. 12. SECOND SCREEN? • An offer to interact through a secondary device than the one you are using as primary screen • Some call it:“Multi screen”,“Companion devices”,“Beaming” • The secondary / companion device: smart phone, tablets and laptops etc • Primary:TV, set-top box, xbox, play station etc. • Second screen for implemented: kiosk systems, remote controlling, multimedia trickplay,“flick-to-screen”.. which I am going to talk about
  14. 14. TRENDS!
  15. 15. TV IS DEAD, LONG LIVETV! “The ways we access television content is changing and evolving, and the multitude of devices on which we can viewTV is ever growing; withTV everywhere on your tablet, mobile or PC, set top boxes and connectedTV’s. How do broadcasters and content providers target audiences appropriately, and where should they be focussing their attentions?” • Usage is different now than before the smart devices, like iPhone,Android and tablets - the want to reach the new audience! ! • second screen for better experience. Some “Smart TVs” and STB are horrible UI and slow on rendering. ! • Make theTV/STB device a stream device, make the UI on second screen
  17. 17. KNOWN PROTOCOLS • DIAL - DIscovery And Launch protocol (Google chromecast) - 2012 • Apple AirPlay • MQTT - weak security (why not use SASL?), pub/sub only in application layer • COAP (HTTP / REST based) • APIs and REST? (many IoT companies does this!) • Others
  18. 18. REQUIREMENTS • Support for many devices! • Routable over the Internet • Secure with authorisation and authentication (because it over the Internet!) • Transport agnostic (diverse devices, software stacks etc) • Do not want to reinvent the wheel!
  19. 19. REQUIREMENTS • Easy Discovery & pairing - how do they find each other? • Roles, authorisation and authentication (diff. devices and users) • Single user or multi - let others do remote stuff. • Control over the infrastructure (no dependencies to silos) • Streaming media or signalling? - most want just signalling • A protocol instead of an API
  20. 20. API’SVS PROTOCOLS •Many IoT vendors only focus on API ! •Via an open standard protocol we would be able to communicate a cross vendors and devices.
  21. 21. DISCOVERY • Scan QR code, click download, wait, launch app = tedious! • Open web browser, type in URL • OpenWRT, DDWrt and others: connect to SSID etc • Have a protocol doing discovery and signalling for you
  22. 22. • Roles - user login / auth or simple devices with no login? • provisioning / re-provisioning devices. • Single user / Multiuser • Do you owe all of the device stack???.. login vs. kiosk system (no login, presentation/buying etc) ROLESVS PROVISIONING
  23. 23. TO CAST ORTO MIRROR? • Mirror: to mirror your display and to transcode it on the second screen device • Cast: to send play and trick play “commands” from the second screen to first screen and let 1’st screen decode etc. • Stream vs. signalling. Content owners have a hard time accepting streaming (airplay mirroring.. transcending on the sending device etc)
  24. 24. SECOND SCREEN ENVIRONMENTS AND LIMITATIONS • To be used on aTV or manyTVs!.. • Many people in a household! • Some native, some are browser based! (only a small limited numbers are native of STB).. a diversity of clients and hardware. need a protocol! • DRM • Easy to use.. people normally have a phone and have a special application installed where they can watchTV and seVOD (video on demand) - no need for discovery of the app! • Pairing should only be done if another customer wants to take control of yourTV experience!
  25. 25. • An Open Standard! yay! • Secure & proven over many years! (many implementations, open source / commercial) • Many routing mechanisms • Federated! (setup your own) • Flexible - beyond Second Screen!
  26. 26. • Transport agnostic • TCP, BOSH - XMPP over HTTP, Websockets • Diff. backends (many servers support native MySQL, REST or own plugins) for login and authentication • Easy to plugin your own business logic and extensions • Many extensions! - for routing possibilities, and others!
  27. 27. • Started as a chat protocol - Google, Facebook, WhatsApp etc. • Also used in gaming (e.g chess, Bingo), synchronisation software, white boarding and collaboration software, taxi hailing etc. • Now used in many other places!
  28. 28. WHY? • Can route over the internet (you do not need to be home to use your second-screen) • Can be used locally (link local - bonjour) as well (instead of client/server model) • Versatile protocol - and not just yet another API! • Open and can federate your data. Move it if you like to your own domain and server
  29. 29. XMPP ARCHITECTURE •Clients (persistent connection) •Servers •Plugins •Components & Gateways
  30. 30. • Normal XMPP architecture involves a server and a lot of clients. • A user can login with several clients • Servers can be federated pr. domain • Server less mode - Link Local (Bonjour) XMPP ARCHITECTURE
  31. 31. XMPP CONNECTION LIFECYCLE • Connect with your user and cred. or anonymous login (authorisation) • Get resource from server (or given from client) • Send initial presence and capabilities • Get presence from your other clients if any • Get roster • Communicate with communication primitives (persistent connection)
  32. 32. XMPP ADDRESSING AND ROUTING • Domain routing like mail (SMTP) • Many diff. mechanisms that can affect a route • bare JID vs. full JID • vs • extensions like: Pub/Sub, MUC, etc.
  33. 33. COMMUNICATION PRIMITIVES • 3 Stanzas, thats it! • <presence/> • <message/> • <iq/> (information / query)
  34. 34. <STANZAS> • You can define the payload it self. • Make an extension! • Define your own <tag> with your own name space. • Embed and use JSON containers or other as payload. easier for browsers to understand
  35. 35.   <message from=‘’ to='francisco@denmark.lit' id=‘foo'>" " " " <body/>"          <json xmlns="urn:xmpp:json:0">"            { "name": "romeo", "age": "421", "status": "single" }"          </json>         "    </message> <STANZAS>
  36. 36. EXTENSIONS
  37. 37. RELEVANT EXTENSIONS • Entity Capabilities - XEP-0115 • Service Discovery - XEP-0030 • BOSH / XMPP over HTTP - XEP-0124 • MUC (Multi User Chat) - XEP-0045 • User Location - Geo/GPS - XEP-0080 • Stream Compression - XEP-0138 • PEP - Personal Eventing Protocol - XEP-0163
  38. 38. RELEVANT EXTENSIONS • Jingle -VOIP/Video - XEP-0166 • Server less Messaging (Bonjour/mDNS) - XEP-0174 • Stream Management (acknowledgement / stream resumption) - XEP-0198 • Message Carbons - XEP-0280 • Remote controlling agents - XEP-146 • MAM - Message Archiving Management - XEP-313 • Message Delivery Receipts - XEP-0184
  39. 39. RELEVANT EXTENSIONS • Event Logging over XMPP - XEP-0337 • IoT: Sensor Data • IoT: Provisioning • IoT: Control • IoT: Concentrators
  40. 40. USING XMPP IN SECOND SCREEN • Almost all OTT (over-the-top) devices like smartTVs and set-top boxes (STB) are browser based. • We have STBs that support HTTP/1.0! • Devices that connect nativelyTCP (like IOS,Android, Microsoft etc) • Customers can use the website to watchTV/Video as well.. that is also connected (via. websockets) customer scenario
  41. 41. • You connect with. second screen (IOS): • • Turn on (smart)TV or STB and it connects auto. • • CONNECTION AND DISCOVERY
  42. 42. COMMUNICATE!   <message from=‘’ to=‘’>" " " " <body/>"            <json xmlns="urn:xmpp:json:0">"            { “play-url": ““ }"            </json>         "    </message>
  43. 43. USAGES • Second Screen stuff (trick play etc) - sending play/stop,ff commands! • Capabilities and service discovery pr. device • Use presence for watching the number of concurrent clients and throttle it (DRM purposes) • Use presence for watching user interaction (channel changes,VOD, follow me etc) • Push notifications via. full JID, bare JID, Pub/Sub (groups) and broadcasts to all
  44. 44. • Remote logging / control the device (customer support) • Let your devices talk together in many topologies - one-to-one, many-to-one , many-to-many • Remote recordings on local storage (PVR) • Exchange bandwidth between devices BEYOND SECOND SCREEN
  45. 45. BEYOND SECOND SCREEN • Instant Context Switching • Instant statistics for online devices (second screen as well asTV etc) • InterconnectedTV with your IoT home! (GW for MQTT, zigbee etc) • Connect with your alarm, turn off theTV etc. • The living room devices gives away a lot of data for recommendation
  46. 46. XMPP AND IOT • We can connect a lot of diverse devices. No matter transport • Create your own gateways if you want to connect to other APIs or protocols • I’ve created a Pub/Sub to MQTT gateway to connect to MQTT connected devices • WebRTC - federation and discovery • UPnP cloud - uses XMPP standard for interconnecting devices • Usage stats! - delivery of a huge amount of real-time data == BIG DATA!
  47. 47. UPNP CLOUD “Cloud support in the UPnP architecture will enable IoT and internet-wide usage by allowing IP-based devices and servers to discover each other and communicate independent of the network they are connected with.”
  48. 48. QUESTIONS? </STREAM> Email / XMPP: Twitter: @zooldk thanks to t.christensen from nordija for inspiration to the slides.