DEVIEW2014 [1D1]신개념 N스크린 웹 앱 프레임워크 PARS

3. 이동영 / LG전자
dongyoung.lee@lge.com
신개념 N스크린 웹 앱 프레임워크 'PARS'

4. 1.멀티스크린 웹 앱 아키텍처의 발전 방향
2.PARS 데모
3.PARS 구현 방법 및 이슈
4.Summary
CONTENTS

5. 1. 멀티스크린 웹 앱 아키텍처의 발전 방향

6. Multiscreen Applications
A user has multiple devices.
And they tend to use them together simultaneously.
Manual coordination or ad-hoc methods.
“Orchestrated” user experience.
Browsing a picture gallery on a smartphone, displaying large pictures on a TV.
Watching TV, with a program guide on a tablet.
Watching a lecture on a TV, answering exercise questions on a tablet.
A map on a TV, with a list of points-of-interest on a smartphone.
Multiplayer games, with TV as a shared screen.
A cooperating set of programs running on a set of devices.

7. Approach 1. Fixed Services
An application uses fixed services on other devices
Role of each device is fixed.
E.g., play a video, display a picture.
UI is fixed, and may be inconsistent across devices.
UPnP, DLNA, AirPlay
Platform features
Discovery
APIs

8. Approach 2. Cooperating Applications
A set of cooperating applications (a dedicated app for each device)
Role of each device is programmable, but still pre-defined.
E.g., display a public view (in a game), display a personal view.
Unified UI across devices.
A user has to pre-install and launch apps on all devices.
Early multiscreen applications.
Platform features
Usually none (generic networking, etc.)

9. Approach 2.5 Cooperating Apps with Launch
A set of cooperating applications (a dedicated app for each device)
Role of each device is programmable, but still pre-defined.
Unified UI across devices.
The platform launches (and sometimes installs) an app on another device.
A user doesn’t have to launch an app on every device.
LG Connect SDK, Samsung Multiscreen SDK, DIAL
Fragmented
Platform features
Discovery, Launching
Installing, Messaging, Time synchronization, etc.

10. Multi-platform / protocol support
Not only for LG TV
Open source
Connect SDK
http://connectsdk.com/

11. Approach 3. A Distributed Application
A distributed application (a single application)
Roles of devices can be re-organized during runtime.
A part of an app can migrate from a device to another.
No installation (of apps).
PARS, COLTRAM, and other works from KAIST, NTT, etc.
Platform features
Discovery, Launch
Migration / Replication / Re-organization (during runtime), State synchronization
Messaging

12. Roles of devices can be re-organized during runtime.
Parts of an app migrate or are replicated across devices.
A cross platform is required.
Composable UI is desirable.
UI adaptation is needed.
No installation.
Sandbox security model allows executing alien code without installation.
A Distributed “Web” Application

13. 2. PARS 데모

14. Demo 1. Map (Single User)

15. Demo 2. Gallery (Multi User)

16. 3. PARS 구현 방법 및 이슈

17. Architecture
A PARS app consists of components
<div>, JS variables and methods
Component migration / replication
State synchronization
Component management UI
Device discovery
Messaging

18. A PARS app consists of components
<div>, JavaScript variables and methods (public functions)
All access to another component must be via remote function (method) calls.
Directly accessing another component is prohibited (DOM, JS objects).
Application layer

19. Application layer
<div PARS = “componentA”>
This is component A.
<p id=“A”> Default 0</p>
</div>
<div PARS = “componentB”>
This is component B.
<button type=“button” onclick=“componentB.myFunction()”>
Click to change
</button>
</div>
<script>
componentA.count = 0;
componentA.setText = function(text) {
componentA.count++;
document.getElementById(“A”).innerHTML = text + “ “ + componentA.count;
}
componentB.myFunction = function() {
PARS.invoke(componentA, setText, “Changed”);
}
</script>

20. How to keep the state of a component synchronized with those of other components on other devices?
A component interacts with another component only via remote function calls.
For components on other devices, relay remote function calls.
Open Issues
Call order, timing, network error, etc.
Local API calls
Framework Layer: State Synchronization

21. Take a snapshot, send it, and restore it.
HTML: innerHTML
JavaScript: toString (JSON)
Open Issues
Canvas: toDataURL
CSS: CSSOM
Cookie, local storage, …
Hooks before and after migration / replication
Framework Layer: Component Migration

22. Framework Layer: Replication
How to keep the states of replicated components in sync?
Replicate remote function calls.
Alternatives
Mutation observer (HTML)
Object.observe (JavaScript)
Multi-User Case
Some components may need to have different states according to the user.
E.g., the viewer and local pictures components in the gallery demo

23. Framework Layer: Network Adaptation
To accommodate any underlying network protocols that provides discovery and messaging.
discovery()
registerMessageHandler()
send

24. Device discovery
Messaging
Remote execution (then we don’t need a daemon)
Time synchronization (esp. for synchronized media playback)
Current implementation is based on node.js.
Each device runs a node service, which discovers other devices and relay messages.
The framework connects to the local node service using socket.io.
Platform Layer: Requirements

25. Webinos (http://webinos.org/)
We use only discovery and messaging features.
Based on node.js.
ParsNet
Each device runs a simple node service, which discovers other devices and relay messages.
The framework connects to the local node service using Web Socket.
Alternatives
Discovery and messaging via a cloud service.
Platform Layer: Implementations

26. Interoperability
Abstract APIs to support various underlying technologies, including existing devices.
Standard protocols for interoperability across vendors.
Security and privacy vs. usability
UA or web app?
Platform Layer: Standards
WebRTC
Network Service Discovery
DIAL
Presentation API
Discovery



P2P messaging


Remote execution


Time synchronization

27. W3C Second Screen CG  WG
Presentation API
/* controller.html */
<script>
function playVideo() {
if (!navigator.presentation.displayAvailable)
return;
var p = navigator.presentation.requestShow('player.html');
p.then(function (display) { display.postMessage('http://example.org/video.mp4', '/'); },
function (error) { console.log('Request to show failed: ' + error.msg); }
);
}
playVideo();
</script
/* player.html */
<video>
<script>
var v = document.querySelector('video');
window.onmessage = function (event) {
v.src = event.data;
v.play();
};
</script>

28. 4. Summary

29. A distributed web app as a solution for multiscreen UX.
You can make your own “PARS”.
E.g., on top of Connect SDK.
Stay tuned for the Presentation API.
Summary

30. Q&A

31. THANK YOU