IETF 96 Hackathon - NUBOMEDIA APIs/SDKs and PaaS

IETF 96 Hackathon
17 July 2016, Berlin

Table of contents
1. Introduction
2. NUBOMEDIA APIs/SDKs
3. NUBOMEDIA PaaS
4. Let's get to work

Table of contents
1. Introduction
− Review
− What is NUBOMEDIA?
− References
2. NUBOMEDIA PaaS

1. Introduction
Review
−Kurento is a media server and a set of APIs
aimed to create applications with advance media
capabilities
−The building blocks for applications are named
media elements, chained into media pipelines

1. Introduction
What is NUBOMEDIA?
−NUBOMEDIA is an open source PaaS
(Platform as a Service) based on Kurento
−NUBOMEDIA exposes to developers the
ability of deploying and leveraging
applications with media capabilities:
• WebRTC, media recording, group
communications, computer vision, augmented
reality…

1. Introduction
What is NUBOMEDIA?
−NUBOMEDIA is the first WebRTC PaaS

1. Introduction
What is NUBOMEDIA?
−From the developer’s perspective,
NUBOMEDIA capabilities are accessed
through a set of APIs/SDKs
−NUBOMEDIA applications can be
deployed using the NUBOMEDIA PaaS
Manager
We are going to see this features
in detail in this presentation

1. Introduction
References
−Home page
http://www.nubomedia.eu/
−Developers guide
http://nubomedia.readthedocs.io/
−GitHub organization
https://github.com/nubomedia/
−Support for developers
https://groups.google.com/forum/#!forum/nubomedia-dev

Table of contents
1. Introduction
− Development model
− NUBOMEDIA APIs
− NUBOMEDIA SDKs
1. NUBOMEDIA PaaS

Development model
−The development model in NUBOMEDIA is the
same than in Kurento
• Three-tier model (inspired in the Web)
Client
Application
Server
Media Server
NUBOMEDIA PaaS

Development model
−Like every application with media capabilities, it is
important to distinguish between the media and
signaling plane
Client
Application
Server
Media Server
NUBOMEDIA PaaS
signaling
media

Development model
NUBOMEDIA
Big Picture

NUBOMEDIA APIs
API Description
Media API Enables developers consuming the Media Server capabilities among which
we can find media transport, media archiving, media processing, media
transcoding, and so on
WebRtcPeer API Abstracts the client WebRTC media capabilities, exposing the media
capture and communication capabilities of a browser in a seamless way
Repository API Makes possible to access an elastic scalable media repository for archiving
media information and meta-information
Signaling API Provides a simple signaling mechanism based on JSON-RPCs for
applications
Room API Enables application developers functionalities to create group
communication applications adapted to real social interactions
Tree API Allows developers to build video broadcasting web applications

NUBOMEDIA APIs - Media API
−Media API allows to Java developers consume
the media services provided by Kurento Media
Server (KMS)
−Concepts:
• Media Element
• Media Pipeline

−KMS instances are provided elastically by NUBOMEDIA
• The number of available KMS instances depends on the PaaS
Manager configuration (next section)
−Each KMS has a total amount of available points to create
Media Pipelines and Media Elements
• The total points depends on the number of VCPUs of the KMS
• The type of the instance can be selected on the PaaS Manager
configuration (next section)
Instance type # VCPUs KMS points
Medium 2 200
Large 4 400

−Each KMS is controlled by an instance of KurentoClient
−With each media session an instance of KurentoClient
should be created
−The number of available points per KMS decreases with
each Media Element creation (scaling in/out)
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-client</artifactId>
</dependency>
<dependency>
<groupId>de.fhg.fokus.nubomedia</groupId>
<artifactId>nubomedia-media-client</artifactId>
</dependency>
Dependencies
(Maven)
KurentoClient kurentoClient = KurentoClient.create();

−Example: nubomedia-magic-mirror
// One KurentoClient instance per session
KurentoClient kurentoClient = KurentoClient.create();
// Media logic (pipeline and media elements connectivity)
MediaPipeline mediaPipeline = kurentoClient.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
FaceOverlayFilter faceOverlayFilter = new
FaceOverlayFilter.Builder(mediaPipeline).build();
faceOverlayFilter.setOverlayedImage("http://files.kurento.org/img/mario-wings.png",
-0.35F,
-1.2F, 1.6F, 1.6F);
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);

NUBOMEDIA APIs - Repository API
−NUBOMEDIA provides access to a Kurento Repository in
order to store and recover multimedia streams (and
metadata)
−The repository is controlled by an instance of
RepositoryClient (Java)
<dependency>
</dependency>
<dependency>
<artifactId>nubomedia-repository-
client</artifactId>
</dependency>
Dependencies
(Maven)
RepositoryClient repositoryClient =
RepositoryClientProvider.create();

−Example: nubomedia-repository-tutorial
The first step is to carry out the
WebRTC loopback communication.
We can see the live media while it is
being recorded in the repository.
After that, we are able to reproduce
the recording, reading the media
stored in the repository.

// KurentoClient
kurentoClient = KurentoClient.create();
// Media pipeline
mediaPipeline = kurentoClient.createMediaPipeline();
// Repository item (recorder)
Map<String, String> metadata = Collections.emptyMap();
repositoryItemRecorder = repositoryClient.createRepositoryItem(metadata);
// Media elements and connectivity
webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
recorderEndpoint = new RecorderEndpoint.Builder(mediaPipeline,
repositoryItemRecorder.getUrl()).withMediaProfile(MediaProfileSpecType.WEBM).build();
webRtcEndpoint.connect(webRtcEndpoint);
webRtcEndpoint.connect(recorderEndpoint);
// WebRTC negotiation
String sdpAnswer = performWebRtcNegotiation(session, sdpOffer);
// Start recording
recorderEndpoint.record();

// KurentoClient
// Media pipeline
// Repository item (player)
RepositoryItemPlayer repositoryItemPlayer =
repositoryClient.getReadEndpoint(repositoryItemRecorder.getId());
// Media elements and connectivity
playerEndpoint = new PlayerEndpoint.Builder(mediaPipeline,
repositoryItemPlayer.getUrl())
.build();
playerEndpoint.connect(webRtcEndpoint);
String sdpAnswer = performWebRtcNegotiation(session, sdpOffer);
// Start playing
playerEndpoint.play();

NUBOMEDIA APIs - WebRtcPeer API
−JavaScript API that abstracts the client RTC media (Media
Capture and PeerConnection)
<dependency>
<artifactId>kurento-utils-js</artifactId>
</dependency>
Maven dependency
<script src="js/kurento-utils.js"></script>
var options = {
localVideo : videoInput,
remoteVideo : videoOutput,
onicecandidate : onIceCandidate
}
webRtcPeer = new
kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
function(error) {
if (error) {
return console.error(error);
}
webRtcPeer.generateOffer(onOffer);
});
}
JavaScript library
Example of use
(send-receive)

NUBOMEDIA APIs - Signaling API
−NUBOMEDIA capabilities can be accessed through any
kind of signaling protocol (e.g. SIP, XMPP or REST)
−NUBOMEDIA provides a very signaling protocol based on
JSON-RPCs over WebSockets
−This API has several components
• Server signaling API
• JavaScript client signaling API
• Android client signaling API
• iOS client signaling API
http://nubomedia.readthedocs.io/en/latest/api/signaling/

NUBOMEDIA APIs - Room API
−The Room API is a high-level communications library that
provides capabilities for managing multi-conference
WebRTC sessions. It has the following components:
• Room Server: a container-based implementation of the server,
uses JSON-RPC over WebSockets for communications with the
clients
• Room JavaScript Client: module implementing a Room client for
Web applications
• Room Client: a client library for Java web applications or Android
clients

−Example: nubomedia-room-tutorial
<dependency>
<artifactId>kurento-room-server</artifactId>
</dependency>
<dependency>
<artifactId>kurento-room-client-
js</artifactId>
</dependency>
Dependencies
(Maven)
public class SingleKmsManager extends KmsManager {
@Override
public KurentoClient getKurentoClient(KurentoClientSessionInfo sessionInfo) throws RoomException {
return KurentoClient.create();
}
@Override
public boolean destroyWhenUnused() {
return true;
}
}
Server-side:
KurentoClient
management

NUBOMEDIA APIs - Tree API
−The NUBOMEDIA Tree API allows developers to build
WebRTC broadcasting applications
−This API is composed by a server and two clients:
• tree-server is a component of the API designed to be deployed
and controlled by clients
• tree-client implements a tree client designed to be used in Java
web applications or Android applications. The client contains an
implementation of JSON-RPC WebSocket protocol implemented
by Tree server
• tree-client-js implements a tree client to be used in Single Page
Applications (SPA)

NUBOMEDIA SDKs
−More info on:
• http://webrtcpeer-android.readthedocs.io/
• http://jsonrpc-ws-android.readthedocs.io/
• http://kurento-room-client-android.readthedocs.io/
• http://kurento-ios.readthedocs.io/
SDK Description
Android SDK Android version of the client-side NUBOMEDIA APIs: WebRtcPeer,
Signaling, Room, Tree
iOS SDK Complete SDK to provide NUBOMEDIA client capabilities to iOS
devices

Table of contents
1. Introduction
3. NUBOMEDIA PaaS
− Introduction
− PaaS GUI

3. NUBOMEDIA PaaS
Introduction
−The NUBOMEDIA PaaS manager is a tool aimed to
control the way in which the NUBOMEDIA applications are
built and deployed inside the NUBOMEDIA PaaS
−The capabilities provided by the Paas Manager can be
used by developers using the PaaS GUI:
• The PaaS Manager GUI is a web application that allows to use
the NUBOMEDIA PaaS Manager

3. NUBOMEDIA PaaS
Introduction
−Internally, the NUBOMEDIA PaaS uses Docker
containers to deploy applications
−Therefore it is a requirement to include a Dockerfile in
GitHub repository to be deployed on NUBOMEDIA
−Example: FROM nubomedia/apps-baseimage:src
MAINTAINER Nubomedia
ADD keystore.jks /
ADD . /home/nubomedia
RUN sudo chown -R nubomedia /home/nubomedia
RUN cd /home/nubomedia && mvn compile
ENTRYPOINT cd /home/nubomedia && mvn exec:java
https://docs.docker.com/engine/reference/builder/

3. NUBOMEDIA PaaS
PaaS GUI
−Web application to manage NUBOMEDIA applications
http://paas-manager.nubomedia.eu:8081/
Credentials
needed to login
Home shows a
summary of the
apps currently
deployed

3. NUBOMEDIA PaaS
PaaS GUI
−A NUBOMEDIA application can be deployed using the
PaaS GUI
−It is done providing the GitHub repository URL and a set
of configuration parameters

3. NUBOMEDIA PaaS
PaaS GUI
−Most important configuration values:
−For the rest of parameters take a look to the doc:
http://nubomedia.readthedocs.io/en/latest/paas/paas-gui/
Number of
KMSs
KMS host type:
-Medium = 2 VCPUs (200 points)
-Large = 4 VCPUs (400 points)

Table of contents
1. Introduction
3. NUBOMEDIA PaaS
− Introduction
− Server-side
− Client-side
− Deployment

Introduction
−Devil is in the details, and so, we are going to see a
complete NUBOMEDIA step by step
−This applications the nubomedia-magic-mirror tutorial
https://github.com/nubomedia/nubomedia-magic-mirror/
http://nubomedia.readthedocs.io/en/latest/tutorial/nubomedia-magic-mirror/

Server-side
−We recommend Spring-Boot as the base technology for
NUBOMEDIA applications
• Spring-Boot embeds a Tomcat server in a simple seamless way
for developers
• The application is packaged as a runnable JAR, and when this
JAR is executed, the Tomcat server is started and the web
application automatically deployed
−We recommend Maven as for managing the life-cycle and
managing the dependencies of our applications

Server-side
−The first step is to clone GitHub repository
−We recommend use an IDE to develop applications. For
example, Eclipse (importing app as Maven project)
git clone https://github.com/nubomedia/nubomedia-magic-mirror

Server-side
−First step is understand the pom.xml file:
<parent>
<artifactId>kurento-parent-pom</artifactId>
<version>6.5.0</version>
</parent>
<properties>

<nubomedia-media-client.version>1.0.2</nubomedia-media-client.version>

<start-class>eu.nubomedia.tutorial.magicmirror.MagicMirrorApp</start-class>
</properties>
Inherit from kurento-parent-pom
makes simpler our pom.xml by
reusing versions defined in the parent
It’s important
declare the
fully qualified
name of the
main class

Server-side
<dependencies>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
</dependency>

<dependency>
<artifactId>kurento-client</artifactId>
</dependency>
<dependency>
<artifactId>kurento-utils-js</artifactId>
</dependency>

<dependency>
<version>${nubomedia-media-
client.version}</version>
</dependency>
</dependencies>
…
…
Dependencies clause is one of
the most important. Notice that
our app depends on Spring,
Kurento, and NUBOMEDIA

Server-side
−We need to define also the way
in which signaling is going to
work in our application
−For the shake of simplicity, we
use a raw WebSocket between
the server and client side
−We need to define the messages
exchanged to carry out the
communication about client and
server
clientserver
start
startResponse
iceCandidate
onIceCandidate
stop
error
notEnoughResources
Starts a media
session
ICE candidates
(WebRTC
negotiation)
Stops a media
session
Error cases

Server-side
−The class diagram of our application is the following:
MagicMirrorApp
MagicMirrorHandler
UserSession
Main class
Message handler
(signaling)
User session
(session id,
media logic)

Server-side
@SpringBootApplication
@EnableWebSocket
public class MagicMirrorApp implements WebSocketConfigurer {
@Bean
public MagicMirrorHandler handler() {
return new MagicMirrorHandler();
}
@Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(handler(), "/magicmirror");
}
public static void main(String[] args) throws Exception {
new SpringApplication(MagicMirrorApp.class).run(args);
}
}
MagicMirrorApp defines the
Spring-Boot application and
the WebSocket used for
signaling ("/magicmirror",
managed by
MagicMirrorHandler)

Server-side
public class MagicMirrorHandler extends TextWebSocketHandler {
private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<>();
@Override
public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
JsonObject jsonMessage = new GsonBuilder().create().fromJson(message.getPayload(),
JsonObject.class);
switch (jsonMessage.get("id").getAsString()) {
case "start":
start(session, jsonMessage);
break;
case "stop":
release(session);
break;
case "onIceCandidate":
onIceCandidate(session, jsonMessage);
break;
default:
error(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
break;
}
}
MagicMirrorHandler
manages signaling
messages. It keeps a
collection of user sessions
(thread-safe map users)

Server-side
public class UserSession {
public String startSession(final WebSocketSession session, String sdpOffer) {
// One KurentoClient instance per session
log.info("Created kurentoClient (session {})", sessionId);
// Media logic (pipeline and media elements connectivity)
log.info("Created Media Pipeline {} (session {})", mediaPipeline.getId(), sessionId);
FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(mediaPipeline).build();
faceOverlayFilter.setOverlayedImage("http://files.kurento.org/img/mario-wings.png", -0.35F,
-1.2F, 1.6F, 1.6F);
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);
webRtcEndpoint.gatherCandidates();
return sdpAnswer;
}
UserSession
handles media logic
(media pipeline,
media elements,
WebRTC
negotiation…)

Server-side
−As of Chrome 47, access to user media can only be done
by secure apps (HTTPS), and so, it is needed a Java
keystore
server.port: 8443
server.ssl.key-store: keystore.jks
server.ssl.key-store-password: kurento
server.ssl.keyStoreType: JKS
server.ssl.keyAlias: kurento-selfsigned
application.properties
http://doc-kurento.readthedocs.io/en/stable/mastering/securing-kurento-applications.html

Client-side
−Client-side dependencies are handled by Bower
−Our bower.json file contains:
−In addition the file .bowerrc configures the target path for
the JavaScript libraries
"dependencies": {
"bootstrap": "~3.3.0",
"ekko-lightbox": "~3.1.4",
"adapter.js": "v0.2.9",
"demo-console": "1.5.1"
}
{
"directory" : "static/bower_components"
}

Client-side
−The app has been implemented as a SPA (single page
application) which contains a single HTML
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" href="bower_components/bootstrap/dist/css/bootstrap.min.css">
<link rel="stylesheet" href="bower_components/ekko-lightbox/dist/ekko-lightbox.min.css">
<link rel="stylesheet" href="bower_components/demo-console/index.css">
<link rel="stylesheet" href="css/styles.css">
<script src="bower_components/jquery/dist/jquery.min.js"></script>
<script src="bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
<script src="bower_components/ekko-lightbox/dist/ekko-lightbox.min.js"></script>
<script src="bower_components/adapter.js/adapter.js"></script>
<script src="bower_components/demo-console/index.js"></script>
<script src="js/kurento-utils.js"></script>
<script src="js/index.js"></script>
<title>NUBOMEDIA Tutorial: WebRTC Magic Mirror</title>
</head>
JavaScript/CSS are
linked here. Using
Bootstrap to provide
responsive design

Client-side
−A JavaScript file contains the client-side logic (message
handling and WebRtcPeer use)
var ws = new WebSocket('wss://' + location.host + '/magicmirror');
ws.onmessage = function(message) {
var parsedMessage = JSON.parse(message.data);
switch (parsedMessage.id) {
case 'startResponse':
startResponse(parsedMessage);
break;
case 'error':
onError("Error message from server: " + parsedMessage.message);
stop(false);
break;
case 'iceCandidate':
webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
if (error) return console.error("Error adding candidate: " + error);
});
break;
// ...
}
}

Client-side
−A JavaScript file contains the client-side logic (message
handling and WebRtcPeer use)
var webRtcPeer;
function start() {
var options = {
localVideo : videoInput,
remoteVideo : videoOutput,
onicecandidate : onIceCandidate
}
webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
if (error) {
return console.error(error);
}
webRtcPeer.generateOffer(onOffer);
});
}

Deployment
−We need a Dockerfile to deploy our app
FROM nubomedia/apps-baseimage:src
MAINTAINER Nubomedia
ADD keystore.jks /
ADD . /home/nubomedia
RUN sudo chown -R nubomedia /home/nubomedia
RUN cd /home/nubomedia && mvn compile
ENTRYPOINT cd /home/nubomedia && mvn exec:java

Deployment
−Finally we can deploy our app in the NUBOMEDIA PaaS
Manager:

Deployment
−The app changes its state, from CREATED to RUNNING (it
takes a couple of minutes to finish):

Deployment
−We can trace two types of logs: build and app
I0708 12:51:02.335080 1 builder.go:41] $BUILD env
var is {"kind":"Build","apiVersion":"v1","metadata":{...}
I0708 12:51:02.423366 1 source.go:96] git ls-remote
https://github.com/nubomedia/nubomedia-magic-mirror
--heads
I0708 12:51:02.423547 1 repository.go:275]
Executing git ls-remote
https://github.com/nubomedia/nubomedia-magic-mirror
--heads
I0708 12:51:04.931566 1 source.go:189] Cloning
source from https://github.com/nubomedia/nubomedia-magic-
mirror
...
Image already exists
latest: digest:
sha256:24ac51ca448edfb1582f0bffc990c95506065f0aa26d5b295d
1cb32d35ce2dfe size: 49623
I0708 12:54:31.392578 1 docker.go:99] Push
successful

Deployment
−We can trace two types of logs: build and app
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------------------------
[INFO] Building NUBOMEDIA Java Tutorial - Magic Mirror 6.5.0
[INFO] ----------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @
nubomedia-magic-mirror ---
`.-:////:-.`
-+shmmmmmmmmmmmmmmhs/-
`/ymmmmmmmmmmmmmmmmmmmmmmmms/`
-smmmmmmmmyo+/:----:/+ohmmmmmmmms.
-ymmmmmmy+. -+hmmmmmmy. -------`
òmmmmmd+` .+dmmmmmo +oooooo-
.hmmmmm+ òmmmmmh` +oooooo-
.dmmmmh. ``````` +oooooo- sdddddds
`dmmmmy .dhhhhhh. ```````` smmmmmmy
./+osyysymmmmy .mmmmmmm. smmmmmmy
`:sdmmmmmmmmmmmmd` .mmmmmmm. -----` +yyyyyy+
`+dmmmmmmdhyyyhdmm+ .hhhhhhh. ooooo- --------
-dmmmmmy:` `:` ooooo- .----. oooooooo`
-mmmmmy. .....` hmmmms oooooooo`
dmmmms yhd- hmmmms oooooooo`
:mmmmm` syy- /++++: ::::::::
/mmmmm
`mmmmm/ -ss: os` os` -s/ -ssssss+- .+syys+. -//. ://` .::///- -/////:. // .//`
+mmmmm: :mdm/ hm. hm. /mo :my---/mm`:md:..:dm/ /o+o` -o+o` /o:.```` :o:``.:o+ oo ò/++
smmmmh :ms:m+ ym. hm. /mo :mh+++smo ym/ :mh /o.o: ò/:o`.oo:::::. :o- /o- oo +o`.o/
/mmmh :ms :moym. hm. /mo :my:::+mh sm+ /my /o.-o./o`/o`.oo.....` :o- +o- oo /o+//+o:
òdh :ms -mmm. +my::/dm- :my///omm`.dm+::+mm- /o. +oo: /o` :o/----. :o/---/o/ oo -o:..../o.
- .+: .++` ./+++:` .++++++:` `:+oo+:` .-` `-- .-` `-----. .------` -- -- `--
...
2016-07-08 12:55:32.702 INFO 6 --- [irrorApp.main()]
s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on
port(s): 8443 (https)
2016-07-08 12:55:32.713 INFO 6 --- [irrorApp.main()]
e.n.tutorial.magicmirror.MagicMirrorApp : Started MagicMirrorApp
in 10.529 seconds (JVM running for 31.135)

Deployment
−Ta da!

IETF 96 Hackathon - NUBOMEDIA APIs/SDKs and PaaS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IETF 96 Hackathon - NUBOMEDIA APIs/SDKs and PaaS

Similar to IETF 96 Hackathon - NUBOMEDIA APIs/SDKs and PaaS (20)

More from Boni García

More from Boni García (16)

Recently uploaded

Recently uploaded (20)

IETF 96 Hackathon - NUBOMEDIA APIs/SDKs and PaaS