Remote Graphical Rendering

Remote Graphical Rendering

Joel Isaacson

Ascender Technologies Ltd.

Copyright 2012 Joel Isaacson

Ascender Technology Ltd Remote Rendering

Cellphone App


Exporting Graphics

Graphics can be exported from any of the four
levels of the graphics stack
– Application level
– Toolkit level
– Rendering level
– Bitmap level


Graphics Stack


Application Level (1)

This is the normal way Linux/Android/Iphone
runs apps.

The application itself is exported and run locally.

Normally applications either come installed with
the system or are added later (e.g. Android
Market).

Applications can run locally without using any
network bandwidth.


Bitmap Level (4)

Common approach to exporting graphics (VNC,
Chromoting).

Works well for any mix of graphics applications.

Simple to implement.

Heavy user of network bandwidth.

Amenable to compression.

Computationally intensive.


Toolkit Level (2)

It is technically very complex. Android, to date,
has 15 different toolkit API variants.

Every application can extend the toolkit with
custom widgets (subclasses of android.view.View).

Clearly impossible.


Toolkit Level (2)

It is technically very complex. Android, to date,
has 15 different toolkit API variants.

Every application can extend the toolkit with
custom widgets (subclasses of android.view.View).

Clearly impossible.

This talk will show that effectively exporting
graphics at the toolkit level and even the
application level is in fact possible.


Android Rendering Level

This work exports graphics at the rendering level.

In Android there are a number of rendering
interfaces that can be used:
– Skia graphics
– OpenGL ES 1.1 or OpenGL ES 2.0
– Android.view.View

We shall see that in effect toolkit and application
level graphics can be exported.


The Rendering Layer Expanded


Android Graphics Stack


Software Rendering Level (Skia)

The rendering level is the graphics layer that
actually “colors” the pixels in the bitmap.

The Skia Graphics Engine is a compact open
source graphics library written in C++.

It was originally developed by Skia Inc., which
was subsequently acquired by Google, who
released the software as open source.

Now known as Skia, it is currently used in Google
Chrome, Chrome OS, Chromium OS, and Android.

Rendering Level (Skia)

Skia is a complete 2D graphic library for drawing
Text, Geometries, and Images. Features include:
− 3x3 matrices w/ perspective
− Antialiasing, transparency, filters
− Shaders, xfermodes, maskfilters,
patheffects


Rendering Level (Skia)

Each Skia call has two components: the primitive
being drawing (SkRect, SkPath, etc.) and
color/style attributes (SkPaint).

For example:
canvas.save();
canvas.rotate(45);
canvas.drawRect(rect, paint);
canvas.drawText(“abc”, 3, x, y, paint);
canvas.restore();


NinePatch Widget


GUI Widget
NinePatch Code
DrawbitmapRect(); // 1
DrawRect()


Android Remote Rendering



We can simply modify the Skia library to export
graphics by adding a small stub to Android to
serialize and send a RPC-like rendering stream.

The local client can be on any system (Linux, MS
Windows, Mac OS).

The prototype developed was on Linux-X11.

Only 17 Skia routines had to be modified to
provide remote Android graphics.


Round Trip Latencies

The US east-west coast round trip is about 70 ms.

Just one round trip delay per frame would reduce
the possible frame rate to 14 FPS.

Potentially every Skia call returns a result. Some
results are critical for “Measure Passes”.

In order to obtain measurements, we run a Skia
renderer in parallel on the remote machine.

We can then send the rendering stream at full
network bandwidth, via a simplex link.

Simplex Network Connection


Compression

Android apps have a frame rate of 30-60 FPS.

For every frame the complete rendering sequence
is sent. Compression is a good idea.


MPEG Compression

Video streams, photographically taken or
photorealistically synthesized, can be compressed
with MPEG standard codecs.

Even though no specific technique used in MPEG
compression is applicable for the compression of
our problem domain (rendering streams), some of
the assumptions about how the data sets are
generated are similar and the compression model
design is similar.


MPEG Stream Assumptions

The material consists of a large number of
sequential images (frames).

The target of the video images is the human visual
system and consists of moving images.

The subject matter is a product of our everyday
visual world and not some random series of
random images.

Apparent smooth motion depends on visual
continuity between frames.

MPEG Compression Techniques

The conversion of RGB images to YUV and sub-
sampling is motivated by the color physiological
opponency theory model of human vision.

Within each frame, the accuracy of spacial changes
with shorter wavelengths are less important than
the accuracy of longer wavelengths.

Inter-frame compression is based on finding
motion vectors of the current frame based on
previous and subsequent frames.


Rendering Stream Assumptions

The rendered material consists of a large number of
sequential images (frames).

The target of the GUI images is the human visual
system.

The subject matter while not being a product of our
everyday visual world is modeled on the visual
contexts of this world.

The GUI images are generated frame after frame
by repeated invocation of GUI procedures.

Repeated Invocation of GUI
Procedures

In practice, the number of unique sequences of
rendering functions in execution paths taken within
the code are bounded. This is because the rendering
commands are generated by a fixed number of GUI
functions and an application running a bounded
amount of code.

The execution paths can be incrementally learned
and entered into a procedure dictionary as the
rendering commands are streamed.


Rendering Compression Techniques

Even if the sequences of rendering functions
themselves are in the dictionary, the data
arguments associated with these functions might be
quite different from one another. Therefore, we
keep a dictionary of the data arguments previously
encountered for this particular sequence.

As a rendering procedural sequence with
associated data is encountered, the data sequence
dictionary for this procedural sequence is searched
for the closest match to the current data sequence.

Android Contact List
(Froyo)


Android Contact


Structured Procedural Compression

A careful examination of the rendering stream
generated by the Android GUI reveals additional
structural information that can be used to improve
the data model.

The rendering stream has balanced save()-restore()
pairs within each frame of the rendering stream.

Each save() is found at the beginning of a GUI
function and a restore() is found at the end of each
GUI function.



This information can be used to reverse engineer
individual GUI and application procedures.

It will also reveal the call-graph of these procedures.

Using this information, the rendering code
dictionary becomes a rendering procedure
dictionary.

The call-graph data is best embedded in the per-
procedure data dictionary.

DEMO


The server (encoder) constructs a “procedure
dictionary” that is identical to the dictionary that is
constructed by the client (decoder). Similar of
LZW compression.

The procedures in this dictionary, surprisingly, are
reversed engineered toolkit and application level
routines.

We in effect have ended up exporting graphics at
the toolkit-application level.


Statistics

Our compression algorithm was tested on the
rendering trace of a 60 frame sequence.

There were 13702 rendering commands for an
average of 228 rendering commands per frame.

Of the 13702 rendering commands, there were
2691 functions (save/restore pairs).

Of these, only 47 were unique. This gives a
compression rate of 1.75% (about 1:57).


Functional Compression
47 Unique Functions From 2691 Total


Statistics

Of the 13702 rendering commands only 354 had
completely unique data parameter sets and 203 had
data sets which are partially different.

Only these 557 data sets must be transmitted,
thereby giving data compression of 4.06% (about
1:25).

If the partially different data sets are differentially
transmitted, a data compression of 3.3% (about
1:30) is obtained.


Functional Compression
557 Unique Data Args From 13702 Total


Entropy Encoding

For the last stage of compression, general
techniques such as run length encoding (RLE) and
entropy encoding (Huffman or Arithmetic) are
used to produce a minimum bit count
representation of the compressed material.

This additional phase should be expected to reduce
the number of bits by a factor of 2-3


ICS GUI Rendering

Under ICS remote rendering can be done in any of
three different interfaces:
– Software rendering – Skia
– Hardware rendering – OpenGL ES 2.0
– Abstract rendering – Canvas class

All three interfaces can use procedural
compression.

The OpenGL rendering stream is much more
verbose (5x) than the equivalent Skia stream.

Uses

So what is all this useful for:
− Cloud computing, remote app server
− App library, subscription model
− App demos
− Remote enterprise applications
− Set-top boxes


Cloud computing

Any Android app can be run on the server.

It is not necessary that the remote server and the
local device have the same architecture, i.e. an
ARM server can provide services for an Intel
device.

For example: You might use Remote Rendering to
provide Android apps to Chrome OS.


App Library / Subscription Model

Currently, apps are loaded into the local device -
either installed at the time of purchase or added
subsequently.

If efficient remote execution of apps can be
supported, then instead of software purchases, a
subscription model becomes possible.

A fixed monthly fee would entitle the subscriber to
access a large library of applications.


App Demos

In a purchase/rental model, apps can be demoed
remotely, prior to purchase.

If the user of the device finds the app to his/her
liking, it can then be bought.


Remote Enterprise Applications

Applications may benefit from running within the
enterprise's data centers, with the obvious benefits
of scalability, security and maintainability.

Enterprise workers typically migrates through
various computing environments (mobile-fixed-
mobile) during a typical day.

Having the application running on a server will
allow the running applications to migrate to
various clients seamlessly.


Set-top Boxes

Many set-top boxes cannot practically run Android
since the set-top box might not have sufficient
resources or might not contain an ARM processor.

A local client application that only performs
remote rendering takes limited resources and needs
no long term persistent state.

Set-top boxes typically don't have complex local
installations. Installing Android apps locally would
present a support challenge to the operator.


Remote Graphical Rendering

More Related Content

What's hot

Viewers also liked

Similar to Remote Graphical Rendering

Recently uploaded

Remote Graphical Rendering