Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Remote Graphical Rendering


Published on

We have developed technology that allows a wide range of graphical interfaces to be streamed efficiently over wide area networks. This is an enabling technology that enables remote graphics akin to the way MPEG compression enables video streaming.

Published in: Technology, Art & Photos
  • Be the first to comment

Remote Graphical Rendering

  1. 1. Remote Graphical Rendering Joel Isaacson Ascender Technologies Ltd. Copyright 2012 Joel IsaacsonAscender Technology Ltd Remote Rendering
  2. 2. Cellphone AppAscender Technology Ltd Remote Rendering
  3. 3. Exporting Graphics  Graphics can be exported from any of the four levels of the graphics stack – Application level – Toolkit level – Rendering level – Bitmap levelAscender Technology Ltd Remote Rendering
  4. 4. Graphics StackAscender Technology Ltd Remote Rendering
  5. 5. Application Level (1)  This is the normal way Linux/Android/Iphone runs apps.  The application itself is exported and run locally.  Normally applications either come installed with the system or are added later (e.g. Android Market).  Applications can run locally without using any network bandwidth.Ascender Technology Ltd Remote Rendering
  6. 6. Bitmap Level (4)  Common approach to exporting graphics (VNC, Chromoting).  Works well for any mix of graphics applications.  Simple to implement.  Heavy user of network bandwidth.  Amenable to compression.  Computationally intensive.Ascender Technology Ltd Remote Rendering
  7. 7. Toolkit Level (2)  It is technically very complex. Android, to date, has 15 different toolkit API variants.  Every application can extend the toolkit with custom widgets (subclasses of android.view.View).  Clearly impossible.Ascender Technology Ltd Remote Rendering
  8. 8. Toolkit Level (2)  It is technically very complex. Android, to date, has 15 different toolkit API variants.  Every application can extend the toolkit with custom widgets (subclasses of android.view.View).  Clearly impossible.  This talk will show that effectively exporting graphics at the toolkit level and even the application level is in fact possible.Ascender Technology Ltd Remote Rendering
  9. 9. Android Rendering Level  This work exports graphics at the rendering level.  In Android there are a number of rendering interfaces that can be used: – Skia graphics – OpenGL ES 1.1 or OpenGL ES 2.0 – Android.view.View  We shall see that in effect toolkit and application level graphics can be exported.Ascender Technology Ltd Remote Rendering
  10. 10. The Rendering Layer ExpandedAscender Technology Ltd Remote Rendering
  11. 11. Android Graphics StackAscender Technology Ltd Remote Rendering
  12. 12. Software Rendering Level (Skia)  The rendering level is the graphics layer that actually “colors” the pixels in the bitmap.  The Skia Graphics Engine is a compact open source graphics library written in C++.  It was originally developed by Skia Inc., which was subsequently acquired by Google, who released the software as open source.  Now known as Skia, it is currently used in Google Chrome, Chrome OS, Chromium OS, and Android.Ascender Technology Ltd Remote Rendering
  13. 13. Rendering Level (Skia)  Skia is a complete 2D graphic library for drawing Text, Geometries, and Images. Features include: − 3x3 matrices w/ perspective − Antialiasing, transparency, filters − Shaders, xfermodes, maskfilters, patheffectsAscender Technology Ltd Remote Rendering
  14. 14. Rendering Level (Skia)  Each Skia call has two components: the primitive being drawing (SkRect, SkPath, etc.) and color/style attributes (SkPaint).  For example:; canvas.rotate(45); canvas.drawRect(rect, paint); canvas.drawText(“abc”, 3, x, y, paint); canvas.restore();Ascender Technology Ltd Remote Rendering
  15. 15. NinePatch WidgetAscender Technology Ltd Remote Rendering
  16. 16. GUI Widget NinePatch Code DrawbitmapRect(); // 1 DrawbitmapRect(); // 2 DrawbitmapRect(); // 3 DrawbitmapRect(); // 4 DrawRect() DrawbitmapRect(); // 5 DrawbitmapRect(); // 6 DrawbitmapRect(); // 7 DrawbitmapRect(); // 8 DrawbitmapRect(); // 9Ascender Technology Ltd Remote Rendering
  17. 17. Android Remote RenderingAscender Technology Ltd Remote Rendering
  18. 18. Android Remote Rendering  We can simply modify the Skia library to export graphics by adding a small stub to Android to serialize and send a RPC-like rendering stream.  The local client can be on any system (Linux, MS Windows, Mac OS).  The prototype developed was on Linux-X11.  Only 17 Skia routines had to be modified to provide remote Android graphics.Ascender Technology Ltd Remote Rendering
  19. 19. Round Trip Latencies  The US east-west coast round trip is about 70 ms.  Just one round trip delay per frame would reduce the possible frame rate to 14 FPS.  Potentially every Skia call returns a result. Some results are critical for “Measure Passes”.  In order to obtain measurements, we run a Skia renderer in parallel on the remote machine.  We can then send the rendering stream at full network bandwidth, via a simplex link.Ascender Technology Ltd Remote Rendering
  20. 20. Android Remote Rendering Simplex Network ConnectionAscender Technology Ltd Remote Rendering
  21. 21. Compression  Android apps have a frame rate of 30-60 FPS.  For every frame the complete rendering sequence is sent. Compression is a good idea.Ascender Technology Ltd Remote Rendering
  22. 22. MPEG Compression  Video streams, photographically taken or photorealistically synthesized, can be compressed with MPEG standard codecs.  Even though no specific technique used in MPEG compression is applicable for the compression of our problem domain (rendering streams), some of the assumptions about how the data sets are generated are similar and the compression model design is similar.Ascender Technology Ltd Remote Rendering
  23. 23. MPEG Stream Assumptions  The material consists of a large number of sequential images (frames).  The target of the video images is the human visual system and consists of moving images.  The subject matter is a product of our everyday visual world and not some random series of random images.  Apparent smooth motion depends on visual continuity between frames.Ascender Technology Ltd Remote Rendering
  24. 24. MPEG Compression Techniques  The conversion of RGB images to YUV and sub- sampling is motivated by the color physiological opponency theory model of human vision.  Within each frame, the accuracy of spacial changes with shorter wavelengths are less important than the accuracy of longer wavelengths.  Inter-frame compression is based on finding motion vectors of the current frame based on previous and subsequent frames.Ascender Technology Ltd Remote Rendering
  25. 25. Rendering Stream Assumptions  The rendered material consists of a large number of sequential images (frames).  The target of the GUI images is the human visual system.  The subject matter while not being a product of our everyday visual world is modeled on the visual contexts of this world.  The GUI images are generated frame after frame by repeated invocation of GUI procedures.Ascender Technology Ltd Remote Rendering
  26. 26. Repeated Invocation of GUI Procedures  In practice, the number of unique sequences of rendering functions in execution paths taken within the code are bounded. This is because the rendering commands are generated by a fixed number of GUI functions and an application running a bounded amount of code.  The execution paths can be incrementally learned and entered into a procedure dictionary as the rendering commands are streamed.Ascender Technology Ltd Remote Rendering
  27. 27. Rendering Compression Techniques  Even if the sequences of rendering functions themselves are in the dictionary, the data arguments associated with these functions might be quite different from one another. Therefore, we keep a dictionary of the data arguments previously encountered for this particular sequence.  As a rendering procedural sequence with associated data is encountered, the data sequence dictionary for this procedural sequence is searched for the closest match to the current data sequence.Ascender Technology Ltd Remote Rendering
  28. 28. Android Contact List (Froyo)Ascender Technology Ltd Remote Rendering
  29. 29. Android ContactAscender Technology Ltd Remote Rendering
  30. 30. Structured Procedural Compression  A careful examination of the rendering stream generated by the Android GUI reveals additional structural information that can be used to improve the data model.  The rendering stream has balanced save()-restore() pairs within each frame of the rendering stream.  Each save() is found at the beginning of a GUI function and a restore() is found at the end of each GUI function.Ascender Technology Ltd Remote Rendering
  31. 31. Structured Procedural Compression  This information can be used to reverse engineer individual GUI and application procedures.  It will also reveal the call-graph of these procedures.  Using this information, the rendering code dictionary becomes a rendering procedure dictionary.  The call-graph data is best embedded in the per- procedure data dictionary.  DEMOAscender Technology Ltd Remote Rendering
  32. 32. Structured Procedural Compression  The server (encoder) constructs a “procedure dictionary” that is identical to the dictionary that is constructed by the client (decoder). Similar of LZW compression.  The procedures in this dictionary, surprisingly, are reversed engineered toolkit and application level routines.  We in effect have ended up exporting graphics at the toolkit-application level.Ascender Technology Ltd Remote Rendering
  33. 33. Structured Procedural Compression Statistics  Our compression algorithm was tested on the rendering trace of a 60 frame sequence.  There were 13702 rendering commands for an average of 228 rendering commands per frame.  Of the 13702 rendering commands, there were 2691 functions (save/restore pairs).  Of these, only 47 were unique. This gives a compression rate of 1.75% (about 1:57).Ascender Technology Ltd Remote Rendering
  34. 34. Functional Compression 47 Unique Functions From 2691 TotalAscender Technology Ltd Remote Rendering
  35. 35. Structured Procedural Compression Statistics  Of the 13702 rendering commands only 354 had completely unique data parameter sets and 203 had data sets which are partially different.  Only these 557 data sets must be transmitted, thereby giving data compression of 4.06% (about 1:25).  If the partially different data sets are differentially transmitted, a data compression of 3.3% (about 1:30) is obtained.Ascender Technology Ltd Remote Rendering
  36. 36. Functional Compression 557 Unique Data Args From 13702 TotalAscender Technology Ltd Remote Rendering
  37. 37. Structured Procedural Compression Entropy Encoding  For the last stage of compression, general techniques such as run length encoding (RLE) and entropy encoding (Huffman or Arithmetic) are used to produce a minimum bit count representation of the compressed material.  This additional phase should be expected to reduce the number of bits by a factor of 2-3Ascender Technology Ltd Remote Rendering
  38. 38. ICS GUI Rendering  Under ICS remote rendering can be done in any of three different interfaces: – Software rendering – Skia – Hardware rendering – OpenGL ES 2.0 – Abstract rendering – Canvas class  All three interfaces can use procedural compression.  The OpenGL rendering stream is much more verbose (5x) than the equivalent Skia stream.Ascender Technology Ltd Remote Rendering
  39. 39. Uses  So what is all this useful for: − Cloud computing, remote app server − App library, subscription model − App demos − Remote enterprise applications − Set-top boxesAscender Technology Ltd Remote Rendering
  40. 40. Cloud computing  Any Android app can be run on the server.  It is not necessary that the remote server and the local device have the same architecture, i.e. an ARM server can provide services for an Intel device.  For example: You might use Remote Rendering to provide Android apps to Chrome OS.Ascender Technology Ltd Remote Rendering
  41. 41. App Library / Subscription Model  Currently, apps are loaded into the local device - either installed at the time of purchase or added subsequently.  If efficient remote execution of apps can be supported, then instead of software purchases, a subscription model becomes possible.  A fixed monthly fee would entitle the subscriber to access a large library of applications.Ascender Technology Ltd Remote Rendering
  42. 42. App Demos  In a purchase/rental model, apps can be demoed remotely, prior to purchase.  If the user of the device finds the app to his/her liking, it can then be bought.Ascender Technology Ltd Remote Rendering
  43. 43. Remote Enterprise Applications  Applications may benefit from running within the enterprises data centers, with the obvious benefits of scalability, security and maintainability.  Enterprise workers typically migrates through various computing environments (mobile-fixed- mobile) during a typical day.  Having the application running on a server will allow the running applications to migrate to various clients seamlessly.Ascender Technology Ltd Remote Rendering
  44. 44. Set-top Boxes  Many set-top boxes cannot practically run Android since the set-top box might not have sufficient resources or might not contain an ARM processor.  A local client application that only performs remote rendering takes limited resources and needs no long term persistent state.  Set-top boxes typically dont have complex local installations. Installing Android apps locally would present a support challenge to the operator.Ascender Technology Ltd Remote Rendering