Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Android graphics path
in depth
The Android graphics path 1 Copyright ©2011-2014, 2net Limited
License
These slides are available under a Creative Commons Attribution-ShareAlike 3.0
license. You can read the full text...
About Chris Simmonds
• Consultant and trainer
• Working with embedded Linux since 1999
• Android since 2009
• Speaker at m...
Overview
• The Android graphics stack changed a lot in Jelly
Bean as a result of project Butter
• This presentation descri...
The big picture
SurfaceFlinger
Activity
OpenGL vendor
libraries
OpenGL ES
GPU driver
HWComposer
ION
Gralloc alloc
FB drive...
Inception of a pixel
• Everything begins when an activity draws to a surface
• 2D applications can use
• drawing functions...
2D rendering path
Activity
Canvas
Surface
SkiaOpenGL ES
Vendor Open GL
GPU driver
Activity
HWUI
The Android graphics path ...
Skia and hwui
• For 2D drawing there are two rendering paths
• hwui: (libwhui.so) hardware accelerated using
OpenGL ES 2.0...
3D rendering path
• An activity can instead create a GLSurfaceView and
use OpenGL ES bindings for Java (the
android.opengl...
3D rendering path
Activity
Surface
OpenGL ES
Vendor Open GL
GPU driver
Activity
Android GL
PixelFlinger
The Android graphi...
Composition
SurfaceFlinger
Wallpaper
Launcher
Navigation
bar
Status
bar
The Android graphics path 11 Copyright ©2011-2014,...
SurfaceFlinger
frameworks/native/services/surfaceflinger
• A high-priority native (C++) daemon, started by init
with UID=s...
SurfaceFlinger binder interfaces
SurfaceFlinger
Client
ISurfaceComposer ISurfaceComposerClient IDisplayEventConnection
e.g...
ISurfaceComposer
• ISurfaceComposer
• Clients use this interface to set up a connection with
SurfaceFlinger
• Client begin...
ISurfaceComposerClient
• ISurfaceComposerClient
• This interface has two methods:
• createSurface() asks SufraceFlinger to...
IDisplayEventConnection
• IDisplayEventConnection
• This interface passes vsync event information from
SurfaceFlinger to t...
BufferQueue
frameworks/native/include/gui/BufferQueue.h
• Mechanism for passing GraphicBuffers to
SurfaceFlinger
• Contain...
BufferQueue state diagram
FREE
QUEUED
DEQUEUEDACQUIRED
IGraphicBufferProducer::
dequeueBuffer()
IGraphicBufferConsumer::
rele...
BufferQueue
• Default number of buffer slots since JB is 3
(previously 2)
• In JB you can compile Layer.cpp with
TARGET_DI...
GraphicBuffer
frameworks/native/include/ui/GraphicBuffer.h
• Represents a buffer, wraps ANativeWindowBuffer
• Attributes i...
Composition
• On a vsync event, SurfaceFlinger calls
handleMessageRefresh() which goes through a
composition cycle:
• preC...
Layer
frameworks/native/services/surfaceflinger/Layer.h
• Each Layer has
• Z order
• Alpha value from 0 to 255
• visibleRe...
HWComposer
hardware/libhardware/include/hardware/hwcomposer.h
• HWComposer is a vendor-supplied library, at run-time
in /s...
prepare() and set()
• SurfaceFlinger calls HWComposer in two stages
• prepare()
• Passes a list of layers
• For each layer...
vsync
• Since JB 4.1 SurfaceFlinger is synchronised to a
60Hz (16.7ms period) vsync event
• If HWComposer present, it is r...
Displays
• HWComposer defines three display types
HWC_DISPLAY_PRIMARY e.g. built-in LCD screen
HWC_DISPLAY_EXTERNAL e.g. HD...
IGraphicBufferAlloc and friends
frameworks/native/include/gui/IGraphicBufferAlloc.h
• Binder interface used by SurfaceFlin...
Buffer usage and pixel format
frameworks/native/include/ui/GraphicBuffer.h
USAGE_HW_TEXTURE OpenGL ES texture
USAGE_HW_REN...
Gralloc
hardware/libhardware/include/hardware/gralloc.h
• Gralloc is a vendor-supplied library, at run-time in
/system/lib...
OpenGL ES
• The Khronos OpenGL ES and EGL APIs are
implemented in these libraries
• /system/lib/libEGL.so
• /system/lib/li...
EGL
• EGL is the Khronos Native Platform Graphics
Interface
• Rendering operations are executed in an EGLContext
• In most...
OpenGL vendor implementation
• The vendor OpenGL libraries form the interface to the
GPU
• Responsible for
• creating disp...
• Questions?
The Android graphics path 33 Copyright ©2011-2014, 2net Limited
Background: fences
The Android graphics path 34 Copyright ©2011-2014, 2net Limited
Buffer synchronisation
• There are many producers and consumers of
graphics buffers
• Pre JB sync was implicit: buffer not...
Synchronisation using fences
• Represented by file handles: can be passed between
applications in binder messages
• Can als...
Timeline and sync point
• Timeline
• Per-device (display, GPU, camera, ...)
• Monotonically increasing 32-bit value
• Incr...
Fence
• Fence
• A collection of one or more sync points, possibly from
different timelines
• Represented by a file descript...
Fence: example
FB
timeline
GPU
timeline
Sync pt
Sync pt
Sync
Fence
The Android graphics path 39 Copyright ©2011-2014, 2net...
Background: ION
The Android graphics path 40 Copyright ©2011-2014, 2net Limited
Memory constraints
• Often necessary for a buffer to be accessed by
hardware
• Example: graphics buffer and display contro...
ION
• Previous memory allocators include pmem
(Qualcomm), cmem (TI), and nvmap (NVIDA)
• ION provides a unified interface f...
Types of heap
• ION_HEAP_TYPE_SYSTEM
• memory allocated via vmalloc
• ION_HEAP_TYPE_SYSTEM_CONTIG
• memory allocated via k...
Heap flags
• ION_FLAG_CACHED
• mappings of this buffer should be cached, ION will do
cache maintenance when the buffer is m...
Upcoming SlideShare
Loading in …5
×

The Android graphics path, in depth

8,039 views

Published on

Slides from Android Builder's Summit 2014 in San Jose, CA

In this talk I describe the internal workings of the Android graphics stack from the Application layer down through the stack to pixels on the screen. It is a fairly complex journey, taking in two different 2D rendering engines, applications calling OpenGL ES directory, passing buffers on to the system compositor, SurfaceFlinger, and then down to the display controller or frame buffer.

Published in: Software, Technology, Art & Photos
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Best ever system overview for newbies thank you
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

The Android graphics path, in depth

  1. 1. The Android graphics path in depth The Android graphics path 1 Copyright ©2011-2014, 2net Limited
  2. 2. License These slides are available under a Creative Commons Attribution-ShareAlike 3.0 license. You can read the full text of the license here http://creativecommons.org/licenses/by-sa/3.0/legalcode You are free to • copy, distribute, display, and perform the work • make derivative works • make commercial use of the work Under the following conditions • Attribution: you must give the original author credit • Share Alike: if you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one (i.e. include this page exactly as it is) • For any reuse or distribution, you must make clear to others the license terms of this work The orginals are at http://2net.co.uk/slides/android-graphics-abs-2014.pdf The Android graphics path 2 Copyright ©2011-2014, 2net Limited
  3. 3. About Chris Simmonds • Consultant and trainer • Working with embedded Linux since 1999 • Android since 2009 • Speaker at many conferences and workshops "Looking after the Inner Penguin" blog at http://2net.co.uk/ https://uk.linkedin.com/in/chrisdsimmonds/ https://google.com/+chrissimmonds The Android graphics path 3 Copyright ©2011-2014, 2net Limited
  4. 4. Overview • The Android graphics stack changed a lot in Jelly Bean as a result of project Butter • This presentation describes the current (JB) graphics stack from top to bottom • Main topics covered • The application layer • SurfaceFlinger, interfaces and buffer queues • The hardware modules HWComposer and Gralloc • OpenGL ES and EGL The Android graphics path 4 Copyright ©2011-2014, 2net Limited
  5. 5. The big picture SurfaceFlinger Activity OpenGL vendor libraries OpenGL ES GPU driver HWComposer ION Gralloc alloc FB driver Gralloc FB Android framework Vendor HAL library Kernel driver Activity Activity Manager Window Manager BufferQueue The Android graphics path 5 Copyright ©2011-2014, 2net Limited
  6. 6. Inception of a pixel • Everything begins when an activity draws to a surface • 2D applications can use • drawing functions in Canvas to write to a Bitmap: android.graphics.Canvas.drawRect(), drawText(), etc • descendants of the View class to draw objects such as buttons and lists • a custom View class to implement your own appearance and behaviour • In all cases the drawing is rendered to a Surface which contains a GraphicBuffer The Android graphics path 6 Copyright ©2011-2014, 2net Limited
  7. 7. 2D rendering path Activity Canvas Surface SkiaOpenGL ES Vendor Open GL GPU driver Activity HWUI The Android graphics path 7 Copyright ©2011-2014, 2net Limited
  8. 8. Skia and hwui • For 2D drawing there are two rendering paths • hwui: (libwhui.so) hardware accelerated using OpenGL ES 2.0 • skia: (libskia.so) software render engine • hwui is the default • Hardware rendering can be disabled per view, window, activity, application or for the whole device • Maybe for comparability reasons: hwui produces results different to skia in some (rare) cases The Android graphics path 8 Copyright ©2011-2014, 2net Limited
  9. 9. 3D rendering path • An activity can instead create a GLSurfaceView and use OpenGL ES bindings for Java (the android.opengl.* classes) • Using either the vendor GPU driver (which must support OpenGL ES 2.0 and optinally 3.0) • Or as a fall-back, using PixelFlinger, a software GPU that implements OpenGL ES 1.0 only • Once again, the drawing is rendered to a Surface The Android graphics path 9 Copyright ©2011-2014, 2net Limited
  10. 10. 3D rendering path Activity Surface OpenGL ES Vendor Open GL GPU driver Activity Android GL PixelFlinger The Android graphics path 10 Copyright ©2011-2014, 2net Limited
  11. 11. Composition SurfaceFlinger Wallpaper Launcher Navigation bar Status bar The Android graphics path 11 Copyright ©2011-2014, 2net Limited
  12. 12. SurfaceFlinger frameworks/native/services/surfaceflinger • A high-priority native (C++) daemon, started by init with UID=system • Services connections from activities via Binder interface ISurfaceComposer • Receives activity status from Activity Manager • Receives window status (visibility, Z-order) from Window Manager • Composits multiple Surfaces into a single image • Passes image to one or more displays • Manages buffer allocation, synchronisation The Android graphics path 12 Copyright ©2011-2014, 2net Limited
  13. 13. SurfaceFlinger binder interfaces SurfaceFlinger Client ISurfaceComposer ISurfaceComposerClient IDisplayEventConnection e.g. activity createConnection() createDisplayEventConnection() createSurface() destroySurface() getDataChannel() The Android graphics path 13 Copyright ©2011-2014, 2net Limited
  14. 14. ISurfaceComposer • ISurfaceComposer • Clients use this interface to set up a connection with SurfaceFlinger • Client begins by calling createConnection() which spawns an ISurfaceComposerClient • Client calls createGraphicBufferAlloc() to create an instance of IGraphicBufferAlloc (discussed later) • Client calls createDisplayEventConnection() to create an instance of IDisplayEventConnection • Other methods include captureScreen() and setTransactionState() The Android graphics path 14 Copyright ©2011-2014, 2net Limited
  15. 15. ISurfaceComposerClient • ISurfaceComposerClient • This interface has two methods: • createSurface() asks SufraceFlinger to create a new Surface • destroySurface() destroys a Surface The Android graphics path 15 Copyright ©2011-2014, 2net Limited
  16. 16. IDisplayEventConnection • IDisplayEventConnection • This interface passes vsync event information from SurfaceFlinger to the client • setVsyncRate() sets the vsync event delivery rate: value of 1 returns all events, 0 returns none • requestNextVsync() schedules the next vsync event: has no effect if the vsync rate is non zero • getDataChannel() returns a BitTube which can be used to receive events The Android graphics path 16 Copyright ©2011-2014, 2net Limited
  17. 17. BufferQueue frameworks/native/include/gui/BufferQueue.h • Mechanism for passing GraphicBuffers to SurfaceFlinger • Contains an array of between 2 and 32 GraphicBuffers • Uses interface IGraphicBufferAlloc to allocate buffers (see later) • Provides two Binder interfaces • IGraphicBufferProducer for the client (Activity) • IGraphicBufferConsumer for the consumer (SurfaceFlinger) • Buffers cycle between producer and consumer The Android graphics path 17 Copyright ©2011-2014, 2net Limited
  18. 18. BufferQueue state diagram FREE QUEUED DEQUEUEDACQUIRED IGraphicBufferProducer:: dequeueBuffer() IGraphicBufferConsumer:: releaseBuffer() IGraphicBufferProducer:: queueBuffer() IGraphicBufferConsumer:: acquireBuffer() IGraphicBufferProducer:: cancelBuffer() The Android graphics path 18 Copyright ©2011-2014, 2net Limited
  19. 19. BufferQueue • Default number of buffer slots since JB is 3 (previously 2) • In JB you can compile Layer.cpp with TARGET_DISABLE_TRIPLE_BUFFERING to return to 2 slots • Call setBufferCount() to change the number of slots • BufferQueue operates in two modes: • Synchronous: client blocks until there is a free slot • Asynchronous: queueBuffer() discards any existing buffers in QUEUED state so the queue only holds the most recent frame The Android graphics path 19 Copyright ©2011-2014, 2net Limited
  20. 20. GraphicBuffer frameworks/native/include/ui/GraphicBuffer.h • Represents a buffer, wraps ANativeWindowBuffer • Attributes including width, height, format, usage inherited from ANativeWindowBuffer The Android graphics path 20 Copyright ©2011-2014, 2net Limited
  21. 21. Composition • On a vsync event, SurfaceFlinger calls handleMessageRefresh() which goes through a composition cycle: • preComposition(): sort layers by Z order and call onPreComposition() for each • doComposition(): loop through displays: if there is a dirty region, mark it to be drawn then call postFameBuffer() to do the drawing • postComposition(): loop through layers in Z order and call onPostComposition() The Android graphics path 21 Copyright ©2011-2014, 2net Limited
  22. 22. Layer frameworks/native/services/surfaceflinger/Layer.h • Each Layer has • Z order • Alpha value from 0 to 255 • visibleRegion • crop region • transformation: rotate 0, 90, 180, 270: flip H, V: scale • SurfaceFlinger composits the layers using • HWComposer, if it supports the operation • Fall back to the GPU, via OpenGL ES (version 1.0 only, for historical reasons) The Android graphics path 22 Copyright ©2011-2014, 2net Limited
  23. 23. HWComposer hardware/libhardware/include/hardware/hwcomposer.h • HWComposer is a vendor-supplied library, at run-time in /system/lib/hw/hwcomposer.[product name].so • Optional: in all cases there are fall-backs if HWC is absent • HWC does several different things • sync framework (vsync callback) • modesetting, display hotplug (e.g. hdmi) • compositing layers together using features of the display controller • displaying frames on the screen The Android graphics path 23 Copyright ©2011-2014, 2net Limited
  24. 24. prepare() and set() • SurfaceFlinger calls HWComposer in two stages • prepare() • Passes a list of layers • For each layer, HWComposer returns • HWC_FRAMEBUFFER: SurfaceFlinger should write this layer (using OpenGL) • HWC_OVERLAY: will be composed by HWComposer • set() • Passes the list of layers for HWComposer to handle • set() is used in place of eglSwapBuffers() The Android graphics path 24 Copyright ©2011-2014, 2net Limited
  25. 25. vsync • Since JB 4.1 SurfaceFlinger is synchronised to a 60Hz (16.7ms period) vsync event • If HWComposer present, it is responsible for vsync • Usually using an interrupt from the display: if no h/w trigger, fake in software • vsync() is a callback registered with HWComposer • Each callback includes a display identifier and a timestamp (in ns) • If no HWComposer, SurfaceFlinger uses 16ms timeout in s/w The Android graphics path 25 Copyright ©2011-2014, 2net Limited
  26. 26. Displays • HWComposer defines three display types HWC_DISPLAY_PRIMARY e.g. built-in LCD screen HWC_DISPLAY_EXTERNAL e.g. HDMI, WiDi HWC_DISPLAY_VIRTUAL not a real display • For each display there is an instance of DisplayDevice in SurfaceFlinger The Android graphics path 26 Copyright ©2011-2014, 2net Limited
  27. 27. IGraphicBufferAlloc and friends frameworks/native/include/gui/IGraphicBufferAlloc.h • Binder interface used by SurfaceFlinger to allocate buffers • Has one function createGraphicBuffer • Implemented by class GraphicBufferAllocator, which wraps the ANativeWindowBuffer class • Uses Gralloc.alloc to the the actual allocation • Underlying buffer is referenced by a buffer_handle_t which is a file descriptor (returned by gralloc alloc) • Binder can pass open file descriptors from process to process • Access buffer data using mmap The Android graphics path 27 Copyright ©2011-2014, 2net Limited
  28. 28. Buffer usage and pixel format frameworks/native/include/ui/GraphicBuffer.h USAGE_HW_TEXTURE OpenGL ES texture USAGE_HW_RENDER OpenGL ES render target USAGE_HW_2D 2D hardware blitter USAGE_HW_COMPOSER used by the HWComposer HAL USAGE_HW_VIDEO_ENCODER HW video encoder frameworks/native/include/ui/PixelFormat.h PIXEL_FORMAT_RGBA_8888 4x8-bit RGBA PIXEL_FORMAT_RGBX_8888 4x8-bit RGB0 PIXEL_FORMAT_RGB_888 3x8-bit RGB PIXEL_FORMAT_RGB_565 16-bit RGB PIXEL_FORMAT_BGRA_8888 4x8-bit BGRA The Android graphics path 28 Copyright ©2011-2014, 2net Limited
  29. 29. Gralloc hardware/libhardware/include/hardware/gralloc.h • Gralloc is a vendor-supplied library, at run-time in /system/lib/hw/gralloc.[product name].so • Does two things • gralloc alloc: allocates graphic buffers • gralloc framebuffer: interface to Linux framebuffer device, e.g. /dev/graphics/fb0 • gralloc alloc allocates all graphic buffers using a kernel memory manager, typically ION • Selects appropriate ION heap based on the buffer usage flags The Android graphics path 29 Copyright ©2011-2014, 2net Limited
  30. 30. OpenGL ES • The Khronos OpenGL ES and EGL APIs are implemented in these libraries • /system/lib/libEGL.so • /system/lib/libGLESv1_CM.so • /system/lib/libGLESv2.so • /system/lib/libGLESv3.so (optional from JB 4.3 onwards: actually a symlink to libGLESv2.so) • In most cases they simply call down to the vendor-supplied libraries in /system/lib/egl The Android graphics path 30 Copyright ©2011-2014, 2net Limited
  31. 31. EGL • EGL is the Khronos Native Platform Graphics Interface • Rendering operations are executed in an EGLContext • In most cases the EGLContext is based on the default display • The mapping from the EGL generic display type is done in frameworks/native/opengl/include/EGL/eglplatform.h typedef struct ANativeWindow* EGLNativeWindowType; • EGLNativeWindowType is defined in system/core/include/system/window.h The Android graphics path 31 Copyright ©2011-2014, 2net Limited
  32. 32. OpenGL vendor implementation • The vendor OpenGL libraries form the interface to the GPU • Responsible for • creating display lists • scheduling work for the GPU • managing buffer synchronisation (typically using fences, see background at the end) • Usually there is a kernel driver which handles low level memory management, DMA and interrupts • The kernel interface is usually a group of ioctl functions The Android graphics path 32 Copyright ©2011-2014, 2net Limited
  33. 33. • Questions? The Android graphics path 33 Copyright ©2011-2014, 2net Limited
  34. 34. Background: fences The Android graphics path 34 Copyright ©2011-2014, 2net Limited
  35. 35. Buffer synchronisation • There are many producers and consumers of graphics buffers • Pre JB sync was implicit: buffer not released until operation complete • Did not encourage parallel processing • JB introduced explicit sync: each buffer has a sync object called a fence • Means a buffer can be passed to the next user before operations complete • The next user waits on the fence before accessing the buffer contents The Android graphics path 35 Copyright ©2011-2014, 2net Limited
  36. 36. Synchronisation using fences • Represented by file handles: can be passed between applications in binder messages • Can also be passed from applications to drivers • Each device driver (display, camera, video codec...) has its own timeline • A fence may have synchronisation points on multiple timelines • Allows buffers to be passed between multiple devices The Android graphics path 36 Copyright ©2011-2014, 2net Limited
  37. 37. Timeline and sync point • Timeline • Per-device (display, GPU, camera, ...) • Monotonically increasing 32-bit value • Incremented after each event (essentially it is a count of the jobs processed by the device) • Sync point • A point on a timeline • Becomes signalled when the timeline passes it The Android graphics path 37 Copyright ©2011-2014, 2net Limited
  38. 38. Fence • Fence • A collection of one or more sync points, possibly from different timelines • Represented by a file descriptor so an application can wait using poll() • Two fences can be merged to create a new fence that depends on all the sync points of the original pair The Android graphics path 38 Copyright ©2011-2014, 2net Limited
  39. 39. Fence: example FB timeline GPU timeline Sync pt Sync pt Sync Fence The Android graphics path 39 Copyright ©2011-2014, 2net Limited
  40. 40. Background: ION The Android graphics path 40 Copyright ©2011-2014, 2net Limited
  41. 41. Memory constraints • Often necessary for a buffer to be accessed by hardware • Example: graphics buffer and display controller or GPU • Hardware may constrain memory access • Example: hardware without IOMMU usually needs physically contiguous memory • To avoid copying, the memory must be allocated for the most constrained device The Android graphics path 41 Copyright ©2011-2014, 2net Limited
  42. 42. ION • Previous memory allocators include pmem (Qualcomm), cmem (TI), and nvmap (NVIDA) • ION provides a unified interface for these needs • Different allocation constraints • Different caching requirements • But the programmer still has to make the right choices The Android graphics path 42 Copyright ©2011-2014, 2net Limited
  43. 43. Types of heap • ION_HEAP_TYPE_SYSTEM • memory allocated via vmalloc • ION_HEAP_TYPE_SYSTEM_CONTIG • memory allocated via kmalloc • ION_HEAP_TYPE_CARVEOUT • memory allocated from a pre reserved carveout heap • allocations are physically contiguous The Android graphics path 43 Copyright ©2011-2014, 2net Limited
  44. 44. Heap flags • ION_FLAG_CACHED • mappings of this buffer should be cached, ION will do cache maintenance when the buffer is mapped for DMA • ION_FLAG_CACHED_NEEDS_SYNC • Cache must be managed manually, e.g. using ION_IOC_SYNC The Android graphics path 44 Copyright ©2011-2014, 2net Limited

×