Outline
               Building Blocks
              Under the Hood
                     Get Dirty
                        Q&A




Introduction to Android Window System

                     Chia-I Wu
                   olv@0xlab.org



                     May 15, 2009




      Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                       Building Blocks
                      Under the Hood
                             Get Dirty
                                Q&A




Building Blocks
   Overview
   Interested Components

Under the Hood
   Random Topics

Get Dirty
   Development
   Code

Q&A



              Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Overview
                         Under the Hood
                                            Interested Components
                                Get Dirty
                                   Q&A


Outline

   Building Blocks
      Overview
      Interested Components

   Under the Hood
      Random Topics

   Get Dirty
      Development
      Code

   Q&A


                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                       Building Blocks
                                         Overview
                      Under the Hood
                                         Interested Components
                             Get Dirty
                                Q&A


System Architecture




              Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                       Building Blocks
                                         Overview
                      Under the Hood
                                         Interested Components
                             Get Dirty
                                Q&A


System Architecture




              Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                       Building Blocks
                                         Overview
                      Under the Hood
                                         Interested Components
                             Get Dirty
                                Q&A


System Architecture




              Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                           Building Blocks
                                             Overview
                          Under the Hood
                                             Interested Components
                                 Get Dirty
                                    Q&A


Building Blocks



   There are more, but we focus on
       SurfaceManager
       WindowManager
       ActivityManager




                  Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Overview
                         Under the Hood
                                            Interested Components
                                Get Dirty
                                   Q&A


SurfaceManager




      frameworks/base/libs/surfaceflinger/
      a.k.a SurfaceFlinger
      Allocate surfaces. Backed by ashmem/pmem/?
      Composite surfaces




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                         Building Blocks
                                           Overview
                        Under the Hood
                                           Interested Components
                               Get Dirty
                                  Q&A


WindowManager


     frameworks/base/services/java/
     com/android/server/WindowManagerService.java
     About 9000 SLOC in one file. Poorly documented, bad
     namings, ...
     (Ask SurfaceManager to) create/layout surfaces on behalf of
     the clients
     Dispatch input events to clients
     Transition animation
     WindowManagerPolicy


                Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Overview
                         Under the Hood
                                            Interested Components
                                Get Dirty
                                   Q&A


ActivityManager



      frameworks/base/services/java/
      com/android/server/am/
      Manage lifecycles of activities
      Manage stacking of activities
      Dispatch intents
      Spawn processes




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Overview
                         Under the Hood
                                            Interested Components
                                Get Dirty
                                   Q&A


Confusions



      An activity has one or more windows (e.g. dialogs)
      A window has one or more surfaces (e.g. surface views)
      However, in window manager, a window is called a session
      A surface is called a window
      And an activity becomes roughly a token




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Overview
                         Under the Hood
                                            Interested Components
                                Get Dirty
                                   Q&A


Special Keys




      HOME key
      BACK key




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                         Under the Hood     Random Topics
                                Get Dirty
                                   Q&A


Outline

   Building Blocks
      Overview
      Interested Components

   Under the Hood
      Random Topics

   Get Dirty
      Development
      Code

   Q&A


                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                         Under the Hood     Random Topics
                                Get Dirty
                                   Q&A


Process View




      SurfaceManager, WindowManager, and SurfaceManager are
      threads of a single process (system server)
      Every application is usually a process of itself




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                             Building Blocks
                            Under the Hood     Random Topics
                                   Get Dirty
                                      Q&A


Zygote




         Is a process started on system initialization
         Preloads java classes and resources
         Forks system server
         Listens silently on /dev/socket/zygote




                    Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                             Building Blocks
                            Under the Hood     Random Topics
                                   Get Dirty
                                      Q&A


Binder




         Early in the lifetime of an application process, thread(s) are
         created and blocked on /dev/binder
         Binder is used mainly for RPC
         Fragile




                    Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Outline

   Building Blocks
      Overview
      Interested Components

   Under the Hood
      Random Topics

   Get Dirty
      Development
      Code

   Q&A


                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                         Building Blocks
                                           Development
                        Under the Hood
                                           Code
                               Get Dirty
                                  Q&A


Build System




      build/core/core/build-system.html
      .   build/envsetup.sh
      showcommands
      export ANDROID JAVA HOME if non-standard




                Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                             Building Blocks
                                               Development
                            Under the Hood
                                               Code
                                   Get Dirty
                                      Q&A


adb



      ADBHOST for transport over TCP/IP
      kill-server
      remount
      pull/push
      logcat
      shell




                    Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                           Building Blocks
                                             Development
                          Under the Hood
                                             Code
                                 Get Dirty
                                    Q&A


hierarchyviewer




       Display view hierarchy
       Display view
       Invalidate/Relayout




                  Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                         Building Blocks
                                           Development
                        Under the Hood
                                           Code
                               Get Dirty
                                  Q&A


Graphics: Memory Management




     SurfaceFlinger has a SurfaceHeapManager
     Every client has a MemoryDealer, as returned by
     SurfaceHeapManager
     Every surface of a client also has dealer(s), from client or GPU




                Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.




      A dealer consists of a heap and an allocator
      A heap represents a sharable big chunk of memory
      An allocator is an algorithm
      Small chunks of memory from the heap are returned




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface
      createSurface calls createNormalSurfaceLocked




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface
      createSurface calls createNormalSurfaceLocked
      A layer is created and setBuffers is called to allocate buffers




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface
      createSurface calls createNormalSurfaceLocked
      A layer is created and setBuffers is called to allocate buffers
      Two dealers are created from client, one for front buffer and
      one for back buffer




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface
      createSurface calls createNormalSurfaceLocked
      A layer is created and setBuffers is called to allocate buffers
      Two dealers are created from client, one for front buffer and
      one for back buffer
      Two LayerBitmaps are created, initialized with the two dealers




                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                                            Development
                         Under the Hood
                                            Code
                                Get Dirty
                                   Q&A


Graphics: Memory Management cont.


  Real flow
      A client asks for a new surface, createSurface
      createSurface calls createNormalSurfaceLocked
      A layer is created and setBuffers is called to allocate buffers
      Two dealers are created from client, one for front buffer and
      one for back buffer
      Two LayerBitmaps are created, initialized with the two dealers
      Heaps of dealers along with info about the layer are returned



                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                         Building Blocks
                                           Development
                        Under the Hood
                                           Code
                               Get Dirty
                                  Q&A


Hello World




      http://people.debian.org.tw/˜olv/surfaceflinger/demo.tar.gz




                Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                        Building Blocks
                                          Development
                       Under the Hood
                                          Code
                              Get Dirty
                                 Q&A


Many Buffers




     Surface is double buffered
     EGLDisplaySurface is double buffered
     Same technique; Different code pathes, different purposes




               Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                         Building Blocks
                                           Development
                        Under the Hood
                                           Code
                               Get Dirty
                                  Q&A


Double Buffering




      Sofware v.s. Hardware
      Memory copy
      Page flipping




                Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                           Building Blocks
                                             Development
                          Under the Hood
                                             Code
                                 Get Dirty
                                    Q&A


Dirty Region




      Associate buffers with dirty regions
      Copy back




                  Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                   Building Blocks
                                     Development
                  Under the Hood
                                     Code
                         Get Dirty
                            Q&A


Frame 0




          Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                   Building Blocks
                                     Development
                  Under the Hood
                                     Code
                         Get Dirty
                            Q&A


Frame 1




          Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                   Building Blocks
                                     Development
                  Under the Hood
                                     Code
                         Get Dirty
                            Q&A


Frame 2




          Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                          Building Blocks
                         Under the Hood
                                Get Dirty
                                   Q&A


Outline

   Building Blocks
      Overview
      Interested Components

   Under the Hood
      Random Topics

   Get Dirty
      Development
      Code

   Q&A


                 Chia-I Wu olv@0xlab.org    Introduction to Android Window System
Outline
                            Building Blocks
                           Under the Hood
                                  Get Dirty
                                     Q&A


Q&A




      Questions?




                   Chia-I Wu olv@0xlab.org    Introduction to Android Window System

Introduction to Android Window System

  • 1.
    Outline Building Blocks Under the Hood Get Dirty Q&A Introduction to Android Window System Chia-I Wu olv@0xlab.org May 15, 2009 Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 2.
    Outline Building Blocks Under the Hood Get Dirty Q&A Building Blocks Overview Interested Components Under the Hood Random Topics Get Dirty Development Code Q&A Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 3.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A Outline Building Blocks Overview Interested Components Under the Hood Random Topics Get Dirty Development Code Q&A Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 4.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A System Architecture Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 5.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A System Architecture Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 6.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A System Architecture Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 7.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A Building Blocks There are more, but we focus on SurfaceManager WindowManager ActivityManager Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 8.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A SurfaceManager frameworks/base/libs/surfaceflinger/ a.k.a SurfaceFlinger Allocate surfaces. Backed by ashmem/pmem/? Composite surfaces Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 9.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A WindowManager frameworks/base/services/java/ com/android/server/WindowManagerService.java About 9000 SLOC in one file. Poorly documented, bad namings, ... (Ask SurfaceManager to) create/layout surfaces on behalf of the clients Dispatch input events to clients Transition animation WindowManagerPolicy Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 10.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A ActivityManager frameworks/base/services/java/ com/android/server/am/ Manage lifecycles of activities Manage stacking of activities Dispatch intents Spawn processes Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 11.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A Confusions An activity has one or more windows (e.g. dialogs) A window has one or more surfaces (e.g. surface views) However, in window manager, a window is called a session A surface is called a window And an activity becomes roughly a token Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 12.
    Outline Building Blocks Overview Under the Hood Interested Components Get Dirty Q&A Special Keys HOME key BACK key Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 13.
    Outline Building Blocks Under the Hood Random Topics Get Dirty Q&A Outline Building Blocks Overview Interested Components Under the Hood Random Topics Get Dirty Development Code Q&A Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 14.
    Outline Building Blocks Under the Hood Random Topics Get Dirty Q&A Process View SurfaceManager, WindowManager, and SurfaceManager are threads of a single process (system server) Every application is usually a process of itself Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 15.
    Outline Building Blocks Under the Hood Random Topics Get Dirty Q&A Zygote Is a process started on system initialization Preloads java classes and resources Forks system server Listens silently on /dev/socket/zygote Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 16.
    Outline Building Blocks Under the Hood Random Topics Get Dirty Q&A Binder Early in the lifetime of an application process, thread(s) are created and blocked on /dev/binder Binder is used mainly for RPC Fragile Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 17.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Outline Building Blocks Overview Interested Components Under the Hood Random Topics Get Dirty Development Code Q&A Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 18.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Build System build/core/core/build-system.html . build/envsetup.sh showcommands export ANDROID JAVA HOME if non-standard Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 19.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A adb ADBHOST for transport over TCP/IP kill-server remount pull/push logcat shell Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 20.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A hierarchyviewer Display view hierarchy Display view Invalidate/Relayout Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 21.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management SurfaceFlinger has a SurfaceHeapManager Every client has a MemoryDealer, as returned by SurfaceHeapManager Every surface of a client also has dealer(s), from client or GPU Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 22.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. A dealer consists of a heap and an allocator A heap represents a sharable big chunk of memory An allocator is an algorithm Small chunks of memory from the heap are returned Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 23.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 24.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface createSurface calls createNormalSurfaceLocked Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 25.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface createSurface calls createNormalSurfaceLocked A layer is created and setBuffers is called to allocate buffers Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 26.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface createSurface calls createNormalSurfaceLocked A layer is created and setBuffers is called to allocate buffers Two dealers are created from client, one for front buffer and one for back buffer Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 27.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface createSurface calls createNormalSurfaceLocked A layer is created and setBuffers is called to allocate buffers Two dealers are created from client, one for front buffer and one for back buffer Two LayerBitmaps are created, initialized with the two dealers Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 28.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Graphics: Memory Management cont. Real flow A client asks for a new surface, createSurface createSurface calls createNormalSurfaceLocked A layer is created and setBuffers is called to allocate buffers Two dealers are created from client, one for front buffer and one for back buffer Two LayerBitmaps are created, initialized with the two dealers Heaps of dealers along with info about the layer are returned Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 29.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Hello World http://people.debian.org.tw/˜olv/surfaceflinger/demo.tar.gz Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 30.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Many Buffers Surface is double buffered EGLDisplaySurface is double buffered Same technique; Different code pathes, different purposes Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 31.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Double Buffering Sofware v.s. Hardware Memory copy Page flipping Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 32.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Dirty Region Associate buffers with dirty regions Copy back Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 33.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Frame 0 Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 34.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Frame 1 Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 35.
    Outline Building Blocks Development Under the Hood Code Get Dirty Q&A Frame 2 Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 36.
    Outline Building Blocks Under the Hood Get Dirty Q&A Outline Building Blocks Overview Interested Components Under the Hood Random Topics Get Dirty Development Code Q&A Chia-I Wu olv@0xlab.org Introduction to Android Window System
  • 37.
    Outline Building Blocks Under the Hood Get Dirty Q&A Q&A Questions? Chia-I Wu olv@0xlab.org Introduction to Android Window System