Accelerating the Android Platform Build
David Rosen, Product Manager
drosen@electric-cloud.com
@adr0sen
OnAndroidConf 2013
What’s this Android buzz all about?

2
Qualcomm Reference Design Program
<60 days TTM, and pushing for faster

Source: http://www.qualcomm.com/connect/analyst-re...
Android Development Landscape
Real-world challenges at large APAC mobile device maker
Electric Cloud Android/Embedded Survey (Q1 2013)

Users of Android by Vertical

• Survey
• Focused on Android
development ...
Android Delivery Process Today
Technical Challenges

Negative Business Impacts

• Slow Android Builds
• Slow, Complex CTS
...
Accelerating the
Android build
Why look at Android builds?

• Android is everywhere
• 68% of global mobile phone market share1
• Explosion of non-phone a...
What is (and is not) covered in this part of the talk?

• Build acceleration techniques fall into two broad categories:
• ...
AOSP LOC Evolution
Jan 2009 – Jan 20131
18,000,000

16,000,000

14,000,000

12,000,000

10,000,000
Linux Kernel
8,000,000
...
AOSP Build Evolution
Release Date

LOC1

LOC
Growth %

GNU Make
Build Time2

Build Time
Growth %

2.2

May 2010

8,837,858...
Common wisdom about Android builds

• I can just…
• … add more cores
• … use distcc
• … use ccache

• “The Android build i...
The common wisdom is wrong.
Sidebar: how much faster is “interesting”?

• Some large mobile device makers run > 50K builds/week
• At 15 minutes per bu...
How fast is GNU make?

• Best time: ~15m
• 48-cores
• 128GB RAM
Problem: Visibility

• Knowing the end-to-end time is great but not actionable
• We need visibility
• Where is the time go...
Solution: ElectricInsight
What can we see from here?

• An ideal build would look like a tightly packed box
• Overall looks almost perfect – well do...
Longest serial chain
Best possible is about 15m

• Best possible runtime is about 15m
Projected runtime with more cores
What else can we learn?
• Lots of non-compile work in the build - ~25% of total time
Problem: long parse time

• Why do we build?
• To transform sources into deliverables (programs, docs, etc).

• Does parsi...
Solution: parse avoidance

• What if we don’t parse (every time)?
• Makefiles don’t change very often, so why reparse ever...
Parse avoidance impact
Build time reduced to about 13m30s
Problem: serializations

• Gaps in visualization suggest serializations
• Q: How many dependencies are there in the
Androi...
Android Dependency Graph

• Dump all makefile rules
• 100,000 files/targets
• 1,990,628 dependencies
Dependencies in Android

• ~19 dependencies per file: why so many?
• Consider a typical build structure:
lib.a: foo.o bar....
Superfluous dependencies in Android

• Do we really need all 1.9M dependencies?
• The filesystem can tell us!
• Collect a ...
Actual dependencies in Android

• 100,000 files/targets
• 288,804 dependencies
Specified vs. Actual dependencies in Android

Specified

Actual
Solution: dependency optimization

• Impossible to manually eliminate superfluous
dependencies
• Electric Make can prune t...
Dependency optimization impact
Build time reduced to about 12m30s
Problem: long serial jobs

• Long serial jobs prohibits further parallelization
• Dominates longest serial chain:

• In An...
Solution: Javadoc Caching

• Javadoc generation can be cached
• Stash output after running javadoc
• Reuse output in futur...
Javadoc Caching impact
Build time reduced to about 9m30s
Summary
• Google has done a great job of optimizing Android builds
• But there’s still room for improvement!
• By caching ...
Questions?
David Rosen, Product Manager
drosen@electric-cloud.com
@adr0sen
http://www.electric-cloud.com/eade
Upcoming SlideShare
Loading in...5
×

OnAndroidConf 2013: Accelerating the Android Platform Build

2,226

Published on

Presented at the OnAndroidConf, October 22 2013, http://www.onandroidconf.com/sessions.html

Abstract:
Optimizing the Android build environment to perform at world-class level is a big challenge for many Android device and chipset makers today. Churning through thousands of platform builds per week requires laser-focus on high-performance infrastructure and tooling. If you’re looking at improving your overall engineering and developer productivity, the software build use case is an obvious area to prioritize.

This technical talk will focus on the following aspects of the Android platform build:

Common Android platform build challenges and opportunities with real-life production references
The various Android build use cases and their needs – full integration and release builds, developer incremental builds
Evolution of the Android build and codebase with trends and statistics
Detailed technical analysis of the Android platform build, highlighting opportunities for improvements
Proposed solutions and technical tricks to optimize an Android software build environment

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,226
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Point out that axis is time in minutes, rows are cores/threads, boxes are individual targets
  • OnAndroidConf 2013: Accelerating the Android Platform Build

    1. 1. Accelerating the Android Platform Build David Rosen, Product Manager drosen@electric-cloud.com @adr0sen OnAndroidConf 2013
    2. 2. What’s this Android buzz all about? 2
    3. 3. Qualcomm Reference Design Program <60 days TTM, and pushing for faster Source: http://www.qualcomm.com/connect/analyst-relations/events/americas-analyst-summit-2012
    4. 4. Android Development Landscape Real-world challenges at large APAC mobile device maker
    5. 5. Electric Cloud Android/Embedded Survey (Q1 2013) Users of Android by Vertical • Survey • Focused on Android development & delivery Military/aero space 3% Other 25% • Global embedded contacts in Electric Cloud marketing lists • 350+ respondents • 175+ Android Automotive 6% Mobile 38% Medical 4% Industrial control and automation 6% Consumer electronics 18%
    6. 6. Android Delivery Process Today Technical Challenges Negative Business Impacts • Slow Android Builds • Slow, Complex CTS Testing • Complex B-T-R Matrix • Custom Testing – Volume, Complexity • Slow Time to Market • Poor Quality of products • Poor productivity of teams Avg. release build time More than 4 hours, 15.3 % 1-4 hours, 27.0 % Avg. time to execute CTS Less than 5 minutes, 7.8 % 5-20 minutes, 21. 0% 20 minutes 1 hour, 28.8% 4-8 hours, 15.0 % More than 8 hours, 5.0% Less than 1 hour, 22.5% 1-4 hours, 57.5 %
    7. 7. Accelerating the Android build
    8. 8. Why look at Android builds? • Android is everywhere • 68% of global mobile phone market share1 • Explosion of non-phone applications • • • • Automotive (Saab, Renault) Entertainment (Ouya, Nvidia) Exercise equipment (NordicTrack) Rice cookers! (Panasonic) • Android is everywhere • … and that means thousands of developers building Android • What if we can make those builds faster? • How would your development process change? • How much more could you achieve? 1 Source: Gartner, February 2013
    9. 9. What is (and is not) covered in this part of the talk? • Build acceleration techniques fall into two broad categories: • Hardware improvements like faster CPU’s and better disks • Software improvements like smarter build tools and faster compilers • In general, hardware and software improvements are complimentary • If you want the fastest possible builds, leverage both! • This talk is about software techniques for build acceleration • Given a fixed hardware platform, how fast can we build Android? • Or, what do you do when you reach the limits of your hardware?
    10. 10. AOSP LOC Evolution Jan 2009 – Jan 20131 18,000,000 16,000,000 14,000,000 12,000,000 10,000,000 Linux Kernel 8,000,000 Android Mozilla Firefox 6,000,000 4,000,000 2,000,000 0 1 Source: http://www.ohloh.net, http://en.wikipedia.org/wiki/Android_version_history (February 2013) Chromium Webkit
    11. 11. AOSP Build Evolution Release Date LOC1 LOC Growth % GNU Make Build Time2 Build Time Growth % 2.2 May 2010 8,837,858 - 28m55s - 2.3.4 April 2011 11,492,324 30% 33m10s 15% 4.0.1 October 2011 12,827,330 12% 1h13m54s 123% 4.1.1 July 2012 15,028,331 17% 1h28m11s 19% 4.2.2 February 2013 15,266,8033 2% 1h32m56s 2% 01:40 +19% +2% 18 16 01:26 +123% +30% 01:12 +17% +2% 14 12 +12% 00:57 10 00:43 8 6 00:28 +15% 4 00:14 2 00:00 0 2.2 1 Millions Android Version 2.3.4 4.0.1 4.1.1 4.2.2 http://www.ohloh.net, 2 Builds performed on 8-core 16GB RAM server, 3 LOC data as of December 2012 Build Time LOC
    12. 12. Common wisdom about Android builds • I can just… • … add more cores • … use distcc • … use ccache • “The Android build is as optimized as it can be!”
    13. 13. The common wisdom is wrong.
    14. 14. Sidebar: how much faster is “interesting”? • Some large mobile device makers run > 50K builds/week • At 15 minutes per build that’s 12,500 hours of build time • A reduction of just one minute would save 800 hours every week • What about wait time? • Faster builds = more iterations • Faster builds = higher quality • Faster builds = lower time to market
    15. 15. How fast is GNU make? • Best time: ~15m • 48-cores • 128GB RAM
    16. 16. Problem: Visibility • Knowing the end-to-end time is great but not actionable • We need visibility • Where is the time going? • What are the bottlenecks? • Are there opportunities for improvement? • How do we get that visibility? • Instrument the build… somehow! strace, echo, hack gmake, or… • ElectricInsight • Visualization and analysis of builds run with ElectricMake • Let’s pump this build through emake and see how it looks!
    17. 17. Solution: ElectricInsight
    18. 18. What can we see from here? • An ideal build would look like a tightly packed box • Overall looks almost perfect – well done, Android team! • But! a couple of things stand out: • Crazy long parse time1 • Gaps at the end of the build, indicative of serializations • Some very long serial chunks at the bottom of graph • We’ll investigate these issues, but first: what if we just use more cores? 1 emake parsing may be slower than gmake parsing
    19. 19. Longest serial chain Best possible is about 15m • Best possible runtime is about 15m
    20. 20. Projected runtime with more cores
    21. 21. What else can we learn? • Lots of non-compile work in the build - ~25% of total time
    22. 22. Problem: long parse time • Why do we build? • To transform sources into deliverables (programs, docs, etc). • Does parsing makefiles transform sources into deliverables? • Nope. • Parsing makefiles is pure overhead • But you have to tell make what to do somehow • Unless you want to manually invoke the compiler, linker, etc.
    23. 23. Solution: parse avoidance • What if we don’t parse (every time)? • Makefiles don’t change very often, so why reparse every time? • Reuse parse results from a previous build, as long as… • Makefiles are unchanged (MD5) • Command-line is unchanged • Environment is unchanged • How do we do it? • Electric Make already has parse results in a reloadable form, just need to add cache management • GNU make doesn’t, but could be modified
    24. 24. Parse avoidance impact Build time reduced to about 13m30s
    25. 25. Problem: serializations • Gaps in visualization suggest serializations • Q: How many dependencies are there in the Android build? • A: More than you think!
    26. 26. Android Dependency Graph • Dump all makefile rules • 100,000 files/targets • 1,990,628 dependencies
    27. 27. Dependencies in Android • ~19 dependencies per file: why so many? • Consider a typical build structure: lib.a: foo.o bar.o foo.o: foo.c foo.h util.h bar.o: bar.c bar.h util.h • Some files will have many dependencies • Most have only a few • What is going on in Android?
    28. 28. Superfluous dependencies in Android • Do we really need all 1.9M dependencies? • The filesystem can tell us! • Collect a list of files actually used to generate a target • Compare to the list of prerequisites specified in the makefile • Example: foo.txt: echo “Hello” > foo.txt bar.txt: foo.txt echo “World” > bar.txt • Why not run foo.txt and bar.txt in parallel?
    29. 29. Actual dependencies in Android • 100,000 files/targets • 288,804 dependencies
    30. 30. Specified vs. Actual dependencies in Android Specified Actual
    31. 31. Solution: dependency optimization • Impossible to manually eliminate superfluous dependencies • Electric Make can prune them automatically • If you use gmake, you can achieve the same effect: • Remove existing dependency specifications • Generate minimal dependency specifications from emake data • Before: $(OBJECTS): %.o: %.cpp $(ALL_HEADERS) $(OTHER_DEPS) $(COMPILE.CC) • After: $(OBJECTS): %.o: %.cpp $(COMPILE.CC) foo.o: foo.h util.h bar.o: bar.h util.h generated.h
    32. 32. Dependency optimization impact Build time reduced to about 12m30s
    33. 33. Problem: long serial jobs • Long serial jobs prohibits further parallelization • Dominates longest serial chain: • In Android, Javadoc generation is a performance albatross • Typically a long, monolithic job • Not decomposable/parallelizable, not incremental
    34. 34. Solution: Javadoc Caching • Javadoc generation can be cached • Stash output after running javadoc • Reuse output in future builds if inputs match • Like ccache, but for javadoc Carving 4 minutes away from job!
    35. 35. Javadoc Caching impact Build time reduced to about 9m30s
    36. 36. Summary • Google has done a great job of optimizing Android builds • But there’s still room for improvement! • By caching parse results and javadoc output while pruning dependencies, we have shown to carve off 5m+ or ~37% from build 21:36 19:30 17:45 19:12 16:48 15:00 14:24 15:00 12:00 11:55 Gmake 09:36 Emake 09:30 07:12 04:48 02:24 00:00 24 32 48
    37. 37. Questions? David Rosen, Product Manager drosen@electric-cloud.com @adr0sen http://www.electric-cloud.com/eade
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×