LCA13: Android Infrastructure Automation Improvements


Published on

Resource: LCA13
Name: Android Infrastructure Automation Improvements
Date: 05-03-2013
Speaker: Tyler Baker

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

LCA13: Android Infrastructure Automation Improvements

  1. 1. ASIA 2013 (LCA13) Tyler Baker – March 2013 LAVA Workshop @ LCA13 A LAVA Success Story
  2. 2. ASIA 2013 (LCA13) Introduction Tyler Baker – Former Android Platform Engineer at Intermec Started working with Android early 2008 Board bring ups on custom hardware designs Boot loader porting Development of BSP and device drivers 1D/2D Imaging solutions System services and application development RIL integration and debugging System administrator for Gerrit, Jenkins, and LAVA Android system internals course instructor at Marakana Studied Computer Science and Applied Mathematics at Western Washington University In my spare time I enjoy designing autonomous aerial drones, snowboarding, and fishing. Located in Bellingham, WA
  3. 3. ASIA 2013 (LCA13) Perspective Working for an OEM designing and developing enterprise products has given me unique perspective The intent of this presentation is to highlight the success I have had utilizing LAVA Takeaway Points Continuous Integration Philosophy What are the pain points associated with existing test frameworks Why LAVA was adopted How LAVA can be easily extended How LAVA can be integrated into CI environments Why everyone here should consider LAVA
  4. 4. ASIA 2013 (LCA13) Continuous Integration Philosophy Imagine CI as the LIM(x->0) of 1/x There is a vertical asymptote at x = 0 The function increases without bound Think of true CI being the asymptote Select a number close to zero, and I will select one that is closer
  5. 5. ASIA 2013 (LCA13) Continuous Integration Philosophy Continued Achieving true CI is very difficult and some say impossible There is always something else that can be automated CI practices and tools are constantly evolving New hardware platforms constantly drive CI software changes Early stages of hardware and software development can be unstable Boot loader CI? What is the return on investment? In my opinion, the best approach to CI is to evolve your strategy as the technologies changes
  6. 6. ASIA 2013 (LCA13) Pain Points Seemingly many companies build their own automated test framework and would like you to purchase it Intellectual property Licensing Closed Source Paid support Costly integration efforts Technical debt incurred to modify the system Integration with corporate IT infrastructure I.E (LDAP) Scalability? Extensibility? Multi platform? Reporting? Dashboards? KPI? There is no perfect solution yet
  7. 7. ASIA 2013 (LCA13) Linaro Automated Validation Architecture I was first introduced to LAVA at ELC 2011 Began to add support for our rugged Android OMAP3 based handset shortly after Why LAVA was adopted Flexible architecture Compartmentalized Easy to extend to meet the any need Automated recovery of bricked devices Well supported by LAVA team Tests already integrated Open Source LAVA itself is an enabler Contributing changes upstream reduces technical debt
  8. 8. ASIA 2013 (LCA13) LAVA Extensions CLI tools are wrapped, exposed as actions to the dispatcher Very simple to extend the existing software once you understand entry points Developers who write services or applications should also be able to take advantage of LAVA The issues that needed to be solved Facilitate downloading, installing and uninstalling applications/system services Download and deploy files to the file system Run commands on the client side over ADB Run unit or functional tests and get the the results
  9. 9. ASIA 2013 (LCA13) LAVA Extensions Continued Extended lava-android-test Download and push files to a specific location on the file system Install, uninstall, and reinstall applications Run client side shell commands insmod, chmod, reboot Download, install, and run any Gtest or Junit binary with correct parse Can be referenced in a job file Was able to successfully test Android OTA updates
  10. 10. ASIA 2013 (LCA13) LAVA and LDAP To manage existing user accounts an extension was needed to allow LDAP authentication Gerrit and Jenkins both support LDAP LAVA server extension written in a single day Used the OpenID authentication implementation as reference Straight forward LAVA can adapt to almost any need
  11. 11. ASIA 2013 (LCA13) LAVA and Jenkins Integration Having Jenkins submit a job to LAVA after a build is trivial It would be nice to have a plug-in, but not necessary LAVA CLI tools work just fine The issues When you submit a job in LAVA it is asynchronous operation LAVA dashboard holds the test results The developer must seek out the test results for his/her build This violates the “rapid feedback loop” CI principle How will LAVA be instructed to download the correct binaries from Jenkins in a dynamic fashion when the job file is static?
  12. 12. ASIA 2013 (LCA13) LAVA and Jenkins Integration Continued Extended lava-tool Created a function called “testjob-run” It schedules the job It blocks like a synchronous operation It monitors the job progress When the job completes, the results are fed to STDOUT There is also a option to write the results out into xUnit format Now the results can be visualized right from Jenkins! Not only just graphs, but the same data you might find in a results bundle This data can be pushed into an existing test results database
  13. 13. ASIA 2013 (LCA13) LAVA and Jenkins Integration Continued
  14. 14. ASIA 2013 (LCA13) LAVA and Jenkins Integration Continued
  15. 15. ASIA 2013 (LCA13) LAVA and Jenkins Integration Continued How are the job files being dynamically created? The job files used are “templates” sed is used to replace “variables” in the LAVA job template This allows the dynamic creation of Jenkins links in the job template submitted to LAVA A bit of a hack, but so far been very robust When the upstream build finishes, the modified job template(s) are downloaded to the downstream LAVA test job These modified job templates are scheduled and results pushed up into Gerrit
  16. 16. ASIA 2013 (LCA13) LAVA and Gerrit Integration Jenkins can listen for Gerrit events to trigger builds Once triggered Jenkins will download the patch set, applied it to the source tree and build This will allow a developer build and test their code before ever merging Since the test results are now being reported to Jenkins It is possible to use the Gerrit SSH API to “code review” the change set that got built and tested A thin bash wrapper parses the STDOUT from the lava-tool extension The failed results get put into the code review message and -1 is applied to the change set If all the results pass a +1 is applied to the change set If the is job status is returned as “incomplete” then 0 is applied to the change set In every case a link to the LAVA test bundle is included in the message The feedback from the build and the test(s) will be reported all in one place This is what CI is all about, giving the developer timely feedback about his/her changes
  17. 17. ASIA 2013 (LCA13) LAVA and Gerrit Integration Continued
  18. 18. ASIA 2013 (LCA13) LAVA and Gerrit Integration Continued
  19. 19. ASIA 2013 (LCA13) CI Architecture Overview
  20. 20. ASIA 2013 (LCA13) LAVA Canary Testing Build machines can build boot loaders, kernel, modules, and Android file system in less than two minutes Change set applied Dirty build, no CCACHE 18 minutes to build all components from a clean 4.2.1 tree Quad Xeon processors with 48GB of RAM per processor Building on a ramdisk All units can recover from a bad kernel or file system Once builds are smoked tested on the canary units, they are pushed out to larger populations as OTA updates Commit, build, and receipt of test results takes just under 8 minutes This includes full OS build, downloading the build to LAVA server, flashing full OS using fastboot, booting, running a quick smoke testing which includes monkey
  21. 21. ASIA 2013 (LCA13) LAVA Canary Testing
  22. 22. ASIA 2013 (LCA13) CI Build Machines
  23. 23. ASIA 2013 (LCA13) LAVA Multi Platform Support Built a extension in a matter of days to support the Windows Mobile platform Flashing of the OS Running tests Reporting results IOS support is being implemented
  24. 24. ASIA 2013 (LCA13) Conclusion Planning to upstream all the extensions mentioned in this presentation for the benefit of the community LAVA may not do everything you want but can be easily extended LAVA is easy to use LAVA has a great team of developers LAVA can be tightly integrated into a CI environment I would personally like to see LAVA emerge as the gold standard for automated testing frameworks
  25. 25. ASIA 2013 (LCA13) Thank you! You can reach me at or on IRC as tyler-baker
  26. 26. LAVA Hands On – Wednesday 4pm - 6pm • Requirements • Laptop At least 4GB Ram Virtualisation support (i.e. Intel i3-i7) • Software VirtualBox 4.1 or later (Linux, Windows and OS X supported LAVA VM Image • Limited space, but some spaces available • E-mail and turn up
  27. 27. More about Linaro Connect: More about Linaro: More about Linaro engineering: ASIA 2013 (LCA13)