Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins


Published on

By Sarah Woodall, NXP Semiconductors

LPCXpresso is a multi-platform IDE for developers of embedded software to run on NXP Semiconductor's ARM-based microcontrollers. NXP needs to test that the debugger can execute programs on numerous different development boards that connect to the USB ports of host computers. Besides building a complex software product, the Jenkins installation drives an automated test farm consisting of home-built software-controlled USB switches ("cows") that control a huge array of combinations of test board, debug probe and host platform. This talk will give a tour of the NXP farm, including video of the cows in action, and will describe the features of Jenkins that are used to make it work, with particular emphasis on dynamic selection of combinations within matrix jobs, parameterized triggers and the Summary Display plugin. Finally, plans to migrate to the new Workflow plugin will be discussed. NXP believes the Workflow plugin will simplify the structure and make it more maintainable.

Published in: Technology
  • Be the first to comment

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins

  1. 1. The Famous Cows of Cambridge A non-standard use-case for Jenkins Sarah Woodall, NXP Semiconductors
  2. 2. #jenkinsconf Footer What this talk is about •  How (and why) we use Jenkins to test our software automatically using a farm of custom hardware •  The plugins and features of Jenkins that help •  Improvements we would love to see •  Why we think the Workflow Plugin will be important 2
  3. 3. #jenkinsconf Setting the scene
  4. 4. #jenkinsconf Some of the world’s most photographed cows 4 Photo by Alex Brown (originally posted to Flickr as Cows and King's) [CC BY 2.0 (http:// licenses/by/2.0)], via Wikimedia Commons
  5. 5. #jenkinsconf The product that our Jenkins builds and tests A software development toolchain based on Eclipse and GCC/GDB •  Designed for ease of use with NXP’s ARM-based LPC microcontrollers •  Hosted on Windows, Mac, Linux (separate installers)
  6. 6. #jenkinsconf LPCXpresso in normal use 6 Host computer running LPCXpresso Debug probe USB Development board Target MCU Stub
  7. 7. #jenkinsconf LPCXpresso’s components Open-source IDE built and tested by the Eclipse Foundation Public releases three times a year GNU tools (GCC compiler, GDB debugger, ...) built and tested by ARM Public releases four times a year Project wizards; Debug perspective; MCU definitions; support for advanced features such as Trace, etc ... Written locally: our responsibility to build and test Debug stubs: let GDB talk to NXP MCUs on target boards via USB Many target/probe/host platform combinations => many stubs! Written locally: our responsibility to build and test Target software: C Libraries; LPC-Open related components; ... Written locally: our responsibility to build and test
  8. 8. #jenkinsconf Testing LPCXpresso’s debug stubs 8
  9. 9. #jenkinsconf Manual testing will not get us very far 9
  10. 10. #jenkinsconf Debug stubs: the scale of the testing problem This matrix is only a subset •  New targets and debug probes are constantly being added •  We have to run the same tests over again for each host platform
  11. 11. #jenkinsconf It’s not enough to run the tests just once Why test frequently? •  It’s much easier to find problems straight away •  Developers see what is wrong while their work is still fresh in their minds Why test everything, everywhere? •  Many of the stubs share common code: change one – break the others! •  Stubs work with the host OS, so they behave differently on each platform 11
  12. 12. #jenkinsconf The solution: automate testing with Jenkins •  Jenkins builds the three LPCXpresso installers –  (How it does that is a whole talk for another day) •  A set of host machines are set up as Jenkins slaves –  Windows 7, Windows 8.1, Ubuntu, Fedora, Mac, … •  TestInstall (a matrix job) –  Uses the Copy Artifact plugin to get the right installer –  Runs it silently and checks the result 12
  13. 13. #jenkinsconf Test a wide range of target/probe combinations •  Using a big matrix job, each testing slave can –  use every kind of debug probe to communicate with target boards –  execute a set of standard images compiled for each kind of target hardware 13
  14. 14. #jenkinsconf The snag: how to power-cycle the test boards? •  In normal use, nobody would expect to have multiple boards and debug probes attached to one computer –  They are not designed to work like that! •  Many of them have to be switched off and on again as well as being booted before use •  How can we simulate this from Jenkins? 14
  15. 15. #jenkinsconf The answer: our test farm has COWs 15 COW = “Collection of Wires”
  16. 16. #jenkinsconf The test farm (much simplified) 16 Jenkins master TestWindows TestMac TestUbuntu ... many similar Jenkins slaves Ermintrude Daisy Poppy Willow USB probe + target ... many similar cows ... and very many more probe + target probe + target probe + target probe + target probe + target probe + target probe + target probe + target USB
  17. 17. #jenkinsconf Part of the Cambridge herd 17
  18. 18. #jenkinsconf An expatriate cow 18 Our Jenkins master drives this laptop in California as a (rather slow) slave
  19. 19. #jenkinsconf How are our cows connected today? •  Cows are physically plugged in to particular test machines •  We run a special job to tell Jenkins whenever a cow has been moved •  All the test jobs begin by copying over the artifact that this job created 19
  20. 20. #jenkinsconf Summary Display shows test coverage available The result of the ConfigureTestFarm job shows us where the gaps in our testing are
  21. 21. #jenkinsconf Test results overview shown in Jenkins Results table created with the Jenkins Summary Display Plugin
  22. 22. #jenkinsconf Tests are defined using Robot Framework •  Open source; easily extended using Python •  Works on all our supported platforms •  Ready-made output display formatted in HTML •  Easy to integrate with Jenkins 22
  23. 23. #jenkinsconf Links in our results table lead to Robot’s reports
  24. 24. #jenkinsconf Drilling down further shows more detail
  25. 25. #jenkinsconf A cow in action 25
  26. 26. #jenkinsconf A cow driving a development board 26
  27. 27. #jenkinsconf More automation Testing other aspects of LPCXpresso
  28. 28. #jenkinsconf Importing and compiling 2000 templates 28
  29. 29. #jenkinsconf Uploading, downloading, … •  Jenkins uploads our build artifacts to a cloud server for easy access by colleagues in remote locations •  A shared cloud slave enables communication with a distant group’s Jenkins instance –  Their Jenkins uploads zipped packages as artifacts –  Our Jenkins fetches them, unpacks them and runs them through our test farm, reporting results by email 29
  30. 30. #jenkinsconf The plugins and features of Jenkins that help us most
  31. 31. #jenkinsconf Our favourite plugins •  Exclusion •  Conditional buildstep •  EnvInject •  Parameterized trigger •  Matrix combinations •  Promoted builds •  Copy artifact •  Node/label parameters •  Summary display •  Description setter •  Build timeout •  Green balls 31
  32. 32. #jenkinsconf Things we do with build parameters •  Check out from a branch of the source •  Name a separate exclusion resource on each slave •  Allow interactive input to Groovy scripts •  Select matrix combinations dynamically 32
  33. 33. #jenkinsconf Selecting matrix combinations dynamically 33 This is part of the configuration for a job called LaunchTestMatrix
  34. 34. #jenkinsconf Matrix combinations example This is generated automatically by a script COMBINATIONS=(PROBE=="LPC-Link"&&(TARGET=="LPC2138"|| TARGET=="LPC11U14_201"||TARGET=="LPC1114_301"|| TARGET=="LPC1549"))||(PROBE=="RedProbe +"&&(TARGET=="LPC2368"||TARGET=="LPC1768"))||(PROBE=="LPC- Link2"&&(TARGET=="LPC4337"||TARGET=="LPC1768"|| TARGET=="LPC4330_SPIFI_1M_64K"||TARGET=="LPC1227_301"|| TARGET=="LPC1830_NGX_SPIFI"||TARGET=="LPC4370_SPIFI"|| TARGET=="LPC1549"||TARGET=="LPC11U68"|| TARGET=="LPC54102J512"||TARGET=="LPC810"))||(PROBE=="RDB- Link"&&(TARGET=="LPC1768"))||(PROBE=="CMSIS- DAP"&&(TARGET=="LPC11U68"||TARGET=="LPC824"|| TARGET=="LPC1549"||TARGET=="LPC4370_SPIFI"|| TARGET=="LPC54102J512"||TARGET=="LPC1768")) 34
  35. 35. #jenkinsconf Exclusion to manage shared resources The whole point of the cows is to have only one target board/probe combination switched on at one time. •  What if more than one job is running on the slave? Our solution: use the Jenkins Exclusion Plugin •  Define a resource called ${SLAVE}_BOARDS •  Claim it before running each test 35
  36. 36. #jenkinsconf Our Jenkins wish-list
  37. 37. #jenkinsconf We do love Jenkins, but it would be nice if … •  You could select one configuration in a matrix job when copying artifacts from a permalink •  There was proper support for source-code branches •  The Cross-platform shell plugin worked better •  Matrix jobs were first-class citizens 37
  38. 38. #jenkinsconf Automated is good, but scripted is better!
  39. 39. #jenkinsconf Our job structure is quite complicated Visualization created with the Jenkins Dependency Graph Plugin 39
  40. 40. #jenkinsconf We think the Workflow plugin will help •  Allows complicated job structures to be expressed concisely as scripts •  Easier to read, to write and to manage But •  We shall need good support for matrix jobs 40
  41. 41. #jenkinsconf What is special about our use-case for Jenkins
  42. 42. #jenkinsconf Our world is a matrix world •  Our test farm is a sparse multi-dimensional matrix •  The hardware is reconfigurable, so the Jenkins model of it has to be flexible, too •  We need build parameters and combination filters to manage this complexity •  The proper functioning of matrix jobs is essential 42
  43. 43. #jenkinsconf • • • Questions?
  44. 44. London, UK 23-24 June