Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Famous Cows of Cambridge
A non-standard use-case for Jenkins
Sarah Woodall, NXP Semiconductors
#jenkinsconf
Footer
What this talk is about
•  How (and why) we use Jenkins to test our software
automatically using a far...
#jenkinsconf
Setting the scene
#jenkinsconf
Some of the world’s most photographed cows
4
Photo by Alex Brown
(originally posted to Flickr
as Cows and Kin...
#jenkinsconf
The product that our Jenkins builds and tests
A software
development
toolchain based
on Eclipse and
GCC/GDB
•...
#jenkinsconf
LPCXpresso in normal use
6
Host computer
running
LPCXpresso
Debug probe
USB
Development board
Target MCU
Stub
#jenkinsconf
LPCXpresso’s components
Open-source IDE built and tested by the Eclipse Foundation
Public releases three time...
#jenkinsconf
Testing LPCXpresso’s
debug stubs
8
#jenkinsconf
Manual testing will not get us very far
9
#jenkinsconf
Debug stubs: the scale of the testing problem
This matrix is only a subset
•  New targets and debug probes
ar...
#jenkinsconf
It’s not enough to run the tests just once
Why test frequently?
•  It’s much easier to find problems straight...
#jenkinsconf
The solution: automate testing with Jenkins
•  Jenkins builds the three LPCXpresso installers
–  (How it does...
#jenkinsconf
Test a wide range of target/probe combinations
•  Using a big matrix job, each testing slave can
–  use every...
#jenkinsconf
The snag: how to power-cycle the test boards?
•  In normal use, nobody would expect to have multiple
boards a...
#jenkinsconf
The answer: our test farm has COWs
15
COW =
“Collection
of Wires”
#jenkinsconf
The test farm (much simplified)
16
Jenkins master
TestWindows TestMac TestUbuntu ... many similar Jenkins sla...
#jenkinsconf
Part of the Cambridge herd
17
#jenkinsconf
An expatriate cow
18
Our Jenkins master drives this laptop in California as a (rather slow) slave
#jenkinsconf
How are our cows connected today?
•  Cows are physically
plugged in to particular
test machines
•  We run a s...
#jenkinsconf
Summary Display shows test coverage available
The result of the
ConfigureTestFarm
job shows us
where the gaps...
#jenkinsconf
Test results overview shown in Jenkins
Results table
created with the
Jenkins Summary
Display Plugin
#jenkinsconf
Tests are defined using Robot Framework
•  Open source; easily extended using Python
•  Works on all our supp...
#jenkinsconf
Links in our results table lead to Robot’s reports
#jenkinsconf
Drilling down further shows more detail
#jenkinsconf
A cow in action
25
#jenkinsconf
A cow driving a development board
26
#jenkinsconf
More automation
Testing other aspects of LPCXpresso
#jenkinsconf
Importing and compiling 2000 templates
28
#jenkinsconf
Uploading, downloading, …
•  Jenkins uploads our build artifacts to a cloud server
for easy access by colleag...
#jenkinsconf
The plugins and features of
Jenkins that help us most
#jenkinsconf
Our favourite plugins
•  Exclusion
•  Conditional buildstep
•  EnvInject
•  Parameterized trigger
•  Matrix c...
#jenkinsconf
Things we do with build parameters
•  Check out from a branch of the source
•  Name a separate exclusion reso...
#jenkinsconf
Selecting matrix combinations dynamically
33
This is part of the
configuration for a job
called LaunchTestMat...
#jenkinsconf
Matrix combinations example
This is generated automatically by a script
COMBINATIONS=(PROBE=="LPC-Link"&&(TAR...
#jenkinsconf
Exclusion to manage shared resources
The whole point of the cows is to have only one target
board/probe combi...
#jenkinsconf
Our Jenkins wish-list
#jenkinsconf
We do love Jenkins, but it would be nice if …
•  You could select one configuration in a matrix job
when copy...
#jenkinsconf
Automated is good,
but scripted is better!
#jenkinsconf
Our job structure is quite complicated
Visualization created with the
Jenkins Dependency Graph Plugin
39
#jenkinsconf
We think the Workflow plugin will help
•  Allows complicated job structures to be expressed
concisely as scri...
#jenkinsconf
What is special about our
use-case for Jenkins
#jenkinsconf
Our world is a matrix world
•  Our test farm is a sparse multi-dimensional matrix
•  The hardware is reconfig...
#jenkinsconf
•  www.lpcware.com/lpcxpresso
•  sarah.woodall@nxp.com
•  sarah@woodall.org.uk
Questions?
London, UK 23-24 June
Upcoming SlideShare
Loading in …5
×

of

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 1

YouTube videos are no longer supported on SlideShare

View original on YouTube

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 3 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 4 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 5 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 6 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 7 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 8 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 9 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 10 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 11 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 12 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 13 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 14 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 15 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 16 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 17 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 18 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 19 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 20 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 21 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 22 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 23 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 24 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 25 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 26 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 27 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 28 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 29 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 30 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 31 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 32 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 33 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 34 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 35 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 36 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 37 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 38 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 39 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 40 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 41 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 42 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 43 JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 44

YouTube videos are no longer supported on SlideShare

View original on YouTube

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins Slide 46
Upcoming SlideShare
Uxdesign & hackathon
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins

Download to read offline

By Sarah Woodall, NXP Semiconductors

LPCXpresso is a multi-platform IDE for developers of embedded software to run on NXP Semiconductor's ARM-based microcontrollers. NXP needs to test that the debugger can execute programs on numerous different development boards that connect to the USB ports of host computers. Besides building a complex software product, the Jenkins installation drives an automated test farm consisting of home-built software-controlled USB switches ("cows") that control a huge array of combinations of test board, debug probe and host platform. This talk will give a tour of the NXP farm, including video of the cows in action, and will describe the features of Jenkins that are used to make it work, with particular emphasis on dynamic selection of combinations within matrix jobs, parameterized triggers and the Summary Display plugin. Finally, plans to migrate to the new Workflow plugin will be discussed. NXP believes the Workflow plugin will simplify the structure and make it more maintainable.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins

  1. 1. The Famous Cows of Cambridge A non-standard use-case for Jenkins Sarah Woodall, NXP Semiconductors
  2. 2. #jenkinsconf Footer What this talk is about •  How (and why) we use Jenkins to test our software automatically using a farm of custom hardware •  The plugins and features of Jenkins that help •  Improvements we would love to see •  Why we think the Workflow Plugin will be important 2
  3. 3. #jenkinsconf Setting the scene
  4. 4. #jenkinsconf Some of the world’s most photographed cows 4 Photo by Alex Brown (originally posted to Flickr as Cows and King's) [CC BY 2.0 (http:// creativecommons.org/ licenses/by/2.0)], via Wikimedia Commons
  5. 5. #jenkinsconf The product that our Jenkins builds and tests A software development toolchain based on Eclipse and GCC/GDB •  Designed for ease of use with NXP’s ARM-based LPC microcontrollers •  Hosted on Windows, Mac, Linux (separate installers)
  6. 6. #jenkinsconf LPCXpresso in normal use 6 Host computer running LPCXpresso Debug probe USB Development board Target MCU Stub
  7. 7. #jenkinsconf LPCXpresso’s components Open-source IDE built and tested by the Eclipse Foundation Public releases three times a year GNU tools (GCC compiler, GDB debugger, ...) built and tested by ARM Public releases four times a year Project wizards; Debug perspective; MCU definitions; support for advanced features such as Trace, etc ... Written locally: our responsibility to build and test Debug stubs: let GDB talk to NXP MCUs on target boards via USB Many target/probe/host platform combinations => many stubs! Written locally: our responsibility to build and test Target software: C Libraries; LPC-Open related components; ... Written locally: our responsibility to build and test
  8. 8. #jenkinsconf Testing LPCXpresso’s debug stubs 8
  9. 9. #jenkinsconf Manual testing will not get us very far 9
  10. 10. #jenkinsconf Debug stubs: the scale of the testing problem This matrix is only a subset •  New targets and debug probes are constantly being added •  We have to run the same tests over again for each host platform
  11. 11. #jenkinsconf It’s not enough to run the tests just once Why test frequently? •  It’s much easier to find problems straight away •  Developers see what is wrong while their work is still fresh in their minds Why test everything, everywhere? •  Many of the stubs share common code: change one – break the others! •  Stubs work with the host OS, so they behave differently on each platform 11
  12. 12. #jenkinsconf The solution: automate testing with Jenkins •  Jenkins builds the three LPCXpresso installers –  (How it does that is a whole talk for another day) •  A set of host machines are set up as Jenkins slaves –  Windows 7, Windows 8.1, Ubuntu, Fedora, Mac, … •  TestInstall (a matrix job) –  Uses the Copy Artifact plugin to get the right installer –  Runs it silently and checks the result 12
  13. 13. #jenkinsconf Test a wide range of target/probe combinations •  Using a big matrix job, each testing slave can –  use every kind of debug probe to communicate with target boards –  execute a set of standard images compiled for each kind of target hardware 13
  14. 14. #jenkinsconf The snag: how to power-cycle the test boards? •  In normal use, nobody would expect to have multiple boards and debug probes attached to one computer –  They are not designed to work like that! •  Many of them have to be switched off and on again as well as being booted before use •  How can we simulate this from Jenkins? 14
  15. 15. #jenkinsconf The answer: our test farm has COWs 15 COW = “Collection of Wires”
  16. 16. #jenkinsconf The test farm (much simplified) 16 Jenkins master TestWindows TestMac TestUbuntu ... many similar Jenkins slaves Ermintrude Daisy Poppy Willow USB probe + target ... many similar cows ... and very many more probe + target probe + target probe + target probe + target probe + target probe + target probe + target probe + target USB
  17. 17. #jenkinsconf Part of the Cambridge herd 17
  18. 18. #jenkinsconf An expatriate cow 18 Our Jenkins master drives this laptop in California as a (rather slow) slave
  19. 19. #jenkinsconf How are our cows connected today? •  Cows are physically plugged in to particular test machines •  We run a special job to tell Jenkins whenever a cow has been moved •  All the test jobs begin by copying over the artifact that this job created 19
  20. 20. #jenkinsconf Summary Display shows test coverage available The result of the ConfigureTestFarm job shows us where the gaps in our testing are
  21. 21. #jenkinsconf Test results overview shown in Jenkins Results table created with the Jenkins Summary Display Plugin
  22. 22. #jenkinsconf Tests are defined using Robot Framework •  Open source; easily extended using Python •  Works on all our supported platforms •  Ready-made output display formatted in HTML •  Easy to integrate with Jenkins 22 robotframework.org
  23. 23. #jenkinsconf Links in our results table lead to Robot’s reports
  24. 24. #jenkinsconf Drilling down further shows more detail
  25. 25. #jenkinsconf A cow in action 25
  26. 26. #jenkinsconf A cow driving a development board 26
  27. 27. #jenkinsconf More automation Testing other aspects of LPCXpresso
  28. 28. #jenkinsconf Importing and compiling 2000 templates 28
  29. 29. #jenkinsconf Uploading, downloading, … •  Jenkins uploads our build artifacts to a cloud server for easy access by colleagues in remote locations •  A shared cloud slave enables communication with a distant group’s Jenkins instance –  Their Jenkins uploads zipped packages as artifacts –  Our Jenkins fetches them, unpacks them and runs them through our test farm, reporting results by email 29
  30. 30. #jenkinsconf The plugins and features of Jenkins that help us most
  31. 31. #jenkinsconf Our favourite plugins •  Exclusion •  Conditional buildstep •  EnvInject •  Parameterized trigger •  Matrix combinations •  Promoted builds •  Copy artifact •  Node/label parameters •  Summary display •  Description setter •  Build timeout •  Green balls 31
  32. 32. #jenkinsconf Things we do with build parameters •  Check out from a branch of the source •  Name a separate exclusion resource on each slave •  Allow interactive input to Groovy scripts •  Select matrix combinations dynamically 32
  33. 33. #jenkinsconf Selecting matrix combinations dynamically 33 This is part of the configuration for a job called LaunchTestMatrix
  34. 34. #jenkinsconf Matrix combinations example This is generated automatically by a script COMBINATIONS=(PROBE=="LPC-Link"&&(TARGET=="LPC2138"|| TARGET=="LPC11U14_201"||TARGET=="LPC1114_301"|| TARGET=="LPC1549"))||(PROBE=="RedProbe +"&&(TARGET=="LPC2368"||TARGET=="LPC1768"))||(PROBE=="LPC- Link2"&&(TARGET=="LPC4337"||TARGET=="LPC1768"|| TARGET=="LPC4330_SPIFI_1M_64K"||TARGET=="LPC1227_301"|| TARGET=="LPC1830_NGX_SPIFI"||TARGET=="LPC4370_SPIFI"|| TARGET=="LPC1549"||TARGET=="LPC11U68"|| TARGET=="LPC54102J512"||TARGET=="LPC810"))||(PROBE=="RDB- Link"&&(TARGET=="LPC1768"))||(PROBE=="CMSIS- DAP"&&(TARGET=="LPC11U68"||TARGET=="LPC824"|| TARGET=="LPC1549"||TARGET=="LPC4370_SPIFI"|| TARGET=="LPC54102J512"||TARGET=="LPC1768")) 34
  35. 35. #jenkinsconf Exclusion to manage shared resources The whole point of the cows is to have only one target board/probe combination switched on at one time. •  What if more than one job is running on the slave? Our solution: use the Jenkins Exclusion Plugin •  Define a resource called ${SLAVE}_BOARDS •  Claim it before running each test 35
  36. 36. #jenkinsconf Our Jenkins wish-list
  37. 37. #jenkinsconf We do love Jenkins, but it would be nice if … •  You could select one configuration in a matrix job when copying artifacts from a permalink •  There was proper support for source-code branches •  The Cross-platform shell plugin worked better •  Matrix jobs were first-class citizens 37
  38. 38. #jenkinsconf Automated is good, but scripted is better!
  39. 39. #jenkinsconf Our job structure is quite complicated Visualization created with the Jenkins Dependency Graph Plugin 39
  40. 40. #jenkinsconf We think the Workflow plugin will help •  Allows complicated job structures to be expressed concisely as scripts •  Easier to read, to write and to manage But •  We shall need good support for matrix jobs 40
  41. 41. #jenkinsconf What is special about our use-case for Jenkins
  42. 42. #jenkinsconf Our world is a matrix world •  Our test farm is a sparse multi-dimensional matrix •  The hardware is reconfigurable, so the Jenkins model of it has to be flexible, too •  We need build parameters and combination filters to manage this complexity •  The proper functioning of matrix jobs is essential 42
  43. 43. #jenkinsconf •  www.lpcware.com/lpcxpresso •  sarah.woodall@nxp.com •  sarah@woodall.org.uk Questions?
  44. 44. London, UK 23-24 June
  • zahernourredine

    Dec. 12, 2018

By Sarah Woodall, NXP Semiconductors LPCXpresso is a multi-platform IDE for developers of embedded software to run on NXP Semiconductor's ARM-based microcontrollers. NXP needs to test that the debugger can execute programs on numerous different development boards that connect to the USB ports of host computers. Besides building a complex software product, the Jenkins installation drives an automated test farm consisting of home-built software-controlled USB switches ("cows") that control a huge array of combinations of test board, debug probe and host platform. This talk will give a tour of the NXP farm, including video of the cows in action, and will describe the features of Jenkins that are used to make it work, with particular emphasis on dynamic selection of combinations within matrix jobs, parameterized triggers and the Summary Display plugin. Finally, plans to migrate to the new Workflow plugin will be discussed. NXP believes the Workflow plugin will simplify the structure and make it more maintainable.

Views

Total views

1,012

On Slideshare

0

From embeds

0

Number of embeds

8

Actions

Downloads

9

Shares

0

Comments

0

Likes

1

×