Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Embedded Recipes 2019 - Testing firmware the devops way


Published on

ITRenew is selling recertified OCP servers under the Sesame brand, those servers come either with their original UEFI BIOS or with LinuxBoot. The LinuxBoot project is pushing the Linux kernel inside bios flash and using userland programs as bootloader.

To achieve quality on our software stack, as any project, we need to test it. Traditional BIOS are tested by hand, this is 2019 we need to do it automatically! We already presented the hardware setup behind the LinuxBoot CI, this talk will focus on the software.

We use u-root for our userland bootloader; this software is written in Go so we naturally choose to use Go for our testing too. We will present how we are using and extending the Go native test framework `go test` for testing embedded systems (serial console) and improving the report format for integration to a CI.
Julien Viard de Galbert

Published in: Software
  • Login to see the comments

Embedded Recipes 2019 - Testing firmware the devops way

  1. 1. Testing Firmware the DevOps way
  2. 2. System Firmware The firmware that starts your computer BIOS UEFI LinuxBoot
  3. 3. LinuxBoot on OCP hardware ITRENEW is selling Open Compute Project hardware under the SESAME brand. Most of our hardware comes from hyper-scale datacenter decommissioning. The recertification process includes updating the firmwares. One option for firmware is LinuxBoot. ITRENEW historical activity is decommissioning
  4. 4. LinuxBoot ? What Linuxboot is placing a Linux kernel in the BIOS flash and using userland to implement a bootloader. Presented last year at ER2018 by Trammel Hudson. We use u-root and it's bootloader implementations Why Maintainability Security microcode update Reduice/Remove UEFI footprint Linux is already part of our security audit... Extra features (nvme, infra integration) Replacing the UEFI (proprietary) firmware, with an open source one gives us those advantages.
  5. 5. Developing LinuxBoot need Testing ! Testing requirements Automated testing with CI, like devops! Keeping the full logs from the console for debugging Up to 3 logs to watch: test output serial console IPMI console Keep all details on first run (long tests) Simple/summary view by default IPMI provides a Serial Over LAN access to the console.
  6. 6. Testing Hardware setup This setup was presented last year at ER2018 by Jean-Marie Verdun. The hardware for testing is composed of servers used as DUT, controlling servers used as CI runners. Each DUT has: a relay for controlling power, a flash emulator to load firmware. So today we will focus on the testing software.
  7. 7. Let's look at it! DEMO time! If you are reading the slides: Features are presented live here, explanations comes later...
  8. 8. Why use Golang to write tests ? u-root is written in Go Go has it's test framework: go test and the testing package. BUT Go test output does not directly fulfil our requirements
  9. 9. go test Files named *_test.go are compiled only for testing. Functions named Test*(t) are test case automatically run by the framework. Inside test functions subtest can be created by calling t.Run. Short example: func TestMyFeature(t *testing.T) { got := MyFeature(/* ... */) want := "OK" /* expected result */ if got != want { t.Fatalf("test failed, got %v, want %v", got, want) } }
  10. 10. go test output Success has very minimal output: ok MyPackage 0.001s Error is more verbose: --- FAIL: TestMyFeature (0.00s) prez_test.go:13: test failed, got KO, want OK FAIL FAIL MyPackage 0.001s Verbose mode: messages on success, error gets flooded :( === RUN TestMyFeature --- PASS: TestMyFeature (0.00s) PASS ok command-line-arguments 0.001s
  11. 11. go test -json There is a JSON output for further processing an event log with one JSON event per line {"Action":"run","Test":"TestMyFeature","Time":"2019-08-12T11:55:41.174693884+02:00"} {"Action":"output","Test":"TestMyFeature","Output":"=== RUN TestMyFeaturen","Time":"2 {"Action":"output","Test":"TestMyFeature","Output":"--- FAIL: TestMyFeature (0.00s)n"," {"Action":"output","Test":"TestMyFeature","Output":" prez_test.go:13: test failed, go {"Action":"fail","Test":"TestMyFeature","Elapsed":0,"Time":"2019-08-12T11:55:41.17486883 {"Action":"output","Output":"FAILn","Time":"2019-08-12T11:55:41.174877223+02:00"} {"Action":"output","Output":"FAILtMyPackaget0.001sn","Time":"2019-08-12T11:55:41.1750 {"Action":"fail","Elapsed":0.001,"Time":"2019-08-12T11:55:41.175039291+02:00"}
  12. 12. GoTestWeb in a nutshell html5/css/js application Parses go test json output client side Integrates asciinema-player for showing console (serial, IPMI) logs parses output event to get asciinema filename events timestamp are used to seek the video on click Renders the verbose test results folded if OK, open if Failed. Keep test hierarchy (renders subtests inside tests)
  13. 13. GoTestWeb development Disclaimer: I'm not a front dev! github: GoTestWeb techno: Bootstrap jQuerry Ractive.js RequireJS
  14. 14. Enough of JS, more of Go!
  15. 15. Tastevin Tastevin is grouping the Go code used to build the tests. Provides packages to ease writing test for remote platform. Includes a gotest command to wrap go test -json save the json and display the summary. Named as a wine tool to go with fiano the project to analyse and build LinuxBoot images from UEFI images.
  16. 16. GoExpect goexpect is an implementation of Expect in go. expect allows to script a behavior on a terminal: wait for a regex send a string wait for another regex send another string ...
  17. 17. GoExpect Spawners command ssh generic (to be customized)
  18. 18. GoExpect custom Spawners serial Spawner connect a serial port to the generic spawner IPMI Spawner wrap ipmitool sol activate in a command spawner QEMU Spawner wrap qemu in a command spawner can be extended as needed!
  19. 19. GoExpect Batcher A sequence of actions (expect/send) can be presented in a go slice BExp expect a regexp with the default timeout BExpT expect a regexp with a given timeout BSnd send a string example for login in a server: []expect.Batcher{ &expect.BExp{R: "username:"}, &expect.BSnd{S: *user + "n"}, &expect.BExp{R: "password:"}, &expect.BSnd{S: *pass + "n"}, }
  20. 20. BExpTLog: BExpT with Log output This custom element can be used in Batcher. It provides the same feature as BExpT: a matching regex and a timeout Adds a Log output to be send when the text is matched. The goal: this log will be a seek point for the asciinema view.
  21. 21. Asciinema Writer goexpect can duplicate the terminal output to a writer interface. it's actually a writecloser interface, it will close at the end of the expect session. The asciinema writer gets it's data for the writer interface adds the timestamp for the current event save it in asciicast v2 format logs message on open/close so that GoTestWeb can catch the details.
  22. 22. Scriptreplay Writer The scriptreplay writer log the data to a plain file record the timestamp in a timing file can be played with scriptreplay -t file.tim file.log logs message on open/close so that GoTestWeb can generate download links.
  23. 23. Tastevin packages Includes: goexpect spawner for serial, IPMI, qemu goexpect batcher extension: BExpTLog console writer: asciicast and scriptreplay control package for the CI system: em100, relay a basic testsuite for LinuxBoot startup a few utility packages
  24. 24. Tastevin command gotest run a wrapper to go test -json saves the json to a file output the package test summary (without error detail) or output the verbose go test output gotest serve for developer: given a JSON, locally serve GoTestWeb to render it can convert json to junit.
  25. 25. CI integration gotest can be used directly in CI systems to save the json and asciicasts gotest gen can generate the html and export the GoTestWeb application as well simply place the json, asciicast, html and GoTestWeb in a CI artifact for later rendering in the browser!
  26. 26. Developer live mode DEMO gotest live cd gotest-live-demo gotest live go test -json .
  27. 27. Developer live mode gotest live run tests as gotest run serve the results as gotest serve use websocket to stream the results between gotest live and the browser asciinema-player can only play plain files so this is unfortunately not streamed.
  28. 28. Tastevin and GoTestWeb are ready For you to use and contribute to! Or contact me for more informations: Julien Viard de Galbert
  29. 29. Questions ?
  30. 30. Links Tastevin and GoTestWeb on github: More on ITRENEW vision by Ali Fenn at OCPSummit19 Unlocking the Power of Circular Datacenters Introducing Sesame Open Hardware Compute Me: Julien Viard de Galbert