Continous integration has gained a lot of popularity in the open source world in recent years, with the loud/web world at the forefront. The kernel world has however been somewhat slow to catch-up, especially in the embedded world, services like KernelCi and the 0-day are incredibly useful services once patches are flowing upstream but are limited in their scope. In this talk we’ll go over how CI best practices can be used when doing kernel development, and how with a little bit of infrastructure and tooling one can setup a “mini-kernelci” to run tests relevant to the current development on actual embedded hardware. Sjoerd SimonsOlivier Crête

1. Open First
Testing on device with LAVA
Olivier Crête
Sjoerd Simons

2. Continuous Integration (CI)
●
Basically, merge early, merge often
– General good Open Source work-flow advise
●
So that’s well covered!

3. ●
What most people think of when they say CI
●
Add-on to CI, do automated testing early and often
●
Popular in the web world, good tooling build into
GitHub/GitLab/etc (e.g. Travis CI)
●
These tend to be build and unit tests run in virtual
machines or docker containers
●
Not useful when hacking on device drivers for an ARM board
CI Testing

4. Why CI testing for Linux
●
Upstream Linux kernel code is literally perfect right?
– Just get your patches upstream
– The review process will ensure it is the best in the world
●
But...
– Code review cannot catch all bugs
– The sooner issues are caught the easier they are to fix

5. ●
Majority of kernel testing still tends to happen this way
●
If you break enough potential users
– High likelihood one of them will test the latest kernels
– or even linux-next and report the issue(s)
●
Does not work for custom hardware not publicly available
●
Evaluation boards with upstream support help some
Non-automated kernel testing needs critical mass

6. kernelci.org
●
Upstream automated kernel testing by Linaro and others
●
Kernelci monitors a collection of kernel trees, builds them
and tries to boot them on a variety of boards
●
Since early 2014 (numbers from https://kernelci.org/stats/)
– Performed >1,8M builds on >275 unique defconfigs
– Performed >3.2M boots on >260 unique boards
– 3 CPU architectures and 34 unique SoCs
– >1,500 builds per day
– >2,600 boots per day

7. ●
Testing breadth (lots of configs, lots of boards), not depth
– Boot test is a network boot of simple kernel + initramfs image executing basic shell commands
●
Ensures a board is bootable, but not much else
●
Still finds a surprising amount of issues!
●
Can always use more support
– Boards
– Build servers
– Engineering
– See the kernelci.org site for more details
kernelci.org

8. Downstream CI testing
●
Not typically very well done either
●
Commonly heard:
– “Oh, our Jenkins has been broken for weeks, we'll fix it for the
next phase of the project”
– “No, we don't do any, we've got a testing department that does
tests once a month”
– “We do build testing”

9. ●
Seen as a box ticking exercise
●
Not enough perceived benefit/trust in tests
●
Setup effort seen too much of a barrier
●
Automation of flashing/booting seen as challenging
– Hardware design doesn't always help here
●
Missing even basic infrastructure (e.g. no golden images, “ ”
builds or standardized build-environment), all developers are
meant to build at their desk “ ”
– Seems especially common for Yocto based projects
Why

10. Make automation easier
●
Consistent workflow is the key
●
Standardized way of building images
●
Standardized build infrastructure (e.g. using docker images)
●
Avoid all manual tweaks ( hacks ) to get things running“ ”
●
Also makes things easier for humans!

11. ●
Close the CI Loop!
– Don't wait for developers to check the dashboard“ ”
– Most CI tooling can send messages on failures (mail, IRC, etc)
●
Infrastructure like GitLab, GitHub, and Phabricator has
support for trigger CI on path submission
– Get automated test feedback early (recommended)
Improve perceived value

12. LAVA – automate on board testing
●
Linaro Automation and Validation Architecture (version 2)
●
Makes testing with boards easier
●
Focuses on management of boards
– Checks board health by running known good tests“ ”
– Schedules test jobs on boards
– Handles console access, power control etc.
●
Packaged for Debian, Docker images also available

13. ●
Not a full CI testing system, but part of your CI loop!
●
Should be combined with another system (e.g. Jenkins) to
build kernels and/or full images to be deployed by LAVA
LAVA

14. LAVA – Adding boards
●
Base requirements:
– A way to reliably turn the board on and off (typically
turning power on/off)
– A way to interact with the bootloader to load custom
kernels (e.g. fastboot, u-boot)
– A way to access a reliable console (e.g. serial)

15. ●
LAVA doesn’t care as long as it can run a command to turn things
on/off
●
LAVA-pdu: Simple network daemon and client to control PDUs
– Supports variety of APC networked PDUs (pricey but can be eBayed)
– Supports Energenie networked power sockets (~70 euros for 4 sockets)
– Supports devantech network relay boards (up to 20 relays, ~140 euros)
– Supports "local commands" for everything else e.g. USB relays
●
https://github.com/Linaro/pdudaemon.git
LAVA - Power on/off

16. LAVA – Access to console
●
Typically serial console for embedded boards
●
Like power on/off as long as it can be accessed via a
command, LAVA van use it
●
Typically used via network console services: conserver,
ser2net, etc

17. ●
LAVA supports various different bootloaders
– U-boot most common for generic embedded boards
– Fastboot for Android-based boards
– Others supported (ipxe, uefi menu) or can be added as needed
●
Bootloader should be configured to support network booting
LAVA - Bootloader

18. LAVA – Device type configuration (u-boot)
●
Configuration is YAML templated with Jinja2
{% extends 'base-uboot.jinja2' %}
{% set device_type = "rk3288-rock2-square" %}
{% set console_device = console_device|default('ttyS2') %}
{% set bootm_kernel_addr = '0x2000000' %}
{% set bootm_ramdisk_addr = '0x4000000' %}
{% set bootm_dtb_addr = '0x1f00000' %}
{% set bootz_kernel_addr = '0x2000000' %}
{% set bootz_ramdisk_addr ='0x4000000' %}
{% set bootz_dtb_addr = '0x1f00000' %}
{% set uboot_mkimage_arch = 'arm' %}

19. LAVA – Device instance configuration
{% extends 'rk3288-rock2-square.jinja2' %}
{% set connection_command = '/usr/bin/console -Mphobos -f rk3288-rock2-square-ehv-0' %}
{% set hard_reset_command = 'pduclient --daemon=phobos --hostname=eth8020-0 --port=6
--command=reboot' %}
{% set power_on_command = 'pduclient --daemon=phobos --hostname=eth8020-0 --port=6
--command=on' %}
{% set power_off_command = 'pduclient --daemon=phobos --hostname=eth8020-0 --port=6
--command=off' %}

20. ●
Test definitions are YAML files as well
●
can be submitted directly through the API or via command line
tools
– lava-tool
– lqa (https://git.collabora.com/cgit/singularity/tools/lqa.git/)
●
We tend to use lqa as it's a bit more friendly to use and can use
jinja2 based templating for yaml files.
LAVA – Submitting tests

21. LAVA – Example kernel boot test (1/3)
job_name: rk3288-rock2-square boot test
timeouts:
job:
minutes: 10
action:
minutes: 5
priority: high
visibility: public

22. LAVA – Example kernel boot test (2/3)
actions:
- deploy:
timeout:
minutes: 2
to: tftp
kernel:
url: https://.../zImage
type: image
ramdisk:
url: https://.../rootfs.cpio.gz
compression: gz
modules:
url: https://.../modules.tar.xz
compression: xz
dtb:
url: https://.../rk3288-rock2-square.dtb
os: oe

23. LAVA – Example kernel boot test (3/3)
- boot:
method: u-boot
commands: ramdisk
type: image
prompts:
- 'linaro-test'
- '/ #'

24. LAVA – Functional testing
- test:
timeout:
minutes: 5
definitions:
- repository: git://git.linaro.org/qa/test-definitions.git
from: git
path: ubuntu/smoke-tests-basic.yaml
name: smoke-tests

25. LAVA – including in the CI loop
●
Can not expect developers to go and look at lava output
●
Trigger can be GitLab Merge Request (or equivalent)
– Feedback can be pushed into the merge request as build status

26. The CI loop
Commit
Submit
LAVA job
Build
artifacts
Jenkins
build
Test
results
Notifi-
cation
LAVA
run
Email,
Slack,
IRC

27. ●
LAVA can send HTTP callbacks
LAVA – notification
notify:
criteria:
status: finished
callback:
url: {{callback_url}}
method: POST
content-type: json
dataset: results

28. LAVA – Receiving notification in Jenkins
●
Using Jenkins webhook plugin for a hook endpoint
[ Stages to build the test binaries and publish them ]
stage("Run job on lava") {
script {
hook = registerWebhook()
withCredentials([file(credentialsId: 'LQARC', variable: 'lqarc') ]) {
sh(script: "/usr/bin/lqa -c ${lqarc} submit -t callback_url:${hook.getURL()} test.yaml")
}
result = waitForWebhook hook
writeFile(file: "results.json", text: result)
sh(script: "python convert-results results.json results.xml")
junit("results.xml")
}
}

29. ●
Setting up CI testing with boards really is not that hard with LAVA
●
Setting a full CI loop requires more work, but once all infrastructure
is in place it can easily be grown with more in depth testing
●
One-time investment that keeps paying out throughout projects
Feel free to ask us for advice or help!
Conclusions

30. Thank you!