Previous introductory tutorials on LAVA have focussed on virtual platforms. This is an end-to-end tutorial as a basis to evaluate LAVA with one or more embedded targets using U-Boot. It integrates both a physical bootloader device with a stand-alone installation of LAVA, along with a simple PDU for target power control which is based on off-the-shelf Arduino components and fully integrated with pdudaemon. It covers device requirements, device configuration for 32- and 64-bit platforms, use of lavatool, tftp, pduclient and logging via the LAVA web interface and /var.

1. Presented by
Date
Event
BKK16-312: Integrating and
controlling embedded
devices in LAVA
Bill Fletcher
Linaro Field Engineering
Linaro Connect BKK16
Bill Fletcher
BKK16-312 March 9, 2016
v2.0

2. Overview
● Practical desk-top LAVA configuration
● Embedded 32 and 64-bit targets
● Power control
● Boot test automation example for real-world devices
● Starting point to leverage the awesomeness of LAVA

3. More about this session
● The slide deck has a lot of detail
● I don’t propose to go line-by-line through
configuration files here
● Kind of a meta-tutorial in the session
● It’s a big invitation to have a go with LAVA and
physical devices

4. Invitations to interrupt
● Any LAVA folks feel free to pitch in on new
dispatcher, supported versions, ...
● Any 96Boards folks feel free to pitch in on U-Boot
status on Hikey and other boards
● Questions & comments welcome

5. All the configs & images
● All material is at: http://people.linaro.org/~bill.
fletcher/
● LAVA configuration files
● Sample drivers and scripts for power control
● Boot images for Hikey and BB

6. Preamble - Brief Intro To LAVA

7. ● The Linaro Automated Validation
Architecture (LAVA wiki link)
● An automation system for deploying kernel,
dtb and rootfs onto physical and virtual
hardware for running tests.
● A collection of participating components
● Very scalable
LAVA is ...

8. YAML test
definition
Basic Elements of LAVA
LAVA Dispatcher
● Executes the test job
● Contains target platform
configuration files
● Locally intelligent and tries to 'do
the right thing'
● Manages communication to the
target device
● Collects and evaluates the job
results
LAVA Server
● Handles job submission& scheduling
● Handles instances of target platforms
● Invokes the dispatcher
● Consolidates the results into bundle
stream
Device
configuration
.conf
JSON job
definition
Dashboard/Admin
● Accepts jobs, shows device status,
job queues etc
● Manages object creation and
permissions
● Reports results
Target
Device
Pre-parsed
results
scheduling
single instance

9. YAML test
definition
Focus for this Session
LAVA Dispatcher
● Executes the test job
● Contains target platform
configuration files
● Locally intelligent and tries to 'do
the right thing'
● Manages communication to the
target device
● Collects and evaluates the job
results
LAVA Server
● Handles job submission& scheduling
● Handles instances of target platforms
● Invokes the dispatcher
● Consolidates the results into bundle
stream
Device
configuration
.conf
JSON job
definition
Dashboard/Admin
● Accepts jobs, shows device status,
job queues etc
● Manages object creation and
permissions
● Reports results
Target
Device
Pre-parsed
results
scheduling
single instance
i.e. interfacing the dispatcher to the target

10. Start now - be aware that LAVA
is evolving ...
● The pipeline will be the new model for the dispatcher
code:
● Submitted jobs converted to a pipeline of discrete
actions
● The entire pipeline is validated before the job starts.
● The model integrates concepts like fail-early, error
identification, avoid defaults, fail and diagnose later
COMING SOON

11. ● LAVA is a Debian package
● Work with supported versions (more info here):
● apt-get install lava
○ Everything you need for a single instance
● Installs latest packaged release
● Find it under /usr/lib/python2.7
Getting Started - Installing

12. 1. Finish configuring the installation
2. Configure a virtual test device (KVM, qemu)
3. Acquire a bootable test image
4. Write a basic json boot test job definition
5. Submit a test job
See these virtual target tutorials:
https://youtu.be/T8jFzXRrFh8
https://youtu.be/0FlfRMxwC00
http://people.linaro.org/~bill.fletcher/SFO15-TR8_Getting_Started_With_LAVA_supporting_material/
After Installing

13. Adding Embedded Devices in
LAVA
(i) Mainly about the hardware

14. Embedded Devices
● 2 worked examples - Beaglebone and Hikey
● Both boot with U-Boot
● Each has a console, network, boots Linux
● Note: a bootloader device is a specific type of target
in LAVA i.e. client_type = bootloader

15. Embedded devices in LAVA
Requirements:
1. An uninterrupted serial console connection
2. A way to put the system in a known state
and reboot
3. TFTP assumed
i.e. serial, network and power control ...

16. Serial Console and Network for TFTP
You need console and network connections:
● ser2net package is a dependency of lava-dispatcher, so will be
installed automatically. In /etc/ser2net.conf add something like:
2000:telnet:0:/dev/tty.beaglebone:115200
8DATABITS NONE 1STOPBIT banner
● The dispatcher relies on TFTP downloads to all be made from
/var/lib/lava/dispatcher/tmp. After installing tftp-hpa, the
configuration file for tftpd-hpa needs to be modified to use the
LAVA directory instead of the default

17. Beaglebone
Comparison vs previously stated requirements:
An uninterrupted serial connection
- BB(W) has FTDI serial via USB peripheral connector that also
supplies power to the board
A way to put the system in a known state and reboot the board
- Removing USB power kills the FTDI serial connection
- Removing 5V power - the device switches to using the USB power
- :(
A bootloader with tftp
- Very mature U-Boot with TFTP via onboard RJ45

18. Where there’s a will there’s a way ...
(hard-wiring the reset line to
a jack for relay control)

19. Hikey
Comparison vs previously stated requirements:
An uninterrupted serial connection
- FTDI serial (a couple of options)
A way to put the system in a known state and reboot the board
- Power cycle 12V input
A bootloader with tftp
- Support in U-Boot upstream
- TFTP Support using KY88772 USB ethernet dongles

20. Rebooting the board:
An Introduction to
PDUs
● PDU = Power Distribution Unit
● “A way to put the system in a known state and reboot the board”
● Power cycle the board under software control
● PDUs tend to be shared between targets (potential access conflicts)
● pdudaemon is a framework that LAVA provides to manage multiple
accesses to PDUs
● LAVA device config file: hard_reset_command =

21. PDU Daemon - elements
pduclient - command line utility to send a command as a TCP request to
the daemon
e.g. pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --
command=reboot
pdusocket listener server - TCP request handler. Puts a request into the
request database.
pdurunner - one runner process for each PDU - loops looking for jobs in
the database for a given pdu.
pdu drivers - class-based drivers to interact with PDUs and now some
other measurement/control hardware, mainly over ethernet. Many PDUs
have an interface via Web, SNMP, and Telnet.

22. Reboot requests
LAVA Dispatcher
PDUclient
listener server
PDU
queue
SQL
adapter
PDUrunner
PDUdriver
PDUdriver
PDUdrivers
TCP
request
handler
PDU
process
insert request
PDU
queuePDU
queues
read next
job
PDU
processPDU
processes
reboot Requests database
PDUdriverPDUdriverPDU
hardware
reboot

23. Trivial Desktop PDU Example
● Arduino with a relay shield
for low-voltage or reset line
switching
● USB control of up to 4
targets
● Access managed via
pdudaemon localcmdline
driver class

24. {
"daemon": {
"hostname": "0.0.0.0",
"port": 16421,
"dbhost": "127.0.0.1",
"dbuser": "pdudaemon",
"dbpass": "pdudaemon",
"dbname": "lavapdu",
"retries": 5,
"logging_level": "INFO"
},
"pdus": {
"127.0.0.1": {
"driver": "localcmdline"
},
"192.168.10.3": {
"driver": "apc7952",
"telnetport": 5023
}
Configuring pdudaemon
● /etc/lavapdu/lavapdu.conf ->
● Example
○ Local daemon
○ 2 local pdus
○ one networked APC (via telnet)
and one localcmdline

25. Adding Embedded Devices in
LAVA
(ii) Mainly about the configuration
● Inform the dispatcher about the devices
● What are the sequences of commands
that the bootloaders should execute

26. /etc/lava-dispatcher/devices/beaglebone_001.conf
Basic device information in the device configuration file:
device_type = beaglebone (predefined type)
hostname = beaglebone_001 (arbitrary)
connection_command = telnet localhost 2000 (see ser2net)
hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --command=reboot
(see previous pdu_daemon slide)
...

27. Boot stanzas
● Boot command stanzas are
predefined Boot Commands which
are included in the device
configuration
● ‘boot_cmds_ramdisk’ is the default
boot stanza for bootloader clients
● This default defined in /python2.
7/dist-
packages/lava_dispatcher/device/
bootloader.py
● The Boot stanza is executed as if
typed at the u-boot command line
● Leverages target’s command line
functionality
● {Filenames} supplied from the job
boot_cmds_ramdisk =
setenv autoload no,
setenv label "'Hikey LAVA U-Boot Tutorial'",
setenv fdt_high "'0xffffffffffffffff'",
setenv kernel_addr_r "'0x00000000'",
setenv initrd_addr_r "'0x02000000'",
setenv fdt_addr_r "'0x01f00000'",
setenv loadkernel "'tftp ${kernel_addr_r}{KERNEL}'",
setenv loadinitrd "'tftp ${initrd_addr_r}{RAMDISK};
setenv initrd_size ${filesize}'",
setenv loadfdt "'tftp ${fdt_addr_r}{DTB}'",
setenv bootargs "'console=ttyAMA3,115200 debug
root=/dev/sda0 rootwait ip=none'",
setenv bootcmd "'usb start; setenv ipaddr 192.168.1.210;
setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd;
run loadfdt; booti ${kernel_addr_r} ${initrd_addr_r}
${fdt_addr_r}'",
boot

28. /etc/lava-dispatcher/devices/beaglebone_001.conf
device_type = beaglebone (predefined type)
hostname = beaglebone_001 (arbitrary)
connection_command = telnet localhost 2000(see ser2net)
hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --command=reboot
(see previous pdu_daemon slide)
boot_cmds_ramdisk = (boot stanza - see previous slide)
setenv autoload no,
setenv label "'Beaglebone LAVA U-Boot Tutorial'",
setenv fdt_high "'0xffffffff'",
setenv kernel_addr_r "'{KERNEL_ADDR}'",
setenv initrd_addr_r "'{RAMDISK_ADDR}'",
setenv fdt_addr_r "'{DTB_ADDR}'",
setenv loadkernel "'tftp ${kernel_addr_r} {KERNEL}'",
setenv loadinitrd "'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'",
setenv loadfdt "'tftp ${fdt_addr_r} {DTB}'",
setenv bootargs "'console=ttyO0,115200n8 root=/dev/ram0 ip=none'",
setenv bootcmd "'setenv ipaddr 192.168.1.200; setenv serverip {SERVER_IP}; run loadkernel; run
loadinitrd; run loadfdt; {BOOTX}'",
boot
(DEVICE CONFIGURATION FILE)

29. /etc/lava-dispatcher/devices/hikey_002.conf
device_type = hikey
hostname = hikey_002
connection_command = telnet localhost 2099
hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=3 --command=reboot
boot_cmds_ramdisk =
setenv autoload no,
setenv label "'Hikey LAVA U-Boot Tutorial'",
setenv fdt_high "'0xffffffffffffffff'",
setenv kernel_addr_r "'0x00000000'",
setenv initrd_addr_r "'0x02000000'",
setenv fdt_addr_r "'0x01f00000'",
setenv loadkernel "'tftp ${kernel_addr_r} {KERNEL}'",
setenv loadinitrd "'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'",
setenv loadfdt "'tftp ${fdt_addr_r} {DTB}'",
setenv bootargs "'console=ttyAMA3,115200 debug root=/dev/sda0 rootwait ip=none'",
setenv bootcmd "'usb start; setenv ipaddr 192.168.1.210; setenv serverip {SERVER_IP}; run loadkernel;
run loadinitrd; run loadfdt; booti ${kernel_addr_r} ${initrd_addr_r} ${fdt_addr_r}'",
boot
(DEVICE CONFIGURATION FILE)

30. Adding Embedded Devices in
LAVA
(iii) Putting it all together

32. Submitting the job file - 2 ways
1.Web Interface 2.lava-tool
● Rather than paste our json job files into the web interface,
we can submit test jobs to LAVA via the command line
using lava-tool.
● Generate an authentication token via the web interface.
lava-tool auth-add http://bill@localhost
● (note that you need to generate the authentication token
logged in to the local web interface as yourself)
● Submit: lava-tool submit-job http:
//bill@localhost
/home/bill/development/lava/boot_bb.json

33. json job file - Beaglebone
{
"actions": [
{
"command": "deploy_linaro_kernel",
"parameters":
{
"dtb": "file:/home/bill/development/lava/images/am335x-bone.dtb",
"kernel": "file:/home/bill/development/lava/images/uImage-bone",
"ramdisk": "file:/home/bill/development/lava/images/rootfs-bone.cpio.gz"
}
},
{
"command": "boot_linaro_image"
}],
"job_name": "boot-bb",
"target": "beaglebone_001",
"timeout": 18000
}

34. json job file - Hikey
{
"actions": [
{
"command": "deploy_linaro_kernel",
"parameters":
{
"dtb": "file:/home/bill/development/lava/images/hi6220-hikey-0202B.dtb",
"kernel": "file:/home/bill/development/lava/images/Image-0202A",
"ramdisk": "file:/home/bill/development/lava/images/rootfs-aarch64-2up-2000000.uboot",
"login_prompt": "aarch64_test login:",
"username": "root"
}
},
{
"command": "boot_linaro_image"
}],
"job_name": "boot-hi-318-0202",
"target": "hikey_002",
"timeout": 18000

35. Adding Embedded Devices in
LAVA
(iv) A quick demo

36. ● All the referenced configuration files are
available at http://people.linaro.org/~bill.
fletcher/bkk16_lava
● Support via mail and IRC. Details at https:
//validation.linaro.org/static/docs/support.html
● Have a go!
Wrap Up

37. Thanks for
listening
Any questions?

38. Screenshot - PDU reboot

39. Screenshot - boot command (BB)

40. Screenshot - boot command (Hikey)

41. lava stores images in /var/lib/lava/dispatcher/tmp
pdudaemon logs stored in /var/log/lavapdu*
Under the hood

42. Dispatcher Device conf files
Inherent defaults for client_type = bootloader configuration in
/usr/lib/python2.7/dist-
packages/lava_dispatcher/default-config/lava-
dispatcher/device-defaults.conf
Default configurations exists in: /usr/lib/python2.7/dist-
packages/lava_dispatcher/default-config/lava-
dispatcher/device-types/device.conf
Local device types can be defined in /etc/lava-
dispatcher/device-types
Specific device configs can be created in /etc/lava-
dispatcher/devices