LCE13: Advanced LAVA Lab configuration

535 views
408 views

Published on

Resource: LCE13
Name: Advanced LAVA Lab configuration
Date: 11-07-2013
Speaker:
Video: https://www.youtube.com/watch?v=KgNSoe2WosY

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
535
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

LCE13: Advanced LAVA Lab configuration

  1. 1. Advanced LAVA Lab Configuration Tyler Baker & Georgy Redkozubov LCE13 - Dublin, July 2013 Taking advantage of advanced configuration options to optimize your LAVA Lab
  2. 2. Education ● Overview of the Cambridge Lab Deployment ● Multi-Machine Deployment ● Guided Tour: LAVA Dispatcher ○ Integrating a Test ○ Jenkins, Metadata, Filters, Image Reports ○ Override Boot Commands ○ Interactive Boot Commands Ideas ● Remote Workers Improvements Feedback ● LAVA Configuration Improvements Community Overview
  3. 3. Cambridge Lab Deployment validation.linaro.org: ● control ● dispatcher01 ● fastmodels01 ● fastmodels02 ● fastmodels03 ● fastmodels04 ● fastmodels05 ● fastmodels06 ● leg01 ● leg02 ● lng01 ● lng02 ● 75 Devices ● 4 Serial Console Servers ● 14 Power Distribution Units ● 5 Switches ● Constantly adding new hardware to keep up with demand ● Mixture of VMs and Bare metal
  4. 4. Cambridge Lab Deployment How do we keep all the LAVA servers running the same software? ● Salt Stack - https://github.com/saltstack/ ○ Salt provides fast, flexible, and easy to use configuration management. ○ LAVA team uses Salt to manage: ■ Deployment of LAVA code into production ● http://validation.linaro.org ■ Installed packages ■ Device configurations ■ User accounts and access control ■ Deployment of Fastmodels ■ Various helper scripts
  5. 5. Cambridge Lab Deployment LAVA Lab Configuration ● It's open source! ○ bzr branch lp:lava-lab ● Scripts for managing our production LAVA instance ○ lava-lab/scripts ● This is great resource for anyone interested in sample device configurations ○ lava-lab/lava/devices ● Want to know how our power cycling scripts work? ○ lava-lab/lava/lab-scripts
  6. 6. Cambridge Lab Deployment There is a salt state for that! ● As an example lets say you want the latest version of adb and fastboot installed. android-tools-adb: pkg: - latest android-tools-fastboot: pkg: - latest
  7. 7. Cambridge Lab Deployment Salt is what keeps all the LAVA remote workers in sync from code to configuration. Now lets look at a multi machine LAVA deployment.
  8. 8. Cambridge Lab Deployment DEMO
  9. 9. Multi-Machine LAVA Deployment LAVA Remote Workers ● Master Instance ○ Typically host the postgres database ○ Can use a remote postgres database if required ● Remote Worker Instance ○ Connects remotely to postgres database ○ Mounts the master instance media directory using sshfs ○ /srv/lava/instances/<instance>/var/lib/lava- server/media ● Room for improvement
  10. 10. LAVA Master Instance Multi-Machine LAVA Deployment Database Logs Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T ... User Interaction
  11. 11. Multi-Machine LAVA Deployment ● Master Instance ○ Is provisioned with salt to provide a user account for LAVA management ■ instance-manager ○ LAVA is installed with LAVA Deployment Tool ■ become-instance-manager ■ ./lava-deployment-tool setup ■ export LAVA_BUILDOUT_CFG=buildout-production.cfg ■ export LAVA_DB_ALLOWREMOTE=yes ■ ./lava-deployment-tool install production
  12. 12. Multi-Machine LAVA Deployment ● Remote Worker Instance ○ Is provisioned with salt to provide a user account for LAVA management ■ instance-manager ○ LAVA is installed with LAVA Deployment Tool ■ become-instance-manager ■ ./lava-deployment-tool setup ■ export LAVA_BUILDOUT_CFG=buildout-production.cfg ■ ./lava-deployment-tool installworker production ○ Salt managed files ■ Device configurations ■ PDU Scripts
  13. 13. Multi-Machine LAVA Deployment DEMO
  14. 14. Guided Tour: Integrating a Test LAVA checks return codes or parses stdout to generate test results ● This is very powerful, as test written in ANY language can be easily integrated. ● What do you need to know? ○ Where the test lives ○ How to compile and run the tests ○ How to interpret the test output ○ Write a regular expression ○ Submit a job
  15. 15. Guided Tour: Integrating a Test metadata: name: mauve format: "Lava-Test-Shell Test Definition 1.0" description: "Java Mauve Tests for Open Embedded" version: 1.0 install: git-repos: - git://git.linaro.org/qa/test-definitions.git - git://git.linaro.org/people/andrew.mcdermott/mauve/aarch64/mauve.git run: steps: - 'DISPLAY=:10' - 'export DISPLAY' - 'Xvfb $DISPLAY &' - 'XVFBPID=$!' - 'PATH=$PATH:/lava/tests/$TESTRUN_ID/test-definitions/openembedded/scripts' - 'rm -rf $HOME/mauve' - 'ln -s /lava/tests/$TESTRUN_ID/mauve $HOME/mauve' - 'cd $HOME/mauve' - 'lava-test-case java.lang --shell mauve-test-pkg java.lang -showpasses' parse: pattern: "(?P<result>(PASS|FAIL|SKIP)):s+(?P<test_case_id>(.*))" fixupdict: PASS: pass FAIL: fail SKIP: skip
  16. 16. Guided Tour: Integrating a Test SKIP: java.lang.UnsupportedOperationException.classInfo.getMethods SKIP: java.lang.VerifyError.classInfo.getConstructors SKIP: java.lang.VerifyError.classInfo.getDeclaredConstructors SKIP: java.lang.VerifyError.classInfo.getDeclaredFields SKIP: java.lang.VerifyError.classInfo.getDeclaredMethods SKIP: java.lang.VerifyError.classInfo.getFields SKIP: java.lang.VerifyError.classInfo.getMethods [dix] Could not init font path element /usr/share/fonts/X11/misc/, removing from list! [dix] Could not init font path element /usr/share/fonts/X11/TTF/, removing from list! [dix] Could not init font path element /usr/share/fonts/X11/OTF/, removing from list! [dix] Could not init font path element /usr/share/fonts/X11/Type1/, removing from list! [dix] Could not init font path element /usr/share/fonts/X11/100dpi/, removing from list! [dix] Could not init font path element /usr/share/fonts/X11/75dpi/, removing from list! PASS: java.lang.reflect.Modifier.toString PASS: java.lang.reflect.Modifier.toString12 PASS: java.lang.reflect.Field.promotion PASS: java.lang.reflect.Field.toString PASS: java.lang.reflect.Field.access PASS: java.lang.reflect.Array.newInstance PASS: java.lang.reflect.Array.set PASS: java.lang.reflect.Method.invoke PASS: java.lang.reflect.Method.equals PASS: java.lang.reflect.Method.toString PASS: java.lang.reflect.Proxy.ExceptionRaising PASS: java.lang.reflect.Proxy.ToString PASS: java.lang.reflect.Proxy.check13 PASS: java.lang.reflect.Proxy.DeclaringClass PASS: java.lang.reflect.InvocationTargetException.Chain PASS: java.lang.reflect.Constructor.newInstance parse: pattern: "(?P<result>(PASS|FAIL|SKIP)):s+(?P<test_case_id>(.*))" fixupdict: PASS: pass FAIL: fail SKIP: skip
  17. 17. Guided Tour: Integrating a Test Many tests already integrated ● http://git.linaro.org/gitweb?p=qa/test-definitions.git;a=tree ● Folder for distribution specific tests ○ Ubuntu ○ Fedora ○ OpenEmbedded ○ Android ○ Contribution is encouraged. Please feel free to help us integrate!
  18. 18. Guided Tour: Jenkins, Metadata, Filters, Image Reports DEMO
  19. 19. Guided Tour: Override Boot Commands ● When might I need to override the default boot commands? ○ FDT ○ Android ○ OE ○ Ubuntu ○ TFTP ○ PXE ○ UEFI
  20. 20. Guided Tour: Override Boot Commands ● Need to add the boot command options to your device-type configuration or device configuration boot_options = boot_cmds [boot_cmds] default = boot_cmds
  21. 21. Guided Tour: Override Boot Commands ● Now add your new boot commands to the device-type configuration or device configuration boot_cmds_pxe = usb start, dhcp, pxe boot
  22. 22. Guided Tour: Override Boot Commands ● Simply add your new boot options to the job file to override the boot commands. { "command": "boot_linaro_image", "parameters": { "options": ["boot_cmds=boot_cmds_pxe"] } },
  23. 23. Guided Tour: Interactive Boot Commands ● Allows you to specify boot commands in the job file ○ sendline/expect stanzas ■ “sendline foo” - This instructs LAVA to send an “foo” to the bootloader prompt. ■ “expect bar” - This instructs LAVA to expect a “bar” from the bootloader prompt. ○ Traditional boot commands ■ This will simply send the commands to the bootloader and proceed. ○ Rolling out to all devices very shortly.
  24. 24. Guided Tour: Interactive Boot Commands ● How do I use them? ○ First set the parameter "interactive_boot_cmds" to true. ○ Add your sendline/expect stanza to the options field. ■ Each line is enclosed in quotations. ■ All lines are comma separated. ■ No need to quote your kernel arguments. ■ Any backslash must be escaped by ""
  25. 25. Guided Tour: Interactive Boot Commands { "command": "boot_linaro_image", "parameters": { "interactive_boot_cmds": true, "options": ["sendline a", "expect Choice:", "sendline 1", "expect Select the Boot Device:", "sendline 2", "expect File path of the EFI Application or the kernel:", "sendline uImage", "expect [a/g/l]", "sendline l", "expect Add an initrd: [y/n]", "sendline y", "expect File path of the initrd:", "sendline uInitrd", "expect Arguments to pass to the binary:", "sendline console=ttyAMA0,38400n8 root=/dev/mmcblk0p2 rootwait ro mem=1024M earlyprink", "expect File path of the local FDT:", "sendline rtsmrtsm_ve-ca15x4-ca7x4.dtb", "expect Description for this new Entry:", "sendline Test Image", "expect Choice:", "sendline 5", "expect Start:", "sendline 2"] } },
  26. 26. Community: Open Distributed Hardware Lab Idea: Provide hardware as a service to the community ● community.validation.linaro.org ○ Will be open to anyone. ○ Connecting board farms from around the world. ○ Community donated hardware (where you are) ● Want to help out? ○ tyler.baker@linaro.org LAVA Master Instance Databas e Log s Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T
  27. 27. LAVA Master Instance Multi-Machine LAVA Deployment Database Logs Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T Worker Instance D U T D U T D U T ... User Interaction
  28. 28. Discussion: Better LAVA Scalability ● LAVA Dispatcher ○ Proxy all commands over ssh to node. ○ Can use ARM boards as dispatcher proxies. ● TODO: More discussion items.
  29. 29. Discussion: Remote Worker Improvements ● The current issues with the remote workers ○ If the remote worker goes offline, the boards appear online in LAVA and submitted jobs are never scheduled. This is very confusing for the LAVA user. ○ If a remote worker goes offline, LAVA should detect this and offline all associated boards. ○ There needs to be a visual way to know which remote workers are connected or disconnected. ○ Writing log files over sshfs is hacky, there must be a better way. ○ Remote database connection security.
  30. 30. Feedback: LAVA Configuration Improvements ● How can we make LAVA easier to configure? ○ Installation from packages ■ apt-get install lava-server ● apt-get install lava-worker ■ yum install lava-server ● yum install lava-worker ■ Effort in this area underway ● How can we make LAVA easier to distribute? ○ Use SSH to tunnel database connection? ● What else can be done to make a large LAVA lab deployment easier?

×