Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Remote testing environment workshop

36 views

Published on



As PC Engines firmware maintainers and aspiring IBV, we face lots of different issues with development. Critical bugs and regressions are our biggest enemies. Narrowing down serious problem requires a lot of repetitive work. The typical process looks like that:

1. modify & build firmware with a potential fix
2. flash affected firmware version
3. boot system
4. verify if the bug still exists

This can take more than 10min and requires human attention not to make a mistake. If you work hours on one issue, this can be boring and at some point, you will make mistakes which can affect results. Automation of point 1 is already done by coreboot-sdk. It also can involve user-developed scripts. Points 2-4 can be automated using RTE.

Another issue is that platforms and quality firmware developers are spread around the world and we would like to have reasonable access to key features that help in remote development. Because of that, we created RTE.

Remote Testing Environment is an open hardware hat designed for Orange Pi Zero board to work with hardware remotely. It takes advantage of using Linux system and open source frameworks such as Robot Framework to let user conveniently develop firmware, test software and/or hardware. With RTE, your everyday work routines become much faster and easier to maintain from places not related to current setup location. Those include flashing firmware, controlling GPIOs and power management for Device Under Test. RTE is released under CERN Open Hardware
License.

RTE applications:
* Remote access / control (hardware functionality)
* Remote industrial application debugging
* Compatibility testing
* Platform enabling
* Performance optimization
* Automation validation
* Manufacturing procedures support

Agenda
-------------

In this workshop, we want to present how RTE can be used with open source firmware and help R&D and Q&A teams to achieve their goals.

RTE workshop main points:

1. RTE features set.
2. Connecting RTE to real hardware (e.g. PC Engines apu2 or MinnowBoard
Dual/Quad).
* validation setup - run a simple test in RobotFramework
3. Comparison of simple procedures:
* firmware flashing (main goal)
* present power control possibilities (direct GPIO and Legacy interface)
4. Development workflow with RTE based on the simple bug.
5. Questionnaire and RTE giveaway.

Additional information
----------------------

* Level: Intermediate
* Workshop will be interactive with examples
* There will be a draw between participants, 3mdeb gives away 3 full RTE
platforms (worth 85 USD each)
* We are not sure we can handle more than 8 attendees
* Please, check out our [GitHub](https://github.com/3mdeb/rte-schematics)
repository.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Remote testing environment workshop

  1. 1. Remote Testing Environment Workshop OSFC 2018 Piotr Król, Michał Żygowski 1 / 41
  2. 2. Introduction Firmware developer routines Remote Testing Environment Infrastructure Hardware setup Automation Tests Fixing bugs Future plans RTE Control API (beta) New RTE revision for RPi Q&A and RTE giveaway Agenda 2 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  3. 3. Piotr Król Michał Żygowski Founder & Embedded Systems Consultant Firmware Engineer michal.zygowski@3mdeb.com linkedin.com/in/michał- żygowski-88954416b @pietrushnic piotr.krol@3mdeb.com linkedin.com/in/krolpiotr facebook.com/piotr.krol.756859 Introduction 3 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  4. 4. Narrowing down serious issue requires a lot of repetitive work: modify & build firmware with potential fix flash affected firmware version boot system verify if the bug still exists Problem with automation and remote development To get rid of these problems we created Remote Testing Environment. Firmware developer routines 4 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  5. 5. Remote Testing Environment 5 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  6. 6. Today, our Devices Under Test are PC Engines APU platforms. For workshop purposes we will need: Additional infrastructure that can be used is Debian machine with iPXE server for supplementary boot option. Infrastructure 6 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  7. 7. connect RTE Hat with Orange Pi Zero board (~3min) Hardware setup 7 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  8. 8. RS232 interface via a null modem cable (~2min) Hardware setup 8 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  9. 9. SPI interface via IDC 2x4 pin cable (~3min) Hardware setup 9 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  10. 10. APU PWR/RST pins with RTE OC buffers (~5min) Hardware setup 10 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  11. 11. DC-DC jack cable (~2min) Hardware setup 11 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  12. 12. take the connected setup to the power/network section (~10min): power up RTE and APU platforms connect Ethernet cable to Orange Pi Zero Hardware setup 12 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  13. 13. Generic test automation framework for acceptance test-driven development (ATDD) Utilizes the keyword-driven testing approach The core is implemented using Python and runs also on Jython (JVM) and IronPython (.NET) Bug fixes need tests too - Test-Driven Bug Fixing (TDBF) Automation tests - RobotFramework 13 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  14. 14. Each test suite should contain 4 sections: *** Settings *** Library SSHLibrary Resource robotframework/variables.robot Suite Setup SSH Connection and Log In ${rte_ip} Suite Teardown Log Out And Close Connections *** Variables *** ${username} root ${password} meta-rte *** Keywords *** Log Out And Close Connections [Documentation] Close all telnet and ssh open connections. Telnet.Close All Connections SSHLibrary.Close All Connections *** Test Cases *** RTE: 1.1 SSH connection ${ssh_info}= SSHLibrary.Get Connection Should Be Equal As Strings ${ssh_info.host} ${rte_ip} Automation tests - test suite structure 14 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  15. 15. First, prepare the environment (~10min): Automation tests - setup environment 15 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  16. 16. First, prepare the environment (~10min): start Debian RTE Workshop VBox and login password: rte-workshop Automation tests - setup environment 16 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  17. 17. First, prepare the environment (~10min): start Debian RTE Workshop VBox and login password: rte-workshop open terminal (Activities/Terminal) Automation tests - setup environment 17 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  18. 18. First, prepare the environment (~10min): start Debian RTE Workshop VBox and login password: rte-workshop open terminal (Activities/Terminal) change directory: cd RTE-Workshop Automation tests - setup environment 18 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  19. 19. First, prepare the environment (~10min): start Debian RTE Workshop VBox and login password: rte-workshop open terminal (Activities/Terminal) change directory: cd RTE-Workshop activate virtualenv: source robot-venv/bin/activate Automation tests - setup environment 19 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  20. 20. RTE-Workshop directory tree 20 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  21. 21. Now we should run script which tests our hardware setup (~5min): Connection validation - part 1 21 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  22. 22. Now we should run script which tests our hardware setup (~5min): robot -v rte_ip:RTE_IP validation.robot where RTE_IP is your RTE IP address Connection validation - part 1 22 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  23. 23. Now we should run script which tests our hardware setup (~5min): robot -v rte_ip:RTE_IP validation.robot where RTE_IP is your RTE IP address Connection validation - part 1 23 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  24. 24. Connection validation - part 2 24 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  25. 25. Connection validation - part 3 25 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  26. 26. Possible failures: ssh timeout - RTE lacks network connection NoValidConnections - bad IP address Telnet timeout - check RS232 cable or close established connection Automation tests - logs 26 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  27. 27. Possible failures: ssh timeout - RTE lacks network connection NoValidConnections - bad IP address Telnet timeout - check RS232 cable or close established connection After each RobotFramework test suite we can track generated log files: Try it by yourself by typing: firefox log.html report.html output.xml Automation tests - logs 27 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  28. 28. Flashing process can be started by flash-apu.robot script (~5min): robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot remember to change fw_file firmware path for appropriate APU2/3/4 binary (if you are not sure, check the sign of life from previous tests) Flashing firmware - automated process 28 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  29. 29. Flashing process can be started by flash-apu.robot script (~5min): robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot remember to change fw_file firmware path for appropriate APU2/3/4 binary (if you are not sure, check the sign of life from previous tests) Possible failures: opening "/tmp/coreboot.rom" failed - SSH key not authenticated No such file or directory - bad firmware file path Flashing firmware - automated process 29 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  30. 30. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: Fixing bugs (~10min) 30 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  31. 31. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: I. First, let's try to update our DUT with custom firmware: Fixing bugs (~10min) 31 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  32. 32. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: I. First, let's try to update our DUT with custom firmware: run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash- apu.robot Fixing bugs (~10min) 32 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  33. 33. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: I. First, let's try to update our DUT with custom firmware: run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash- apu.robot Fixing bugs (~10min) 33 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  34. 34. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: II. Assumes that we made appropriate changes and build the new binary, flash APU with the newest v4.8.0.3 firmware: Fixing bugs (~10min) 34 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  35. 35. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: II. Assumes that we made appropriate changes and build the new binary, flash APU with the newest v4.8.0.3 firmware: run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.rom flash-apu.robot Fixing bugs (~10min) 35 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  36. 36. We have prepared 2 binaries to present the RTE and RobotFramework capability to track down unexpected bugs and flash APUs firmware: II. Assumes that we made appropriate changes and build the new binary, flash APU with the newest v4.8.0.3 firmware: run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.rom flash-apu.robot Fixing bugs (~10min) 36 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  37. 37. To open rest API service, type: firefox RTE_IP:8000 (~3min) Our plans - RTE Control API (beta) 37 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  38. 38. New RTE Hat compatible with 40-pin header RaspberryPi 2/3/Zero Our plans - New revision for RPi 38 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  39. 39. New revision during testing phase Our plans - New revision for RPi 39 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  40. 40. RTE website & shop 40 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski
  41. 41. Q&A and lottery Cards received at the entrance will serve as numbers to select the winner of the lottery. The prizes are three full RTE sets! Where can I order the RTE for myself? The online store is starting soon, so please submit any orders to: contact@3mdeb.com Please visit our RTE website: 3mdeb.com/rte 41 / 41 OSFC 2018 CC BY 4.0 | Piotr Król, Michał Żygowski

×