Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
7th WhiteRabbit Workshop - November 2012
Agenda

About Igalia
Virtual vs Real
Linux support and making of FMC TDC board
Testing, Virtual Hardware and WhiteRabbit
T...
Who is Igalia?

Free and open software company
50 hackers around the globe
Hacking in different areas
kernel
virtualizatio...
Virtual vs Real

Using virtual HW is all about having the possibility to test
low level software, such as firmwares, HDL co...
Linux support and making of FMC TDC board

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC Linux support

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC Linux support

Time-To-Digital-Converter (TDC) board
Used for measuring the time of arrival of each incoming
pulse...
Making of FMC TDC Linux support

We face the typical issues when developing drivers for non
mature hardware
Hardware not a...
Making of FMC TDC Linux support
In order to minimize the previous issues we explored the
virtual way with the following re...
Testing, Virtual Hardware and WhiteRabbit

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing, Virtual Hardware and WhiteRabbit
Some thoughs ...
It is not possible catching all bugs in a complex system
A comp...
Testing, Virtual Hardware and WhiteRabbit

Potential WhiteRabbits components to test
Tools and compilers (ARM, LM32)
Builr...
Testing, Virtual Hardware and WhiteRabbit
Testing with Virtual Hardware would be possible on the
following components
No v...
Technical details and demo

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Our work

Virtualized models
Industry Pack
TEWS TPCI-200.
GE IP-OCTAL-232.

FMC TDC
SPEC board (only needed bits)
FMC TDC ...
FMC TDC board

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC board
It was needed to test error conditions on the driver
Generic stuff
The virtual board in being detected by th...
What was virtualized?

SPEC
PCI: (sub)Vendor ID, (sub)Device ID. No MSI support.
Memory Map: BAR addresses.

FMC TDC Firmw...
How we did that

QEMU (http://wiki.qemu.org)
Open source machine emulator and virtualizer.

Buildbot (http://trac.buildbot...
Testing suite design

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing suite design
Launch QEMU with some setups:
Choose memory size, virtual machine image, network, etc.
Snapshot mode....
The result (I)

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (II)

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (III)

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (IV)

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Potential improvements

Improve the virtual model.
Do a more realistic model of the device.
Add VHDL virtualization.

QEMU...
Testing and HW Virtualization

Questions & Answers

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias G...
Testing and HW Virtualization

Thank you!

Testing and HW Virtualization

Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Links

http://www.ohwr.org/projects/spec
http://www.ohwr.org/projects/fmc-tdc
http://www.ohwr.org/projects/fmc-tdc-sw
http...
Upcoming SlideShare
Loading in...5
×

Testing and HW Virtualization (7th White Rabbit Workshop)

139

Published on

By Javier Muñoz and Samuel Iglesias.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
139
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Testing and HW Virtualization (7th White Rabbit Workshop)

  1. 1. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez 7th WhiteRabbit Workshop - November 2012
  2. 2. Agenda About Igalia Virtual vs Real Linux support and making of FMC TDC board Testing, Virtual Hardware and WhiteRabbit Technical details and demo Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  3. 3. Who is Igalia? Free and open software company 50 hackers around the globe Hacking in different areas kernel virtualization graphics/rendering compilers distros ... www.igalia.com Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  4. 4. Virtual vs Real Using virtual HW is all about having the possibility to test low level software, such as firmwares, HDL cores or kernel/drivers, without real HW. It is not about replacing real hardware at all. The idea is analysing every test case on real hardware in order to know if we can run it on virtual hardware. This approach saves resources and time. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  5. 5. Linux support and making of FMC TDC board Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  6. 6. FMC TDC Linux support Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  7. 7. FMC TDC Linux support Time-To-Digital-Converter (TDC) board Used for measuring the time of arrival of each incoming pulse in order to save the timestamp into the memory, ready to be read later CERN/Igalia effort to support FMC-TDC board in Linux drivers, user-space library and test cases developed Integration with ZIO framework, use FMC bus and SPEC drivers Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  8. 8. Making of FMC TDC Linux support We face the typical issues when developing drivers for non mature hardware Hardware not available Buggy firmwares delaying development Is it a hardware or software bug? Where is it? Flashing firmware, running tests... are time-intensive tasks Hardware specification evolving along the project ... Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  9. 9. Making of FMC TDC Linux support In order to minimize the previous issues we explored the virtual way with the following results ... No real hardware bottle-neck, parallel development No buggy firmwares issues Got a virtual baseline in order to test old and new specifications/firmwares, bug identification really easy Saved time in real flashing and testing approaches Developed continuous integration and automatic testing tooling for software (kernel, libraries, etc) on top of virtual hardware Got good-enough virtual models (buses, I/O spaces, memory maps, etc) to cover goals. Further emulation (VHDL, firmware, etc) possible but it wasn’t required for this project Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  10. 10. Testing, Virtual Hardware and WhiteRabbit Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  11. 11. Testing, Virtual Hardware and WhiteRabbit Some thoughs ... It is not possible catching all bugs in a complex system A complex system based on hardware/software require different, but integrated, testing approaches Pragmatic testing is a must. Keep all possible testing in the soft side. It is more affordable, fast and flexible It doesn’t matter where the bug is (software or hardware). If it appears in a critical system then it is a critical bug In the long, continuous integration and automatic testing is the way Flexible and robust testing approach is key to get great releases At the end, final testing should always run in the real system Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  12. 12. Testing, Virtual Hardware and WhiteRabbit Potential WhiteRabbits components to test Tools and compilers (ARM, LM32) Builroot Barebox (bootloader) IPL Linux kernel and drivers (versions, configurations, etc) User space applications (ptpd, hal, libs, etc) VHDL and LM32 binaries Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  13. 13. Testing, Virtual Hardware and WhiteRabbit Testing with Virtual Hardware would be possible on the following components No virtual Hardware required Tools and compilers (ARM, LM32) Builroot Virtual Hardware required Linux kernel and drivers code (similar FMC TDC approach) LM32 binaries (real-time) VHDL (behaviour) Not explored Barebox (bootloader) IPL User space applications (ptpd, hal, libs, etc) Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  14. 14. Technical details and demo Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  15. 15. Our work Virtualized models Industry Pack TEWS TPCI-200. GE IP-OCTAL-232. FMC TDC SPEC board (only needed bits) FMC TDC board Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  16. 16. FMC TDC board Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  17. 17. FMC TDC board It was needed to test error conditions on the driver Generic stuff The virtual board in being detected by the driver as a genuine device. Normal mode: no error injection. DMA DMA error: TDC_DMA_STAT_ERR and TDC_DMA_STAT_ABORT. DMA timeout. Input pulses Filter events with pulse width lower than 100 ns (requirement). Test different input pulse arrival rate. Test timestamp threshold setup. Test time threshold setup. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  18. 18. What was virtualized? SPEC PCI: (sub)Vendor ID, (sub)Device ID. No MSI support. Memory Map: BAR addresses. FMC TDC Firmware Memory map: status and configuration registers, Event’s circular buffer. DMA core: memory map, registers. Input pulse generation Different parameters setup: input pulse arrival rate, pulse width, etc. Error injection As it was previously mentioned. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  19. 19. How we did that QEMU (http://wiki.qemu.org) Open source machine emulator and virtualizer. Buildbot (http://trac.buildbot.net/) Continuous integration system designed to automate the build/test cycle. Python. Web interface to check the logs. Our own testing suite Based on Sam’s experience working with PTS in the past. Needed more flexibility: different setups for the same test. Developed in Python, as PTS. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  20. 20. Testing suite design Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  21. 21. Testing suite design Launch QEMU with some setups: Choose memory size, virtual machine image, network, etc. Snapshot mode. Shared folders to interchange data. No screen. The kernel is pass as a parameter to QEMU -> Not using the one in the virtual machine. Timeout. If QEMU is still running after a given time, something wrong happened. Forward kernel logs to QEMU’s serial port. We can read dmesg info from host. Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  22. 22. The result (I) Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  23. 23. The result (II) Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  24. 24. The result (III) Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  25. 25. The result (IV) Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  26. 26. Potential improvements Improve the virtual model. Do a more realistic model of the device. Add VHDL virtualization. QEMU configuration: use config files? Buildbot: fix issues. Testing suite Add new features (hooks, etc). Choose Red Hat’s autotest framework? Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  27. 27. Testing and HW Virtualization Questions & Answers Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  28. 28. Testing and HW Virtualization Thank you! Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  29. 29. Links http://www.ohwr.org/projects/spec http://www.ohwr.org/projects/fmc-tdc http://www.ohwr.org/projects/fmc-tdc-sw http://blogs.igalia.com/berto/2012/10/03/ industrypack-qemu-and-linuxcon/ http://events.linuxfoundation.org/images/ stories/pdf/lceu2012_garcia.pdf Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×