The document discusses testing embedded software and highlights several challenges and opportunities. It notes that while embedded systems are diverse, there are now many open hardware platforms like Arduino, TinyOS, Android and ROS that enable more academic research. However, properties to test for are difficult to define as embedded systems interact continuously with the physical world. The document advocates using tools like static analysis, contracts and model checking to test for stack overflow, interface correctness, memory safety and application-level distributed properties. Many bugs were found this way in existing systems like TinyOS.

1. Testing EmbeddedSoftwareJohn RegehrUniversity of Utah

2. “Over 15 billion ARM based chips shipped to date”[ARM web site, 2011]“The microcontroller market is forecast to reach over $16 billion worldwide in 2011”[Microcontroller Market Tracker, 2011]2

3. 3

4. Diverse!4

5. Diverse!I have 6 pins and 32 bytes of RAM5

6. Diverse!6

7. Diverse!7

8. Diverse!I am quad core @ 1.5 GHz and have a GPU 8

9. Usually there are multiple processorsOn-chip networksIn-device networksDistributed systemsResource constraints are…Severe – to minimize unit costHard – failure if system runs out of…TimeRAM – stack or heapEnergy9

10. Continuously interact with the world through I/O devicesMay be little abstraction of HWProbably using both interrupt handlers and threadsOften there are fault tolerance and security requirements10

11. Sensor network -> 103–105 LOCModern airplane -> 106–107 LOC Hybrid vehicle -> 107–108 LOCHow do we get these right?Mostly testing11

12. Software on many individual processors is smallPermits aggressive analysis and testingConstrained domain simplifies testingEmbedded systems are (by definition) special-purpose devices12

13. The “Real System Problem”Many interesting embedded codes are proprietaryNecessary tools may be expensive or nonexistentCompilers, debuggers, simulatorsMay not be able to run it in the labOften lacks specifications and oracles13

14. Consequently, academic embedded work may be…Forced to use small, contrived examplesOut of tune with industry14

15. Consequently, academic embedded work may be…Forced to use small, contrived examplesOut of tune with industry15Solution: Ubiquitous open embedded platforms

16. ArduinoArduino Uno:8-bit AVR processor @ 16 MHz2 KB RAM~$30Emphasis is on interfacing16

17. ArduinoNice IDE + libraries + C/C++Minimal abstraction of the embedded processor18 new books in 201117

18. Simulators and model checkers for AVR code exist

19. Very few Arduino tool papers exist

20. This is a big opportunityArduinoNice IDE + libraries + C/C++Minimal abstraction of the embedded processor18 new books in 201118

21. TinyOSOS and middleware support for sensor networksSensingCollection and disseminationLocalizationApplications are in nesC, a C dialect19

22. TinyOS“Motes” based on a variety of MCUsCost $50 – $200Good simulators existThere are a few books20

23. TinyOS ~100 tool papers

24. Many open problems“Motes” based on a variety of MCUsCost $50 – $200Good simulators existThere are a few books21

25. AndroidOS + middleware for smart phones / tabletsARM based hardware running LinuxMuch less constrained than motes and Arduino22

26. AndroidApplication code in JavaGreat toolsTons of books23

27. Android < 100 tool papers

28. Most are very recent

29. This is not a scary platformApplication code in JavaGreat toolsTons of books24

30. ROS – Robot Operating SystemLinux-based infrastructure for programming robotsPrimary abstraction is graph of communicating processesLocal and distributed25

31. ROS – Robot Operating System Very few ROS tool papers existLinux-based infrastructure for programming robotsPrimary abstraction is graph of communicating processesLocal and distributed26

32. Plenty of other open embedded platforms existFreeRTOSContikiPacemaker ChallengeEtc.Embarrassment of richesStill, huge room for improvementWhere’s the open automobile?27

33. So, let’s test some embedded softwareBut what are we testing for?28

34. Properties / OraclesTemporal safetyDeadlinesOr just responsivenessMemory safetyContracts / assertionsReference implementation29

35. Worst-Case Execution TimeWhat is the upper bound on execution time for a piece of code?We care because the world has deadlinesStatic analysis of WCET is extremely difficult if there is…A cachePreemptionAn aggressive processor30

36. True WCETNumber of executionsExecution timeConservative WCETLongest observed ET #2Longest observed ET #131

37. 32+=printf()pthread_attr_setstacksize (&attr, &mystacksize);HANDLE WINAPI CreateThread( LPSECURITY_ATTRIBUTES lpThreadAttributes, SIZE_T dwStackSize, LPTHREAD_START_ROUTINE lpStartAddress, LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD lpThreadId );

38. Stack Overflow in TinyOS334 KB

39. Stack Overflow in TinyOS34main()4 KB

40. Stack Overflow in TinyOS35main()irq 04 KB

41. Stack Overflow in TinyOS36main()irq 04 KBirq 1

42. Stack Overflow in TinyOS37main()irq 04 KBirq 1

43. Stack Overflow in TinyOS38Not the same thing as buffer overflow!Type safe language doesn’t solve this problemmain()irq 04 KBirq 1

44. Eliminating Stack OverflowTesting is hardNeed to drive code to its WC stack depthInterrupt coincidences are rareApproach: Static analysis of compiled codeCan’t estimate stack depth of source39

45. Estimate WC stack depth of each sequential flow, handlingIndirect branchesRecursionLoads into the stack pointerCompute “interrupt preemption graph”Find longest cycle in this graph40

46. 41in r24, 0x3f ; r24 <- CPU status registercli ; disable interruptsadc r24, r24 ; carry bit <- prev interrupt statuseor r24, r24 ; r24 <- 0adc r24, r24 ; r24 <- carry bitmov r18, r24 ; r18 <- r24... critical section ...and r18, r18 ; test r18 for zerobreq .+2 ; if zero, skip next instructionsei ; enable interruptsret ; return from function

47. Stack analysis tool deployed in the TinyOS distributionResults are typically much larger than worst observed stack depthsBut, we validated its results by randomly firing interrupts42

48. Need… more… oracles…43

49. TinyOS applications are built using componentsInterface requirements documented but not checkedInterface misuse often silent44

50. We augmented nesC with contractsDynamic checking reasonable efficientFound some long-standing bugs45

51. nesC is not type safeMemory safety bugs in TinyOS are difficultWe ported an existing safe C dialectFound some otherwise-impossible bugsMain problem was getting overhead under controlWhole-program optimization46

52. 47Code size35%13%-11%

53. 48Increasing AvailabilityNormal TinyOS:0% average availabilityArray Out-of-boundsNormalTinyOSSafe TinyOS:95% average availabilityArray Out-of-boundsRebuildSoft stateSafeTinyOSReboot

54. What about application-level sensornet properties?All the interesting ones are distributedWe adapted TOSSIM, a non-cycle-accurate simulator, to be…A random testerA depth-bounded model checkerOracles: Type safety checksApplication-level properties49

55. Application-Level PropertiesEventually…Each send buffer is unlockedNo cycles in the routing treeAll nodes become part of the collection treeAll nodes have consistent values6 out of 8 of these properties require global knowledge50

56. Found 12 previously unknown bugs in TinyOS 2.010 safety, 2 livenessRandom testing outperformed depth-bounded model checkingEven after a lot of work on PORBut required work to shorten long error traces51

57. ConclusionsOpen embedded platforms existSome have steep learning curvesFinding oracles is hardGenerating valid input is hardEmbedded systems are fun and important and rewarding52