Testing EmbeddedSoftware<br />John Regehr<br />University of Utah<br />
“Over 15 billion ARM based chips shipped to date”<br />[ARM web site, 2011]<br />“The microcontroller market is forecast t...
3<br />
Diverse!<br />4<br />
Diverse!<br />I have 6 pins and 32 bytes of RAM<br />5<br />
Diverse!<br />6<br />
Diverse!<br />7<br />
Diverse!<br />I am quad core @ 1.5 GHz and have a GPU <br />8<br />
Usually there are multiple processors<br />On-chip networks<br />In-device networks<br />Distributed systems<br />Resource...
Continuously interact with the world through I/O devices<br />May be little abstraction of HW<br />Probably using both int...
Sensor network  -> 103–105 LOC<br />Modern airplane -> 106–107 LOC <br />Hybrid vehicle -> 107–108 LOC<br />How do we get ...
Software on many individual processors is small<br />Permits aggressive analysis and testing<br />Constrained domain simpl...
The “Real System Problem”<br />Many interesting embedded codes are proprietary<br />Necessary tools may be expensive or no...
Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<b...
Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<b...
Arduino<br />Arduino Uno:<br />8-bit AVR processor @ 16 MHz<br />2 KB RAM<br />~$30<br />Emphasis is on interfacing<br />1...
Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded processor<br />18 new books in 2011<br ...
<ul><li> Simulators and model checkers for AVR code exist
 Very few Arduino tool papers exist
 This is a big opportunity</li></ul>Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded pro...
TinyOS<br />OS and middleware support for sensor networks<br />Sensing<br />Collection and dissemination<br />Localization...
TinyOS<br />“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There are a few books<...
TinyOS<br /><ul><li> ~100 tool papers
 Many open problems</li></ul>“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There...
Android<br />OS + middleware for smart phones / tablets<br />ARM based hardware running Linux<br />Much less constrained t...
Android<br />Application code in Java<br />Great tools<br />Tons of books<br />23<br />
Android<br /><ul><li> < 100 tool papers
 Most are very recent
 This is not a scary platform</li></ul>Application code in Java<br />Great tools<br />Tons of books<br />24<br />
ROS – Robot Operating System<br />Linux-based infrastructure for programming robots<br />Primary abstraction is graph of c...
ROS – Robot Operating System<br /><ul><li> Very few ROS tool papers exist</li></ul>Linux-based infrastructure for programm...
Plenty of other open embedded platforms exist<br />FreeRTOS<br />Contiki<br />Pacemaker Challenge<br />Etc.<br />Embarrass...
So, let’s test some embedded software<br />But what are we testing for?<br />28<br />
Properties / Oracles<br />Temporal safety<br />Deadlines<br />Or just responsiveness<br />Memory safety<br />Contracts / a...
Worst-Case Execution Time<br />What is the upper bound on execution time for a piece of code?<br />We care because the wor...
True WCET<br />Number of executions<br />Execution time<br />Conservative WCET<br />Longest observed ET #2<br />Longest ob...
32<br />+<br />=<br />printf()<br />pthread_attr_setstacksize (&attr, &mystacksize);<br />HANDLE WINAPI CreateThread( <br ...
Stack Overflow in TinyOS<br />33<br />4 KB<br />
Stack Overflow in TinyOS<br />34<br />main()<br />4 KB<br />
Stack Overflow in TinyOS<br />35<br />main()<br />irq 0<br />4 KB<br />
Stack Overflow in TinyOS<br />36<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
Stack Overflow in TinyOS<br />37<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
Stack Overflow in TinyOS<br />38<br />Not the same thing as buffer overflow!<br />Type safe language doesn’t solve this pr...
Eliminating Stack Overflow<br />Testing is hard<br />Need to drive code to its WC stack depth<br />Interrupt coincidences ...
Estimate WC stack depth of each sequential flow, handling<br />Indirect branches<br />Recursion<br />Loads into the stack ...
41<br />in      r24, 0x3f    ; r24 <- CPU status <br />                       register<br />cli                  ; disable...
Stack analysis tool deployed in the TinyOS distribution<br />Results are typically much larger than worst observed stack d...
Need… more… oracles…<br />43<br />
TinyOS applications are built using components<br />Interface requirements documented but not checked<br />Interface misus...
We augmented nesC with contracts<br />Dynamic checking reasonable efficient<br />Found some long-standing bugs<br />45<br />
nesC is not type safe<br />Memory safety bugs in TinyOS are difficult<br />We ported an existing safe C dialect<br />Found...
47<br />Code size<br />35%<br />13%<br />-11%<br />
Upcoming SlideShare
Loading in …5
×

Issta11

4,460
-1

Published on

slides from John Regehr's keynote at ISSTA 2011

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,460
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Issta11

  1. 1. Testing EmbeddedSoftware<br />John Regehr<br />University of Utah<br />
  2. 2. “Over 15 billion ARM based chips shipped to date”<br />[ARM web site, 2011]<br />“The microcontroller market is forecast to reach over $16 billion worldwide in 2011”<br />[Microcontroller Market Tracker, 2011]<br />2<br />
  3. 3. 3<br />
  4. 4. Diverse!<br />4<br />
  5. 5. Diverse!<br />I have 6 pins and 32 bytes of RAM<br />5<br />
  6. 6. Diverse!<br />6<br />
  7. 7. Diverse!<br />7<br />
  8. 8. Diverse!<br />I am quad core @ 1.5 GHz and have a GPU <br />8<br />
  9. 9. Usually there are multiple processors<br />On-chip networks<br />In-device networks<br />Distributed systems<br />Resource constraints are…<br />Severe – to minimize unit cost<br />Hard – failure if system runs out of…<br />Time<br />RAM – stack or heap<br />Energy<br />9<br />
  10. 10. Continuously interact with the world through I/O devices<br />May be little abstraction of HW<br />Probably using both interrupt handlers and threads<br />Often there are fault tolerance and security requirements<br />10<br />
  11. 11. Sensor network -> 103–105 LOC<br />Modern airplane -> 106–107 LOC <br />Hybrid vehicle -> 107–108 LOC<br />How do we get these right?<br />Mostly testing<br />11<br />
  12. 12. Software on many individual processors is small<br />Permits aggressive analysis and testing<br />Constrained domain simplifies testing<br />Embedded systems are (by definition) special-purpose devices<br />12<br />
  13. 13. The “Real System Problem”<br />Many interesting embedded codes are proprietary<br />Necessary tools may be expensive or nonexistent<br />Compilers, debuggers, simulators<br />May not be able to run it in the lab<br />Often lacks specifications and oracles<br />13<br />
  14. 14. Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<br />14<br />
  15. 15. Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<br />15<br />Solution: <br />Ubiquitous open embedded platforms<br />
  16. 16. Arduino<br />Arduino Uno:<br />8-bit AVR processor @ 16 MHz<br />2 KB RAM<br />~$30<br />Emphasis is on interfacing<br />16<br />
  17. 17. Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded processor<br />18 new books in 2011<br />17<br />
  18. 18. <ul><li> Simulators and model checkers for AVR code exist
  19. 19. Very few Arduino tool papers exist
  20. 20. This is a big opportunity</li></ul>Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded processor<br />18 new books in 2011<br />18<br />
  21. 21. TinyOS<br />OS and middleware support for sensor networks<br />Sensing<br />Collection and dissemination<br />Localization<br />Applications are in nesC, a C dialect<br />19<br />
  22. 22. TinyOS<br />“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There are a few books<br />20<br />
  23. 23. TinyOS<br /><ul><li> ~100 tool papers
  24. 24. Many open problems</li></ul>“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There are a few books<br />21<br />
  25. 25. Android<br />OS + middleware for smart phones / tablets<br />ARM based hardware running Linux<br />Much less constrained than motes and Arduino<br />22<br />
  26. 26. Android<br />Application code in Java<br />Great tools<br />Tons of books<br />23<br />
  27. 27. Android<br /><ul><li> < 100 tool papers
  28. 28. Most are very recent
  29. 29. This is not a scary platform</li></ul>Application code in Java<br />Great tools<br />Tons of books<br />24<br />
  30. 30. ROS – Robot Operating System<br />Linux-based infrastructure for programming robots<br />Primary abstraction is graph of communicating processes<br />Local and distributed<br />25<br />
  31. 31. ROS – Robot Operating System<br /><ul><li> Very few ROS tool papers exist</li></ul>Linux-based infrastructure for programming robots<br />Primary abstraction is graph of communicating processes<br />Local and distributed<br />26<br />
  32. 32. Plenty of other open embedded platforms exist<br />FreeRTOS<br />Contiki<br />Pacemaker Challenge<br />Etc.<br />Embarrassment of riches<br />Still, huge room for improvement<br />Where’s the open automobile?<br />27<br />
  33. 33. So, let’s test some embedded software<br />But what are we testing for?<br />28<br />
  34. 34. Properties / Oracles<br />Temporal safety<br />Deadlines<br />Or just responsiveness<br />Memory safety<br />Contracts / assertions<br />Reference implementation<br />29<br />
  35. 35. Worst-Case Execution Time<br />What is the upper bound on execution time for a piece of code?<br />We care because the world has deadlines<br />Static analysis of WCET is extremely difficult if there is…<br />A cache<br />Preemption<br />An aggressive processor<br />30<br />
  36. 36. True WCET<br />Number of executions<br />Execution time<br />Conservative WCET<br />Longest observed ET #2<br />Longest observed ET #1<br />31<br />
  37. 37. 32<br />+<br />=<br />printf()<br />pthread_attr_setstacksize (&attr, &mystacksize);<br />HANDLE WINAPI CreateThread( <br /> LPSECURITY_ATTRIBUTES lpThreadAttributes, <br /> SIZE_T dwStackSize, <br /> LPTHREAD_START_ROUTINE lpStartAddress, <br /> LPVOID lpParameter, <br /> DWORD dwCreationFlags, <br /> LPDWORD lpThreadId );<br />
  38. 38. Stack Overflow in TinyOS<br />33<br />4 KB<br />
  39. 39. Stack Overflow in TinyOS<br />34<br />main()<br />4 KB<br />
  40. 40. Stack Overflow in TinyOS<br />35<br />main()<br />irq 0<br />4 KB<br />
  41. 41. Stack Overflow in TinyOS<br />36<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  42. 42. Stack Overflow in TinyOS<br />37<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  43. 43. Stack Overflow in TinyOS<br />38<br />Not the same thing as buffer overflow!<br />Type safe language doesn’t solve this problem<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  44. 44. Eliminating Stack Overflow<br />Testing is hard<br />Need to drive code to its WC stack depth<br />Interrupt coincidences are rare<br />Approach: Static analysis of compiled code<br />Can’t estimate stack depth of source<br />39<br />
  45. 45. Estimate WC stack depth of each sequential flow, handling<br />Indirect branches<br />Recursion<br />Loads into the stack pointer<br />Compute “interrupt preemption graph”<br />Find longest cycle in this graph<br />40<br />
  46. 46. 41<br />in r24, 0x3f ; r24 <- CPU status <br /> register<br />cli ; disable interrupts<br />adc r24, r24 ; carry bit <- prev<br /> interrupt status<br />eor r24, r24 ; r24 <- 0<br />adc r24, r24 ; r24 <- carry bit<br />mov r18, r24 ; r18 <- r24<br />... critical section ...<br />and r18, r18 ; test r18 for zero<br />breq .+2 ; if zero, skip next <br /> instruction<br />sei ; enable interrupts<br />ret ; return from function<br />
  47. 47. Stack analysis tool deployed in the TinyOS distribution<br />Results are typically much larger than worst observed stack depths<br />But, we validated its results by randomly firing interrupts<br />42<br />
  48. 48. Need… more… oracles…<br />43<br />
  49. 49. TinyOS applications are built using components<br />Interface requirements documented but not checked<br />Interface misuse often silent<br />44<br />
  50. 50. We augmented nesC with contracts<br />Dynamic checking reasonable efficient<br />Found some long-standing bugs<br />45<br />
  51. 51. nesC is not type safe<br />Memory safety bugs in TinyOS are difficult<br />We ported an existing safe C dialect<br />Found some otherwise-impossible bugs<br />Main problem was getting overhead under control<br />Whole-program optimization<br />46<br />
  52. 52. 47<br />Code size<br />35%<br />13%<br />-11%<br />
  53. 53. 48<br />Increasing Availability<br />Normal TinyOS:<br />0% average <br />availability<br />Array <br />Out-of-bounds<br />Normal<br />TinyOS<br />Safe TinyOS:<br />95% average <br />availability<br />Array <br />Out-of-bounds<br />Rebuild<br />Soft state<br />Safe<br />TinyOS<br />Reboot<br />
  54. 54. What about application-level sensornet properties?<br />All the interesting ones are distributed<br />We adapted TOSSIM, a non-cycle-accurate simulator, to be…<br />A random tester<br />A depth-bounded model checker<br />Oracles: <br />Type safety checks<br />Application-level properties<br />49<br />
  55. 55. Application-Level Properties<br />Eventually…<br />Each send buffer is unlocked<br />No cycles in the routing tree<br />All nodes become part of the collection tree<br />All nodes have consistent values<br />6 out of 8 of these properties require global knowledge<br />50<br />
  56. 56. Found 12 previously unknown bugs in TinyOS 2.0<br />10 safety, 2 liveness<br />Random testing outperformed depth-bounded model checking<br />Even after a lot of work on POR<br />But required work to shorten long error traces<br />51<br />
  57. 57. Conclusions<br />Open embedded platforms exist<br />Some have steep learning curves<br />Finding oracles is hard<br />Generating valid input is hard<br />Embedded systems are fun and important and rewarding<br />52<br />

×