Your SlideShare is downloading. ×
0
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Issta11
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Issta11

4,404

Published on

slides from John Regehr's keynote at ISSTA 2011

slides from John Regehr's keynote at ISSTA 2011

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,404
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Testing EmbeddedSoftware<br />John Regehr<br />University of Utah<br />
  • 2. “Over 15 billion ARM based chips shipped to date”<br />[ARM web site, 2011]<br />“The microcontroller market is forecast to reach over $16 billion worldwide in 2011”<br />[Microcontroller Market Tracker, 2011]<br />2<br />
  • 3. 3<br />
  • 4. Diverse!<br />4<br />
  • 5. Diverse!<br />I have 6 pins and 32 bytes of RAM<br />5<br />
  • 6. Diverse!<br />6<br />
  • 7. Diverse!<br />7<br />
  • 8. Diverse!<br />I am quad core @ 1.5 GHz and have a GPU <br />8<br />
  • 9. Usually there are multiple processors<br />On-chip networks<br />In-device networks<br />Distributed systems<br />Resource constraints are…<br />Severe – to minimize unit cost<br />Hard – failure if system runs out of…<br />Time<br />RAM – stack or heap<br />Energy<br />9<br />
  • 10. Continuously interact with the world through I/O devices<br />May be little abstraction of HW<br />Probably using both interrupt handlers and threads<br />Often there are fault tolerance and security requirements<br />10<br />
  • 11. Sensor network -> 103–105 LOC<br />Modern airplane -> 106–107 LOC <br />Hybrid vehicle -> 107–108 LOC<br />How do we get these right?<br />Mostly testing<br />11<br />
  • 12. Software on many individual processors is small<br />Permits aggressive analysis and testing<br />Constrained domain simplifies testing<br />Embedded systems are (by definition) special-purpose devices<br />12<br />
  • 13. The “Real System Problem”<br />Many interesting embedded codes are proprietary<br />Necessary tools may be expensive or nonexistent<br />Compilers, debuggers, simulators<br />May not be able to run it in the lab<br />Often lacks specifications and oracles<br />13<br />
  • 14. Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<br />14<br />
  • 15. Consequently, academic embedded work may be…<br />Forced to use small, contrived examples<br />Out of tune with industry<br />15<br />Solution: <br />Ubiquitous open embedded platforms<br />
  • 16. Arduino<br />Arduino Uno:<br />8-bit AVR processor @ 16 MHz<br />2 KB RAM<br />~$30<br />Emphasis is on interfacing<br />16<br />
  • 17. Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded processor<br />18 new books in 2011<br />17<br />
  • 18. <ul><li> Simulators and model checkers for AVR code exist
  • 19. Very few Arduino tool papers exist
  • 20. This is a big opportunity</li></ul>Arduino<br />Nice IDE + libraries + C/C++<br />Minimal abstraction of the embedded processor<br />18 new books in 2011<br />18<br />
  • 21. TinyOS<br />OS and middleware support for sensor networks<br />Sensing<br />Collection and dissemination<br />Localization<br />Applications are in nesC, a C dialect<br />19<br />
  • 22. TinyOS<br />“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There are a few books<br />20<br />
  • 23. TinyOS<br /><ul><li> ~100 tool papers
  • 24. Many open problems</li></ul>“Motes” based on a variety of MCUs<br />Cost $50 – $200<br />Good simulators exist<br />There are a few books<br />21<br />
  • 25. Android<br />OS + middleware for smart phones / tablets<br />ARM based hardware running Linux<br />Much less constrained than motes and Arduino<br />22<br />
  • 26. Android<br />Application code in Java<br />Great tools<br />Tons of books<br />23<br />
  • 27. Android<br /><ul><li> < 100 tool papers
  • 28. Most are very recent
  • 29. This is not a scary platform</li></ul>Application code in Java<br />Great tools<br />Tons of books<br />24<br />
  • 30. ROS – Robot Operating System<br />Linux-based infrastructure for programming robots<br />Primary abstraction is graph of communicating processes<br />Local and distributed<br />25<br />
  • 31. ROS – Robot Operating System<br /><ul><li> Very few ROS tool papers exist</li></ul>Linux-based infrastructure for programming robots<br />Primary abstraction is graph of communicating processes<br />Local and distributed<br />26<br />
  • 32. Plenty of other open embedded platforms exist<br />FreeRTOS<br />Contiki<br />Pacemaker Challenge<br />Etc.<br />Embarrassment of riches<br />Still, huge room for improvement<br />Where’s the open automobile?<br />27<br />
  • 33. So, let’s test some embedded software<br />But what are we testing for?<br />28<br />
  • 34. Properties / Oracles<br />Temporal safety<br />Deadlines<br />Or just responsiveness<br />Memory safety<br />Contracts / assertions<br />Reference implementation<br />29<br />
  • 35. Worst-Case Execution Time<br />What is the upper bound on execution time for a piece of code?<br />We care because the world has deadlines<br />Static analysis of WCET is extremely difficult if there is…<br />A cache<br />Preemption<br />An aggressive processor<br />30<br />
  • 36. True WCET<br />Number of executions<br />Execution time<br />Conservative WCET<br />Longest observed ET #2<br />Longest observed ET #1<br />31<br />
  • 37. 32<br />+<br />=<br />printf()<br />pthread_attr_setstacksize (&attr, &mystacksize);<br />HANDLE WINAPI CreateThread( <br /> LPSECURITY_ATTRIBUTES lpThreadAttributes, <br /> SIZE_T dwStackSize, <br /> LPTHREAD_START_ROUTINE lpStartAddress, <br /> LPVOID lpParameter, <br /> DWORD dwCreationFlags, <br /> LPDWORD lpThreadId );<br />
  • 38. Stack Overflow in TinyOS<br />33<br />4 KB<br />
  • 39. Stack Overflow in TinyOS<br />34<br />main()<br />4 KB<br />
  • 40. Stack Overflow in TinyOS<br />35<br />main()<br />irq 0<br />4 KB<br />
  • 41. Stack Overflow in TinyOS<br />36<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  • 42. Stack Overflow in TinyOS<br />37<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  • 43. Stack Overflow in TinyOS<br />38<br />Not the same thing as buffer overflow!<br />Type safe language doesn’t solve this problem<br />main()<br />irq 0<br />4 KB<br />irq 1<br />
  • 44. Eliminating Stack Overflow<br />Testing is hard<br />Need to drive code to its WC stack depth<br />Interrupt coincidences are rare<br />Approach: Static analysis of compiled code<br />Can’t estimate stack depth of source<br />39<br />
  • 45. Estimate WC stack depth of each sequential flow, handling<br />Indirect branches<br />Recursion<br />Loads into the stack pointer<br />Compute “interrupt preemption graph”<br />Find longest cycle in this graph<br />40<br />
  • 46. 41<br />in r24, 0x3f ; r24 <- CPU status <br /> register<br />cli ; disable interrupts<br />adc r24, r24 ; carry bit <- prev<br /> interrupt status<br />eor r24, r24 ; r24 <- 0<br />adc r24, r24 ; r24 <- carry bit<br />mov r18, r24 ; r18 <- r24<br />... critical section ...<br />and r18, r18 ; test r18 for zero<br />breq .+2 ; if zero, skip next <br /> instruction<br />sei ; enable interrupts<br />ret ; return from function<br />
  • 47. Stack analysis tool deployed in the TinyOS distribution<br />Results are typically much larger than worst observed stack depths<br />But, we validated its results by randomly firing interrupts<br />42<br />
  • 48. Need… more… oracles…<br />43<br />
  • 49. TinyOS applications are built using components<br />Interface requirements documented but not checked<br />Interface misuse often silent<br />44<br />
  • 50. We augmented nesC with contracts<br />Dynamic checking reasonable efficient<br />Found some long-standing bugs<br />45<br />
  • 51. nesC is not type safe<br />Memory safety bugs in TinyOS are difficult<br />We ported an existing safe C dialect<br />Found some otherwise-impossible bugs<br />Main problem was getting overhead under control<br />Whole-program optimization<br />46<br />
  • 52. 47<br />Code size<br />35%<br />13%<br />-11%<br />
  • 53. 48<br />Increasing Availability<br />Normal TinyOS:<br />0% average <br />availability<br />Array <br />Out-of-bounds<br />Normal<br />TinyOS<br />Safe TinyOS:<br />95% average <br />availability<br />Array <br />Out-of-bounds<br />Rebuild<br />Soft state<br />Safe<br />TinyOS<br />Reboot<br />
  • 54. What about application-level sensornet properties?<br />All the interesting ones are distributed<br />We adapted TOSSIM, a non-cycle-accurate simulator, to be…<br />A random tester<br />A depth-bounded model checker<br />Oracles: <br />Type safety checks<br />Application-level properties<br />49<br />
  • 55. Application-Level Properties<br />Eventually…<br />Each send buffer is unlocked<br />No cycles in the routing tree<br />All nodes become part of the collection tree<br />All nodes have consistent values<br />6 out of 8 of these properties require global knowledge<br />50<br />
  • 56. Found 12 previously unknown bugs in TinyOS 2.0<br />10 safety, 2 liveness<br />Random testing outperformed depth-bounded model checking<br />Even after a lot of work on POR<br />But required work to shorten long error traces<br />51<br />
  • 57. Conclusions<br />Open embedded platforms exist<br />Some have steep learning curves<br />Finding oracles is hard<br />Generating valid input is hard<br />Embedded systems are fun and important and rewarding<br />52<br />

×