Issta11

Testing EmbeddedSoftware
John Regehr
University of Utah

“Over 15 billion ARM based chips shipped to date”
[ARM web site, 2011]
“The microcontroller market is forecast to reach over $16 billion worldwide in 2011”
[Microcontroller Market Tracker, 2011]
2

Diverse!
4

Diverse!
I have 6 pins and 32 bytes of RAM
5

Diverse!
6

Diverse!
7

Diverse!
I am quad core @ 1.5 GHz and have a GPU
8

Usually there are multiple processors
On-chip networks
In-device networks
Distributed systems
Resource constraints are…
Severe – to minimize unit cost
Hard – failure if system runs out of…
Time
RAM – stack or heap
Energy
9

Continuously interact with the world through I/O devices
May be little abstraction of HW
Probably using both interrupt handlers and threads
Often there are fault tolerance and security requirements
10

Sensor network -> 103–105 LOC
Modern airplane -> 106–107 LOC
Hybrid vehicle -> 107–108 LOC
How do we get these right?
Mostly testing
11

Software on many individual processors is small
Permits aggressive analysis and testing
Constrained domain simplifies testing
Embedded systems are (by definition) special-purpose devices
12

The “Real System Problem”
Many interesting embedded codes are proprietary
Necessary tools may be expensive or nonexistent
Compilers, debuggers, simulators
May not be able to run it in the lab
Often lacks specifications and oracles
13

Consequently, academic embedded work may be…
Forced to use small, contrived examples
Out of tune with industry
14

Consequently, academic embedded work may be…
Forced to use small, contrived examples
Out of tune with industry
15
Solution:
Ubiquitous open embedded platforms

Arduino
Arduino Uno:
8-bit AVR processor @ 16 MHz
2 KB RAM
~$30
Emphasis is on interfacing
16

Arduino
Nice IDE + libraries + C/C++
Minimal abstraction of the embedded processor
18 new books in 2011
17

Simulators and model checkers for AVR code exist
Very few Arduino tool papers exist
This is a big opportunity
Arduino
Nice IDE + libraries + C/C++
Minimal abstraction of the embedded processor
18 new books in 2011
18

TinyOS
OS and middleware support for sensor networks
Sensing
Collection and dissemination
Localization
Applications are in nesC, a C dialect
19

TinyOS
“Motes” based on a variety of MCUs
Cost $50 – $200
Good simulators exist
There are a few books
20

TinyOS
~100 tool papers
Many open problems
“Motes” based on a variety of MCUs
Cost $50 – $200
Good simulators exist
There are a few books
21

Android
OS + middleware for smart phones / tablets
ARM based hardware running Linux
Much less constrained than motes and Arduino
22

Android
Application code in Java
Great tools
Tons of books
23

Android
< 100 tool papers
Most are very recent
This is not a scary platform
Application code in Java
Great tools
Tons of books
24

ROS – Robot Operating System
Linux-based infrastructure for programming robots
Primary abstraction is graph of communicating processes
Local and distributed
25

ROS – Robot Operating System
Very few ROS tool papers exist
Linux-based infrastructure for programming robots
Primary abstraction is graph of communicating processes
Local and distributed
26

Plenty of other open embedded platforms exist
FreeRTOS
Contiki
Pacemaker Challenge
Etc.
Embarrassment of riches
Still, huge room for improvement
Where’s the open automobile?
27

So, let’s test some embedded software
But what are we testing for?
28

Properties / Oracles
Temporal safety
Deadlines
Or just responsiveness
Memory safety
Contracts / assertions
Reference implementation
29

Worst-Case Execution Time
What is the upper bound on execution time for a piece of code?
We care because the world has deadlines
Static analysis of WCET is extremely difficult if there is…
A cache
Preemption
An aggressive processor
30

True WCET
Number of executions
Execution time
Conservative WCET
Longest observed ET #2
Longest observed ET #1
31

32
+
=
printf()
pthread_attr_setstacksize (&attr, &mystacksize);
HANDLE WINAPI CreateThread(
LPSECURITY_ATTRIBUTES lpThreadAttributes,
SIZE_T dwStackSize,
LPTHREAD_START_ROUTINE lpStartAddress,
LPVOID lpParameter,
DWORD dwCreationFlags,
LPDWORD lpThreadId );

Stack Overflow in TinyOS
33
4 KB

34
main()
4 KB

35
main()
irq 0
4 KB

36
main()
irq 0
4 KB
irq 1

37
main()
irq 0
4 KB
irq 1

38
Not the same thing as buffer overflow!
Type safe language doesn’t solve this problem
main()
irq 0
4 KB
irq 1

Eliminating Stack Overflow
Testing is hard
Need to drive code to its WC stack depth
Interrupt coincidences are rare
Approach: Static analysis of compiled code
Can’t estimate stack depth of source
39

Estimate WC stack depth of each sequential flow, handling
Indirect branches
Recursion
Loads into the stack pointer
Compute “interrupt preemption graph”
Find longest cycle in this graph
40

41
in r24, 0x3f ; r24 <- CPU status
register
cli ; disable interrupts
adc r24, r24 ; carry bit <- prev
interrupt status
eor r24, r24 ; r24 <- 0
adc r24, r24 ; r24 <- carry bit
mov r18, r24 ; r18 <- r24
... critical section ...
and r18, r18 ; test r18 for zero
breq .+2 ; if zero, skip next
instruction
sei ; enable interrupts
ret ; return from function

Stack analysis tool deployed in the TinyOS distribution
Results are typically much larger than worst observed stack depths
But, we validated its results by randomly firing interrupts
42

Need… more… oracles…
43

TinyOS applications are built using components
Interface requirements documented but not checked
Interface misuse often silent
44

We augmented nesC with contracts
Dynamic checking reasonable efficient
Found some long-standing bugs
45

nesC is not type safe
Memory safety bugs in TinyOS are difficult
We ported an existing safe C dialect
Found some otherwise-impossible bugs
Main problem was getting overhead under control
Whole-program optimization
46

47
Code size
35%
13%
-11%

48
Increasing Availability
Normal TinyOS:
0% average
availability
Array
Out-of-bounds
Normal
TinyOS
Safe TinyOS:
95% average
availability
Array
Out-of-bounds
Rebuild
Soft state
Safe
TinyOS
Reboot

What about application-level sensornet properties?
All the interesting ones are distributed
We adapted TOSSIM, a non-cycle-accurate simulator, to be…
A random tester
A depth-bounded model checker
Oracles:
Type safety checks
Application-level properties
49

Application-Level Properties
Eventually…
Each send buffer is unlocked
No cycles in the routing tree
All nodes become part of the collection tree
All nodes have consistent values
6 out of 8 of these properties require global knowledge
50

Found 12 previously unknown bugs in TinyOS 2.0
10 safety, 2 liveness
Random testing outperformed depth-bounded model checking
Even after a lot of work on POR
But required work to shorten long error traces
51

Conclusions
Open embedded platforms exist
Some have steep learning curves
Finding oracles is hard
Generating valid input is hard
Embedded systems are fun and important and rewarding
52

http://blog.regehr.org	3522
http://news.int80.biz	25
http://dashboard.bloglines.com	7
http://feeds.feedburner.com	6
http://www.hanrss.com	3
http://newsblur.com	3
http://translate.googleusercontent.com	3
http://webcache.googleusercontent.com	2
http://feeds.churchturing.org	2
http://www.ljr-embed.ru	1
http://www.netvibes.com	1
http://snap.diclonius.org	1
https://twitter.com	1
http://xianguo.com	1

Issta11

by regehr

Accessibility

Categories

Upload Details

Usage Rights

14 Embeds 3,578

Statistics

Issta11 Presentation Transcript