Issta11
Upcoming SlideShare
Loading in...5
×
 

Issta11

on

  • 4,571 views

slides from John Regehr's keynote at ISSTA 2011

slides from John Regehr's keynote at ISSTA 2011

Statistics

Views

Total Views
4,571
Views on SlideShare
863
Embed Views
3,708

Actions

Likes
3
Downloads
16
Comments
0

14 Embeds 3,708

http://blog.regehr.org 3652
http://news.int80.biz 25
http://dashboard.bloglines.com 7
http://feeds.feedburner.com 6
http://www.hanrss.com 3
http://newsblur.com 3
http://translate.googleusercontent.com 3
http://webcache.googleusercontent.com 2
http://feeds.churchturing.org 2
http://www.ljr-embed.ru 1
http://www.netvibes.com 1
http://snap.diclonius.org 1
https://twitter.com 1
http://xianguo.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Issta11 Issta11 Presentation Transcript

  • Testing EmbeddedSoftware
    John Regehr
    University of Utah
  • “Over 15 billion ARM based chips shipped to date”
    [ARM web site, 2011]
    “The microcontroller market is forecast to reach over $16 billion worldwide in 2011”
    [Microcontroller Market Tracker, 2011]
    2
  • 3
  • Diverse!
    4
  • Diverse!
    I have 6 pins and 32 bytes of RAM
    5
  • Diverse!
    6
  • Diverse!
    7
  • Diverse!
    I am quad core @ 1.5 GHz and have a GPU
    8
  • Usually there are multiple processors
    On-chip networks
    In-device networks
    Distributed systems
    Resource constraints are…
    Severe – to minimize unit cost
    Hard – failure if system runs out of…
    Time
    RAM – stack or heap
    Energy
    9
  • Continuously interact with the world through I/O devices
    May be little abstraction of HW
    Probably using both interrupt handlers and threads
    Often there are fault tolerance and security requirements
    10
  • Sensor network -> 103–105 LOC
    Modern airplane -> 106–107 LOC
    Hybrid vehicle -> 107–108 LOC
    How do we get these right?
    Mostly testing
    11
  • Software on many individual processors is small
    Permits aggressive analysis and testing
    Constrained domain simplifies testing
    Embedded systems are (by definition) special-purpose devices
    12
  • The “Real System Problem”
    Many interesting embedded codes are proprietary
    Necessary tools may be expensive or nonexistent
    Compilers, debuggers, simulators
    May not be able to run it in the lab
    Often lacks specifications and oracles
    13
  • Consequently, academic embedded work may be…
    Forced to use small, contrived examples
    Out of tune with industry
    14
  • Consequently, academic embedded work may be…
    Forced to use small, contrived examples
    Out of tune with industry
    15
    Solution:
    Ubiquitous open embedded platforms
  • Arduino
    Arduino Uno:
    8-bit AVR processor @ 16 MHz
    2 KB RAM
    ~$30
    Emphasis is on interfacing
    16
  • Arduino
    Nice IDE + libraries + C/C++
    Minimal abstraction of the embedded processor
    18 new books in 2011
    17
    • Simulators and model checkers for AVR code exist
    • Very few Arduino tool papers exist
    • This is a big opportunity
    Arduino
    Nice IDE + libraries + C/C++
    Minimal abstraction of the embedded processor
    18 new books in 2011
    18
  • TinyOS
    OS and middleware support for sensor networks
    Sensing
    Collection and dissemination
    Localization
    Applications are in nesC, a C dialect
    19
  • TinyOS
    “Motes” based on a variety of MCUs
    Cost $50 – $200
    Good simulators exist
    There are a few books
    20
  • TinyOS
    • ~100 tool papers
    • Many open problems
    “Motes” based on a variety of MCUs
    Cost $50 – $200
    Good simulators exist
    There are a few books
    21
  • Android
    OS + middleware for smart phones / tablets
    ARM based hardware running Linux
    Much less constrained than motes and Arduino
    22
  • Android
    Application code in Java
    Great tools
    Tons of books
    23
  • Android
    • < 100 tool papers
    • Most are very recent
    • This is not a scary platform
    Application code in Java
    Great tools
    Tons of books
    24
  • ROS – Robot Operating System
    Linux-based infrastructure for programming robots
    Primary abstraction is graph of communicating processes
    Local and distributed
    25
  • ROS – Robot Operating System
    • Very few ROS tool papers exist
    Linux-based infrastructure for programming robots
    Primary abstraction is graph of communicating processes
    Local and distributed
    26
  • Plenty of other open embedded platforms exist
    FreeRTOS
    Contiki
    Pacemaker Challenge
    Etc.
    Embarrassment of riches
    Still, huge room for improvement
    Where’s the open automobile?
    27
  • So, let’s test some embedded software
    But what are we testing for?
    28
  • Properties / Oracles
    Temporal safety
    Deadlines
    Or just responsiveness
    Memory safety
    Contracts / assertions
    Reference implementation
    29
  • Worst-Case Execution Time
    What is the upper bound on execution time for a piece of code?
    We care because the world has deadlines
    Static analysis of WCET is extremely difficult if there is…
    A cache
    Preemption
    An aggressive processor
    30
  • True WCET
    Number of executions
    Execution time
    Conservative WCET
    Longest observed ET #2
    Longest observed ET #1
    31
  • 32
    +
    =
    printf()
    pthread_attr_setstacksize (&attr, &mystacksize);
    HANDLE WINAPI CreateThread(
    LPSECURITY_ATTRIBUTES lpThreadAttributes,
    SIZE_T dwStackSize,
    LPTHREAD_START_ROUTINE lpStartAddress,
    LPVOID lpParameter,
    DWORD dwCreationFlags,
    LPDWORD lpThreadId );
  • Stack Overflow in TinyOS
    33
    4 KB
  • Stack Overflow in TinyOS
    34
    main()
    4 KB
  • Stack Overflow in TinyOS
    35
    main()
    irq 0
    4 KB
  • Stack Overflow in TinyOS
    36
    main()
    irq 0
    4 KB
    irq 1
  • Stack Overflow in TinyOS
    37
    main()
    irq 0
    4 KB
    irq 1
  • Stack Overflow in TinyOS
    38
    Not the same thing as buffer overflow!
    Type safe language doesn’t solve this problem
    main()
    irq 0
    4 KB
    irq 1
  • Eliminating Stack Overflow
    Testing is hard
    Need to drive code to its WC stack depth
    Interrupt coincidences are rare
    Approach: Static analysis of compiled code
    Can’t estimate stack depth of source
    39
  • Estimate WC stack depth of each sequential flow, handling
    Indirect branches
    Recursion
    Loads into the stack pointer
    Compute “interrupt preemption graph”
    Find longest cycle in this graph
    40
  • 41
    in r24, 0x3f ; r24 <- CPU status
    register
    cli ; disable interrupts
    adc r24, r24 ; carry bit <- prev
    interrupt status
    eor r24, r24 ; r24 <- 0
    adc r24, r24 ; r24 <- carry bit
    mov r18, r24 ; r18 <- r24
    ... critical section ...
    and r18, r18 ; test r18 for zero
    breq .+2 ; if zero, skip next
    instruction
    sei ; enable interrupts
    ret ; return from function
  • Stack analysis tool deployed in the TinyOS distribution
    Results are typically much larger than worst observed stack depths
    But, we validated its results by randomly firing interrupts
    42
  • Need… more… oracles…
    43
  • TinyOS applications are built using components
    Interface requirements documented but not checked
    Interface misuse often silent
    44
  • We augmented nesC with contracts
    Dynamic checking reasonable efficient
    Found some long-standing bugs
    45
  • nesC is not type safe
    Memory safety bugs in TinyOS are difficult
    We ported an existing safe C dialect
    Found some otherwise-impossible bugs
    Main problem was getting overhead under control
    Whole-program optimization
    46
  • 47
    Code size
    35%
    13%
    -11%
  • 48
    Increasing Availability
    Normal TinyOS:
    0% average
    availability
    Array
    Out-of-bounds
    Normal
    TinyOS
    Safe TinyOS:
    95% average
    availability
    Array
    Out-of-bounds
    Rebuild
    Soft state
    Safe
    TinyOS
    Reboot
  • What about application-level sensornet properties?
    All the interesting ones are distributed
    We adapted TOSSIM, a non-cycle-accurate simulator, to be…
    A random tester
    A depth-bounded model checker
    Oracles:
    Type safety checks
    Application-level properties
    49
  • Application-Level Properties
    Eventually…
    Each send buffer is unlocked
    No cycles in the routing tree
    All nodes become part of the collection tree
    All nodes have consistent values
    6 out of 8 of these properties require global knowledge
    50
  • Found 12 previously unknown bugs in TinyOS 2.0
    10 safety, 2 liveness
    Random testing outperformed depth-bounded model checking
    Even after a lot of work on POR
    But required work to shorten long error traces
    51
  • Conclusions
    Open embedded platforms exist
    Some have steep learning curves
    Finding oracles is hard
    Generating valid input is hard
    Embedded systems are fun and important and rewarding
    52