Your SlideShare is downloading. ×
Enabling and Supporting the Debugging of Field Failures (Job Talk)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Enabling and Supporting the Debugging of Field Failures (Job Talk)

198
views

Published on

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
198
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ENABLING AND SUPPORTINGTHE DEBUGGINGOF FIELD FAILURESJames ClauseGeorgia Institute of Technology
  • 2. Jazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ICSE 07 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptRESEARCH OVERVIEW
  • 3. Efficient instrumentationJazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ICSE 07 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptRESEARCH OVERVIEW
  • 4. Efficient instrumentationJazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ICSE 07 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptDynamic taintingbased analysesRESEARCH OVERVIEW
  • 5. Efficient instrumentationJazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ICSE 07 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptDynamic taintingbased analysesEnabling moreefficient debuggingRESEARCH OVERVIEW
  • 6. Efficient instrumentationJazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptICSE 07Dynamic taintingbased analysesEnabling moreefficient debuggingRESEARCH OVERVIEW
  • 7. OVERALL PICTURE
  • 8. OVERALL PICTURE
  • 9. OVERALL PICTURE
  • 10. OVERALL PICTURE
  • 11. OVERALL PICTUREField failures:Anomalous behavior (or crashes)of deployed software that occur on user machines• Difficult to debug• Relevant to users
  • 12. CURRENT PRACTICE
  • 13. CURRENT PRACTICEAsk the user
  • 14. CURRENT PRACTICEAsk the userI frobbed thethingummy like the guytold me. Then I spunthe doodadwiddershins and a littlethinger popped up andit just stopped working.
  • 15. CURRENT PRACTICEAsk the userI opened my webbrowser.Specifically, I clicked onthe dock icon. Itbounced twice beforecrashing.Please help.
  • 16. CURRENT PRACTICEGather static information
  • 17. CURRENT PRACTICEGather static informationDifficult to reproducethe failure
  • 18. CURRENT PRACTICEGather static informationDifficult to reproducethe failureLocations onlycorrelated with thefailureLiblit et al. 03Tucek et al. 07Chilimbi et al. 09...
  • 19. OUR SOLUTION
  • 20. OUR SOLUTIONRecordfailing executionsin the field
  • 21. OUR SOLUTIONRecordfailing executionsin the fieldReplayfailing executionsin house+
  • 22. OUR SOLUTIONRecordfailing executionsin the fieldReplayfailing executionsin houseDebugfield failureseffectively+
  • 23. In the fieldIn houseUSAGE SCENARIO✘Replay / DebugDevelop RecordCapturedfailure
  • 24. In the fieldIn houseUSAGE SCENARIO✘Replay / DebugDevelop RecordCapturedfailureOracle
  • 25. In the fieldIn houseUSAGE SCENARIO✘Replay / DebugDevelop RecordCapturedfailure
  • 26. PRACTICALITY ISSUES✘
  • 27. 345PRACTICALITY ISSUESLarge in size✘
  • 28. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformation✘
  • 29. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformationMinimize✘ ✘
  • 30. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformationMinimize Anonymize✘ ✘
  • 31. In the fieldIn houseReplay / DebugDevelop RecordCapturedfailureMinimizeAnonymizeUSAGE SCENARIO✘✘
  • 32. ✘OUTLINERecording / ReplayingMinimizationAnonymization
  • 33. COMINGSOON✘OUTLINERecording / ReplayingMinimizationAnonymizationFuture work
  • 34. COMINGSOON✘OUTLINERecording / ReplayingMinimizationAnonymizationFuture work
  • 35. COMINGSOON✘OUTLINERecording / ReplayingMinimizationAnonymizationFuture work
  • 36. EXISTING RECORD/REPLAYAPPROACHES
  • 37. EXISTING RECORD/REPLAYAPPROACHESChen et al. 01, King et al. 05Narayanasamy et al. 05,Netzer and Weaver 94,Srinivasan et al. 04,VMWareExactly replay everything
  • 38. EXISTING RECORD/REPLAYAPPROACHESNot amenable tominimization oranonymizationUnacceptableruntime overheadChen et al. 01, King et al. 05Narayanasamy et al. 05,Netzer and Weaver 94,Srinivasan et al. 04,VMWareExactly replay everything
  • 39. Not amenable tominimization oranonymizationUnacceptableruntime overheadRecord low-level events• numerous• high interdependenceRECORD & REPLAY
  • 40. Record high-level events• fewer in number• low interdependenceAmenable tominimization oranonymizationAcceptableruntime overheadRECORD & REPLAY
  • 41. ENVIRONMENT INTERACTIONS
  • 42. ENVIRONMENT INTERACTIONSStreams
  • 43. ENVIRONMENT INTERACTIONSStreamsFiles
  • 44. ENVIRONMENT INTERACTIONSStreamsFilesInteraction Events:FILE — interaction with a filePOLL — checks for availability of data on a streamPULL — read data from a stream
  • 45. Environment data (streams):Event log:Environment data (files):
  • 46. Environment data (streams):Event log:Environment data (files):
  • 47. Environment data (streams):Event log:Environment data (files):FILE foo.1
  • 48. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1
  • 49. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1
  • 50. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1POLL KEYBOARD NOK
  • 51. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1POLL KEYBOARD NOK
  • 52. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOARD NOK
  • 53. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOARD NOK
  • 54. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOK
  • 55. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOK
  • 56. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}
  • 57. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}❙
  • 58. Environment data (streams):Event log:Environment data (files):FILE foo.1foo.1KEYBOARD: {5680}helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}❙
  • 59. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  • 60. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  • 61. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  • 62. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  • 63. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  • 64. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔
  • 65. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔
  • 66. Environment data (files):Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔✔✔✔✔✔✔✔✔✔✔✔✔
  • 67. EVALUATIONAcceptableruntime overheadAmenable tominimization oranonymization
  • 68. EVALUATIONPrototype implementation:• maps libc function calls tointeraction eventsSubjects:• several cpu intensiveapplications (e.g., bzip, gcc)Results:• negligible overheads• (i.e. less than 10%)• data size is acceptable• application dependent
  • 69. COMINGSOONOUTLINE✘Recording / ReplayingMinimizationAnonymizationFuture work
  • 70. ✘ 345Large in sizeMINIMIZATION✘Goal: focus developer effort
  • 71. MINIMIZATION✘24:15
  • 72. MINIMIZATION✘Timeminimization 2:5524:15
  • 73. MINIMIZATION✘✂Dataminimization 2:55Timeminimization 2:5524:15
  • 74. MINIMIZATION✘✂Dataminimization 2:55Timeminimization 2:5524:15Oracle Oracle
  • 75. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...
  • 76. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle time
  • 77. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle time
  • 78. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle timeRemove delays
  • 79. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle timeRemove delays
  • 80. DATA MINIMIZATIONEnvironment data (files):foo.1 foo.2 bar.1Whole entitiesChunksAtoms
  • 81. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 82. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 83. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Whole entitiesChunksAtoms
  • 84. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 85. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 86. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 87. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 88. DATA MINIMIZATIONEnvironment data (files):bar.1sadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 89. DATA MINIMIZATIONEnvironment data (files):bar.1sadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  • 90. DATA MINIMIZATIONEnvironment data (files):bar.1Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.
  • 91. DATA MINIMIZATIONEnvironment data (files):bar.1Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.foo.2
  • 92. DATA MINIMIZATIONEnvironment data (files):Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.foo.2
  • 93. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?
  • 94. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?Pine email and news client• two real field failures• 20 failing executions, 10 per failure
  • 95. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?Pine email and news client• two real field failures• 20 failing executions, 10 per failureMinimized executions generated by• randomly generating interaction scripts• manually performing the scripts (while recording)• minimizing the captured executions
  • 96. RESULTSHeader-color fault Address book fault0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimization
  • 97. RESULTSHeader-color fault Address book faultResults are likely to be conservative; recorded executionsonly contain the minimal amount of data needed to perform an action.0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimization
  • 98. RESULTSHeader-color fault Address book faultResults are likely to be conservative; recorded executionsonly contain the minimal amount of data needed to perform an action.0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimizationInputs can be minimized in a reasonableamount of time (less then 75 minutes)
  • 99. RESULTSHEADER COLOR FAULT
  • 100. RESULTSHEADER COLOR FAULT1. color is enabled2. one or more colors are added3. all colors are removedCrash when:
  • 101. RESULTSHEADER COLOR FAULT1. color is enabled2. one or more colors are added3. all colors are removedCrash when:Recorded execution:34files and streams≈800kb
  • 102. Minimized execution:1stream4files≈72kb(partial)RESULTSHEADER COLOR FAULT1. color is enabled2. one or more colors are added3. all colors are removedCrash when:Recorded execution:34files and streams≈800kb
  • 103. COMINGSOONOUTLINE✘Recording / ReplayingMinimizationAnonymizationFuture work
  • 104. ANONYMIZATION345Contain sensitiveinformation✘Goal: help address privacy concerns
  • 105. Sensitiveinput (I)that causes FInput domainANONYMIZATION
  • 106. Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATION
  • 107. Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATIONAnonymizedinput (I’)that alsocauses F
  • 108. Inputs that satisfyF’s path condition Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATIONAnonymizedinput (I’)that alsocauses F
  • 109. PATH CONDITION GENERATIONPath condition: set of constraints on a program’sinputs that encode the conditions necessary for aspecific path to be executed.
  • 110. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION
  • 111. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION
  • 112. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0
  • 113. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0(sensitive)
  • 114. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0(sensitive)
  • 115. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 116. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 117. Path Condition:i1 <= 5Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 118. Path Condition:i1 <= 5Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 119. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 120. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  • 121. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10(sensitive)
  • 122. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10(sensitive)
  • 123. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10∧ i3 == 0(sensitive)
  • 124. CHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  • 125. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  • 126. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  • 127. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0 i1 == 5i2 == 3i3 == 0
  • 128. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0 i1 == 5i2 == 3i3 == 0boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}5 3 0
  • 129. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  • 130. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0
  • 131. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0(breakable)
  • 132. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0(breakable)
  • 133. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0i1 == 4i2 == 10i3 == 0(breakable)
  • 134. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  • 135. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  • 136. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  • 137. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  • 138. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  • 139. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array reads
  • 140. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array reads
  • 141. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsx.equals(y);
  • 142. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsx.equals(y);// x = “abc”// y = “abd”
  • 143. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsx.equals(y);Traditional:x0 == y0∧ x1 == y1∧ x2 != y2// x = “abc”// y = “abd”
  • 144. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsx.equals(y);Traditional:x0 == y0∧ x1 == y1∧ x2 != y2Relaxed:x0 != y0∨ x1 != y1∨ x2 != y2// x = “abc”// y = “abd”
  • 145. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array reads
  • 146. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array reads
  • 147. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}
  • 148. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}// x = 5
  • 149. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}Traditional:x == 5// x = 5
  • 150. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}Traditional:x == 5Relaxed:x == 5∨ x == 3// x = 5
  • 151. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}Traditional:x == 5Relaxed:x == 5∨ x == 3// x = 10
  • 152. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}Traditional:Relaxed:x == 5∨ x == 3// x = 10x == 10
  • 153. PATH CONDITION RELAXATION1. Array inequalities 3. Multi-clause conditionals2. Switch statements 4. Array readsswitch(x) {case 1:...break;case 3:case 5:...break;default:...}Traditional:Relaxed:// x = 10x == 10x != 1∧ x != 3∧ x != 5
  • 154. EVALUATIONFeasibilityCan the approach generate, in areasonable amount of time, anonymizedinputs that reproduce the failure?StrengthHow much information about theoriginal inputs is revealed?EffectivenessAre the anonymized inputs safe to sendto developers?
  • 155. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faults(20 faults, total)
  • 156. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faultsSelect sensitive failure-inducing inputs• manually generated or included with subject• several 100 bytes to 5MB in size(20 faults, total)
  • 157. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faultsSelect sensitive failure-inducing inputs• manually generated or included with subject• several 100 bytes to 5MB in size(Assume all of each input is potentially sensitive)(20 faults, total)
  • 158. RQ1: FEASIBILITY0150300450600ExecutionTime(s)05101520columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nanoxml14nanoxml15nanoxml16SolverTime(s)
  • 159. RQ1: FEASIBILITY0150300450600ExecutionTime(s)05101520columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nanoxml14nanoxml15nanoxml16SolverTime(s)Inputs can be anonymized in a reasonableamount of time (easily done overnight)
  • 160. Average % Bits Revealed Average % ResidueRQ2: STRENGTH
  • 161. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionLittleinformation revealed
  • 162. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionLots ofinformation revealed
  • 163. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionMeasures how much of theanonymized input is identicalto the original inputAAAAAAsecretAAAAAA...AAAAAABBBBBBsecretBBBBBB...BBBBBBI’Lots ofinformation revealedI
  • 164. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionMeasures how much of theanonymized input is identicalto the original inputAAAAAAsecretAAAAAA...AAAAAABBBBBBsecretBBBBBB...BBBBBBI’Lots ofinformation revealedI
  • 165. RQ2: STRENGTH02550751000255075100columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nenoxml14nanoxml15nanoxml16Average%BitsRevealedAverage%Residue
  • 166. RQ2: STRENGTH02550751000255075100columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nenoxml14nanoxml15nanoxml16Average%BitsRevealedAverage%ResidueAnonymized inputs reveal, on average, between60% (worst case) and 2% (best case) of theinformation in the original inputs
  • 167. RQ3: EFFECTIVENESSNANOXML<!DOCTYPE Foo [   <!ELEMENT Foo (ns:Bar)>   <!ATTLIST Foo       xmlns CDATA #FIXED http://nanoxml.n3.net/bar       a     CDATA #REQUIRED>   <!ELEMENT ns:Bar (Blah)>   <!ATTLIST ns:Bar       xmlns:ns CDATA #FIXED http://nanoxml.n3.net/bar>   <!ELEMENT Blah EMPTY>   <!ATTLIST Blah       x    CDATA #REQUIRED       ns:x CDATA #REQUIRED>]><!-- comment --><Foo a=very b=secret c=stuff>vaz   <ns:Bar>       <Blah x="1" ns:x="2"/>   </ns:Bar></Foo>
  • 168. RQ3: EFFECTIVENESSNANOXML<!DOCTYPE [   <! >   <!ATTLIST        #FIXED         >   <!E >   <!ATTLIST        #FIXED >   <!E >   <!ATTLIST        #        : # >]><!-- -->< = = = >   < : >       < =" " : =" "/>   </ :
  • 169. Wayne,Bartley,Bartley,Wayne,wbartly@acp.com,,Ronald,Kahle,Kahle,Ron,ron.kahle@kahle.com,,Wilma,Lavelle,Lavelle,Wilma,,lavelle678@aol.com,Jesse,Hammonds,Hammonds,Jesse,,hamj34@comcast.com,Amy,Uhl,Uhl,Amy,uhla@corp1.com,uhla@gmail.com,Hazel,Miracle,Miracle,Hazel,hazel.miracle@corp2.com,,Roxanne,Nealy,Nealy,Roxie,,roxie.nearly@gmail.com,Heather,Kane,Kane,Heather,kaneh@corp2.com,,Rosa,Stovall,Stovall,Rosa,,sstoval@aol.com,Peter,Hyden,Hyden,Pete,,peteh1989@velocity.net,Jeffrey,Wesson,Wesson,Jeff,jwesson@corp4.com,,Virginia,Mendoza,Mendoza,Ginny,gmendoza@corp4.com,,Richard,Robledo,Robledo,Ralph,ralphrobledo@corp1.com,,Edward,Blanding,Blanding,Ed,,eblanding@gmail.com,Sean,Pulliam,Pulliam,Sean,spulliam@corp2.com,,Steven,Kocher,Kocher,Steve,kocher@kocher.com,,Tony,Whitlock,Whitlock,Tony,,tw14567@aol.com,Frank,Earl,Earl,Frankie,,,Shelly,Riojas,Riojas,Shelly,srojas@corp6.com,,RQ3: EFFECTIVENESSCOLUMBA, , , , ,,, , , , ,,, , , ,, ,, , , ,, ,, , , , , ,, , , , ,,, , , ,, ,, , , , ,,, , , ,, ,, , , ,, ,, , , , ,,, , , , ,,, , , , ,,, , , ,, ,, , , , ,,, , , , ,,, , , ,, ,
  • 170. RQ3: EFFECTIVENESSCOLUMBA, , , , ,,, , , , ,,, , , ,, ,, , , ,, ,, , , , , ,, , , , ,,, , , ,, ,, , , , ,,, , , ,, ,, , , ,, ,, , , , ,,, , , , ,,, , , , ,,, , , ,, ,, , , , ,,, , , , ,,, , , ,, ,
  • 171. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>
  • 172. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>
  • 173. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>The portions of the inputs that remain afteranonymization tend to be structural in nature andtherefore are safe to send to developers
  • 174. COMINGSOONOUTLINE✘Recording / ReplayingMinimizationAnonymizationFuture work
  • 175. COMINGSOON-ISHFUTURE WORKCOMINGSOON
  • 176. COMINGSOON-ISHFUTURE WORKCOMINGSOON
  • 177. COMINGSOON-ISHFUTURE WORKCOMINGSOONImproved minimization
  • 178. COMINGSOON-ISHFUTURE WORKCOMINGSOONImproved minimizationLeverage passing executions
  • 179. COMINGSOON-ISHFUTURE WORKCOMINGSOONImproved minimizationLeverage passing executionsDebugging for developers
  • 180. Foo512BBar1KBBaz1.5GBIMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING
  • 181. 1 Taint inputsFoo512BBar1KBBaz1.5GBIMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING
  • 182. 1 Taint inputsFoo512BBar1KBBaz1.5GBIMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING1234567890
  • 183. 1 Taint inputs2 Propagatetaint marksFoo512BBar1KBBaz1.5GBIMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING1234567890
  • 184. 1 Taint inputs2 Propagatetaint marksFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...IMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING1234567890
  • 185. 1 Taint inputs2 Propagatetaint marks3 Identifyrelevant inputsFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...IMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING1234567890
  • 186. 1 Taint inputs2 Propagatetaint marks3 Identifyrelevant inputsFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...IMPROVED MINIMIZATIONLEVERAGE DYNAMICTAINTING1234567890
  • 187. In the fieldIn houseLEVERAGE PASSING EXECUTIONSReplay / DebugDevelop Record✘MinimizeSanitize✘
  • 188. In the fieldIn houseLEVERAGE PASSING EXECUTIONSReplay / DebugDevelop Record✘MinimizeSanitize✘✔
  • 189. In the fieldIn houseLEVERAGE PASSING EXECUTIONSReplay / DebugDevelop Record✘MinimizeSanitize✘✔
  • 190. LEVERAGE PASSING EXECUTIONS✔
  • 191. LEVERAGE PASSING EXECUTIONS✔“Fuzz” to create failing executions
  • 192. LEVERAGE PASSING EXECUTIONS✔“Fuzz” to create failing executionsAugment in-house test suites
  • 193. LEVERAGE PASSING EXECUTIONS✔“Fuzz” to create failing executionsAugment in-house test suitesGuide in house testing
  • 194. DEBUGGING FOR DEVELOPERSMost debugging tools are:By us,for us
  • 195. DEBUGGING FOR DEVELOPERSMost debugging tools are:Limited industrial impactBy us,for us
  • 196. DEBUGGING FOR DEVELOPERSMost debugging tools are:Limited industrial impactWith developers,for developers
  • 197. DEBUGGING FOR DEVELOPERSMost debugging tools are:With developers,for developersLots of industrial impact
  • 198. DEBUGGING FOR DEVELOPERSMost debugging tools are:With developers,for developersLots of industrial impact ?
  • 199. Efficient instrumentationJazz: A Tool for Demand-Driven StructuralTestingJonathan Misurda1, Jim Clause1, Juliya Reed1, Bruce R. Childers1, and MaryLou So a21University of Pittsburgh, Pittsburgh PA 15260, USA,{jmisurda,clausej,juliya,childers}@cs.pitt.edu2University of Virginia, Charlottesville VA 22904, USA,soffa@cs.virginia.eduAbstract. Software testing to produce reliable and robust software hasbecome vitally important. Testing is a process by which quality can beassured through the collection of information about software. While test-ing can improve software quality, current tools typically are inflexibleand have high overheads, making it a challenge to test large projects.We describe a new scalable and flexible tool, called Jazz, that uses ademand-driven structural testing approach. Jazz has a low overhead ofonly 17.6% for branch testing.1 IntroductionIn the last several years, the importance of producing high quality and robustsoftware has become paramount. Testing is an important process to supportquality assurance by gathering information about the software being developedor modified. It is, in general, extremely labor and resource intensive, accountingfor 50-60% of the total cost of software development [1]. The increased emphasison software quality and robustness mandates improved testing methodologies.To test software, a number of techniques can be applied. One class of tech-niques is structural testing, which checks that a given coverage criterion is sat-isfied. For example, branch testing checks that a certain percentage of branchesare executed. Other structural tests include def-use testing in which pairs ofvariable definitions and uses are checked for coverage and node testing in whichnodes in a program’s control flow graph are checked.Unfortunately, structural testing is often hindered by the lack of scalableand flexible tools. Current tools are not scalable in terms of both time andmemory, limiting the number and scope of the tests that can be applied to largeprograms. These tools often modify the software binary to insert instrumentationfor testing. In this case, the tested version of the application is not the sameversion that is shipped to customers and errors may remain. Testing tools areusually inflexible and only implement certain types of testing. For example, manytools implement branch testing, but do not implement node or def-use testing.In this paper, we describe a new tool for structural testing, called Jazz, thataddresses these problems. Jazz uses a novel demand-driven technique to applyABSTRACTProducing reliable and robust software has become oneof the most important software development concerns inrecent years. Testing is a process by which softwarequality can be assured through the collection of infor-mation. While testing can improve software reliability,current tools typically are inflexible and have high over-heads, making it challenging to test large softwareprojects. In this paper, we describe a new scalable andflexible framework for testing programs with a noveldemand-driven approach based on execution paths toimplement test coverage. This technique uses dynamicinstrumentation on the binary code that can be insertedand removed on-the-fly to keep performance and mem-ory overheads low. We describe and evaluate implemen-tations of the framework for branch, node and def-usetesting of Java programs. Experimental results forbranch testing show that our approach has, on average, a1.6 speed up over static instrumentation and also usesless memory.Categories and Subject DescriptorsD.2.5. [Software Engineering]: Testing and Debug-ging—Testing tools; D.3.3. [Programming Lan-guages]: Language Constructs and Features—Programinstrumentation, run-time environmentsGeneral TermsExperimentation, Measurement, VerificationKeywordsTesting, Code Coverage, Structural Testing, Demand-Driven Instrumentation, Java Programming Language1. INTRODUCTIONIn the last several years, the importance of produc-ing high quality and robust software has become para-mount [15]. Testing is an important process to supportquality assurance by gathering information about thebehavior of the software being developed or modified. Itis, in general, extremely labor and resource intensive,accounting for 50-60% of the total cost of softwaredevelopment [17]. Given the importance of testing, it isimperative that there are appropriate testing tools andframeworks. In order to adequately test software, anumber of different testing techniques must be per-formed. One class of testing techniques used extensivelyis structural testing in which properties of the softwarecode are used to ensure a certain code coverage.Struc-tural testing techniques include branch testing, nodetesting, path testing, and def-use testing [6,7,8,17,19].Typically, a testing tool targets one type of struc-tural test, and the software unit is the program, file orparticular methods. In order to apply various structuraltesting techniques, different tools must be used. If a toolfor a particular type of structural testing is not available,the tester would need to either implement it or not usethat testing technique. The tester would also be con-strained by the region of code to be tested, as deter-mined by the tool implementor. For example, it may notbe possible for the tester to focus on a particular regionof code, such as a series of loops, complicated condi-tionals, or particular variables if def-use testing isdesired. The user may want to have higher coverage onfrequently executed regions of code. Users may want todefine their own way of testing. For example, allbranches should be covered 10 times rather than once inall loops.In structural testing, instrumentation is placed atcertain code points (probes). Whenever such a programpoint is reached, code that performs the function for thetest (payload) is executed. The probes in def-use testingare dictated by the definitions and uses of variables andthe payload is to mark that a definition or use in a def-use pair has been covered. Thus for each type of struc-tural testing, there is a testing “plan”. A test plan is aPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage andthat copies bear this notice and the full citation on the first page. Tocopy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee.ICSE05, May 15-21, 2005, St. Louis, Missouri, USA.Copyright 2005 ACM 1-58113-963-2/05/0005...$5.00.Demand-Driven Structural Testing with DynamicInstrumentationJonathan Misurda†, James A. Clause†, Juliya L. Reed†, Bruce R. Childers†, andMary Lou Soffa‡†Department of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260{jmisurda, clausej, juliya, childers}@cs.pitt.edu‡Department of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904soffa@cs.virginia.edu156A Technique for Enabling and Supporting Debugging of Field FailuresJames Clause and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause, orso}@cc.gatech.eduAbstractIt is difficult to fully assess the quality of software in-house, outside the actual time and context in which it willexecute after deployment. As a result, it is common forsoftware to manifest field failures, failures that occur onuser machines due to untested behavior. Field failures aretypically difficult to recreate and investigate on developerplatforms, and existing techniques based on crash report-ing provide only limited support for this task. In this pa-per, we present a technique for recording, reproducing, andminimizing failing executions that enables and supports in-house debugging of field failures. We also present a toolthat implements our technique and an empirical study thatevaluates the technique on a widely used e-mail client.1. IntroductionQuality-assurance activities, such as software testing andanalysis, are notoriously difficult, expensive, and time-consuming. As a result, software products are often re-leased with faults or missing functionality. In fact, real-world examples of field failures experienced by users be-cause of untested behaviors (e.g., due to unforeseen us-ages), are countless. When field failures occur, it is im-portant for developers to be able to recreate and investigatethem in-house. This pressing need is demonstrated by theemergence of several crash-reporting systems, such as Mi-crosoft’s error reporting systems [13] and Apple’s CrashReporter [1]. Although these techniques represent a firstimportant step in addressing the limitations of purely in-house approaches to quality assurance, they work on lim-ited data (typically, a snapshot of the execution state) andcan at best identify correlations between a crash report anddata on other known failures.In this paper, we present a novel technique for reproduc-ing and investigating field failures that addresses the limita-tions of existing approaches. Our technique works in threephases, intuitively illustrated by the scenario in Figure 1. Inthe recording phase, while users run the software, the tech-nique intercepts and logs the interactions between applica-tion and environment and records portions of the environ-ment that are relevant to these interactions. If the executionterminates with a failure, the produced execution recordingis stored for later investigation. In the minimization phase,using free cycles on the user machines, the technique re-plays the recorded failing executions with the goal of au-tomatically eliminating parts of the executions that are notrelevant to the failure. In the replay and debugging phase,developers can use the technique to replay the minimizedfailing executions and investigate the cause of the failures(e.g., within a debugger). Being able to replay and debugreal field failures can give developers unprecedented insightinto the behavior of their software after deployment and op-portunities to improve the quality of their software in waysthat were not possible before.To evaluate our technique, we implemented it in a proto-type tool, called ADDA (Automated Debugging of DeployedApplications), and used the tool to perform an empiricalstudy. The study was performed on PINE [19], a widely-used e-mail client, and involved the investigation of failurescaused by two real faults in PINE. The results of the studyare promising. Our technique was able to (1) record all ex-ecutions of PINE (and two other subjects) with a low timeand space overhead, (2) completely replay all recorded exe-cutions, and (3) perform automated minimization of failingexecutions and obtain shorter executions that manifested thesame failures as the original executions. Moreover, we wereable to replay the minimized executions within a debugger,which shows that they could have actually been used to in-vestigate the failures.The contributions of this paper are:• A novel technique for recording and later replaying exe-cutions of deployed programs.• An approach for minimizing failing executions and gen-erating shorter executions that fail for the same reasons.• A prototype tool that implements our technique.• An empirical study that shows the feasibility and effec-tiveness of the approach.29th International Conference on Software Engineering (ICSE07)0-7695-2828-7/07 $20.00 © 2007Dytan: A Generic Dynamic Taint Analysis FrameworkJames Clause, Wanchun Li, and Alessandro OrsoCollege of ComputingGeorgia Institute of Technology{clause|wli7|orso}@cc.gatech.eduABSTRACTDynamic taint analysis is gaining momentum. Techniques basedon dynamic tainting have been successfully used in the context ofapplication security, and now their use is also being explored in dif-ferent areas, such as program understanding, software testing, anddebugging. Unfortunately, most existing approaches for dynamictainting are defined in an ad-hoc manner, which makes it difficultto extend them, experiment with them, and adapt them to new con-texts. Moreover, most existing approaches are focused on data-flowbased tainting only and do not consider tainting due to control flow,which limits their applicability outside the security domain. Toaddress these limitations and foster experimentation with dynamictainting techniques, we defined and developed a general frameworkfor dynamic tainting that (1) is highly flexible and customizable, (2)allows for performing both data-flow and control-flow based taint-ing conservatively, and (3) does not rely on any customized run-time system. We also present DYTAN, an implementation of ourframework that works on x86 executables, and a set of preliminarystudies that show how DYTAN can be used to implement differenttainting-based approaches with limited effort. In the studies, wealso show that DYTAN can be used on real software, by using FIRE-FOX as one of our subjects, and illustrate how the specific char-acteristics of the tainting approach used can affect efficiency andaccuracy of the taint analysis, which further justifies the use of ourframework to experiment with different variants of an approach.Categories and Subject Descriptors: D.2.5 [Software Engineer-ing]: Testing and Debugging;General Terms: Experimentation, SecurityKeywords: Dynamic tainting, information flow, general framework1. INTRODUCTIONDynamic taint analysis (also known as dynamic information flowanalysis) consists, intuitively, in marking and tracking certain datain a program at run-time. This type of dynamic analysis is be-coming increasingly popular. In the context of application secu-rity, dynamic-tainting approaches have been successfully used toprevent a wide range of attacks, including buffer overruns (e.g., [8,17]), format string attacks (e.g., [17, 21]), SQL and command in-jections (e.g., [7, 19]), and cross-site scripting (e.g., [18]). Morerecently, researchers have started to investigate the use of tainting-based approaches in domains other than security, such as programunderstanding, software testing, and debugging (e.g., [11, 13]).Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’07, July 9–12, 2007, London, England, United Kingdom.Copyright 2007 ACM 978-1-59593-734-6/07/0007 ...$5.00.Unfortunately, most existing techniques and tools for dynamictaint analysis are defined in an ad-hoc manner, to target a specificproblem or a small class of problems. It would be difficult to ex-tend or adapt such techniques and tools so that they can be used inother contexts. In particular, most existing approaches are focusedon data-flow based tainting only, and do not consider tainting dueto the control flow within an application, which limits their generalapplicability. Also, most existing techniques support either a sin-gle taint marking or a small, fixed number of markings, which isproblematic in applications such as debugging. Finally, almost noexisting technique handles the propagation of taint markings in atruly conservative way, which may be appropriate for the specificapplications considered, but is problematic in general. Because de-veloping support for dynamic taint analysis is not only time con-suming, but also fairly complex, this lack of flexibility and gener-ality of existing tools and techniques is especially limiting for thistype of dynamic analysis.To address these limitations and foster experimentation with dy-namic tainting techniques, in this paper we present a framework fordynamic taint analysis. We designed the framework to be generaland flexible, so that it allows for implementing different kinds oftechniques based on dynamic taint analysis with little effort. Userscan leverage the framework to quickly develop prototypes for theirtechniques, experiment with them, and investigate trade-offs of dif-ferent alternatives. For a simple example, the framework could beused to investigate the cost effectiveness of considering differenttypes of taint propagation for an application.Our framework has several advantages over existing approaches.First, it is highly flexible and customizable. It allows for easilyspecifying which program data should be tainted and how, how taintmarkings should be propagated at run-time, and where and howtaint markings should be checked. Second, it allows for performingdata-flow and both data-flow and control-flow based tainting. Third,from a more practical standpoint, it works on binaries, does notneed access to source code, and does not rely on any customizedhardware or operating system, which makes it broadly applicable.We also present DYTAN, an implementation of our frameworkthat works on x86 binaries, and a set of preliminary studies per-formed using DYTAN. In the first set of studies, we report on ourexperience in using DYTAN to implement two tainting-based ap-proaches presented in the literature. Although preliminary, our ex-perience shows that we were able to implement these approachescompletely and with little effort. The second set of studies illus-trates how the specific characteristics of a tainting approach canaffect efficiency and accuracy of the taint analysis. In particular, weinvestigate how ignoring control-flow related propagation and over-looking some data-flow aspects can lead to unsafety. These resultsfurther justify the usefulness of experimenting with different varia-tions of dynamic taint analysis and assessing their tradeoffs, whichcan be done with limited effort using our framework. The secondset of studies also shows the practical applicability of DYTAN, bysuccessfully running it on the FIREFOX web browser.196Effective Memory Protection Using Dynamic TaintingJames Clause, Ioannis Doudalis, Alessandro Orso, and Milos PrvulovicCollege of ComputingGeorgia Institute of Technology{clause|idoud|orso|milos}@cc.gatech.eduABSTRACTPrograms written in languages that provide direct access to memorythrough pointers often contain memory-related faults, which maycause non-deterministic failures and even security vulnerabilities.In this paper, we present a new technique based on dynamic taint-ing for protecting programs from illegal memory accesses. Whenmemory is allocated, at runtime, our technique taints both the mem-ory and the corresponding pointer using the same taint mark. Taintmarks are then suitably propagated while the program executes andare checked every time a memory address m is accessed through apointer p; if the taint marks associated with m and p differ, the ex-ecution is stopped and the illegal access is reported. To allow for alow-overhead, hardware-assisted implementation of the approach,we make several key technical and engineering decisions in thedefinition of our technique. In particular, we use a configurable,low number of reusable taint marks instead of a unique mark foreach area of memory allocated, which reduces the overhead of theapproach without limiting its flexibility and ability to target mostmemory-related faults and attacks known to date. We also definethe technique at the binary level, which lets us handle the (very)common case of applications that use third-party libraries whosesource code is unavailable. To investigate the effectiveness andpracticality of our approach, we implemented it for heap-allocatedmemory and performed a preliminary empirical study on a set ofprograms. Our results show that (1) our technique can identify alarge class of memory-related faults, even when using only twounique taint marks, and (2) a hardware-assisted implementation ofthe technique could achieve overhead in the single digits.Categories and Subject Descriptors: D.2.5 [Software Engineering]: Test-ing and Debugging; C.0 [General]: Hardware/Software Interfaces;General Terms: Performance, SecurityKeywords: Illegal memory accesses, dynamic tainting, hardware support1. INTRODUCTIONMemory-related faults are a serious problem for languages thatallow direct memory access through pointers. An important classof memory-related faults are what we call illegal memory accesses.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ASE’07, November 5–9, 2007, Atlanta, Georgia, USA.Copyright 2007 ACM 978-1-59593-882-4/07/0011 ...$5.00.In languages such as C and C++, when memory allocation is re-quested, a currently-free area of memory m of the specified sizeis reserved. After m has been allocated, its initial address can beassigned to a pointer p, either immediately (e.g., in the case ofheap allocated memory) or at a later time (e.g., when retrievingand storing the address of a local variable). From that point on,the only legal accesses to m through a pointer are accesses per-formed through p or through other pointers derived from p. (InSection 3, we clearly define what it means to derive a pointer fromanother pointer.) All other accesses to m are Illegal Memory Ac-cesses (IMAs), that is, accesses where a pointer is used to accessmemory outside the bounds of the memory area with which it wasoriginally associated.IMAs are especially relevant for several reasons. First, they arecaused by typical programming errors, such as array-out-of-boundsaccesses and NULL pointer dereferences, and are thus widespreadand common. Second, they often result in non-deterministic fail-ures that are hard to identify and diagnose; the specific effects of anIMA depend on several factors, such as memory layout, that mayvary between executions. Finally, many security concerns such asviruses, worms, and rootkits use IMAs as their injection vectors.In this paper, we present a new dynamic technique for protectingprograms against IMAs that is effective against most known typesof illegal accesses. The basic idea behind the technique is to usedynamic tainting (or dynamic information flow) [8] to keep trackof which memory areas can be accessed through which pointers,as follows. At runtime, our technique taints both allocated mem-ory and pointers using taint marks. Dynamic taint propagation, to-gether with a suitable handling of memory-allocation and deallo-cation operations, ensures that taint marks are appropriately prop-agated during execution. Every time the program accesses somememory through a pointer, our technique checks whether the ac-cess is legal by comparing the taint mark associated with the mem-ory and the taint mark associated with the pointer used to access it.If the marks match, the access is considered legitimate. Otherwise,the execution is stopped and an IMA is reported.In defining our approach, our final goal is the development of alow-overhead, hardware-assisted tool that is practical and can beused on deployed software. A hardware-assisted tool is a tool thatleverages the benefits of both hardware and software. Typically,some performance critical aspects are moved to the hardware toachieve maximum efficiency, while software is used to perform op-erations that would be too complex to implement in hardware.There are two main characteristics of our approach that were de-fined to help achieve our goal of a hardware-assisted implementa-tion. The first characteristic is that our technique only uses a small,configurable number of reusable taint marks instead of a uniquemark for each area of memory allocated. Using a low number of283Penumbra: Automatically Identifying Failure-RelevantInputs Using Dynamic TaintingJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing automated debugging techniques focus on re-ducing the amount of code to be inspected and tend to ig-nore an important component of software failures: the in-puts that cause the failure to manifest. In this paper, wepresent a new technique based on dynamic tainting for au-tomatically identifying subsets of a program’s inputs thatare relevant to a failure. The technique (1) marks programinputs when they enter the application, (2) tracks them asthey propagate during execution, and (3) identifies, for anobserved failure, the subset of inputs that are potentiallyrelevant for debugging that failure. To investigate feasibil-ity and usefulness of our technique, we created a prototypetool, penumbra, and used it to evaluate our technique onseveral failures in real programs. Our results are promising,as they show that penumbra can point developers to inputsthat are actually relevant for investigating a failure and canbe more practical than existing alternative approaches.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsAlgorithms, Experimentation, ReliabilityKeywordsFailure-relevant inputs, automated debugging, dynamic in-formation flow, dynamic tainting1. INTRODUCTIONDebugging is known to be a labor-intensive, time-consum-ing task that can be responsible for a large portion of soft-ware development and maintenance costs [21,23]. Commoncharacteristics of modern software, such as increased con-figurability, larger code bases, and increased input sizes, in-troduce new challenges for debugging and exacerbate exist-ing problems. In response, researchers have proposed manyPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00.semi- and fully-automated techniques that attempt to re-duce the cost of debugging (e.g., [8,9,11–13,18,24,25,27]).The majority of these techniques are code-centric in thatthey focus exclusively on one aspect of debugging—tryingto identify the faulty statements responsible for a failure.Although code-centric approaches can work well in somecases (e.g., for isolated faults that involve a single state-ment), they are often inadequate for more complex faults [4].Faults of omission, for instance, where part of a specificationhas not been implemented, are notoriously problematic fordebugging techniques that attempt to identify potentiallyfaulty statements. The usefulness of code-centric techniquesis also limited in the case of long-running programs and pro-grams that process large amounts of information; failures inthese types of programs are typically di⌅cult to understandwithout considering the data involved in such failures.To debug failures more e ectively, it is necessary to pro-vide developers with not only a relevant subset of state-ments, but also a relevant subset of inputs. There are onlya few existing techniques that attempt to identify relevantinputs [3, 17, 25], with delta debugging [25] being the mostknown of these. Although delta debugging has been shownto be an e ective technique for automatic debugging, it alsohas several drawbacks that may limit its usefulness in prac-tice. In particular, it requires (1) multiple executions of theprogram being debugged, which can involve a long runningtime, and (2) complex oracles and setup, which can resultin a large amount of manual e ort [2].In this paper, we present a novel debugging technique thataddresses many of the limitations of existing approaches.Our technique can complement code-centric debugging tech-niques because it focuses on identifying program inputs thatare likely to be relevant for a given failure. It also overcomessome of the drawbacks of delta debugging because it needsa single execution to identify failure-relevant inputs and re-quires minimal manual e ort.Given an observable faulty behavior and a set of failure-inducing inputs (i.e., a set of inputs that cause such behav-ior), our technique automatically identifies failure-relevantinputs (i.e., a subset of failure-inducing inputs that are ac-tually relevant for investigating the faulty behavior). Ourapproach is based on dynamic tainting. Intuitively, the tech-nique works by tracking the flow of inputs along data andcontrol dependences at runtime. When a point of failureis reached, the tracked information is used to identify andpresent to developers the failure-relevant inputs. At thispoint, developers can use the identified inputs to investigatethe failure at hand.LEAKPOINT: Pinpointing the Causes of Memory LeaksJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTMost existing leak detection techniques for C and C++ applicationsonly detect the existence of memory leaks. They do not provideany help for fixing the underlying memory management errors. Inthis paper, we present a new technique that not only detects leaks,but also points developers to the locations where the underlyingerrors may be fixed. Our technique tracks pointers to dynamically-allocated areas of memory and, for each memory area, records sev-eral pieces of relevant information. This information is used toidentify the locations in an execution where memory leaks occur.To investigate our technique’s feasibility and usefulness, we devel-oped a prototype tool called LEAKPOINT and used it to performan empirical evaluation. The results of this evaluation show thatLEAKPOINT detects at least as many leaks as existing tools, reportszero false positives, and, most importantly, can be effective at help-ing developers fix the underlying memory management errors.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and DebuggingGeneral TermsPerformance, ReliabilityKeywordsLeak detection, Dynamic tainting1. INTRODUCTIONMemory leaks are a type of unintended memory consumptionthat can adversely impact the performance and correctness of anapplication. In programs written in languages such as C and C++,memory is allocated using allocation functions, such as mallocand new. Allocation functions reserve a currently free area ofmemory m and return a pointer p that points to m’s starting ad-dress. Typically, the program stores and then uses p, or anotherThis work was supported in part by NSF awards CCF-0725202and CCF-0541080 to Georgia Tech.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.pointer derived from p, to interact with m. When m is no longerneeded, the program should pass p to a deallocation function (e.g.,free or delete) to deallocate m. A leak occurs if, due to amemory management error, m is not deallocated at the appropri-ate time. There are two types of memory leaks: lost memory andforgotten memory. Lost memory refers to the situation where m be-comes unreachable (i.e., the program overwrites or loses p and allpointers derived from p) without first being deallocated. Forgottenmemory refers to the situation where m remains reachable but isnot deallocated or accessed in the rest of the execution.Memory leaks are relevant for several reasons. First, they are dif-ficult to detect. Unlike many other types of failures, memory leaksdo not immediately produce an easily visible symptom (e.g., a crashor the output of a wrong value); typically, leaks remain unobserveduntil they consume a large portion of the memory available to a sys-tem. Second, leaks have the potential to impact not only the appli-cation that leaks memory, but also every other application runningon the system; because the overall amount of memory is limited,as the memory usage of a leaking program increases, less memoryis available to other running applications. Consequently, the per-formance and correctness of every running application can be im-pacted by a program that leaks memory. Third, leaks are common,even in mature applications. For example, in the first half of 2009,over 100 leaks in the Firefox web-browser were reported [18].Because of the serious consequences and common occurrence ofmemory leaks, researchers have created many static and dynamictechniques for detecting them (e.g., [1,2,4,7–14,16,17,20–23,25,27,28]). The adoption of static techniques has been limited by sev-eral factors, including the lack of scalable, precise heap modeling.Dynamic techniques are therefore more widely used in practice. Ingeneral, dynamic techniques provide one main piece of informa-tion: the location in an execution where a leaked area of memory isallocated. This location is supposed to serve as a starting point forinvestigating the leak. However, in many situations, this informa-tion does not provide any insight on where or how to fix the mem-ory management error that causes the leak: the allocation locationand the location of the memory management error are typically incompletely different parts of the application’s code.To address this limitation of existing approaches, we proposea new memory leak detection technique. Our technique providesthe same information as existing techniques but also identifies thelocations in an execution where leaks occur. In the case of lostmemory, the location is defined as the point in an execution wherethe last pointer to an unallocated memory area is lost or overwritten.In the case of forgotten memory, the location is defined as the lastpoint in an execution where a pointer to a leaked area of memorywas used (e.g., when it is dereferenced to read or write memory,passed as a function argument, returned from a function, or used asCamouflage: Automated Sanitization of Field DataJames ClauseCollege of ComputingGeorgia Institute of Technologyclause@cc.gatech.eduAlessandro OrsoCollege of ComputingGeorgia Institute of Technologyorso@cc.gatech.eduABSTRACTPrivacy and security concerns have adversely a ected theusefulness of many types of techniques that leverage infor-mation gathered from deployed applications. To address thisissue, we present a new approach for automatically sanitiz-ing failure-inducing inputs. Given an input I that causesa failure f, our technique can generate a sanitized input Ithat is di erent from I but still causes f. I can then be sentto the developers to help them debug f, without revealingthe possibly sensitive information contained in I. We im-plemented our approach in a prototype tool, camouflage,and performed an empirical evaluation. In the evaluation,we applied camouflage to a large set of failure-inducinginputs for several real applications. The results of the eval-uation are promising; they show that camouflage is bothpractical and e ective at generating sanitized inputs. In par-ticular, for the inputs that we considered, I and I sharedno sensitive information.1. INTRODUCTIONInvestigating techniques that capture data from deployedapplications to support in-house software engineering tasksis an increasingly active and successful area of research (e.g.,[1,3–5,13,14,17,21,22,26,27,29]). However, privacy and se-curity concerns have prevented widespread adoption of manyof these techniques and, because they rely on user partici-pation, have ultimately limited their usefulness. Many ofthe earlier proposed techniques attempt to sidestep theseconcerns by collecting only limited amounts of information(e.g., stack traces and register dumps [1, 3, 5] or sampledbranch profiles [26,27]) and providing a privacy policy thatspecifies how the information will be used (e.g., [2,8]). Be-cause the types of information collected by these techniquesare unlikely to be sensitive, users are more willing to trustdevelopers. Moreover, because only a small amount of infor-mation is collected, it is feasible for users to manually inspectand sanitize such information before it is sent to developers.Unfortunately, recent research has shown that the e ec-tiveness of these techniques increases when they can lever-age large amounts of detailed information (e.g., completeexecution recordings [4, 14] or path profiles [13, 24]). Sincemore detailed information is bound to contain sensitive data,users will most likely be unwilling to let developers collectsuch information. In addition, collecting large amounts ofinformation would make it infeasible for users to sanitizethe collected information by hand. To address this prob-lem, some of these techniques suggest using an input mini-mization approach (e.g., [6, 7, 35]) to reduce the number offailure-inducing inputs and, hopefully, eliminate some sensi-tive information. Input-minimization techniques, however,were not designed to specifically reduce sensitive inputs, sothey can only eliminate sensitive data by chance. In or-der for techniques that leverage captured field informationto become widely adopted and achieve their full potential,new approaches for addressing privacy and security concernsmust be developed.In this paper, we present a novel technique that addressesprivacy and security concerns by sanitizing information cap-tured from deployed applications. Our technique is designedto be used in conjunction with an execution capture/replaytechnique (e.g., [4, 14]). Given an execution recording thatcontains a captured failure-inducing input I = i1, i2, . . . in⇥and terminates with a failure f, our technique replays theexecution recording and leverages a specialized version ofsymbolic-execution to automatically produce I , a sanitizedversion of I, such that I (1) still causes f and (2) reveals aslittle information about I as possible. A modified executionrecording where I replaces I can then be constructed andsent to the developers, who can use it to debug f.It is, in general, impossible to construct I such that itdoes not reveal any information about I while still caus-ing the same failure f. Typically, the execution of f woulddepend on the fact that some elements of I have specificvalues (e.g., i1 must be 0 for the failing path to be taken).However, this fact does not prevent the technique from be-ing useful in practice. In our evaluation, we found that theinformation revealed by the sanitized inputs was not sensi-tive and tended to be structural in nature (e.g., a specificportion of the input must be surrounded by double quotes).Conversely, the parts of the inputs that were more likely tobe sensitive (e.g., values contained inside the double quotes)were not revealed (see Section 4).To evaluate the e ectiveness of our technique, we imple-mented it in a prototype tool, called camouflage, and car-ried out an empirical evaluation of 170 failure-inducing in-1CC 05 ICSE 05 ISSTA 07 ASE 07 ISSTA 09 ICSE 10 Tech ReptICSE 07Dynamic taintingbased analysesEnabling moreefficient debuggingQUESTIONS?