Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Discovering Flaws in Security-Focused Static Analysis Tools for Android using Systematic Mutation

37 views

Published on

Mobile application security has been one of the major areas of security research in the last decade. Numerous application analysis tools have been proposed in response to malicious, curious, or vulnerable apps. However, existing tools, and specifically, static analysis tools, trade soundness of the analysis for precision and performance, and are hence soundy. Unfortunately, the specific unsound choices or flaws in the design of these tools are often not known or well-documented, leading to a misplaced confidence among researchers, developers, and users. This paper proposes the Mutation-based soundness evaluation (μSE) framework, which systematically evaluates Android static analysis tools to discover, document, and fix, flaws, by leveraging the well-founded practice of mutation analysis. We implement μSE as a semi-automated framework, and apply it to a set of prominent Android static analysis tools that detect private data leaks in apps. As the result of an in-depth analysis of one of the major tools, we discover 13 undocumented flaws. More importantly, we discover that all 13 flaws propagate to tools that inherit the flawed tool. We successfully fix one of the flaws in cooperation with the tool developers. Our results motivate the urgent need for systematic discovery and documentation of unsound choices in soundy tools, and demonstrate the opportunities in leveraging mutation testing in achieving this goal.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Discovering Flaws in Security-Focused Static Analysis Tools for Android using Systematic Mutation

  1. 1. Richie Bonett, Kaushal Kafle, Kevin Moran, Adwait Nadkarni & Denys Poshyvanyk Discovering Flaws in Security-Focused Static AnalysisTools for Android using Systematic Mutation Friday,Aug 17th, 2018
  2. 2. MOTIVATION 2
  3. 3. MOTIVATION Flowdroid IccTA DroidSafe CryptoLint MalloDroid Argus-SAF Taintdroid DidFail BlueSeal End User Safer Apps 2
  4. 4. Detecting SSL Vulnerabilities OAuth-token Tracking Data Leak Detection Intent spoofing Permission misuse Password tracking • Security tools have diverse security goals • Security analysis of apps is highly beneficial to end users • Keeps the ecosystem clear of malicious or vulnerable apps SECURITY ANALYSIS OF APPLICATIONS 3
  5. 5. Detecting SSL Vulnerabilities OAuth-token Tracking Data Leak Detection Intent spoofing Permission misuse Password tracking • Security tools have diverse security goals • Security analysis of apps is highly beneficial to end users • Keeps the ecosystem clear of malicious or vulnerable apps Q: Do we really know how well these tools work? SECURITY ANALYSIS OF APPLICATIONS 3
  6. 6. • 2015: Soundiness manifesto1 • Static analysis tools are implicitly expected to be sound (i.e., they over-approximate) • In practice, all tools are soundy:A sound core, but with some unsound assumptions to be practical; e.g. JNI, Reflection • Soundy tools are practical • However, developers might not document unsound choices for various reasons SOUNDINESS [1] Livshits, Benjamin, et al. "In defense of soundiness: a manifesto." Communications of the ACM 58.2 (2015): 44-46. Static Analysis tool t Sound core Dynamic code loading …JNI Java Reflection 4
  7. 7. • 2015: Soundiness manifesto1 • Static analysis tools are implicitly expected to be sound (i.e., they over-approximate) • In practice, all tools are soundy:A sound core, but with some unsound assumptions to be practical; e.g. JNI, Reflection • Soundy tools are practical • However, developers might not document unsound choices for various reasons SOUNDINESS [1] Livshits, Benjamin, et al. "In defense of soundiness: a manifesto." Communications of the ACM 58.2 (2015): 44-46. Static Analysis tool t Sound core Dynamic code loading …JNI Java Reflection • We want to discover the extent of the unsound decisions 4
  8. 8. • The scope of the soundiness manifesto is language features • We target security analysis of mobile apps (e.g., data leak detection, SSL vuln, etc. ) • This paper:A general discussion on the design/ implementation choices in the context of the target platform, i.e.,Android, and its unique abstractions: • Application model • Inter-component communication • Asynchronous invocation and component lifecycles SOUNDINESS OF MOBILE SECURITY TOOLS Activities Intent messages BroadcastReceiver Fragments XML Resource Files Callbacks 5
  9. 9. A framework that enables systematic evaluation of existing security tools to identify and document unsound decisions, eventually expanding the sound core OUR VISION • Benefits: • Researchers: discover undocumented flaws in tools • Developers: build more effective tools by discovering easily fixed but evasive bugs • Users: benefit from better detection, and hence a better application ecosystem 6
  10. 10. μSE: MUTATION-BASED SOUNDNESS EVALUATION • μSE leverages mutation analysis for systematic evaluation of security tools • Contextualizes mutation analysis to security • Develops the abstractions of 1. Security operators and 2. Mutation schemes 7
  11. 11. MUTATION ANALYSIS BACKGROUND Mutation Engine Mutation operators Test Suite to evaluate Mutated Apks Killed Mutants Unkilled Mutants 8
  12. 12. μSE OVERVIEW • μSE leverages mutation analysis for systematic evaluation of security tools Static Analysis tool t Sound core 9
  13. 13. μSE OVERVIEW • μSE leverages mutation analysis for systematic evaluation of security tools Analyze Apps App 1 App 2 …. App nStatic Analysis tool t Sound core 9
  14. 14. μSE OVERVIEW • μSE leverages mutation analysis for systematic evaluation of security tools Analyze Apps App 1 App 2 …. App nStatic Analysis tool t Sound core Mutate apps μSE Mutants Mutation Scheme Security Operators 9
  15. 15. μSE OVERVIEW • μSE leverages mutation analysis for systematic evaluation of security tools Analyze Apps App 1 App 2 …. App n Analyze Uncaught Mutants Improved tool t’ Sound core Static Analysis tool t Sound core Mutate apps μSE Mutants Mutation Scheme Security Operators 9
  16. 16. μSE DESIGN • Basic Components and their definitions: • Security operator: What anomaly/mutation to insert in the app • Mutation scheme: Where to place/seed it 10
  17. 17. μSE DESIGN: SECURITY OPERATORS • Challenges: • Too fine-grained —> Not scalable • Too generic —> Not effective as different tools have different security focus • μSE defines security operator in terms of security goals of the tools • Scalable to tools with similar security goals (e.g., data leak detection) 11
  18. 18. μSE DESIGN: SECURITY OPERATORS boolean isServerTrusted() { return true } dataLeak = Location.read() log.d(dataLeak) 1. Operator for data leak detectors 2. Operator for SSL vulnerability detectors 12
  19. 19. • Multiple strategies with different objectives 1. Reachability analysis 2. Android Abstractions 3. Security goals μSE DESIGN: MUTATION SCHEME 13
  20. 20. 1. Reachability analysis • Placing operator at the start of every method • Helps in the evaluation of the coverage of flaws • Simplest mutation scheme for operator placement μSE DESIGN: MUTATION SCHEME 14
  21. 21. 2. Android abstractions • Model unique aspects of Android platform • Mutants are built specifically for Android by choosing its unique abstractions as the starting point • Activity & Fragment Lifecycles • Callbacks • Intent messages • Android Resource files μSE DESIGN: MUTATION SCHEME 15
  22. 22. 2. Android abstractions • Model unique aspects of Android platform • Mutants are built specifically for Android by choosing its unique abstractions as the starting point • Activity & Fragment Lifecycles • Callbacks • Intent messages • Android Resource files μSE DESIGN: MUTATION SCHEME BroadcastReceiver Leak onReceive() 15
  23. 23. 2. Android abstractions • Model unique aspects of Android platform • Mutants are built specifically for Android by choosing its unique abstractions as the starting point • Activity & Fragment Lifecycles • Callbacks • Intent messages • Android Resource files onReceive() BroadcastReceiver μSE DESIGN: MUTATION SCHEME BroadcastReceiver Leak onReceive() 15
  24. 24. 3. Security goal • Accounting for the specific objective of the tool under scrutiny • Can be applied to other tools with similar goals • E.g., a taint-based scheme for data leak detection tools μSE DESIGN: MUTATION SCHEME 16
  25. 25. 3. Security goal • Accounting for the specific objective of the tool under scrutiny • Can be applied to other tools with similar goals • E.g., a taint-based scheme for data leak detection tools μSE DESIGN: MUTATION SCHEME onStart() Source onResume() Sink 16
  26. 26. IMPLEMENTATION Security operator(s) Mutation scheme(s) Step 1: Specification Android Abstractions Security Goals 17
  27. 27. IMPLEMENTATION Mutation Engine Step 2: Mutation Security operator(s) Mutation scheme(s) Step 1: Specification Android Abstractions Security Goals 17
  28. 28. IMPLEMENTATION Mutation Engine Step 2: Mutation Test tool(s) on mutants Manual analysis Vulnerability Documentation Software Patches Step 3: Analysis Security operator(s) Mutation scheme(s) Step 1: Specification Android Abstractions Security Goals 17
  29. 29. IMPLEMENTATION Mutation Engine Step 2: Mutation Test tool(s) on mutants Manual analysis Vulnerability Documentation Software Patches Step 3: Analysis Mutation Engine Execution Engine Non-executing mutants Security operator(s) Mutation scheme(s) Step 1: Specification Android Abstractions Security Goals 17
  30. 30. IMPLEMENTATION Mutation Engine Step 2: Mutation Test tool(s) on mutants Manual analysis Vulnerability Documentation Software Patches Step 3: Analysis Mutation Engine Execution Engine Non-executing mutants Generated Mutants Execution Engine Security Tool Manual Analysis NumberofMutants Security operator(s) Mutation scheme(s) Step 1: Specification Android Abstractions Security Goals 17
  31. 31. EVALUATION • We evaluate the effectiveness of μSE using a case study •Security goal we chose for our case study: Data leak detection 18
  32. 32. EVALUATION:TESTING DATA LEAK DETECTORS • 7,584 mutants in total, 2,026 verified as executable • 3 data leak detection tools evaluated using 2,026 mutants TOOLS UNDETECTED LEAKS Flowdroid 2.0 987/2026 (48.7%) Argus-SAF 1480/2026 (73.1%) DroidSafe 83/2026 (4.1%) 19
  33. 33. EVALUATION:TESTING DATA LEAK DETECTORS • 7,584 mutants in total, 2,026 verified as executable • 3 data leak detection tools evaluated using 2,026 mutants TOOLS UNDETECTED LEAKS Flowdroid 2.0 987/2026 (48.7%) Argus-SAF 1480/2026 (73.1%) DroidSafe 83/2026 (4.1%) • Impact: Cited over 900 times • Immediate response and benefits: Flowdroid is actively being maintained 19
  34. 34. EVALUATION: FLAWS DISCOVERED VULNERABILITY CLASS (VC) EXAMPLE FLAW IN VC DESCRIPTION OF THE FLAW 1 Missing Callbacks (5 flaws) Fragments Doesn't model Android Fragments correctly. 2 Missing Implicit Calls (2 flaws) RunOnUIThread Misses a path to Runnable.run()for runnables passed into Activity.runOnUIThread() 3 Incorrect Modeling of Anonymous Classes (2 flaws) BroadcastReceiver Misses the onReceive() callback of a BroadcastReceiver implemented programmatically and registered within another programmatically defined BroadcastReceiver's onReceive() callback. 4 Incorrect Modeling of Asynchronous Methods (4 flaws) LocationListenerTaint Misses the flow from a source in the onStatusChanged() callback to a sink in the onLocationChanged() callback of the LocationListener interface, despite recognizing leaks wholly contained in either. 20
  35. 35. EVALUATION: FLAW PROPAGATION FLAW FD 2.5.1 FD 2.5.0 FD 2.0 BLUESEAL ICCTA HORNDROID ARGUS DROIDSAFE DIDFAIL 1 DialogFragmentShow ✘ ✘ ✘ ✓ ✘ ✘ ✓ ✓ ✘ 2 PhoneStateListener ✘ ✘ ✘ ✓ ✘ ✘ ✓ ✓ ✘ 3 NavigationView ✘ ✘ ✘ - ✘ - ✘ - ✘ 4 SQLiteOpenHelper ✘ ✘ ✘ ✓ ✘ ✘ ✘ ✓ ✘ 5 Fragments ✘ ✘ ✘ ✘ ✘ ✘ ✘ - ✘ 6 RunOnUIThread ✘ ✘ ✘ ✓ ✘ ✘ ✘ ✓ ✘ 7 ExecutorService ✘ ✘ ✘ ✓ ✘ ✘ ✘ ✓ ✘ 8 ButtonOnClickToDialogOnClick ✘ ✘ ✘ ✓ ✘ ✓ ✓ ✘ ✘ 9 BroadcastReceiver ✘ ✘ ✘ ✓ ✘ ✓ ✓ ✓ ✘ 10 LocationListenerTaint ✘ ✘ ✘ ✓ ✘ ✓ ✓ ✓ ✘ 11 NSDManager ✘ ✘ ✘ ✓ ✘ ✓ ✘ ✓ ✘ 12 ListViewCallbackSequential ✘ ✘ ✘ ✓ ✘ ✓ ✓ ✓ ✘ 13 ThreadTaint ✘ ✘ ✘ ✓ ✘ ✓ ✓ ✓ ✘ • Inheriting flowdroid as a black box - IccTA (13/13), DidFail (13/13) • Motivated by flowdroid’s design (but augmented to their need) - Argus-SAF (6/13) • Implementing their own methodologies - BlueSeal (1/13), HornDroid (6/13), DroidSafe (1/13) ✘ - Fails to detect 21
  36. 36. RECALL: EXPANDING THE SOUND CORE • We could fix one of the problems (fragment, FlowDroid 2.0) • However, fixing flaws is significantly challenging • Some flaws are design-choices that are hard to immediately fix (e.g. Runnable) • Some are unsolved research challenges (e.g., BroadcastReceiver) • μSE effectively serves the function of discovering/documenting these for future research 22
  37. 37. CAVEATS • μSE doesn’t claim soundness • Aims to increase the confidence in the results of soundy tools by discovering and documenting unsound choices • μSE doesn’t replace formal verification • Rather a framework for systematically uncovering flaws in security tools • Significant advancement over manually curated toolkits 23
  38. 38. CONCLUDING REMARKS • μSE demonstrates the effectiveness of mutation analysis at discovering undocumented flaws in security tools • Flaws not only affect individual tools, but propagate to future research • Android evolves, and μSE is a significant improvement over manually curated benchmarks that need keep up with Android’s fast-paced evolution • μSE allows patching of easily fixable but evasive flaws; however, this is a hard problem in general 24
  39. 39. Thank you! Kaushal Kafle William & Mary kkafle@cs.wm.edu Code and data at: https://muse-security-evaluation.github.io/ 25
  40. 40. ADDITIONAL SLIDES 26
  41. 41. • 92 minutes total time • Crashcope: systematic exploration of the application 27
  42. 42. EVALUATION FLAWS DISCOVERED 28
  43. 43. CONCLUDING REMARKS • μSE demonstrates the effectiveness of mutation analysis at discovering undocumented flaws in security tools • Flaws not only affect individual tools, but propagate to future research • Android evolves, and μSE is a significant improvement over manually curated benchmarks that need keep up with Android’s fast-paced evolution • μSE allows patching of easily fixable but evasive flaws. However, this is a hard problem in general.29

×