Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Масштабируемый и эффективный фаззинг Google Chrome

3,708 views

Published on

Ведущий: Макс Мороз

Обзор системы ClusterFuzz, позволяющей осуществить проверку браузера Chrome на наличие уязвимостей в режиме реального времени и получить воспроизводимые результаты исследования каждого конкретного сбоя. Будут продемонстрированы преимущества использования различных санитайзеров и LibFuzzer, библиотеки для направленного фаззинга. Будет приведена подробная статистика видов уязвимостей, найденных в Chrome. Слушатели узнают о подводных камнях распределенного фаззинга; о том, как можно запустить свои собственные фаззеры в инфраструктуре Google и получить вознаграждение за найденные уязвимости.

Published in: Technology

Масштабируемый и эффективный фаззинг Google Chrome

  1. 1. Scalable and Effective Fuzzing of Google Chrome Browser Positive Hack Days VI Moscow, 17-18 May 2016 Max Moroz, Google
  2. 2. ● Google Chrome Security team, Bugs-- ● BalalaikaCr3w, LC↯BC ● CTF, BugBounty, etc Bio 2
  3. 3. 1. ClusterFuzz overview 2. Memory Debugging Tools 3. LibFuzzer 4. Bugs statistics 5. Vulnerability Reward Program 6. Get some $$$ with ClusterFuzz Agenda 3
  4. 4. A software testing technique, often automated or semi-automated, that involves passing invalid, unexpected or random input to a program and monitor result for crashes, failed assertions, races, leaks, etc. Definition: Fuzzing 4
  5. 5. The Big 3s in Fuzzing 5 Fuzzers Memory Tools Scale Know WHERE to reach Know WHEN you reached Know HOW to reach FASTER
  6. 6. 6 ClusterFuzz overview
  7. 7. Suppliers of Chrome vulns 7
  8. 8. ClusterFuzz 8 BackendFrontend Bots Builder Bots 1. Trunk (Release (- O2+-O1), Debug) 2. Stable, Beta Sync Bots using GlusterFS 1. Fuzzer Data bundles 2. Build Cache 3. Code Coverage Local Storage 1. Tests 2. Fuzzers 3. Some Fuzzer Data Bundles 4. Builds Google Cloud Storage Blobstore 1. Custom Binaries 2. Crash testcases 3. User uploaded testcases 4. Fuzzers 5. Small data bundles Task Pull Queue High replication datastore All the metadata - Jobs, Crash List, Crash Stats, Fuzzers, Bot tasks, Bot heartbeats, etc Tasks 1. Fuzz 2. Minimize 3. Impact 4. Regression 5. Fixed 6. Sheriff 7. Coverage ClusterFuzz UI Appengine Google Compute Engine and Chrome Lab
  9. 9. 1. Automated crash detection, analysis and management 2. Fully reproducible and minimized testcases 3. Real-time regression and fixed testing ClusterFuzz: Goals 9
  10. 10. ClusterFuzz: Goals 10
  11. 11. ClusterFuzz: Life of a Crash 11 Find Actionable, Reproducible and Minimized Tests Verify Test Deduplication, Regresse, Production ImpactAnalyze Fixed, Merged, Released Prevent Regressions
  12. 12. 12 Goal 1: Crash automation & management
  13. 13. Fuzzer Types 13 Generation Based Generate from scratch with no prior state Example https://bugs.webkit.org/show_bug.cgi?id=60831 <script> document.body = document.createElement(‘iframe’); </script>
  14. 14. Fuzzer Types 14 Mutation Based Mutate existing state based on some rules Example crbug.com/552046 --- orig.pdf +++ crash.pdf @@ -57,7 +57,7 @@ /DecodeParms [null 8 0 R] /Type /XObject /Width 1760 -/Filter [/FlateDecode /DCTDecode] +/Filter [/JBIG2Decode /DCTDecode] /Height 1248 /Length 2277
  15. 15. Fuzzer Types 15 Evolutionary Generation or mutation based or both, in-process with code coverage feedback Example crbug.com/575205 SELECT'xef(xfb; DSx1aLEETABLxfeES'REGEXP';0ttC LE|A* (xc8*.+!*)*h*00x0b$T''&'
  16. 16. ● Setup build {trunk}, run application with test ● Choose params ● Gestures, tool settings, timeout multiplier, window location+size, etc. ● Figure out resource dependencies ● Test for reproducibility ● Test for duplicates ● Store crash, coverage, stats, etc Infrastructure 16
  17. 17. ● Disable inline frames (-inlining in llvm-symbolizer) ● Crash_type - e.g. “Heap-buffer-overflow READ 2” ● Crash_state - top 3 frames ● +namespaces, -line_numbers ● E.g. ● WebCore::SVGDocumentExtensions::removeAnimationElement ● WebCore::SVGSMILElement::removedFrom ● WebCore::ContainerNode::removeChild ● security_flag Testcase Duplication Check 17
  18. 18. Code Coverage: Aggregate View Sample 18
  19. 19. Scale, Scale and Scale (in cores) 19 12k 5k Linux 4k Android x86 200+ Android devices 3k Windows 20 Mac {wip} iOS
  20. 20. 20 Goal 2: Fully reproducible and minimized testcases
  21. 21. ● Same bot configuration as crash ● Multi-threaded minimization based on Delta Debugging ● Custom minimizers for some file types ● + Gesture minimization ● + Resource minimization ● + Tool settings correction (redzone {asan}, history size {tsan}, origins {msan}, etc) Fully reproducible & minimized tests 21
  22. 22. ● Tokenize the input ● Generate hypotheses that certain groups of tokens are not required for the crash ● Test hypotheses by running the test with the tokens from the hypothesis removed ● If it crashes, removing them was fine ● If not, try breaking it into smaller groups How does the minimizer work? 22
  23. 23. ● Stack trace with default redzone (128) ==9485== ERROR: AddressSanitizer heap-use-after-free on address 0x7f8f653ff11e at pc 0x7f8f849fbb10 bp 0x7f8f5514a0a0 sp 0x7f8f5514a098 READ of size 2 at 0x7f8f653ff11e thread T14 #0 0x7f8f849fbb10 in WTF::charactersToIntStrict(unsigned short const*, unsigned long, bool*, int) #1 0x7f8f8589d863 in WebCore::InlineTextBox::isLineBreak() const #2 0x7f8f858a771d in WebCore::InlineTextBox::containsCaretOffset(int) ● Actual stack trace with bigger redzone (1024 / 2048) ==14334== ERROR: AddressSanitizer heap-buffer-overflow on address 0x7f7e42b9b81c at pc 0x7f7e8f79a6ca bp 0x7f7e3cc30040 sp 0x7f7e3cc30038 READ of size 2 at 0x7f7e42b9b81c thread T15 #0 0x7f7e8f79a6ca in WebCore::InlineTextBox::isLineBreak() const #1 0x7f7e8f7abddd in WebCore::InlineTextBox::containsCaretOffset(int) const #2 0x7f7e8e06b19d in WebCore::Position::inRenderedText() const Tools Settings Correction: ASAN example crbug.com/118662 23
  24. 24. 24 Goal 3: Real-time regression and fixed testing
  25. 25. ● Use LKGR builds archived on Google Cloud ● Account for bad builds / startup crashes ● Use a LOOK_BEHIND_WINDOW ● If previous step failed, then use binary bisect ● Use FindIt to find culprit changelist Real-time regression and fixed testing 25
  26. 26. ● Manual owner triage is usually … ● Slow ● Inaccurate ● Complex ● From factors such as refactorings, size of regression range, etc FindIt: Culprit CL Finder 26
  27. 27. 1. Parse the stacktrace FindIt: How it works 27
  28. 28. 2. Parse the changelog(s) in the regression range FindIt: How it works 28
  29. 29. 3. Generate a list of suspected CLs, and sort / filter the results FindIt: How it works 29
  30. 30. 4. Show blame information if no results are available FindIt: How it works 30
  31. 31. FindIt: How it works 31
  32. 32. 32 ClusterFuzz: Sample
  33. 33. Testcase Report 33
  34. 34. FindIt: Culprit CL Finder 34
  35. 35. Filed bug: crbug.com/430925 35
  36. 36. Patched + Verified: < 1 day 36
  37. 37. Or not patched... 37
  38. 38. 38 Memory Debugging Tools
  39. 39. Memory Debugging Tools 39 Tool OS Android Chrome OS Linux OS X Windows ASan + + + + + MSan + UBSan + TSan + LSan + CFI + SyzyASAN + DrFuzz + DrMemory +
  40. 40. ● AddressSanitizer (aka ASan) ● Detects use-after-free, buffer overflows (heap, stack, globals), stack-use-after-return, container-overflow ● Cpu: 2x, memory 1.5x-3x ● ThreadSanitizer (aka TSan) ● Detects data races, esp on use-after-frees. Object vtpr ● Cpu: 4x-10x, memory: 5x-8x Memory Debugging Tools 40
  41. 41. ● MemorySanitizer (aka MSan) ● Detects uninitialized memory reads ● Cpu: 3x, memory: 2x ● Special mode: origins ● UndefinedBehaviorSanitizer (aka UBSan) ● Detects several classes of bugs (19?), esp on type confusion, signed-integer-overflow, undefined shift, etc. ● Cpu: 10-50% ● Memory: ~1x (no allocator, no shadow) Memory Debugging Tools 41
  42. 42. ● Others ● SyzyASAN ● DrFuzz ● DrMemory Memory Debugging Tools 42
  43. 43. ● Container-overflow (ASan): #include <vector> #include <assert.h> typedef long T; int main() { std::vector<T> v; v.push_back(0); v.push_back(1); v.push_back(2); assert(v.capacity() >= 4); assert(v.size() == 3); T *p = &v[0]; // Here the memory is accessed inside a heap-allocated buffer // but outside of the region `[v.begin(), v.end())`. return p[3]; // OOPS. } Memory Debugging Tools: Examples 43
  44. 44. 44 LibFuzzer
  45. 45. ● Fuzzing very wide scope ● Cannot fuzz specific function ● Hard to fuzz network protocols ● Speed of regular fuzzers (html, css, dom, etc mutators): ● From ~30K to ~2.5M testcases per week ● Regular fuzzers are great, but not enough... Problems of regular fuzzing of browsers 45
  46. 46. ● More granulated fuzzing ● Faster fuzzing ● Smarter fuzzing ● Easier fuzzer writing Goals 46
  47. 47. ● In-process guided fuzz testing ● Very effective at function / protocol level ● It’s easy not hard to write a “target function” ● Can be checked along with unit-tests ● Maintained by Googlers :) LibFuzzer 47
  48. 48. libFuzzer example [code] 48 #include <stdint.h> #include "third_party/icu/source/i18n/unicode/regex.h" // Entry point for LibFuzzer. extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) { UParseError pe = { 0 }; UErrorCode status = U_ZERO_ERROR; URegularExpression* re = uregex_open(reinterpret_cast<const UChar*>(data), static_cast<int>(size) / sizeof(UChar), 0, &pe, &status); if (re) uregex_close(re); return 0; }
  49. 49. libFuzzer example [crash] 49 ==19494==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4d306d in icu_54::RegexCompile::doParseActions(int) third_party/icu/source/i18n/regexcmp.cpp:1702:13 #1 0x4a1f3c in icu_54::RegexCompile::compile(UText*, UParseError&, UErrorCode&) third_party/icu/source/i18n/regexcmp.cpp:228:13 #2 0x5fc98e in icu_54::RegexPattern::compile(UText*, unsigned int, UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:357:5 #3 0x601be7 in uregex_open_54 third_party/icu/source/i18n/uregex.cpp:155:20 #4 0x49dc3f in LLVMFuzzerTestOneInput testing/libfuzzer/fuzzers/icu_uregex_open_fuzzer.cc:13:28 Uninitialized value was created by an allocation of compiler in the stack frame of function _ZN6icu_5412RegexPattern7compileEP5UTextjR11UParseErrorR10UErrorCode #0 0x5fc220 in icu_54::RegexPattern::compile(UText*, unsigned int, UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:325
  50. 50. ● Dictionaries ● Seed corpus ● Custom mutators ● Coverage visualisation Ways to improve a libFuzzer 50
  51. 51. ● dictionary and custom options for ICU uregex_open() fuzzer: ○ coverage ↑ 25+% ● dictionary for libxml fuzzer: ○ 5 new security bugs ● dictionary and seed corpus for libpng fuzzer: ○ coverage ↑↑↑ 70+% Dictionaries are awesome! 51
  52. 52. 52
  53. 53. 53
  54. 54. 54
  55. 55. 55
  56. 56. ● Timeout (DoS) ● Heap-buffer-overflow ● Use-of-uninitialized-value ● Heap-buffer-overflow ● Heap-buffer-overflow sqlite3 fuzzer with dictionary at ClusterFuzz 56
  57. 57. ● ASan, MSan, UBSan builds ● Corpus Synchronization with Google Cloud Storage ● Support for dictionaries and custom runtime options ● Automated exploitability scoring for crashes ● Statistics dashboard LibFuzzer & ClusterFuzz 57
  58. 58. ● ~85 target functions for Chromium ● 88 bugs filed ○ 46 security bugs ● 700 GCE bots running target functions at ClusterFuzz 24/7 ● ~310 billion testcases daily ● ~8,000 corpus synchronization jobs daily ● From ~200k to ~800k corpus files ● Best coverage: 100.00% of funcs, 89.32% of edges Numbers 58
  59. 59. ● 200+ bugs in third party, Google, other software ● http://llvm.org/docs/LibFuzzer.html#trophies: Trophies of LibFuzzer outside of Chrome 59
  60. 60. ● Slow inputs and timeouts ● Out-of-memory with valid inputs ● You have to improve / refresh some fuzzers ● Stuck at easy crashes Problems 60
  61. 61. Example: 10 lines fuzzer 61 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { auto var = xxxxxxxxxxxxxx(data, size, NULL, 0); if (var) { xxxxxxxxxx_Free(var); } return 0; }
  62. 62. Example: 10 lines fuzzer ⇨ 10+ bugs 62 ● Memory leak ● 7 × Heap-buffer-overflow ● Timeout (DoS) ● Heap-based buffer-underreads ● Heap-based buffer overread ● Undesired side effects
  63. 63. Self-fuzzing! 63 WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x56b2594 in fuzzer::Fuzzer::AlarmCallback() third_party/llvm/lib/Fuzzer/FuzzerLoop.cpp:138:7 <...> #9 0x56abbb7 in fuzzer::FileToVector() third_party/llvm/lib/Fuzzer/FuzzerIO.cpp: 65:3 <...> #14 0x56a08a8 in main third_party/llvm/lib/Fuzzer/FuzzerMain.cpp:25:10 Uninitialized value was created by an allocation of 'F.i' in the stack frame of function '_ZN6fuzzer12FuzzerDriverEiPPcPFiPKhmE' #0 0x56a1040 in fuzzer::FuzzerDriver() third_party/llvm/lib/Fuzzer/FuzzerDriver.cpp:395
  64. 64. 64
  65. 65. ● The same target function ● Shared corpus: ● Need to be pruned or minimized on regular basis ● Different mutations: ● When one fuzzer got stuck, another one may help ● PoC is ready: ● https://github.com/llvm-mirror/llvm/blob/master/lib/Fuzzer/afl/afl_driver.cpp ● Work in progress Multiple fuzzing engines approach 65
  66. 66. 66 return 0;
  67. 67. 67 Bugs statistics
  68. 68. Bugs statistics 68
  69. 69. 69 Bugs statistics
  70. 70. 70 Bugs statistics
  71. 71. 71 Vulnerability Reward Program
  72. 72. ● Reporter submits a bug on crbug.com ● Bug gets fixed ● Reward panel determines reward based on table and precedent ● Reporter gets cash money Chromium VRP: high level process 72
  73. 73. Reward amounts 73 https://www.google.com/about/appsecurity/chrome-rewards/index.html
  74. 74. Recent updates 74 https://www.google.com/about/appsecurity/chrome-rewards/index.html ● Max reward: $50,000 ⇨ $100,000: ● compromise a Chromebook or Chromebox with device persistence in guest mode (i.e. guest to guest persistence with interim reboot, delivered via a web page) ● Download Protection bypass (i.e. SafeBrowsing) *: ● Baseline: $0 - $500 ● High-quality report: $1,000 * Landing a blacklisted test binary on disk where a typical user could execute it, on Mac or Windows. The file type on disk must lead to non-sandboxed code execution after minimal user interaction with the file.
  75. 75. Chromium VRP stats 75 ● Over $500,000 paid out last year ● More than $2,500,000 in rewards since 2010 ● Median payment went from $2,000 to $3,000 ● Top 0x05 reporters take ~70% of the earnings
  76. 76. 76 Trusted Researcher Program
  77. 77. ClusterFuzz’s first days... 77
  78. 78. ClusterFuzz now 78
  79. 79. ● Invitation-only ● Skilled fuzzer developers ● Fuzz at Google scale ● Get 100% of the reward for bugs + $500 bonus: ● Bugs shouldn’t be found by our* fuzzers within 48 hours ● *our - written by Googlers ● Target functions for LibFuzzer are also in scope! How to get an invite? ● Submit at least one bug found by one of your fuzzers ● mailto:mmoroz@chromium.org Trusted Researcher program 79 https://www.google.com/about/appsecurity/chrome-rewards/index.html http://www.chromium.org/Home/chromium-security/bugs/developing-fuzzers-for-clusterfuzz
  80. 80. ● More than $120,000 paid out for 2014-2015 ● Rewards in range: $500 - $4,500 Stats 80
  81. 81. Thank you Contacts ● email: mmoroz@chromium.org ● twitter: @dor3s ● telegram: @dor1s Q & A 81

×