Масштабируемый и эффективный фаззинг Google Chrome

Scalable and Effective Fuzzing
of Google Chrome Browser
Positive Hack Days VI
Moscow, 17-18 May 2016
Max Moroz, Google

● Google Chrome Security team, Bugs--
● BalalaikaCr3w, LC↯BC
● CTF, BugBounty, etc
Bio
2

1. ClusterFuzz overview
2. Memory Debugging Tools
3. LibFuzzer
4. Bugs statistics
5. Vulnerability Reward Program
6. Get some $$$ with ClusterFuzz
Agenda
3

A software testing technique, often
automated or semi-automated,
that involves passing invalid,
unexpected or random input to a
program and monitor result for
crashes, failed assertions, races,
leaks, etc.
Definition: Fuzzing
4

The Big 3s in Fuzzing
5
Fuzzers Memory Tools Scale
Know
WHERE
to
reach
Know
WHEN
you
reached
Know
HOW
to
reach
FASTER

ClusterFuzz
8
BackendFrontend
Bots
Builder Bots
1. Trunk (Release (-
O2+-O1), Debug)
2. Stable, Beta
Sync Bots using
GlusterFS
1. Fuzzer Data
bundles
2. Build Cache
3. Code Coverage
Local Storage
1. Tests
2. Fuzzers
3. Some Fuzzer
Data Bundles
4. Builds
Google Cloud
Storage
Blobstore
1. Custom Binaries
2. Crash testcases
3. User uploaded testcases
4. Fuzzers
5. Small data bundles
Task Pull
Queue
High replication datastore
All the metadata - Jobs, Crash
List, Crash Stats, Fuzzers, Bot
tasks, Bot heartbeats, etc
Tasks
1. Fuzz
2. Minimize
3. Impact
4. Regression
5. Fixed
6. Sheriff
7. Coverage
ClusterFuzz
UI
Appengine Google Compute Engine and Chrome Lab

1. Automated crash detection, analysis and management
2. Fully reproducible and minimized testcases
3. Real-time regression and fixed testing
ClusterFuzz: Goals
9

ClusterFuzz: Life of a Crash
11
Find Actionable, Reproducible and Minimized Tests
Verify
Test Deduplication, Regresse, Production ImpactAnalyze
Fixed, Merged, Released
Prevent Regressions

12
Goal 1:
Crash automation & management

Fuzzer Types
13
Generation Based
Generate from scratch with
no prior state
Example
https://bugs.webkit.org/show_bug.cgi?id=60831
<script>
document.body = document.createElement(‘iframe’);
</script>

Fuzzer Types
14
Mutation Based
Mutate existing state
based on some rules
Example
crbug.com/552046
--- orig.pdf
+++ crash.pdf
@@ -57,7 +57,7 @@
/DecodeParms [null 8 0 R]
/Type /XObject
/Width 1760
-/Filter [/FlateDecode /DCTDecode]
+/Filter [/JBIG2Decode /DCTDecode]
/Height 1248
/Length 2277

Fuzzer Types
15
Evolutionary
Generation or mutation
based or both, in-process
with code coverage
feedback
Example
crbug.com/575205
SELECT'xef(xfb;
DSx1aLEETABLxfeES'REGEXP';0ttC LE|A*
(xc8*.+!*)*h*00x0b$T''&'

● Setup build {trunk}, run application with test
● Choose params
● Gestures, tool settings, timeout multiplier, window
location+size, etc.
● Figure out resource dependencies
● Test for reproducibility
● Test for duplicates
● Store crash, coverage, stats, etc
Infrastructure
16

● Disable inline frames (-inlining in llvm-symbolizer)
● Crash_type - e.g. “Heap-buffer-overflow READ 2”
● Crash_state - top 3 frames
● +namespaces, -line_numbers
● E.g.
● WebCore::SVGDocumentExtensions::removeAnimationElement
● WebCore::SVGSMILElement::removedFrom
● WebCore::ContainerNode::removeChild
● security_flag
Testcase Duplication Check
17

Code Coverage: Aggregate View Sample
18

Scale, Scale and Scale (in cores)
19
12k
5k
Linux
4k
Android x86
200+
Android devices
3k
Windows
20
Mac
{wip}
iOS

20
Goal 2:
Fully reproducible and minimized
testcases

● Same bot configuration as crash
● Multi-threaded minimization based on Delta Debugging
● Custom minimizers for some file types
● + Gesture minimization
● + Resource minimization
● + Tool settings correction (redzone {asan}, history size
{tsan}, origins {msan}, etc)
Fully reproducible & minimized tests
21

● Tokenize the input
● Generate hypotheses that certain groups of tokens are not
required for the crash
● Test hypotheses by running the test with the tokens from the
hypothesis removed
● If it crashes, removing them was fine
● If not, try breaking it into smaller groups
How does the minimizer work?
22

● Stack trace with default redzone (128)
==9485== ERROR: AddressSanitizer heap-use-after-free on address
0x7f8f653ff11e at
pc 0x7f8f849fbb10 bp 0x7f8f5514a0a0 sp 0x7f8f5514a098
READ of size 2 at 0x7f8f653ff11e thread T14
#0 0x7f8f849fbb10 in WTF::charactersToIntStrict(unsigned short const*,
unsigned long, bool*, int)
#1 0x7f8f8589d863 in WebCore::InlineTextBox::isLineBreak() const
#2 0x7f8f858a771d in WebCore::InlineTextBox::containsCaretOffset(int)
● Actual stack trace with bigger redzone (1024 / 2048)
==14334== ERROR: AddressSanitizer heap-buffer-overflow on address
0x7f7e42b9b81c
at pc 0x7f7e8f79a6ca bp 0x7f7e3cc30040 sp 0x7f7e3cc30038
READ of size 2 at 0x7f7e42b9b81c thread T15
#0 0x7f7e8f79a6ca in WebCore::InlineTextBox::isLineBreak() const
#1 0x7f7e8f7abddd in WebCore::InlineTextBox::containsCaretOffset(int)
const
#2 0x7f7e8e06b19d in WebCore::Position::inRenderedText() const
Tools Settings Correction: ASAN example
crbug.com/118662
23

24
Goal 3:
Real-time regression and fixed testing

● Use LKGR builds archived on Google Cloud
● Account for bad builds / startup crashes
● Use a LOOK_BEHIND_WINDOW
● If previous step failed, then use binary bisect
● Use FindIt to find culprit changelist
Real-time regression and fixed testing
25

● Manual owner triage is usually …
● Slow
● Inaccurate
● Complex
● From factors such as refactorings, size of regression range, etc
FindIt: Culprit CL Finder
26

1. Parse the stacktrace
FindIt: How it works
27

2. Parse the changelog(s) in the regression range
28

3. Generate a list of suspected CLs, and sort / filter the results
29

4. Show blame information if no results are available
30

Filed bug: crbug.com/430925
35

Patched + Verified: < 1 day
36

Memory Debugging Tools
39
Tool OS Android Chrome OS Linux OS X Windows
ASan + + + + +
MSan +
UBSan +
TSan +
LSan +
CFI +
SyzyASAN +
DrFuzz +
DrMemory +

● AddressSanitizer (aka ASan)
● Detects use-after-free, buffer overflows (heap, stack, globals),
stack-use-after-return, container-overflow
● Cpu: 2x, memory 1.5x-3x
● ThreadSanitizer (aka TSan)
● Detects data races, esp on use-after-frees. Object vtpr
● Cpu: 4x-10x, memory: 5x-8x
40

● MemorySanitizer (aka MSan)
● Detects uninitialized memory reads
● Cpu: 3x, memory: 2x
● Special mode: origins
● UndefinedBehaviorSanitizer (aka UBSan)
● Detects several classes of bugs (19?), esp on type confusion,
signed-integer-overflow, undefined shift, etc.
● Cpu: 10-50%
● Memory: ~1x (no allocator, no shadow)
41

● Others
● SyzyASAN
● DrFuzz
● DrMemory
42

● Container-overflow (ASan):
#include <vector>
#include <assert.h>
typedef long T;
int main() {
std::vector<T> v;
v.push_back(0);
v.push_back(1);
v.push_back(2);
assert(v.capacity() >= 4);
assert(v.size() == 3);
T *p = &v[0];
// Here the memory is accessed inside a heap-allocated buffer
// but outside of the region `[v.begin(), v.end())`.
return p[3]; // OOPS.
}
Memory Debugging Tools: Examples
43

● Fuzzing very wide scope
● Cannot fuzz specific function
● Hard to fuzz network protocols
● Speed of regular fuzzers (html, css, dom, etc mutators):
● From ~30K to ~2.5M testcases per week
● Regular fuzzers are great, but not enough...
Problems of regular fuzzing of browsers
45

● More granulated fuzzing
● Faster fuzzing
● Smarter fuzzing
● Easier fuzzer writing
Goals
46

● In-process guided fuzz testing
● Very effective at function / protocol level
● It’s easy not hard to write a “target function”
● Can be checked along with unit-tests
● Maintained by Googlers :)
LibFuzzer
47

libFuzzer example [code]
48
#include <stdint.h>
#include "third_party/icu/source/i18n/unicode/regex.h"
// Entry point for LibFuzzer.
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
UParseError pe = { 0 };
UErrorCode status = U_ZERO_ERROR;
URegularExpression* re = uregex_open(reinterpret_cast<const UChar*>(data),
static_cast<int>(size) / sizeof(UChar),
0, &pe, &status);
if (re)
uregex_close(re);
return 0;
}

libFuzzer example [crash]
49
==19494==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x4d306d in icu_54::RegexCompile::doParseActions(int)
third_party/icu/source/i18n/regexcmp.cpp:1702:13
#1 0x4a1f3c in icu_54::RegexCompile::compile(UText*, UParseError&,
UErrorCode&) third_party/icu/source/i18n/regexcmp.cpp:228:13
#2 0x5fc98e in icu_54::RegexPattern::compile(UText*, unsigned int,
UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:357:5
#3 0x601be7 in uregex_open_54 third_party/icu/source/i18n/uregex.cpp:155:20
#4 0x49dc3f in LLVMFuzzerTestOneInput
testing/libfuzzer/fuzzers/icu_uregex_open_fuzzer.cc:13:28
Uninitialized value was created by an allocation of compiler in the stack frame
of function _ZN6icu_5412RegexPattern7compileEP5UTextjR11UParseErrorR10UErrorCode
#0 0x5fc220 in icu_54::RegexPattern::compile(UText*, unsigned int,
UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:325

● Dictionaries
● Seed corpus
● Custom mutators
● Coverage visualisation
Ways to improve a libFuzzer
50

● dictionary and custom options for ICU uregex_open() fuzzer:
○ coverage ↑ 25+%
● dictionary for libxml fuzzer:
○ 5 new security bugs
● dictionary and seed corpus for libpng fuzzer:
○ coverage ↑↑↑ 70+%
Dictionaries are awesome!
51

● Timeout (DoS)
● Heap-buffer-overflow
● Use-of-uninitialized-value
sqlite3 fuzzer with dictionary at ClusterFuzz
56

● ASan, MSan, UBSan builds
● Corpus Synchronization with Google Cloud Storage
● Support for dictionaries and custom runtime options
● Automated exploitability scoring for crashes
● Statistics dashboard
LibFuzzer & ClusterFuzz
57

● ~85 target functions for Chromium
● 88 bugs filed
○ 46 security bugs
● 700 GCE bots running target functions at ClusterFuzz 24/7
● ~310 billion testcases daily
● ~8,000 corpus synchronization jobs daily
● From ~200k to ~800k corpus files
● Best coverage: 100.00% of funcs, 89.32% of edges
Numbers
58

● 200+ bugs in third party, Google, other software
● http://llvm.org/docs/LibFuzzer.html#trophies:
Trophies of LibFuzzer outside of Chrome
59

● Slow inputs and timeouts
● Out-of-memory with valid inputs
● You have to improve / refresh some fuzzers
● Stuck at easy crashes
Problems
60

Example: 10 lines fuzzer
61
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
auto var = xxxxxxxxxxxxxx(data, size, NULL, 0);
if (var) {
xxxxxxxxxx_Free(var);
}
return 0;
}

Example: 10 lines fuzzer ⇨ 10+ bugs
62
● Memory leak
● 7 × Heap-buffer-overflow
● Timeout (DoS)
● Heap-based buffer-underreads
● Heap-based buffer overread
● Undesired side effects

Self-fuzzing!
63
WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x56b2594 in fuzzer::Fuzzer::AlarmCallback()
third_party/llvm/lib/Fuzzer/FuzzerLoop.cpp:138:7
<...>
#9 0x56abbb7 in fuzzer::FileToVector() third_party/llvm/lib/Fuzzer/FuzzerIO.cpp:
65:3
<...>
#14 0x56a08a8 in main third_party/llvm/lib/Fuzzer/FuzzerMain.cpp:25:10
Uninitialized value was created by an allocation of 'F.i' in the stack frame of
function '_ZN6fuzzer12FuzzerDriverEiPPcPFiPKhmE'
#0 0x56a1040 in fuzzer::FuzzerDriver()
third_party/llvm/lib/Fuzzer/FuzzerDriver.cpp:395

● The same target function
● Shared corpus:
● Need to be pruned or minimized on regular basis
● Different mutations:
● When one fuzzer got stuck, another one may help
● PoC is ready:
● https://github.com/llvm-mirror/llvm/blob/master/lib/Fuzzer/afl/afl_driver.cpp
● Work in progress
Multiple fuzzing engines approach
65

71
Vulnerability Reward Program

● Reporter submits a bug on crbug.com
● Bug gets fixed
● Reward panel determines reward based on table and precedent
● Reporter gets cash money
Chromium VRP: high level process
72

Reward amounts
73 https://www.google.com/about/appsecurity/chrome-rewards/index.html

Recent updates
74 https://www.google.com/about/appsecurity/chrome-rewards/index.html
● Max reward: $50,000 ⇨ $100,000:
● compromise a Chromebook or Chromebox with device persistence in guest
mode (i.e. guest to guest persistence with interim reboot, delivered via a web page)
● Download Protection bypass (i.e. SafeBrowsing) *:
● Baseline: $0 - $500
● High-quality report: $1,000
* Landing a blacklisted test binary on disk where a typical user could execute it, on Mac
or Windows. The file type on disk must lead to non-sandboxed code execution after
minimal user interaction with the file.

Chromium VRP stats
75
● Over $500,000 paid out last year
● More than $2,500,000 in rewards since 2010
● Median payment went from $2,000 to $3,000
● Top 0x05 reporters take ~70% of the earnings

ClusterFuzz’s first days...
77

● Invitation-only
● Skilled fuzzer developers
● Fuzz at Google scale
● Get 100% of the reward for bugs + $500 bonus:
● Bugs shouldn’t be found by our* fuzzers within 48 hours
● *our - written by Googlers
● Target functions for LibFuzzer are also in scope!
How to get an invite?
● Submit at least one bug found by one of your fuzzers
● mailto:mmoroz@chromium.org
Trusted Researcher program
79
https://www.google.com/about/appsecurity/chrome-rewards/index.html
http://www.chromium.org/Home/chromium-security/bugs/developing-fuzzers-for-clusterfuzz

● More than $120,000 paid out for 2014-2015
● Rewards in range: $500 - $4,500
Stats
80

Thank you
Contacts
● email: mmoroz@chromium.org
● twitter: @dor3s
● telegram: @dor1s
Q & A
81

Масштабируемый и эффективный фаззинг Google Chrome

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Масштабируемый и эффективный фаззинг Google Chrome

Similar to Масштабируемый и эффективный фаззинг Google Chrome (20)

More from Positive Hack Days

More from Positive Hack Days (20)

Recently uploaded

Recently uploaded (20)

Масштабируемый и эффективный фаззинг Google Chrome