Making a Scalable Automated Hacking System by Artem Dinaburg
In this presentation, I’ll tell the story of our Cyber Grand Challenge adventure, explain how to automatically find and patch bugs in binary code, and discuss what’s next for our bug finding system. The story will describe how our small team of internationally distributed engineers made an automated bug finding system that placed 2nd in vulnerability discovery. I will cover both the fun parts and the necessary-but-boring-parts of automated bug finding. Fun parts include combining existing fuzzing and symbolic execution tools into one coherent system, and making fuzzing fast by identifying and eliminating performance bottlenecks. The necessary-but-boring-parts include automated testing, deployment, and configuration management, otherwise known as devops. Second, I’ll talk about how to patch bugs by translating binaries to LLVM bitcode, patching the bitcode, and re-emitting working patched binaries. I will cover different patching strategies and the requirements for each approach. I will also discuss instrumentation techniques, transformation operations, and analysis passes that are enabled by LLVM translation. Finally, I will talk about how researchers should fundamentally change the way bug finding tools are developed. Currently each tools is its own discrete island. However, there are quantifiable benefits to be gained by applying the Unix philosophy of discrete, communicating tools to the problem of bug finding.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Making a Scalable Automated Hacking System by Artem Dinaburg
Similar to Making a Scalable Automated Hacking System by Artem Dinaburg (20)
More from Shakacon
More from Shakacon (20)
Recently uploaded
Recently uploaded (20)
Making a Scalable Automated Hacking System by Artem Dinaburg
- 8. Outline • Part 1: Learn how our automated bug finding system works & how to make your own • Part 2: Understand our approach to automated binary patching • What’s Next: the future of automated vulnerability discovery
- 11. IntroducLon • Trail of Bits was a compeLtor in DARPA’s Cyber Grand Challenge • We built Cyberdyne, an automated bug finding and patching system • It worked preFy well…
- 12. 2nd in Bugs Found 77 65 57 57 44 39 23 12 12 9 0 10 20 30 40 50 60 70 80 Team A Cyberdyne Team B Team C Team D Team E Team F Team G Team H Team I Confirmed Bugs Found
- 14. IntroducLon • Trail of Bits was a compeLtor in DARPA’s Cyber Grand Challenge • We built Cyberdyne, an automated bug finding and patching system • It worked preFy well… • But didn’t qualify 😢
- 22. Under-Approximate Analyses: Theory • All tools operate over the same domain • All discoveries are equally true • What if tools could share discoveries? © flickr user Jean-Pierre Dalbéra
- 31. Analysis BoosLng • Inputs generated by one tool feed into all others Inputs Analyses New Inputs Merge Knowledge
- 34. Analysis BoosLng: ImplementaLon • Two symbolic execuLon engines – KLEE (via an x86 to LLVM translator) – PySymEmu • Grr: a really awesome DBT-based fuzzer • Merge knowledge via MinSet – Minimal set of maximal coverage inputs
- 37. PracLcal ConsideraLons: MinSet • Minimizing the input set is essenLal for analysis boosLng to be pracLcal. • MinSet needs a fast method to measure and compare coverage. • Doesn’t have to be perfect.
- 46. Building Your Own • Step 8: Control and Provision MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage MessageQueue Distributed Storage Control & Provision
- 47. Obligatory AFL Comparison 78 69 68 65 61 16 14 0 10 20 30 40 50 60 70 80 Boosted (current) Boosted (CQE) AFL (Driller Paper) Union (KLEE,PSE,Grr) Grr (Fuzzing) PSE (Symbolic) KLEE (Symbolic) Bugs Found in CQE Binaries By Method (approximate)
- 51. Patching • Make the binary not crash • Patching Method – What do you “fix”? • Patching Mechanics – How do you apply the “fix”?
- 52. Patching • Make the binary not crash • Patching Method – What do you “fix”? • Patching Mechanics – How do you apply the “fix”? • Patch LocaLon – Where do you fix, if you aren’t sure of the bug?
- 56. Patching • Patching Method – Prevent access to invalid memory • Patching Mechanics – Translate binary to LLVM, patch the bitcode • Patch LocaLon – Specific vs. Generic
- 57. Patch LocaLon: Specific Patching • So, we have this thing that finds bugs… – Lets just patch those • Insert check for memory validity • Problems – Wrong bug – MulLple code paths
- 58. Patch LocaLon: Generic Patching • Fix every bug of a certain class • Examples: – Stack Cookies – ASLR – CFI • Problem: slow
- 63. Patching Montage • Cross-Block Dead Store EliminaLon • Unused register analysis • Dominator Tree Traversal
- 64. Patching Montage • Cross-Block Dead Store EliminaLon • Unused register analysis • Dominator Tree Traversal • Data Flow from inputs
- 65. Patching Montage • Cross-Block Dead Store EliminaLon • Unused register analysis • Dominator Tree Traversal • Data Flow from inputs • CombinaLons of the above
- 67. What’s Next? • Cyber Grand Challenge conLnues! • Finals are August 4th, co-located with DEFCON – Free to enter • Winner will play the winning human CTF team at DEFCON
- 70. What’s Next? • Lets automate (or semi-automate) boring assessment work. • We are doing a pilot audit of zlib for the Mozilla FoundaLon. • Eventually we would like to automaLcally audit open source sosware.