Testing curl for security
•0 likes•1,246 views
Report
Share
Download to read offline
Daniel Stenberg's presentation done on June 30, 2020 about curl testing, CI, QA, fuzzing and more.
More Related Content
What's hot
What's hot(20)
Similar to Testing curl for security
Similar to Testing curl for security(20)
More from Daniel Stenberg
More from Daniel Stenberg(15)
Recently uploaded
Recently uploaded(20)
Testing curl for security
- 4. … ten billion installations across the world curl runs in... @bagder@bagder
- 5. …… devices all of you heredevices all of you here today own and use!today own and use! curl runs in @bagder@bagder
- 6. Writing safe codeWriting safe code @bagder@bagder
- 7. Safe code is not a coincidence Doesn’t happen randomly, we need to work on it! We care about it! It is about time we all address this topic! Everything I’ll tell you today, you already know! @bagder@bagder
- 8. Writing Help users use the tool and APIs safelyHelp users use the tool and APIs safely Defaults!Defaults! Document. Everything, detailed and a lot.Document. Everything, detailed and a lot. Get rid of warningsGet rid of warnings UseUse commentscomments assert()assert() Releasing the code asReleasing the code as open sourceopen source @bagder@bagder
- 9. Review All codeAll code shallshall get reviewedget reviewed CodeCode shallshall be easy to readbe easy to read and understandand understand Use theUse the same code stylesame code style everywhereeverywhere Commit message template andCommit message template and qualityquality Make sure style and templates are followed!Make sure style and templates are followed! @bagder@bagder
- 10. Test! Unit tests System tests Integration tests Test the documentation Manual tests Fixed a bug? Add a test. Or two. @bagder@bagder
- 11. Torment Valgrind, address/UB/integer sanitizersValgrind, address/UB/integer sanitizers Static code analysersStatic code analysers FuzzersFuzzers @bagder@bagder
- 12. All that, all the time For every commit For every PR All. The. Time @bagder@bagder
- 13. curl CI 80+ builds and test “rounds” per commit Tests code style, indenting etc Thousands of tests per build Builds and tests on tens of platforms 20-25 hours of CI per commit @bagder@bagder
- 14. @bagder@bagder
- 15. @bagder@bagder
- 16. The curl project’s CI sponsors Parts of the CI system run of free tiers Teamviewer sponsors Appveyor powers Travis gives us more power Azure Pipelines gives us more power @bagder@bagder
- 17. @bagder@bagder
- 18. @bagder@bagder
- 19. Writing test cases should be easy A curl test case is a single file in a human readable well-documented format • conditions and features needed to run • what command (line) to run • what the test wants returned from a server • how the protocol exchange should look like • stdout and stderr contents • expected error code • … and more @bagder@bagder
- 20. Custom test servers We avoid using “real” servers for testing Test servers are as dumb as possible Controlled from the test case what it should send and expect Allows “crazy” behaviors and send/receive “anything” Makes the test servers smaller and simpler Helps with test suite portability TLS is done by stunnel-fronting Servers run on random port numbers @bagder@bagder
- 21. Tools of the trade Valgrind Clang address, undefined, signed- integer-overflow sanitizers Clang tidy “torture tests” Scan-build Lgtm Codacy CodeQL Monocle AI Deepcode AI Coverity Travis CI Appveyor Cirrus CI Github Actions Azure Pipelines Buildbots OSS-Fuzz CI-fuzz @bagder@bagder
- 22. Torture tests – error injection Build with a “debug” option Add wrapper functions for common “fallible” functions (that allocate and free resources) Each wrapper function logs what it does and can optionally return error. The complete individual test case is first run once count how many fallible functions rerun the test case that number of times and for each iteration make next fallible function return error Verify nothing crashed and no memory leaked Repeat for all tests @bagder@bagder
- 23. Source code policy Fix all warnings (eye roll)Fix all warnings (eye roll) No defects leftNo defects left Use the strictest and most picky optionsUse the strictest and most picky options As many tests as possibleAs many tests as possible Fix security issues as soon as possibleqFix security issues as soon as possibleq @bagder@bagder
- 24. A million build combos, 70+ OSes and 20 CPUs Testing all combinations is simply not possible Test the common setups Test on as many platforms as possible Test on several different CPU architectures “white spots” in test coverage handled by review Users keep finding untested areas and build combinations @bagder@bagder
- 25. The security of curl is the sum of all components combined. - Lean on the right set! @bagder@bagder
- 26. 32 third party dependencies I/O layer libcurl URL parser libidn2winidn HTTPHTTPS OpenSSL Mesalink gskit mbedTLS wolfSSL Schannel SecureTransport GnuTLS NSS boringssl libressl AmiSSL SFTP SCP LDAP WinLDAP OpenLDAP RTMP librtmp Name resolver c-ares compression libz brotli cookies libpsl IMAP SMTP POP3 HTTP/2 nghttp2 authentication winsspi Heimdal MIT kerberos HTTP/3 quiche ngtcp2 HTTP/1 SSH wolfSSH libssh2 libssh @bagder@bagder BearSSL nghttp3
- 27. Fuzz Testing in wolfSSL ● Best Tested TLS due to our extensive internal testing process ● Fuzz testing in our internal testing process is one of the reasons why ● Fuzz testing bombards the program with invalid, unexpected, and random data that then allows for observing if there is potential memory leaks or logic errors ● Allows us to catch bugs that could turn into potential vulnerabilities ● We only use highly respected external testers: ● Guido Vranken in Holland ● Robert Horr of T-Systemsin Germany ● Internally, we test using several different software fuzzers such as: ● in-memory fuzzer ● a network fuzzer ● OSS-fuzz ● libfuzzer ● tlsfuzzer ● AFL ● To maximize security we run 7 fuzz testers internally every night on our latest code on over 20 platforms! @bagder@bagder
- 28. The curl bug-bounty https://hackerone.com/curl Pays rewards using donated money Rewards up to 12,000 USD for a critical flaw Has paid 2,300 USD so far Increasing bounty amounts @bagder@bagder
- 29. @bagder@bagder
- 30. University researchers looking for a class of bugs Get involved! Corporate security researchers who’s companies consume curl Pen testers who want to find something big and widespread Security enthusiasts looking for a reward and “fame” @bagder@bagder
- 31. There’s one more thing curl has @bagder@bagder
- 33. Bad things will happen We are not immune, no matter how hard we try! Act immediately Own the problem Fix it and announce it Learn from it Make it harder to do the same mistake again @bagder@bagder
- 34. But does it work? 10 billion installations is no proof ✔ Decreasing number of CVEs over time ✔ Decreasing number of OSS-Fuzz reports over time ✔ Increasing bounty rewards @bagder@bagder
- 35. Reported security vulnerabilities per year @bagder@bagder
- 37. Clean code Review Test Torment Act on mistakes @bagder@bagder
- 38. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
- 39. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder