Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing curl for security

Daniel Stenberg's presentation done on June 30, 2020 about curl testing, CI, QA, fuzzing and more.

  • Be the first to comment

  • Be the first to like this

Testing curl for security

  1. 1. DanielStenberg-June30th 2020 for security! Testing
  2. 2. Daniel Stenberg @bagder
  3. 3. Daniel Stenberg @bagder
  4. 4. … ten billion installations across the world curl runs in... @bagder@bagder
  5. 5. …… devices all of you heredevices all of you here today own and use!today own and use! curl runs in @bagder@bagder
  6. 6. Writing safe codeWriting safe code @bagder@bagder
  7. 7. Safe code is not a coincidence Doesn’t happen randomly, we need to work on it! We care about it! It is about time we all address this topic! Everything I’ll tell you today, you already know! @bagder@bagder
  8. 8. Writing Help users use the tool and APIs safelyHelp users use the tool and APIs safely Defaults!Defaults! Document. Everything, detailed and a lot.Document. Everything, detailed and a lot. Get rid of warningsGet rid of warnings UseUse commentscomments assert()assert() Releasing the code asReleasing the code as open sourceopen source @bagder@bagder
  9. 9. Review All codeAll code shallshall get reviewedget reviewed CodeCode shallshall be easy to readbe easy to read and understandand understand Use theUse the same code stylesame code style everywhereeverywhere Commit message template andCommit message template and qualityquality Make sure style and templates are followed!Make sure style and templates are followed! @bagder@bagder
  10. 10. Test! Unit tests System tests Integration tests Test the documentation Manual tests Fixed a bug? Add a test. Or two. @bagder@bagder
  11. 11. Torment Valgrind, address/UB/integer sanitizersValgrind, address/UB/integer sanitizers Static code analysersStatic code analysers FuzzersFuzzers @bagder@bagder
  12. 12. All that, all the time For every commit For every PR All. The. Time @bagder@bagder
  13. 13. curl CI 80+ builds and test “rounds” per commit Tests code style, indenting etc Thousands of tests per build Builds and tests on tens of platforms 20-25 hours of CI per commit @bagder@bagder
  14. 14. @bagder@bagder
  15. 15. @bagder@bagder
  16. 16. The curl project’s CI sponsors Parts of the CI system run of free tiers Teamviewer sponsors Appveyor powers Travis gives us more power Azure Pipelines gives us more power @bagder@bagder
  17. 17. @bagder@bagder
  18. 18. @bagder@bagder
  19. 19. Writing test cases should be easy A curl test case is a single file in a human readable well-documented format • conditions and features needed to run • what command (line) to run • what the test wants returned from a server • how the protocol exchange should look like • stdout and stderr contents • expected error code • … and more @bagder@bagder
  20. 20. Custom test servers We avoid using “real” servers for testing Test servers are as dumb as possible Controlled from the test case what it should send and expect Allows “crazy” behaviors and send/receive “anything” Makes the test servers smaller and simpler Helps with test suite portability TLS is done by stunnel-fronting Servers run on random port numbers @bagder@bagder
  21. 21. Tools of the trade Valgrind Clang address, undefined, signed- integer-overflow sanitizers Clang tidy “torture tests” Scan-build Lgtm Codacy CodeQL Monocle AI Deepcode AI Coverity Travis CI Appveyor Cirrus CI Github Actions Azure Pipelines Buildbots OSS-Fuzz CI-fuzz @bagder@bagder
  22. 22. Torture tests – error injection Build with a “debug” option Add wrapper functions for common “fallible” functions (that allocate and free resources) Each wrapper function logs what it does and can optionally return error. The complete individual test case is first run once count how many fallible functions rerun the test case that number of times and for each iteration make next fallible function return error Verify nothing crashed and no memory leaked Repeat for all tests @bagder@bagder
  23. 23. Source code policy Fix all warnings (eye roll)Fix all warnings (eye roll) No defects leftNo defects left Use the strictest and most picky optionsUse the strictest and most picky options As many tests as possibleAs many tests as possible Fix security issues as soon as possibleqFix security issues as soon as possibleq @bagder@bagder
  24. 24. A million build combos, 70+ OSes and 20 CPUs Testing all combinations is simply not possible Test the common setups Test on as many platforms as possible Test on several different CPU architectures “white spots” in test coverage handled by review Users keep finding untested areas and build combinations @bagder@bagder
  25. 25. The security of curl is the sum of all components combined. - Lean on the right set! @bagder@bagder
  26. 26. 32 third party dependencies I/O layer libcurl URL parser libidn2winidn HTTPHTTPS OpenSSL Mesalink gskit mbedTLS wolfSSL Schannel SecureTransport GnuTLS NSS boringssl libressl AmiSSL SFTP SCP LDAP WinLDAP OpenLDAP RTMP librtmp Name resolver c-ares compression libz brotli cookies libpsl IMAP SMTP POP3 HTTP/2 nghttp2 authentication winsspi Heimdal MIT kerberos HTTP/3 quiche ngtcp2 HTTP/1 SSH wolfSSH libssh2 libssh @bagder@bagder BearSSL nghttp3
  27. 27. Fuzz Testing in wolfSSL ● Best Tested TLS due to our extensive internal testing process ● Fuzz testing in our internal testing process is one of the reasons why ● Fuzz testing bombards the program with invalid, unexpected, and random data that then allows for observing if there is potential memory leaks or logic errors   ● Allows us to catch bugs that could turn into potential vulnerabilities ● We only use highly respected external testers: ● Guido Vranken in Holland   ● Robert Horr of T-Systemsin Germany ● Internally, we test using several different software fuzzers such as: ● in-memory fuzzer ● a network fuzzer  ● OSS-fuzz  ● libfuzzer  ● tlsfuzzer  ● AFL ● To maximize security we run 7 fuzz testers internally every night on our latest code on over 20 platforms!   @bagder@bagder
  28. 28. The curl bug-bounty Pays rewards using donated money Rewards up to 12,000 USD for a critical flaw Has paid 2,300 USD so far Increasing bounty amounts @bagder@bagder
  29. 29. @bagder@bagder
  30. 30. University researchers looking for a class of bugs Get involved! Corporate security researchers who’s companies consume curl Pen testers who want to find something big and widespread Security enthusiasts looking for a reward and “fame” @bagder@bagder
  31. 31. There’s one more thing curl has @bagder@bagder
  32. 32. Full-time curl maintainer @bagder@bagder
  33. 33. Bad things will happen We are not immune, no matter how hard we try! Act immediately Own the problem Fix it and announce it Learn from it Make it harder to do the same mistake again @bagder@bagder
  34. 34. But does it work? 10 billion installations is no proof ✔ Decreasing number of CVEs over time ✔ Decreasing number of OSS-Fuzz reports over time ✔ Increasing bounty rewards @bagder@bagder
  35. 35. Reported security vulnerabilities per year @bagder@bagder
  36. 36. CRTTA @bagder@bagder
  37. 37. Clean code Review Test Torment Act on mistakes @bagder@bagder
  38. 38. Daniel Stenberg @bagder Thank you!Thank you! Questions?Questions? @bagder@bagder
  39. 39. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: @bagder@bagder