Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The state of curl 2019

610 views

Published on

Presentation from curl up 2019. The state of the curl project. In English.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The state of curl 2019

  1. 1. The state of curl 2019
  2. 2. The curl project 2019
  3. 3. Stats Mostly from 2010 or later Due to data availability Represents “the modern curl project”
  4. 4. Number of lines of “product code” 2010-02-09 2010-06-16 2010-10-12 2011-02-17 2011-04-22 2011-09-13 2011-11-17 2012-03-22 2012-07-27 2012-11-20 2013-04-12 2013-08-11 2013-12-16 2014-03-26 2014-07-16 2014-11-05 2015-02-25 2015-04-28 2015-08-11 2015-12-01 2016-02-08 2016-05-17 2016-07-21 2016-09-07 2016-11-02 2016-12-22 2017-02-24 2017-06-14 2017-08-13 2017-10-23 2018-01-23 2018-05-15 2018-09-04 2018-12-12 0 20000 40000 60000 80000 100000 120000 140000 160000 180000
  5. 5. Is a 160K a lot or a little? A dozen TLS backends Two SSH backends Three name resolver backends Feature packed; 221 command line options and 267 setopt() options More portable than most More compliant than most More feature-packed than most 25% comments
  6. 6. C! Efficient and portable! Some security problems could be avoided using something else Lots of “reach” would then also be avoided Mitigation: readable code, reviews, tests, fuzzing, static code analyzing
  7. 7. Coverity on curl – fixed defects
  8. 8. Coverity on curl – defects over time
  9. 9. OSS-Fuzz reports over time 2017-06 2017-07 2017-08 2017-09 2017-10 2017-11 2017-12 2018-01 2018-02 2018-03 2018-04 2018-05 2018-06 2018-07 2018-08 2018-09 2018-10 2018-11 2018-12 2019-01 2019-02 2019-03 0 2 4 6 8 10 12 14 16
  10. 10. Test cases over time 2010-02-09 2010-06-16 2010-10-12 2011-02-17 2011-04-22 2011-09-13 2011-11-17 2012-03-22 2012-07-27 2012-11-20 2013-04-12 2013-08-11 2013-12-16 2014-03-26 2014-07-16 2014-11-05 2015-02-25 2015-04-28 2015-08-11 2015-12-01 2016-02-08 2016-05-17 2016-07-21 2016-09-07 2016-11-02 2016-12-22 2017-02-24 2017-06-14 2017-08-13 2017-10-23 2018-01-23 2018-05-15 2018-09-04 2018-12-12 0 200 400 600 800 1000 1200 1400
  11. 11. Source vs tests over time 2010-02-09 2010-08-11 2011-02-17 2011-06-23 2011-11-17 2012-05-24 2012-11-20 2013-06-22 2013-12-16 2014-05-20 2014-11-05 2015-04-22 2015-08-11 2016-01-27 2016-05-17 2016-08-03 2016-11-02 2017-02-22 2017-06-14 2017-10-04 2018-01-23 2018-07-11 2018-12-12 0 20000 40000 60000 80000 100000 120000 140000 160000 -100 100 300 500 700 900 1100 1300 Test cases Lines of code Linesofcode Numberoftestcases
  12. 12. Source lines per test file since 20107.20.0 7.20.1 7.21.0 7.21.1 7.21.2 7.21.3 7.21.4 7.21.5 7.21.6 7.21.7 7.22.0 7.23.0 7.23.1 7.24.0 7.25.0 7.26.0 7.27.0 7.28.0 7.28.1 7.29.0 7.30.0 7.31.0 7.32.0 7.33.0 7.34.0 7.35.0 7.36.0 7.37.0 7.37.1 7.38.0 7.39.0 7.40.0 7.41.0 7.42.0 7.42.1 7.43.0 7.44.0 7.45.0 7.46.0 7.47.0 7.47.1 7.48.0 7.49.0 7.49.1 7.50.0 7.50.1 7.50.2 7.50.3 7.51.0 7.52.0 7.52.1 7.53.0 7.53.1 7.54.0 7.54.1 7.55.0 7.55.1 7.56.0 7.56.1 7.57.0 7.58.0 7.59.0 7.60.0 7.61.0 7.61.1 7.62.0 7.63.0 7.64.0 7.64.1 120 130 140 150 160 170 180 The y-axis is not zero-based!
  13. 13. Test coverage Good to know, hard to measure 72 - 78% on coveralls.io For a single TLS – SSH – resolver – config setup! Some tests too slow for coverage runs in the cloud (torture) Some code paths still hard to test with existing test suite
  14. 14. Daniel’s share of curl commits 2010-01-13 2010-08-10 2011-04-16 2011-11-01 2012-06-19 2013-02-06 2013-07-09 2013-12-22 2014-05-04 2014-10-29 2015-02-19 2015-08-20 2016-03-29 2016-10-24 2017-04-29 2017-09-15 2018-05-02 2018-11-23 0 10 20 30 40 50 60 70 80
  15. 15. Commits per release since 2010 7.20.0 7.20.1 7.21.0 7.21.1 7.21.2 7.21.3 7.21.4 7.21.5 7.21.6 7.21.7 7.22.0 7.23.0 7.23.1 7.24.0 7.25.0 7.26.0 7.27.0 7.28.0 7.28.1 7.29.0 7.30.0 7.31.0 7.32.0 7.33.0 7.34.0 7.35.0 7.36.0 7.37.0 7.37.1 7.38.0 7.39.0 7.40.0 7.41.0 7.42.0 7.43.0 7.44.0 7.45.0 7.46.0 7.47.0 7.47.1 7.48.0 7.49.0 7.49.1 7.50.0 7.50.1 7.50.2 7.50.3 7.51.0 7.52.0 7.52.1 7.53.0 7.53.1 7.54.0 7.54.1 7.55.0 7.55.1 7.56.0 7.56.1 7.57.0 7.58.0 7.59.0 7.60.0 7.61.0 7.61.1 7.62.0 7.63.0 7.64.0 7.64.1 0 50 100 150 200 250 300 350 400 450 500
  16. 16. Commits per year 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 0 200 400 600 800 1000 1200 1400 1600 1800
  17. 17. Commit authors in curl since 20102010-01 2010-03 2010-05 2010-07 2010-09 2010-11 2011-01 2011-03 2011-05 2011-07 2011-09 2011-11 2012-01 2012-03 2012-05 2012-07 2012-09 2012-11 2013-01 2013-03 2013-05 2013-07 2013-09 2013-11 2014-01 2014-03 2014-05 2014-07 2014-09 2014-11 2015-01 2015-03 2015-05 2015-07 2015-09 2015-11 2016-01 2016-03 2016-05 2016-07 2016-09 2016-11 2017-01 2017-03 2017-05 2017-07 2017-09 2017-11 2018-01 2018-03 2018-05 2018-07 2018-09 2018-11 2019-01 2019-03 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 0 100 200 300 400 500 600 700 First Commit Authors Total count Date Authorspermonth Authorsoveralltime
  18. 18. Authors per month, excluding first-timers 2010-01 2010-04 2010-07 2010-10 2011-01 2011-04 2011-07 2011-10 2012-01 2012-04 2012-07 2012-10 2013-01 2013-04 2013-07 2013-10 2014-01 2014-04 2014-07 2014-10 2015-01 2015-04 2015-07 2015-10 2016-01 2016-04 2016-07 2016-10 2017-01 2017-04 2017-07 2017-10 2018-01 2018-04 2018-07 2018-10 2019-01 0 2 4 6 8 10 12 14 16 18 20
  19. 19. Top-10 commit author share since forever Marc Hoersken Kamil Dudka Patrick Monnerat Jay Satiro Gisle Vanem Guenter Knauf Dan Fandrich Steve Holme Yang Tse (The rest) Daniel Stenberg 0 10 20 30 40 50 60
  20. 20. Top-10 commit author share since 2017 Kamil Dudka Viktor Szakats Johannes Schindelin Michael Kaufmann Daniel Gustafsson Dan Fandrich Patrick Monnerat Jay Satiro Marcel Raad (The rest) Daniel Stenberg 0 10 20 30 40 50 60
  21. 21. Days between curl releases since 2010 Average: 50 Median: 56 7.20.0 7.20.1 7.21.0 7.21.1 7.21.2 7.21.3 7.21.4 7.21.5 7.21.6 7.21.7 7.22.0 7.23.0 7.23.1 7.24.0 7.25.0 7.26.0 7.27.0 7.28.0 7.28.1 7.29.0 7.30.0 7.31.0 7.32.0 7.33.0 7.34.0 7.35.0 7.36.0 7.37.0 7.37.1 7.38.0 7.39.0 7.40.0 7.41.0 7.42.0 7.43.0 7.44.0 7.45.0 7.46.0 7.47.0 7.47.1 7.48.0 7.49.0 7.49.1 7.50.0 7.50.1 7.50.2 7.50.3 7.51.0 7.52.0 7.52.1 7.53.0 7.53.1 7.54.0 7.54.1 7.55.0 7.55.1 7.56.0 7.56.1 7.57.0 7.58.0 7.59.0 7.60.0 7.61.0 7.61.1 7.62.0 7.63.0 7.64.0 7.64.1 0 10 20 30 40 50 60 70 80 90 Max: 83 Min: 2
  22. 22. Bug-fixes per release since 20107.20.0 7.20.1 7.21.0 7.21.1 7.21.2 7.21.3 7.21.4 7.21.5 7.21.6 7.21.7 7.22.0 7.23.0 7.23.1 7.24.0 7.25.0 7.26.0 7.27.0 7.28.0 7.28.1 7.29.0 7.30.0 7.31.0 7.32.0 7.33.0 7.34.0 7.35.0 7.36.0 7.37.0 7.37.1 7.38.0 7.39.0 7.40.0 7.41.0 7.42.0 7.42.1 7.43.0 7.44.0 7.45.0 7.46.0 7.47.0 7.47.1 7.48.0 7.49.0 7.49.1 7.50.0 7.50.1 7.50.2 7.50.3 7.51.0 7.52.0 7.52.1 7.53.0 7.53.1 7.54.0 7.54.1 7.55.0 7.55.1 7.56.0 7.56.1 7.57.0 7.58.0 7.59.0 7.60.0 7.61.0 7.61.1 7.62.0 7.63.0 7.64.0 7.64.1 0 20 40 60 80 100 120 140
  23. 23. Bug-fixes per day since 2010February92010 April142010 June162010 August112010 October132010 December152010 February172011 April172011 April222011 June232011 September132011 November152011 November172011 January242012 March222012 May242012 July272012 October102012 November202012 February62013 April122013 June222013 August122013 October142013 December172013 January292014 March262014 May212014 July162014 September102014 November52014 January82015 February252015 April222015 April292015 June172015 August122015 October72015 December22015 January272016 February82016 March232016 May182016 May302016 July212016 August32016 September72016 September142016 November22016 December212016 December232016 February222017 February242017 April192017 June142017 August92017 August142017 October42017 October232017 November292017 January242018 March142018 May162018 July112018 September52018 October312018 December122018 February62019 March272019 0 1 2 3 4 5 6
  24. 24. Vulnerability reports since 2010 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 0 5 10 15 20 25
  25. 25. Lessons from past vulnerabilities? Integer overflows are tricky things – different architectures make them more so Most flaws linger in the code a long time until detected Fuzzing is king Fixing the flaws is usually straight-forward Bug bounties can help
  26. 26. Top-20 changed source files since 2010 lib/url.c lib/vtls/openssl.c lib/imap.c lib/http2.c lib/smtp.c lib/multi.c lib/pop3.c include/curl/curl.h src/tool_getparam.c lib/transfer.c src/tool_operate.c lib/http.c lib/connect.c lib/urldata.h lib/ssh.c include/curl/curlver.h lib/ftp.c lib/curl_sasl.c lib/vtls/darwinssl.c lib/vtls/nss.c 0 50 100 150 200 250
  27. 27. Annual user survey What is used, what is ignored What is good, what is bad What should be added, what should be removed How are we doing
  28. 28. How good is the project to handle 2014 2015 2016 2017 2018 3 3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5 security credit patches bug reports information newcomers minorities (According to the annual user survey)
  29. 29. curl’s top-5 areas according to users the libcurl API the support of many protocols documentation its availability and functionality on many platforms the quality of the products, curl/libcurl 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 2017 2018
  30. 30. curl’s worst-5 areas according to users project web site and infrastructure welcoming to new users and contributors the libcurl API its build environment/setup documentation 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 2017 2018
  31. 31. User survey 2019 Around May time frame Very much interested in feedback on where to take it and what to ask for Received 670 responses 2018 https://daniel.haxx.se/blog/2018/06/12/curl-survey-2018-analysis/
  32. 32. Web site traffic 2019 Fastly makes our lives easier 1.5 million requests/day (from 1.8) 41.6 TB the last 12 months Fast web site, close to most users No logs, no tracking, very little stats
  33. 33. [curl] 34,550 times [libcurl] 2,510 times
  34. 34. Google trends, worldwide search Wget rsync curl Includes wget and rsync only to provide references with similar projects
  35. 35. CII Best Practices https://bestpractices.coreinfrastructure.org/en/projects/63 100% passing 96% Silver 26% Gold “SHOULD have a legal mechanism where all developers of non-trivial amounts of project software assert that they are legally authorized to make these contributions”
  36. 36. Everyone uses curl 2019 Apps: Youtube, Instagram, Skype, Spotify, ... OS: iOS, macOS, Windows, Linux, ChromeOS, AOSP, ... Cars: Mercedes, BMW, Toyota, Nissan, Volkswagen, … Game consoles: PS4, Nintendo Switch, ... Games: Fortnite, Red Dead Redemption 2, Spider Man, … Estimate: 6 billion installationsEstimate: 6 billion installations
  37. 37. Done the last 12 months
  38. 38. Defaults (1/4) multiplexing enabled by default defaults to "2TLS" leave secure cookies alone high resolution timestamps on Windows headers output in bold
  39. 39. New features (2/4) DNS-over-HTTPS support URL parsing API curl_easy_upkeep() --resolve supports wildcard hosts trailing headers support for chunked transfer uploads alt-svc
  40. 40. Improvements (3/4) %{stderr} and %{stdout} for --write-out support for HTTP Bearer tokens IMAP changed from "FETCH" to "UID FETCH" MesaLink is a new TLS backend microsecond resolution timers for seven getinfo intervals
  41. 41. New setopts (4/4) CURLOPT_CURLU CURLOPT_UPLOAD_BUFFERSIZE CURLOPT_TLS13_CIPHERS and CURLOPT_PROXY_TLS13_CIPHERS CURLOPT_DISALLOW_USERNAME_IN_URL CURLOPT_HAPROXYPROTOCOL CURLOPT_DNS_SHUFFLE_ADDRESSES (the alt-svc pair)
  42. 42. Everything curl 70K words, 10K lines 332 pages (PDF version) “95.1% complete” https://ec.haxx.se/
  43. 43. Everything curl – printed https://curl.haxx.se/book.html
  44. 44. Less good Flaky tests/CI Slow CI tests Vulnerabilities are still reported Still regressions, but less frequently? Could use more people who stick around
  45. 45. FutureFuture
  46. 46. Planning I can’t tell what “we” will do I have some ideas about what to do next Things change all time time Tell us what you want!
  47. 47. Version 8 Release every 56 days 7.65.0 is next A bump in every release gives us 35 * 56 = 1960 days until version 7.100 I want to avoid reaching 7.100 due to confusions it’ll create 1960 days == 5 years and 4.5 months == September 2024 Evolutionary, not revolutionary?
  48. 48. libcurl work to consider Keep up with browsers HTTP/3 and QUIC ESNI Hardcode localhost Refuse HTTP => HTTPS redirects Option to let CURLOPT_CUSTOMREQUEST be overridden on redirect HSTS "menu config"-style build feature selection
  49. 49. New APIs? Config file reader
  50. 50. curl tool work to consider Parallel transfers Support for HTTP/2 Push Master/slave mode Make --retry resume This list is identical to last year’s curl tool list!
  51. 51. Finally

×