Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

curl better

237 views

Published on

Daniel Stenberg's talk at foss-north 2020.

Published in: Technology
  • Be the first to comment

curl better

  1. 1. curl bettercurl better March 30, foss-north 2020March 30, foss-north 2020
  2. 2. Daniel Stenberg @bagderhttps://daniel.haxx.se
  3. 3. Daniel Stenberg @bagder
  4. 4. what curl iswhat curl is option how-tooption how-to Q&AQ&A @bagder@bagder
  5. 5. An open source project that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
  6. 6. Features! DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
  7. 7. @bagder@bagder
  8. 8. 2,200 contributors Who makes curl curl 770 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
  9. 9. @bagder@bagder Conquered the world Right time Open Source Portable Backwards compatible Stable
  10. 10. Many places curl has been around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
  11. 11. Many flavors Millions of build combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
  12. 12. Many protocols and options Today we only talk about the tool named curl curl knows many protocols curl features 230 command line options I stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
  13. 13. Number of command line options @bagder@bagder First curl release 24 230
  14. 14. HTTP RequestRequest - method + path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
  15. 15. Under the hood GET / HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
  16. 16. Minimal behavior by default Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
  17. 17. Long and short options All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
  18. 18. $ curl example.com <html> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
  19. 19. $ curl -i https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
  20. 20. $ curl https://example.com/json | jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
  21. 21. $ curl -I https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
  22. 22. $ curl -I https://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
  23. 23. $ curl -I -L https://example.com/redi rected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT ... HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
  24. 24. URL Globbing $ curl https://example.com/[1-9].html $ curl https://example.com/[01-99].html $ curl https://example.com/[a-z].html $ curl https://example.com/[1-9:2].html $ curl https://example.com/[a-z:3].html $ curl https://example.com/[1-9].html -o save_#1.html $ curl https://example.com/{ham,cheese,pineapple}.jpg -o hawaii_#1.jpg $ curl http://archive.example/issue[1996-1999]/vol[1-4]/part{a,b,c}.html
  25. 25. $ curl -v https://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
  26. 26. One -v is enough. More adds nothing!
  27. 27. $ curl https://example.com/ -H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
  28. 28. $ curl -d name=Daniel -i https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
  29. 29. $ curl -d @file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
  30. 30. $ curl -T localfile -i https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
  31. 31. $ curl -T localfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
  32. 32. curl -X in vain leads to pain https://xkcd.com/386/ @bagder@bagder
  33. 33. $ curl -d postdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
  34. 34. @bagder@bagder Bad -X -X means use this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
  35. 35. @bagder@bagder Avoid -XAvoid -X
  36. 36. @bagder@bagder Cookies are name value pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
  37. 37. $ curl -c cookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
  38. 38. $ curl -c cookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login
  39. 39. $ curl https://127.0.0.1/ $ curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
  40. 40. curl -k really?
  41. 41. @bagder@bagder Avoid -kAvoid -k
  42. 42. $ curl https://example.com/ -d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
  43. 43. Extracting transfer meta-data --write-out (-w) is your friend --write-out ‘text and %{variables} to see’ Variables include:  content_type  http_version  remote_ip  local_ip  speed_download %{json}%{json} Coming soon! Coming soon!
  44. 44. $ curl https://example.com/ -o saved -w ‘%{json}’ | jq @bagder@bagder --write-out %{json} { "url_effective": "https://example.com/", "http_code": 200, "response_code": 200, "http_connect": 0, "time_total": 0.658048, "time_namelookup": 0.001, "time_connect": 0.153556, "time_appconnect": 0.501105, "time_pretransfer": 0.501223, "time_starttransfer": 0.657869, "size_header": 331, "size_request": 77, "size_download": 1256,
  45. 45. $ curl https://example.com/ --libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
  46. 46. @bagder@bagder Mimic that browser – copy as curl
  47. 47. $ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
  48. 48. SSLKEYLOGFILE Why trust your tools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
  49. 49. $ export SSLKEYLOGFILE=$HOME/tmp/tlskey $ curl https://example.com @bagder@bagder Snoop on curl
  50. 50. The Wireshark end @bagder@bagder
  51. 51. Many HTTP versions – but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 shipped in 2015 HTTP/3 is “coming soon” curl pretends all headers look and work like HTTP/1 @bagder@bagder
  52. 52. $ curl –-http1.0 https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
  53. 53. $ curl -d user=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
  54. 54. All this, and much much more... @bagder@bagder
  55. 55. https://curl.haxx.se/book.html @bagder@bagder
  56. 56. 56 You can help!You can help! @bagder@bagder
  57. 57. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
  58. 58. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder

×