Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Just curl it!

385 views

Published on

Daniel's keynote at Nordic APIs Platform Summit 2019. October 22, 2019.

Published in: Software
  • Be the first to comment

Just curl it!

  1. 1. Just curl itJust curl it October 22, Platform Summit 2019October 22, Platform Summit 2019
  2. 2. Daniel Stenberg @bagder
  3. 3. Daniel Stenberg @bagder
  4. 4. An open source project that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
  5. 5. Features! DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
  6. 6. @bagder@bagder
  7. 7. 2,000 contributors Who makes curl curl 730 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
  8. 8. @bagder@bagder Conquered the world Right time Open Source Portable Backwards compatible Stable
  9. 9. Many places curl has been around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
  10. 10. Many flavors Millions of build combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
  11. 11. Many protocols and options Today we only talk about the tool named curl curl knows many protocols curl features 226 command line options We stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
  12. 12. HTTP RequestRequest - method + path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
  13. 13. Under the hood GET / HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
  14. 14. Minimal behavior by default Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
  15. 15. Long and short options All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
  16. 16. $ curl example.com <html> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
  17. 17. $ curl -i https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
  18. 18. $ curl https://example.com/json | jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
  19. 19. $ curl -I https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
  20. 20. $ curl -I https://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
  21. 21. $ curl -I -L https://example.com/redirected HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
  22. 22. $ curl -v https://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
  23. 23. $ curl https://example.com/ -H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
  24. 24. $ curl -d name=Daniel https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
  25. 25. $ curl -d @file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
  26. 26. $ curl -T localfile https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
  27. 27. $ curl -T localfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
  28. 28. curl -X in vain leads to pain https://xkcd.com/386/ @bagder@bagder
  29. 29. $ curl -d postdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
  30. 30. @bagder@bagder Bad -X -X means use this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
  31. 31. @bagder@bagder Avoid -XAvoid -X
  32. 32. @bagder@bagder Cookies are name value pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
  33. 33. $ curl -c cookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
  34. 34. $ curl -c cookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login procedure
  35. 35. $ curl https://127.0.0.1/ $ curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
  36. 36. curl -k really?
  37. 37. @bagder@bagder Avoid -kAvoid -k
  38. 38. $ curl https://example.com/ -d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
  39. 39. $ curl https://example.com/ --libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
  40. 40. @bagder@bagder Mimic that browser – copy as curl
  41. 41. $ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
  42. 42. SSLKEYLOGFILE Why trust your tools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
  43. 43. $ export SSLKEYLOGFILE=$HOME/tmp/tlskey $ curl https://example.com @bagder@bagder Snoop on curl
  44. 44. The Wireshark end @bagder@bagder
  45. 45. Many HTTP versions – but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 came in 2015 HTTP/3 is “coming soon” curl makes HTTP/2 and HTTP/3 headers look and work like HTTP/1 @bagder@bagder
  46. 46. $ curl –-http1.0 https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
  47. 47. $ curl -d user=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
  48. 48. All this, and much much more... @bagder@bagder
  49. 49. https://curl.haxx.se/book.html @bagder@bagder
  50. 50. 51 You can help!You can help! @bagder@bagder
  51. 51. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
  52. 52. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder

×