curl better

curl bettercurl better
March 30, foss-north 2020March 30, foss-north 2020

Daniel Stenberg
@bagderhttps://daniel.haxx.se

what curl iswhat curl is
option how-tooption how-to
Q&AQ&A
@bagder@bagder

An open source project that
makes a command line tool
and a library for transferring
data using Internet protocols
@bagder@bagder

Features!
DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS,
LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP,
SMB, SMBS, SMTP, SMTPS, Telnet and TFTP
TLS certificates, HTTP POST, HTTP PUT, FTP uploading,
HTTP form based upload, HTTP/HTTPS/SOCKS proxy,
cookies, authentication (Basic, Digest, NTLM, Negotiate,
Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file
transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP
compression and much more
@bagder@bagder

2,200 contributors
Who makes curl
curl
770 authors
150 authors per year
12 regulars
Daniel
@bagder@bagder
(The boxes are not drawn to scale)

@bagder@bagder
Conquered the world
Right time
Open Source
Portable
Backwards compatible
Stable

Many places
curl has been around since spring 1998
MacOS (since 2001)
Windows 10 (since 2018)
Linux distros (since 1999)
BSDs, other Unixes, MS-DOS, older systems
Side-loaded - from the official site or elsewhere
Built from source
@bagder@bagder

Many flavors
Millions of build combinations
Thirty different dependencies in combinations
Frequent releases
Rarely the same in two places!
@bagder@bagder

Many protocols and options
Today we only talk about the tool named curl
curl knows many protocols
curl features 230 command line options
I stick to HTTP(S) in this presentation
Using a small subset of flags
@bagder@bagder

Number of command line options
@bagder@bagder
First curl
release
24
230

HTTP
RequestRequest
- method + path- method + path
- headers- headers
- body- body
ResponseResponse
- response code- response code
- headers- headers
- body- body
@bagder@bagder

Under the hood
GET / HTTP/1.1
Host: www.example.com
Accept: */*
User-Agent: HTTP-eats-the-world/2019
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2018 14:49:00 GMT
Server: my-favorite v3
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
Content-Length: 12345
Set-Cookie: this-is-simple=yeah-really;
Content-Type: text/html
[content]
@bagder@bagder

Minimal behavior by default
Basic set of headers
No fancy functionality
Let users enable more when wanted
Toggle features on/off one-by-one
@bagder@bagder

Long and short options
All short options have a long alternative
-s can also be set with –-silent
I will use the short options today
Many options only exist as long options
@bagder@bagder

$ curl example.com
<html>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
@bagder@bagder
Plain curl HTTP GET

$ curl -i https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
</body>
@bagder@bagder
GET and show me the response headers

$ curl https://example.com/json | jq
{
"id": "kr09ddfgbfsf",
"name": "Issues, PRs, Dashboard, Projects",
"status": "operational",
"created_at": "2017-01-31T20:01:46.638Z",
"updated_at": "2019-09-26T14:00:45.562Z",
"position": 3,
"description": "Fake JSON for an example",
"group": false,
"only_show_if_degraded": false
},
@bagder@bagder
GET and show JSON nicely

$ curl -I https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-length: 306
server: server/3000
@bagder@bagder
HEAD only shows the response headers
HEAD returns no body!

$ curl -I https://example.com/redirected
HTTP/1.1 302 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
location: /a/separate/place
content-length: 306
server: server/3000
@bagder@bagder
Curl doesn’t follow redirects by default

$ curl -I -L https://example.com/redi rected
HTTP/1.1 302 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
...
HTTP/1.1 200 OK
date: Wed, 19 Oct 2019 11:23:06 GMT
content-length: 4676
server: server/3000
@bagder@bagder
Please follow redirects

URL Globbing
$ curl https://example.com/[1-9].html
$ curl https://example.com/[01-99].html
$ curl https://example.com/[a-z].html
$ curl https://example.com/[1-9:2].html
$ curl https://example.com/[a-z:3].html
$ curl https://example.com/[1-9].html -o save_#1.html
$ curl https://example.com/{ham,cheese,pineapple}.jpg -o hawaii_#1.jpg
$ curl http://archive.example/issue[1996-1999]/vol[1-4]/part{a,b,c}.html

$ curl -v https://example.com/ -o /dev/null
* Trying 93.184.216.34:443...
* Connected to example.com (93.184.216.34) port 443 (#0)
…
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org
…
> GET / HTTP/2
> Host: example.com
> User-Agent: curl/7.66.0
> Accept: */*
>
< HTTP/2 200
< date: Wed, 09 Oct 2019 11:16:06 GMT
< content-type: text/html
< content-length: 306
< server: server/3000
@bagder@bagder
Verbose shows more from under the hood

One -v is enough. More adds nothing!

$ curl https://example.com/ -H "Magic: disc0"
$ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7"
$ curl https://example.com/ -H "User-agent:"
$ curl https://example.com/ -H "User-agent;"
@bagder@bagder
Pass in custom HTTP headers

$ curl -d name=Daniel -i https://example.com/receiver
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-length: 306
server: server/3000
<body>
</body>
@bagder@bagder
POST some basic data to the remote

$ curl -d @file https://example.com/receiver -o saved
$ ls -l | curl -d @- https://example.com/receiver -o saved
$ ls -l | curl --data-binary @- https://example.com/receiver -o saved
$ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com
@bagder@bagder
POST a file

$ curl -T localfile -i https://example.com/remote_name
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-length: 306
server: server/3000
<body>
</body>
@bagder@bagder
PUT a file

$ curl -T localfile -X SWOOSH https://example.com/remote_name -o save
@bagder@bagder
With -X we change the method string

curl -X in vain leads to pain
https://xkcd.com/386/
@bagder@bagder

$ curl -d postdata -X POST https://example.com/remote_name -o save
$ curl -d postdata -X POST https://example.com/remote_name -o save -L
@bagder@bagder
Bad -X leads to pain
Probably not what you wanted!

@bagder@bagder
Bad -X
-X means use this method in all requests
-L means follow redirects
Redirects instruct the client what method to use in the
next request
-X overrides the method the server wants the client to use
Use -X if you want a different method than curl would use

@bagder@bagder
Avoid -XAvoid -X

@bagder@bagder
Cookies are name value pairs
Cookies are name=value pairs sent by servers
Sent back by clients when the URL matches
curl doesn’t know cookies unless you tell it to
curl has separate options for reading and
writing cookie files

$ curl -c cookiejar.txt https://example.com/
$ curl -b cookiejar.txt https://example.com/
$ cat cookiejar.txt
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
example.com TRUE / FALSE 1602699857 duidd182180ebab48
@bagder@bagder
Cookies!

$ curl -c cookiejar.txt https://example.com/login_form
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel
-d password=1234
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile
@bagder@bagder
Cookies in a login

$ curl https://127.0.0.1/
$ curl -k https://127.0.0.1/
$ curl -k https://127.0.0.1/ -H "Host: example.com"
$ curl https://example.com/ --resolve example.com:443:127.0.0.1
$ curl https://example.com/ --connect-to example.com:443:host.tld:8443
@bagder@bagder
HTTPS obstacles, locally hosted named site
Causes certificate problems
Ducks for the cert problem
Doesn’t work with cookies
Works with cookies
Doesn’t work with virtual servers
Does TLS and certs
correctly
Works with cookies
Works with virtual
servers

@bagder@bagder
Avoid -kAvoid -k

$ curl https://example.com/ -d sendthisdata --trace-ascii -
…
=> Send header, 143 bytes (0x8f)
0000: POST / HTTP/2
000f: Host: example.com
0022: User-Agent: curl/7.66.0
003b: Accept: */*
0048: Content-Length: 12
005c: Content-Type: application/x-www-form-urlencoded
008d:
=> Send SSL data, 5 bytes (0x5)
0000: ....&
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send data, 12 bytes (0xc)
0000: sendthisdata
== Info: We are completely uploaded and fine
…
@bagder@bagder
More details!

Extracting transfer meta-data
--write-out (-w) is your friend
--write-out ‘text and %{variables} to see’
Variables include:
 content_type
 http_version
 remote_ip
 local_ip
 speed_download
%{json}%{json}
Coming soon!
Coming soon!

$ curl https://example.com/ -o saved -w ‘%{json}’ | jq
@bagder@bagder
--write-out %{json}
{
"url_effective": "https://example.com/",
"http_code": 200,
"response_code": 200,
"http_connect": 0,
"time_total": 0.658048,
"time_namelookup": 0.001,
"time_connect": 0.153556,
"time_appconnect": 0.501105,
"time_pretransfer": 0.501223,
"time_starttransfer": 0.657869,
"size_header": 331,
"size_request": 77,
"size_download": 1256,

$ curl https://example.com/ --libcurl sourcecode.c
$ gcc sourcecode.c -lcurl -o ./myapp
$ ./myapp
$ cat sourcecode.c
@bagder@bagder
Convert it into an application?
/********* Sample code generated by the curl command line tool **********
* All curl_easy_setopt() options are documented at:
* https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
************************************************************************/
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts");
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);
...

@bagder@bagder
Mimic that browser – copy as curl

$ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/
20100101 Firefox/69.0' -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-
US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-
Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb-
59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers'
$ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H
'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/
OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile
Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/
*;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip,
deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb-
59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed
@bagder@bagder
Mimic that browser – copy as curl

SSLKEYLOGFILE
Why trust your tools when you can snoop on them
Wireshark is your friend
Tell curl or your browser to store its secrets in
SSLKEYLOGFILE
Tell Wireshark where the secrets are located
Run curl or your browser
@bagder@bagder

$ export SSLKEYLOGFILE=$HOME/tmp/tlskey
$ curl https://example.com
@bagder@bagder
Snoop on curl

The Wireshark end
@bagder@bagder

Many HTTP versions – but they look the same
HTTP/1.0 shipped in 1996
We’ve learned how HTTP headers look like
HTTP/2 shipped in 2015
HTTP/3 is “coming soon”
curl pretends all headers look and work like HTTP/1
@bagder@bagder

$ curl –-http1.0 https://example.com
$ curl –-http1.1 https://example.com
$ curl –-http2 https://example.com
$ curl –-http3 https://example.com
@bagder@bagder
HTTP versions
HTTP/3 support is experimental!

$ curl -d user=daniel https://example.com https://another.example.com
$ curl -d user=daniel https://example.com –-next https://another.example.com
$ curl https://example.com –-next -d user=daniel https://another.example.com
@bagder@bagder
Different requests on the same command line!

All this, and much much more...
@bagder@bagder

https://curl.haxx.se/book.html
@bagder@bagder

56
You can help!You can help!
@bagder@bagder

Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!Thank you!
Questions?Questions?
@bagder@bagder

License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder

curl better

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to curl better

Similar to curl better (20)

More from Daniel Stenberg

More from Daniel Stenberg (14)

Recently uploaded

Recently uploaded (20)

curl better