SlideShare a Scribd company logo
1 of 52
Download to read offline
Just curl itJust curl it
October 22, Platform Summit 2019October 22, Platform Summit 2019
Daniel Stenberg
@bagder
Daniel Stenberg
@bagder
An open source project that
makes a command line tool
and a library for transferring
data using Internet protocols
@bagder@bagder
Features!
DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS,
LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP,
SMB, SMBS, SMTP, SMTPS, Telnet and TFTP
TLS certificates, HTTP POST, HTTP PUT, FTP uploading,
HTTP form based upload, HTTP/HTTPS/SOCKS proxy,
cookies, authentication (Basic, Digest, NTLM, Negotiate,
Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file
transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP
compression and much more
@bagder@bagder
@bagder@bagder
2,000 contributors
Who makes curl
curl
730 authors
150 authors per year
12 regulars
Daniel
@bagder@bagder
(The boxes are not drawn to scale)
@bagder@bagder
Conquered the world
Right time
Open Source
Portable
Backwards compatible
Stable
Many places
curl has been around since spring 1998
MacOS (since 2001)
Windows 10 (since 2018)
Linux distros (since 1999)
BSDs, other Unixes, MS-DOS, older systems
Side-loaded - from the official site or elsewhere
Built from source
@bagder@bagder
Many flavors
Millions of build combinations
Thirty different dependencies in combinations
Frequent releases
Rarely the same in two places!
@bagder@bagder
Many protocols and options
Today we only talk about the tool named curl
curl knows many protocols
curl features 226 command line options
We stick to HTTP(S) in this presentation
Using a small subset of flags
@bagder@bagder
HTTP
RequestRequest
- method + path- method + path
- headers- headers
- body- body
ResponseResponse
- response code- response code
- headers- headers
- body- body
@bagder@bagder
Under the hood
GET / HTTP/1.1
Host: www.example.com
Accept: */*
User-Agent: HTTP-eats-the-world/2019
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2018 14:49:00 GMT
Server: my-favorite v3
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
Content-Length: 12345
Set-Cookie: this-is-simple=yeah-really;
Content-Type: text/html
[content]
@bagder@bagder
Minimal behavior by default
Basic set of headers
No fancy functionality
Let users enable more when wanted
Toggle features on/off one-by-one
@bagder@bagder
Long and short options
All short options have a long alternative
-s can also be set with –-silent
I will use the short options today
Many options only exist as long options
@bagder@bagder
$ curl example.com
<html>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
@bagder@bagder
Plain curl HTTP GET
$ curl -i https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
GET and show me the response headers
$ curl https://example.com/json | jq
{
"id": "kr09ddfgbfsf",
"name": "Issues, PRs, Dashboard, Projects",
"status": "operational",
"created_at": "2017-01-31T20:01:46.638Z",
"updated_at": "2019-09-26T14:00:45.562Z",
"position": 3,
"description": "Fake JSON for an example",
"group": false,
"only_show_if_degraded": false
},
@bagder@bagder
GET and show JSON nicely
$ curl -I https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
@bagder@bagder
HEAD only shows the response headers
HEAD returns no body!
$ curl -I https://example.com/redirected
HTTP/1.1 302 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
location: /a/separate/place
content-length: 306
server: server/3000
@bagder@bagder
Curl doesn’t follow redirects by default
HEAD returns no body!
$ curl -I -L https://example.com/redirected
HTTP/1.1 200 OK
date: Wed, 19 Oct 2019 11:23:06 GMT
content-type: text/html
content-length: 4676
server: server/3000
@bagder@bagder
Please follow redirects
HEAD returns no body!
$ curl -v https://example.com/ -o /dev/null
* Trying 93.184.216.34:443...
* Connected to example.com (93.184.216.34) port 443 (#0)
…
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org
…
> GET / HTTP/2
> Host: example.com
> User-Agent: curl/7.66.0
> Accept: */*
>
< HTTP/2 200
< date: Wed, 09 Oct 2019 11:16:06 GMT
< content-type: text/html
< content-length: 306
< server: server/3000
@bagder@bagder
Verbose shows more from under the hood
$ curl https://example.com/ -H "Magic: disc0"
$ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7"
$ curl https://example.com/ -H "User-agent:"
$ curl https://example.com/ -H "User-agent;"
@bagder@bagder
Pass in custom HTTP headers
$ curl -d name=Daniel https://example.com/receiver
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
POST some basic data to the remote
$ curl -d @file https://example.com/receiver -o saved
$ ls -l | curl -d @- https://example.com/receiver -o saved
$ ls -l | curl --data-binary @- https://example.com/receiver -o saved
$ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com
@bagder@bagder
POST a file
$ curl -T localfile https://example.com/remote_name
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
PUT a file
$ curl -T localfile -X SWOOSH https://example.com/remote_name -o save
@bagder@bagder
With -X we change the method string
curl -X in vain leads to pain
https://xkcd.com/386/
@bagder@bagder
$ curl -d postdata -X POST https://example.com/remote_name -o save
$ curl -d postdata -X POST https://example.com/remote_name -o save -L
@bagder@bagder
Bad -X leads to pain
Probably not what you wanted!
@bagder@bagder
Bad -X
-X means use this method in all requests
-L means follow redirects
Redirects instruct the client what method to use in the
next request
-X overrides the method the server wants the client to use
Use -X if you want a different method than curl would use
@bagder@bagder
Avoid -XAvoid -X
@bagder@bagder
Cookies are name value pairs
Cookies are name=value pairs sent by servers
Sent back by clients when the URL matches
curl doesn’t know cookies unless you tell it to
curl has separate options for reading and
writing cookie files
$ curl -c cookiejar.txt https://example.com/
$ curl -b cookiejar.txt https://example.com/
$ cat cookiejar.txt
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
example.com TRUE / FALSE 1602699857 duidd182180ebab48
@bagder@bagder
Cookies!
$ curl -c cookiejar.txt https://example.com/login_form
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel
-d password=1234
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile
@bagder@bagder
Cookies in a login procedure
$ curl https://127.0.0.1/
$ curl -k https://127.0.0.1/
$ curl -k https://127.0.0.1/ -H "Host: example.com"
$ curl https://example.com/ --resolve example.com:443:127.0.0.1
$ curl https://example.com/ --connect-to example.com:443:host.tld:8443
@bagder@bagder
HTTPS obstacles, locally hosted named site
Causes certificate problems
Ducks for the cert problem
Doesn’t work with cookies
Works with cookies
Doesn’t work with virtual servers
Does TLS and certs
correctly
Works with cookies
Works with virtual
servers
curl -k really?
@bagder@bagder
Avoid -kAvoid -k
$ curl https://example.com/ -d sendthisdata --trace-ascii -
…
=> Send header, 143 bytes (0x8f)
0000: POST / HTTP/2
000f: Host: example.com
0022: User-Agent: curl/7.66.0
003b: Accept: */*
0048: Content-Length: 12
005c: Content-Type: application/x-www-form-urlencoded
008d:
=> Send SSL data, 5 bytes (0x5)
0000: ....&
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send data, 12 bytes (0xc)
0000: sendthisdata
== Info: We are completely uploaded and fine
…
@bagder@bagder
More details!
$ curl https://example.com/ --libcurl sourcecode.c
$ gcc sourcecode.c -lcurl -o ./myapp
$ ./myapp
$ cat sourcecode.c
@bagder@bagder
Convert it into an application?
/********* Sample code generated by the curl command line tool **********
* All curl_easy_setopt() options are documented at:
* https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
************************************************************************/
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts");
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);
...
@bagder@bagder
Mimic that browser – copy as curl
$ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/
20100101 Firefox/69.0' -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-
US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-
Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb-
59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers'
$ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H
'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/
OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile
Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/
*;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip,
deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb-
59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed
@bagder@bagder
Mimic that browser – copy as curl
SSLKEYLOGFILE
Why trust your tools when you can snoop on them
Wireshark is your friend
Tell curl or your browser to store its secrets in
SSLKEYLOGFILE
Tell Wireshark where the secrets are located
Run curl or your browser
@bagder@bagder
$ export SSLKEYLOGFILE=$HOME/tmp/tlskey
$ curl https://example.com
@bagder@bagder
Snoop on curl
The Wireshark end
@bagder@bagder
Many HTTP versions – but they look the same
HTTP/1.0 shipped in 1996
We’ve learned how HTTP headers look like
HTTP/2 came in 2015
HTTP/3 is “coming soon”
curl makes HTTP/2 and HTTP/3 headers look and work like
HTTP/1
@bagder@bagder
$ curl –-http1.0 https://example.com
$ curl –-http1.1 https://example.com
$ curl –-http2 https://example.com
$ curl –-http3 https://example.com
@bagder@bagder
HTTP versions
HTTP/3 support is experimental!
$ curl -d user=daniel https://example.com https://another.example.com
$ curl -d user=daniel https://example.com –-next https://another.example.com
$ curl https://example.com –-next -d user=daniel https://another.example.com
@bagder@bagder
Different requests on the same command line!
All this, and much much more...
@bagder@bagder
https://curl.haxx.se/book.html
@bagder@bagder
51
You can help!You can help!
@bagder@bagder
Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!Thank you!
Questions?Questions?
@bagder@bagder
License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder

More Related Content

What's hot

HTTP/3 is next generation HTTP
HTTP/3 is next generation HTTPHTTP/3 is next generation HTTP
HTTP/3 is next generation HTTPDaniel Stenberg
 
curl - a hobby project that conquered the world
curl - a hobby project that conquered the worldcurl - a hobby project that conquered the world
curl - a hobby project that conquered the worldDaniel Stenberg
 
common mistakes when using libcurl
common mistakes when using libcurlcommon mistakes when using libcurl
common mistakes when using libcurlDaniel Stenberg
 
Getting started with libcurl
Getting started with libcurlGetting started with libcurl
Getting started with libcurlDaniel Stenberg
 
Testing curl for security
Testing curl for securityTesting curl for security
Testing curl for securityDaniel Stenberg
 
HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!Daniel Stenberg
 
Efficient HTTP Apis
Efficient HTTP ApisEfficient HTTP Apis
Efficient HTTP ApisAdrian Cole
 
HTTP/2 What's inside and Why
HTTP/2 What's inside and WhyHTTP/2 What's inside and Why
HTTP/2 What's inside and WhyAdrian Cole
 
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchaginstackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat KorchaginNETWAYS
 
HTTP/2: What no one is telling you
HTTP/2: What no one is telling youHTTP/2: What no one is telling you
HTTP/2: What no one is telling youFastly
 
So that was HTTP/2, what's next?
So that was HTTP/2, what's next?So that was HTTP/2, what's next?
So that was HTTP/2, what's next?Daniel Stenberg
 
HTTP/2 for Developers
HTTP/2 for DevelopersHTTP/2 for Developers
HTTP/2 for DevelopersSvetlin Nakov
 
Http/2 - What's it all about?
Http/2  - What's it all about?Http/2  - What's it all about?
Http/2 - What's it all about?Andy Davies
 
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0Cory Forsyth
 

What's hot (20)

HTTP/3 in curl 2020
HTTP/3 in curl 2020HTTP/3 in curl 2020
HTTP/3 in curl 2020
 
HTTP/3 is next generation HTTP
HTTP/3 is next generation HTTPHTTP/3 is next generation HTTP
HTTP/3 is next generation HTTP
 
curl - a hobby project that conquered the world
curl - a hobby project that conquered the worldcurl - a hobby project that conquered the world
curl - a hobby project that conquered the world
 
common mistakes when using libcurl
common mistakes when using libcurlcommon mistakes when using libcurl
common mistakes when using libcurl
 
Landing code in curl
Landing code in curlLanding code in curl
Landing code in curl
 
Getting started with libcurl
Getting started with libcurlGetting started with libcurl
Getting started with libcurl
 
Testing curl for security
Testing curl for securityTesting curl for security
Testing curl for security
 
HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!
 
Efficient HTTP Apis
Efficient HTTP ApisEfficient HTTP Apis
Efficient HTTP Apis
 
The state of curl 2020
The state of curl 2020The state of curl 2020
The state of curl 2020
 
HTTP/2 What's inside and Why
HTTP/2 What's inside and WhyHTTP/2 What's inside and Why
HTTP/2 What's inside and Why
 
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchaginstackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
 
Curl with rust
Curl with rustCurl with rust
Curl with rust
 
HTTP/2: What no one is telling you
HTTP/2: What no one is telling youHTTP/2: What no one is telling you
HTTP/2: What no one is telling you
 
Http2
Http2Http2
Http2
 
So that was HTTP/2, what's next?
So that was HTTP/2, what's next?So that was HTTP/2, what's next?
So that was HTTP/2, what's next?
 
HTTP/2 for Developers
HTTP/2 for DevelopersHTTP/2 for Developers
HTTP/2 for Developers
 
Http2 right now
Http2 right nowHttp2 right now
Http2 right now
 
Http/2 - What's it all about?
Http/2  - What's it all about?Http/2  - What's it all about?
Http/2 - What's it all about?
 
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
 

Similar to Just curl it!

mastering the curl command line.pdf
mastering the curl command line.pdfmastering the curl command line.pdf
mastering the curl command line.pdfDanielStenberg7
 
CouchDB Day NYC 2017: Introduction to CouchDB 2.0
CouchDB Day NYC 2017: Introduction to CouchDB 2.0CouchDB Day NYC 2017: Introduction to CouchDB 2.0
CouchDB Day NYC 2017: Introduction to CouchDB 2.0IBM Cloud Data Services
 
Front End Website Optimization
Front End Website OptimizationFront End Website Optimization
Front End Website OptimizationGerard Sychay
 
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio Lopes
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio LopesHTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio Lopes
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio LopesCaelum
 
The Case for HTTP/2
The Case for HTTP/2The Case for HTTP/2
The Case for HTTP/2Andy Davies
 
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."Dongwook Lee
 
S3 & Glacier - The only backup solution you'll ever need
S3 & Glacier - The only backup solution you'll ever needS3 & Glacier - The only backup solution you'll ever need
S3 & Glacier - The only backup solution you'll ever needMatthew Boeckman
 
Honing headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextHoning headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextFastly
 
Honing headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextHoning headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextFastly
 
20190516 web security-basic
20190516 web security-basic20190516 web security-basic
20190516 web security-basicMksYi
 
Securing Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultSecuring Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultBram Vogelaar
 

Similar to Just curl it! (20)

mastering the curl command line.pdf
mastering the curl command line.pdfmastering the curl command line.pdf
mastering the curl command line.pdf
 
CouchDB Day NYC 2017: Introduction to CouchDB 2.0
CouchDB Day NYC 2017: Introduction to CouchDB 2.0CouchDB Day NYC 2017: Introduction to CouchDB 2.0
CouchDB Day NYC 2017: Introduction to CouchDB 2.0
 
Front End Website Optimization
Front End Website OptimizationFront End Website Optimization
Front End Website Optimization
 
Curl Tutorial
Curl Tutorial Curl Tutorial
Curl Tutorial
 
Next Level Curl
Next Level CurlNext Level Curl
Next Level Curl
 
Sword v2 at UKCoRR
Sword v2 at UKCoRRSword v2 at UKCoRR
Sword v2 at UKCoRR
 
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio Lopes
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio LopesHTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio Lopes
HTTP/2, SPDY e Otimizações Web - Front In Maceió 2014 - Sérgio Lopes
 
The Case for HTTP/2
The Case for HTTP/2The Case for HTTP/2
The Case for HTTP/2
 
Intro to JavaScript
Intro to JavaScriptIntro to JavaScript
Intro to JavaScript
 
CouchDB Day NYC 2017: Mango
CouchDB Day NYC 2017: MangoCouchDB Day NYC 2017: Mango
CouchDB Day NYC 2017: Mango
 
Copy of cgi
Copy of cgiCopy of cgi
Copy of cgi
 
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."
IBM dwLive, "Internet & HTTP - 잃어버린 패킷을 찾아서..."
 
S3 & Glacier - The only backup solution you'll ever need
S3 & Glacier - The only backup solution you'll ever needS3 & Glacier - The only backup solution you'll ever need
S3 & Glacier - The only backup solution you'll ever need
 
Honing headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextHoning headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertext
 
Honing headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertextHoning headers for highly hardened highspeed hypertext
Honing headers for highly hardened highspeed hypertext
 
RESTFul IDEAS
RESTFul IDEASRESTFul IDEAS
RESTFul IDEAS
 
20190516 web security-basic
20190516 web security-basic20190516 web security-basic
20190516 web security-basic
 
RESTful design
RESTful designRESTful design
RESTful design
 
Intro to JavaScript
Intro to JavaScriptIntro to JavaScript
Intro to JavaScript
 
Securing Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultSecuring Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp Vault
 

More from Daniel Stenberg

mastering libcurl part 2
mastering libcurl part 2mastering libcurl part 2
mastering libcurl part 2Daniel Stenberg
 
mastering libcurl part 1
mastering libcurl part 1mastering libcurl part 1
mastering libcurl part 1Daniel Stenberg
 
curl - openfourm europe.pdf
curl - openfourm europe.pdfcurl - openfourm europe.pdf
curl - openfourm europe.pdfDaniel Stenberg
 
curl experiments - curl up 2022
curl experiments - curl up 2022curl experiments - curl up 2022
curl experiments - curl up 2022Daniel Stenberg
 
curl security - curl up 2022
curl security - curl up 2022curl security - curl up 2022
curl security - curl up 2022Daniel Stenberg
 
HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022Daniel Stenberg
 
Let me tell you about curl
Let me tell you about curlLet me tell you about curl
Let me tell you about curlDaniel Stenberg
 
Writing safe and secure code
Writing safe and secure codeWriting safe and secure code
Writing safe and secure codeDaniel Stenberg
 
Säker kod (secure code)
Säker kod (secure code)Säker kod (secure code)
Säker kod (secure code)Daniel Stenberg
 
You know what's cool? Running on a billion devices
You know what's cool? Running on a billion devicesYou know what's cool? Running on a billion devices
You know what's cool? Running on a billion devicesDaniel Stenberg
 

More from Daniel Stenberg (12)

mastering libcurl part 2
mastering libcurl part 2mastering libcurl part 2
mastering libcurl part 2
 
mastering libcurl part 1
mastering libcurl part 1mastering libcurl part 1
mastering libcurl part 1
 
curl - openfourm europe.pdf
curl - openfourm europe.pdfcurl - openfourm europe.pdf
curl - openfourm europe.pdf
 
curl experiments - curl up 2022
curl experiments - curl up 2022curl experiments - curl up 2022
curl experiments - curl up 2022
 
curl security - curl up 2022
curl security - curl up 2022curl security - curl up 2022
curl security - curl up 2022
 
HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022
 
The state of curl 2022
The state of curl 2022The state of curl 2022
The state of curl 2022
 
Let me tell you about curl
Let me tell you about curlLet me tell you about curl
Let me tell you about curl
 
Writing safe and secure code
Writing safe and secure codeWriting safe and secure code
Writing safe and secure code
 
The state of curl 2019
The state of curl 2019The state of curl 2019
The state of curl 2019
 
Säker kod (secure code)
Säker kod (secure code)Säker kod (secure code)
Säker kod (secure code)
 
You know what's cool? Running on a billion devices
You know what's cool? Running on a billion devicesYou know what's cool? Running on a billion devices
You know what's cool? Running on a billion devices
 

Recently uploaded

Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfkalichargn70th171
 
The State of the Green IT at the beginning of 2024
The State of the Green IT at the beginning of 2024The State of the Green IT at the beginning of 2024
The State of the Green IT at the beginning of 2024Artur Skowroński
 
What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxTechnogeeks
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jNeo4j
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
Preparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesPreparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesAke Koomsin
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisDEEPRAJ PATHAK
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapIshara Amarasekera
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...OnePlan Solutions
 
ManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...kalichargn70th171
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxRemote DBA Services
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Milind Agarwal
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUsamruddhijedgule2004
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 

Recently uploaded (20)

Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
 
The State of the Green IT at the beginning of 2024
The State of the Green IT at the beginning of 2024The State of the Green IT at the beginning of 2024
The State of the Green IT at the beginning of 2024
 
What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docx
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
Preparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesPreparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple Architectures
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business Analysis
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery Roadmap
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
 
ManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide Deck
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptx
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 

Just curl it!

  • 1. Just curl itJust curl it October 22, Platform Summit 2019October 22, Platform Summit 2019
  • 4. An open source project that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
  • 5. Features! DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
  • 7. 2,000 contributors Who makes curl curl 730 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
  • 8. @bagder@bagder Conquered the world Right time Open Source Portable Backwards compatible Stable
  • 9. Many places curl has been around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
  • 10. Many flavors Millions of build combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
  • 11. Many protocols and options Today we only talk about the tool named curl curl knows many protocols curl features 226 command line options We stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
  • 12. HTTP RequestRequest - method + path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
  • 13. Under the hood GET / HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
  • 14. Minimal behavior by default Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
  • 15. Long and short options All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
  • 16. $ curl example.com <html> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
  • 17. $ curl -i https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
  • 18. $ curl https://example.com/json | jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
  • 19. $ curl -I https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
  • 20. $ curl -I https://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
  • 21. $ curl -I -L https://example.com/redirected HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
  • 22. $ curl -v https://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
  • 23. $ curl https://example.com/ -H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
  • 24. $ curl -d name=Daniel https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
  • 25. $ curl -d @file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
  • 26. $ curl -T localfile https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
  • 27. $ curl -T localfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
  • 28. curl -X in vain leads to pain https://xkcd.com/386/ @bagder@bagder
  • 29. $ curl -d postdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
  • 30. @bagder@bagder Bad -X -X means use this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
  • 32. @bagder@bagder Cookies are name value pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
  • 33. $ curl -c cookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
  • 34. $ curl -c cookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login procedure
  • 35. $ curl https://127.0.0.1/ $ curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
  • 38. $ curl https://example.com/ -d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
  • 39. $ curl https://example.com/ --libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
  • 41. $ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
  • 42. SSLKEYLOGFILE Why trust your tools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
  • 43. $ export SSLKEYLOGFILE=$HOME/tmp/tlskey $ curl https://example.com @bagder@bagder Snoop on curl
  • 45. Many HTTP versions – but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 came in 2015 HTTP/3 is “coming soon” curl makes HTTP/2 and HTTP/3 headers look and work like HTTP/1 @bagder@bagder
  • 46. $ curl –-http1.0 https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
  • 47. $ curl -d user=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
  • 48. All this, and much much more... @bagder@bagder
  • 50. 51 You can help!You can help! @bagder@bagder
  • 51. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
  • 52. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder