9. 92 operating systems
@bagde
r
Syllable OS TPF
Tizen
Symbian
Tru64
SunOS
tvOS ucLinux
Genode Hurd
iOS
Integrity
Illumos
HP-UX
HardenedBSD
Haiku
z/OS
Nintendo
Switch
NonStop OS
NetWare
MorphOS MPE/iX MS-DOS NCR MP-RAS NetBSD
RISC OS
Redox
ReactOS Sailfish OS
SCO Unix Serenity SINIX-Z
Qubes OS
UnixWare
WebOS
vxWorks
VMS
Windows
UNICOS
Windows CE
Wii System
Software
AmigaOS Blackberry 10
BeOS
Android
Blackberry
Tablet OS
AIX Cell OS
Aros
IRIX
RTEMS
Mbed
Micrium
macOS
Mac OS 9
Linux Lua RTOS
eCOS
FreeRTOS
FreeBSD
FreeDOS
Fuchsia
DragonFly
BSD
Cygwin
Cisco IOS
OpenBSD OS/2 OS/400
Ultrix
ipadOS
NuttX
Solaris
Xbox
System
Chrome OS
MINIX
Garmin OS
QNX
PlayStation
Portable
Plan 9
OS21
OpenStep Orbis OS
z/TPF z/VM z/VSE
Operating systems known to have run curl
Atari FreeMiNT
DR DOS
OmniOS
Zephyr
watchOS
Xenix
DG/UX
17. Parallel transfers
by default, URLs are transferred serially, one by one
-Z (--parallel)
By default up to 50 simultaneous
Change with --parallel‐max [num]
Works for downloads and uploads
@bagder
server
client
Upload
Download
$ curl -Z -O https://example.com/[1-1000].jpg
18. --no-clobber
does not overwrite destination, adds a number instead
@bagder
$ touch file
$ curl -O --no-clobber https://example.com/file
$ ls -l file.1
-rw-r--r-- 1 user user 1256 Oct 18 13:27 file.1
19. --remove-on-error
do not save leftovers on error
@bagder
$ curl -O --remove-on-error --max-time 2 https://example.com/file
$ curl: (28) Operation timed out after 2000 …
$ ls -l file
ls: cannot access 'file': No such file or directory
20. transfer controls
stop slow transfers
--speed‐limit <speed> --speed‐time <seconds>
transfer rate limiting
curl --limit-rate 100K https://example.com
no more than this number of transfer starts per time unit
curl --rate 2/s https://example.com/[1-20].jpg
curl --rate 3/h https://example.com/[1-20].html
curl --rate 14/m https://example.com/day/[1-365]/fun.html
Truly limit the maximum file size accepted:
curl --max-filesize 238M https://example.com/the-biggie -O
@bagder
22. Massaging URL “queries”
Queries are often name=value pairs separated by amperands (&)
name=daniel & tool=curl & age=old
Add query parts to the URL with --url-query [content]
content is (for example) “name=value”
“value” gets URL encoded to keep the URL fine
@bagder
scheme://user:password@host:1234/path?query#fragment
24. config file
“command lines in a file”
one option (plus argument) per line
$HOME/.curlrc is used by default
-K [file] or --config [file]
can be read from stdin
can be generated (and huge)
10MB line length limit
@bagder
25. Variables in a config file
user = “$USER:$SECRET” - cannot could not be done
introducing this syntax risks breaking countless existing files
what if you would rather read that info from a separate file?
@bagder
26. --variable
Sets a curl variable on command line or in a config file
--variable name=content
variable = name=content
There can be an unlimited amount of variables.
A variable can hold up to 10M of content
Variables are set in a left to right order as it parses the command line or config file.
@bagder
27. Setting variables
Content for the variable can be read from a file:
--variable name@file
Content for the variable can be read from stdin:
--variable name@-
@bagder
28. Environment variables
A variable can be imported from the environment. Error if non-existing:
--variable %name
Import a name from the environment, but if not set use a default value:
--variable %name=default
Import a name from the environment, but if not set read the default value from
a file:
--variable %name@filename
@bagder
29. Expanding variables
Variables can be used in command line option arguments
Must be explicitly asked for
Introducing the --expand- prefix
Reference a variable as {{name}}
A non-existing variable will expand as blank/nothing
Trying to show a variable with a null byte causes error
Examples:
--expand-data “{{content}}”
--expand-url “https://{{host}}/user/{{user}}”
@bagder
30. Expansion functions
When expanding a variable, functions can be applied
{{name:function}}
Functions alter how the variable is expanded
Multiple functions can be applied in a left-to-right order:
{{name:func1:func2:func3}}
Provided functions:
trim, json, url and b64
@bagder
$ curl --variable %DATA --expand-data ‘{{DATA:trim:b64}}’
https://example.com/
33. --write-out
outputs text, information and HTTP headers after a transfer is completed
curl -w "formatted string" http://example.com/
curl -w @filename http://example.com/
curl -w @- http://example.com/
Information from over 50 “variables”
curl -w "Type: %{content_type}nCode: %{response_code}n" …
Show the information as JSON:
curl -w ‘%{json}’ …
Show HTTP response header contents
curl -w "Server: %header{server}nDate: %header{date}n" …
@bagder
37. trace even more
@bagder
--trace-config tells curl what more to include in the trace
“all”, “HTTP/2, “HTTP/3”, “TLS”, …
This was previously only possible in debug builds
$ curl --trace dump --trace-config all https://example.com/one
40. etags
do the transfer only if…
The remote resource is “different”
curl --etag-save remember.txt -O https://example.com/file
curl --etag-compare remember.txt -O https://example.com/file
Both can be used at once for convenient updates:
curl --etag-save remember.txt --etag-compare remember.txt -O
https://example.com/file
Better than time conditions
@bagder
41. HTTP alt-svc
server tells client: there is one or more alternatives at "another place"
The Alt-Svc: response header
Each entry has an expiry time
Only recognized over HTTPS
curl --alt-svc altcache.txt https://example.com/
The alt-svc cache is a text based readable file
@bagder
42. HTTP HSTS
HSTS - HTTP Strict Transport Security
This host name requires HTTPS going forward
The Strict-Transport-Security: response header
Only recognized over HTTPS
Each entry has an expiry time
curl --hsts hsts.txt http://example.com/
The HSTS cache is a text based readable file
@bagder
43. HTTP version 3 changed how data is sent over the wire - again
HTTP/3 is done over QUIC, a new transport protocol
QUIC replaces TCP + TLS, and runs over UDP
curl hides protocol differences from users
HTTP/3 is experimental in curl
HTTP/3 is only for HTTPS, there is no clear text version
with --http3 you can ask curl to attempt HTTP/3
--http3 races HTTP/3 against HTTP/1+2 and picks the winner
With HTTP/3, curl can do multiplexed transfers with -Z
@bagder
44. @bagder
racing
curl.se
client
curl.se has address 151.101.129.91
curl.se has address 151.101.193.91
curl.se has address 151.101.1.91
curl.se has address 151.101.65.91
curl.se has IPv6 address 2a04:4e42:800::347
curl.se has IPv6 address 2a04:4e42:a00::347
curl.se has IPv6 address 2a04:4e42:c00::347
curl.se has IPv6 address 2a04:4e42:e00::347
curl.se has IPv6 address 2a04:4e42::347
curl.se has IPv6 address 2a04:4e42:200::347
curl.se has IPv6 address 2a04:4e42:400::347
curl.se has IPv6 address 2a04:4e42:600::347
DNS
2a04:4e42:800::347
2a04:4e42:a00::347
2a04:4e42:c00::347
2a04:4e42:e00::347
2a04:4e42::347
2a04:4e42:200::347
2a04:4e42:400::347
2a04:4e42:600::347
151.101.129.91
151.101.193.91
151.101.1.91
151.101.65.91
h2 IPv6
h2 IPv4
h3 IPv4
h3 IPv6
47. Going next?
curl is 25 years old
curl development speed is increasing
the Internet does not stop or slow down
protocols and new ways of doing Internet transfers keep popping up
curl will keep expanding, get new features, get taught new things
we, the community, make it do what we think it should do
you can affect what’s next for curl
@bagder