@bagder
curl with rust
Daniel Stenberg – January 27, 2022
Daniel Stenberg
@bagder
https://daniel.haxx.se
Daniel Stenberg
@bagder
@bagder
Agenda
curl is C
Stable API and ABI
Backends
Making an HTTP backend
… and other backends
Challenges
Current status
Future
@bagder
@bagder
Q&A
Immediately
following
@bagder
@bagder
curl is C
@bagder
@bagder
When curl started there
was no choice
C89
Only now alternatives
appear for libraries
C keeps curl extremely
portable
C code will remain a build
option
@bagder
@bagder
Stable API and ABI
@bagder
@bagder
Reliable and predictable
libcurl is forwards API compatible
libcurl does not break the ABI – ever
libcurl exists everywhere
Compiled code from 2001 still works
The armored front we can’t tamper with
@bagder
10,000,000,000
installations
@bagder
Vulnerabilities
50% of past curl vulnerabilities are “C mistakes”
Memory-safe languages could help
Also doing other things to reduce risk
The future is more Internet-connected devices
@bagder
@bagder
Backends and third parties
@bagder
@bagder
Backends
(Build-time) selectable alternative
implementations
@bagder
@bagder
Backends
Backends are selectable and deselectable
Often platform dependent
Often use 3rd
party libraries
… which can differ in features, licensing and maturity
Can be done in other programming languages
The internal APIs are never exposed externally
@bagder
third party world map February 2022
I/O layer
URL parser libidn2
winidn
HTTP
TLS
OpenSSL
gskit
mbedTLS
wolfSSL
Schannel
Secure
Transport
GnuTLS
NSS
boringssl
libressl
AmiSSL
SFTP SCP LDAP
WinLDAP
OpenLDAP
RTMP
librtmp
Name resolver c-ares
compression
libz brotli
cookies
libpsl
IMAP SMTP POP3
HTTP/2
nghttp2
authentication
winsspi Heimdal
MIT
kerberos
HTTP/3
quiche
ngtcp2
HTTP/1
SSH
wolfSSH
libssh2
libssh
@bagder
BearSSL
nghttp3
zstd
FTP
Hyper
FTPS IMAPS POP3S SMBS SMTPS
GOPHERS HTTPS LDAPS RTMPS
libgsasl
rustls
@bagder
Many different backends
IDN
Name resolving
TLS
SSH
HTTP/3
HTTP content encoding
HTTP
@bagder
@bagder
The making of an HTTP backend
libcurl API
Generic transfer engine
HTTP
Creating and reusing connections
@bagder
HTTP involves a lot of things
Authentication
Creating a set of headers
Proxy specifics
Parsing content headers
Parsing transfer headers
HTTP/1 and HTTP/2 transmission
@bagder
Hyper is transmission-focused
Authentication
Creating a set of headers
Proxy specifics
Parsing content headers
Parsing transfer headers
HTTP/1 and HTTP/2 transmission
@bagder
Splitting up HTTP
Authentication
Creating a set of headers
Proxy specifics
Parsing content headers
Parsing transfer headers
HTTP/1 and HTTP/2 transmission
Parsing transfer headers
HTTP/1 and HTTP/2 transmission
Hyper Built-in + nghttp2
@bagder
build curl with hyper
$ ./configure --with-hyper=...
https://github.com/curl/curl/blob/master/docs/HYPER.md
@bagder
HTTP backends
libcurl API
Generic transfer engine
High-level HTTP
Creating and reusing connections
built-in
HTTP/1
nghttp2
HTTP/2
Hyper
HTTP/1 + 2
@bagder
libcurl backends application
HTTP
API
libcurl
Public API
Hyper
built-in
nghttp2
@bagder
rustls
@bagder
@bagder
rustls provides a C API in rustls-ffi
curl builds with different TLS libraries
build curl to use rustls-ffi
… profit!
another TLS backend
@bagder
quiche
@bagder
@bagder
quiche provides a C API
curl builds with different QUIC libraries
build curl to use quiche
… profit!
another HTTP/3 and QUIC backend
@bagder
libcurl backends
libidn2
winidn
Hyper
built-in
threaded
c-ares
sync
quiche
ngtcp2 +
nghttp3
libssh2
wolfSSH
libssh
BearSSL
Gskit
GnuTLS
mbedSSL NSS
OpenSSL Schannel
wolfSSL
Secure Transport
rustls
application
HTTP
API
TLS
API
IDN API
SSH API
HTTP/3
API
Resolver
API
Public API
libcurl
Content encoding
API
brotli
zstd
zlib
= just one
= one or more
nghttp2
@bagder
libcurl backends
libidn2
winidn
Hyper
built-in
threaded
c-ares
sync
quiche
ngtcp2 +
nghttp3
libssh2
wolfSSH
libssh
BearSSL
Gskit
GnuTLS
mbedSSL NSS
OpenSSL Schannel
wolfSSL
Secure Transport
rustls
application
HTTP
API
TLS
API
IDN API
SSH API
HTTP/3
API
Resolver
API
Public API
libcurl
Content encoding
API
brotli
zstd
zlib
= just one
= one or more
nghttp2
@bagder
Challenges
@bagder
@bagder
Using hyper
First user of the Hyper C API
Headers-only, no C docs yet
Splitting HTTP for built-in + Hyper took thinking
Keeping behavior with different API paradigms
Memory leaks are tricky to hunt down
Rust use is mostly seamless (and not the first use)
@bagder
@bagder
Using rustls
Early user of rustls-ffi
Headers-only, no C docs yet
Mostly done by others so easier for me!
The TLS backend situation was already mature
Not yet feature-complete
@bagder
@bagder
Using quiche
Early user of quiche
Headers-only, no C docs yet
Messy TLS situation (for QUIC)
Not yet feature-complete
@bagder
@bagder
Using rust
Updating rust and cargo “all the time”
Suitable for replacing one-component-at-a-time
I am a total rust rookie
@bagder
@bagder
Current status
@bagder
@bagder
Status
Using curl and Hyper from main dev branches
“experimental” and opt-in
HTTP/1 and HTTP/2
HTTP and HTTPS (TLS backend agnostic)
HTTP(S) proxies
Identical HTTP requests over the wire
98% test case success rate (~800 test cases)
@bagder
@bagder
Status now
hyper
“experimental” and opt-in
HTTP/1 and HTTP/2
HTTP and HTTPS (TLS backend agnostic)
HTTP(S) proxies
Identical HTTP requests over the wire
98% test case success rate (~800 test cases)
@bagder
rustls
“experimental” and opt-in
12 test cases disabled
no IP-addresses in cert?
quiche
“experimental” and opt-in
no HTTP/3 tests yet
QUIC and HTTP/3 are not “there” yet anyway
rust in curl
works perfectly well
depends entirely on the C APIs
… which are under-documented still
no stable OOM == panic (will abort)
You can
help!
@bagder
Future
@bagder
@bagder
Coming up
Make all test cases succeed
Verify libcurl API corner cases
Encourage users and developers to use rust backends
Provide binary builds
Enable by default?
More rust components?
More backend flavors?
@bagder
more curl!
Commercial curl support!
@bagder
Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!
Questions?

Curl with rust