mastering libcurl part 1

@bagder
/* set the options (I left out a few, you will get the point anyway) */
curl_easy_setopt(handles[HTTP_HANDLE], CURLOPT_URL, "https://example.com");
curl_easy_setopt(handles[FTP_HANDLE], CURLOPT_URL, "ftp://example.com");
curl_easy_setopt(handles[FTP_HANDLE], CURLOPT_UPLOAD, 1L);
/* init a multi stack */
multi_handle = curl_multi_init();
/* add the individual transfers */
for(i = 0; i<HANDLECOUNT; i++)
curl_multi_add_handle(multi_handle, handles[i]);
while(still_running) {
CURLMcode mc = curl_multi_perform(multi_handle, &still_running);
if(still_running)
/* wait for activity, timeout or "nothing" */
mc = curl_multi_poll(multi_handle, NULL, 0, 1000, NULL);
if(mc)
break;
}
/* See how the transfers went */
while((msg = curl_multi_info_read(multi_handle, &msgs_left))) {
if(msg->msg == CURLMSG_DONE) {
int idx;
/* Find out which handle this message is about */
for(idx = 0; idx<HANDLECOUNT; idx++) {
int found = (msg->easy_handle == handles[idx]);
if(found)
break;
}
switch(idx) {
case HTTP_HANDLE:
printf("HTTP transfer completed with status %dn", msg->data.result);
break;
case FTP_HANDLE:
printf("FTP transfer completed with status %dn", msg->data.result);
break;
}
}
}
/* remove the transfers and cleanup the handles */
for(i = 0; i<HANDLECOUNT; i++) {
curl_multi_remove_handle(multi_handle, handles[i]);
curl_easy_cleanup(handles[i]);
}
#include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
/* example.com is redirected, so we tell libcurl to follow redirection */
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %sn",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
mastering libcurl
November 16, 2023 Daniel Stenberg
more libcurl source code and details in a single video than you ever saw before
part one

Daniel Stenberg @bagder
@mastodon.social
https://daniel.haxx.se

Setup - November 16 + 20, 2023
Live-streamed
Expected to last multiple hours
Recorded
Lots of material never previously presented
There will be LOTS of source code on display
https://github.com/bagder/mastering-libcurl
@bagder

@bagder
Just ask!
Live questions from non-visible chat
@bagder

mastering libcurl
The project
Getting it
API and ABI
Architecture
API fundamentals
Setting up
@bagder
Transfers
Share API
TLS
Proxies
HTTP
Header API
URL API
WebSocket
Future
Part 1 Part 2
November 16, 2023 November 20, 2023

The project
@bagder
slide 7 of 88

November 1996: httpget
August 1997: urlget
March 1998: curl
August 2000: libcurl
@bagder

Name
client for URLs
“see URL”
curl URL Request Library
Capable Ubiquitous Reliable Libre
@bagder

Main products
curl - command line tool for client-side internet transfers with URLs
libcurl - library for client-side internet transfers with URLs
★ Always and only client-side
★ An internet transfer: upload or download or both
★ Endpoint described with a URL
@bagder

Open Source
Everything in the curl project is open source
Every discussion and decision are held and done in the open
Open source means everyone can reshare and change
curl is (essentially) MIT licensed
@bagder
Free
Open
Gratis

COPYRIGHT AND PERMISSION NOTICE
Copyright © 1996 - 2023, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file.
All rights reserved.
Permission to use, copy, modify, and distribute this software for any purpose
with or without fee is hereby granted, provided that the above copyright
notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not
be used in advertising or otherwise to promote the sale, use or other dealings
in this Software without prior written authorization of the copyright holder.
@bagder

Development
curl is developed by “everyone”
only Daniel works on curl full-time
everyone can provide their proposed changes as “pull requests”
No paperwork required
Changes are reviewed and tested thoroughly
A small team of maintainers can accept and merge changes
@bagder

Releases
We do releases every 8 weeks (or sooner when necessary)
At 252 releases
We release what is in the master branch at the time
@bagder

Releases
We never break existing functionality
@bagder

Issues and pull requests
Problems are submitted as issues
Changes are proposed as pull requests
https://github.com/curl/curl
@bagder

Learn more
man pages
https://curl.se/libcurl
Everything curl
@bagder

Asking for help
Mailing lists
library/development/debugging questions: curl-library
https://lists.haxx.se/mailman/listinfo/curl-library
Web “forum”
https://github.com/curl/curl/discussions
@bagder

Paying for curl help
support@wolfssl.com
@bagder

DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP,
IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP,
SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS
TLS certiﬁcates, HTTP POST, HTTP PUT, FTP upload, HTTP form
based upload, proxies (SOCKS4, SOCKS5, HTTP and HTTPS),
HTTP/2, HTTP/3, cookies, user+password authentication (Basic,
Plain, Digest, CRAM-MD5, SCRAM-SHA, NTLM, Negotiate, Kerberos,
Bearer tokens and AWS Sigv4), ﬁle transfer resume, proxy
tunneling, HSTS, Alt-Svc, unix domain sockets, HTTP compression
(gzip, brotli and zstd), etags, parallel transfers, DNS-over-HTTPS
and much more
@bagder

Runs everywhere
Linux
Windows
macOS
FreeBSD
OpenBSD
VMS
…
@bagder

101 operating systems
@bagder
Syllable OS TPF
Tizen
Symbian Tru64
SunOS tvOS
ucLinux
Genode Hurd
iOS
Integrity
Illumos
HP-UX
HardenedBSD
Haiku
z/OS
Nintendo
Switch
NonStop OS
NetWare
MorphOS MPE/iX MS-DOS
NCR MP-RAS NetBSD
RISC OS
Redox
ReactOS
Sailﬁsh OS SCO Unix Serenity SINIX-Z
Qubes OS
UnixWare WebOS
vxWorks
VMS
Windows
UNICOS
Windows CE
Wii System
Software
AmigaOS Blackberry 10
BeOS
Android
Blackberry
Tablet OS
AIX
Cell OS
Aros
IRIX
RTEMS
Mbed Micrium
macOS
Mac OS 9
Linux Lua RTOS
eCOS
FreeRTOS
FreeBSD
FreeDOS
Fuchsia
DragonFly
BSD
ROS
Cisco IOS
OpenBSD
OS/2 OS/400
Ultrix
ipadOS
NuttX
Solaris
Xbox
System
Chrome OS
MINIX
Garmin OS
QNX
PlayStation
Portable
Plan 9
OS21
OpenStep
Orbis OS
z/TPF
z/VM z/VSE Operating systems known to have run curl
Atari FreeMiNT
DR DOS
Sortix
Zephyr
watchOS
Xenix
DG/UX
ArcaOS
Wii U
SkyOS
Wear OS
Meego
Maemo Moblin
NextStep
CheriBSD

>20,000,000,000
installations
@bagder

getting
@bagder
slide 30 of 88

Installing
Linux distros
BSD
Homebrew / macports
vcpkg / Chocolately / conan
…
@bagder
Debian and Ubuntu
apt install libcurl4-openssl-dev
apt install libcurl4-gnutls-dev
Redhat and Centos
yum install libcurl-devel
Nix
nix-env -i curl
SUSE and OpenSUSE
zypper install libcurl-devel
Arch
pacman -S curl
FreeBSD
pkg install curl
macOS Homebrew
brew install curl
Windows MSYS2
pacman -Sy mingw-w64-x86_64-curl
Windows vcpkg
vcpkg.exe install curl:x64-windows
Gentoo
emerge curl

Build it yourself
conﬁgure
cmake
MSVC
needs third party development packages installed
@bagder
autotools
./configure with-openssl …
make -sj
make install
cmake
mkdir build
cd build
cmake .
make -sj
MSVC
cd winbuild
name /f Makefile.vc mode=dll WITH_SSL=dll
Visual C++ project ﬁles
<projects/Windows/VC*/curl-all.sln>

Building custom versions
enable/disable features, protocols and 3rd parties
libcurl API functions remain, but may return different values
Common when you build devices / control the environment
@bagder

Third party dependencies
Many features require 3rd party libraries
TLS
SSH
LDAP
RTMP
HTTP compression (gzip, brotli, zstd)
HTTP/2
HTTP/3
Asynchronous name resolving without threads
Auth - kerberos, SASL
@bagder

Target requirements
a minimal libcurl build with smallest TLS lands on around 100kB
minimum run-time memory requirements for libcurl: 20kB
add memory requirements for the TLS library
the API remains the same
@bagder

API and ABI
@bagder
slide 36 of 88

compatibility
The same API on all platforms
You can always upgrade to the next version without breakage
You may not be able to downgrade
Options and functions are clearly documented when they were added
@bagder

versions
libcurl 7.1 released August 7, 2000
libcurl 8.4.0 released October 11, 2023
Differ in features, protocols, performance
Later versions have more functions
@bagder

The libcurl API is for C
libcurl provides a C interface
works for C++ as well
We will only speak C today
Users of other languages get to use bindings
@bagder

header ﬁles
@bagder

compiling libcurl programs
gcc mycode.c -lcurl
clang mycode.c -lcurl
Your other C compilers work too:
1. Specify where the curl header ﬁle is
2. Tell the linker to link with libcurl
@bagder

Documentation
We take documentation seriously
Every function has a man page
Every option has a man page
120 stand-alone examples
https://curl.se/ has them all (updated)
https://everything.curl.dev/ provides more resources
Tip: just google the function/option name
@bagder

Architecture
@bagder
slide 43 of 88

@bagder
When curl started there
was no choice
C89
Only now alternatives
appear for libraries
C keeps curl extremely
portable
C code will remain a build
option
@bagder

@bagder
libcurl backends
libidn2
winidn
threaded
c-ares
sync
libssh2
wolfSSH
libssh
application
Public API
libcurl
HTTP
HTTP/3
TLS
SSH
IDN
Resolver
Content encoding
brotli
zstd
zlib
= just one
= one or more
Hyper
built-in
nghttp2
msh3
nghttp3
quiche
BearSSL
AWS-LC
GnuTLS
mbedSSL
OpenSSL
Schannel
wolfSSL
Secure Transport
rustls
BoringSSL
libressl
AmiSSL

@bagder
non-blocking
parallel name resolves
parallel connection establishments
parallel TLS handshakes
parallel transfers
with different latencies and bandwidths
in a single thread

API fundamentals
@bagder
slide 47 of 88

@bagder
U
R
L
s
c
a
l
l
b
a
c
k
s
opaque handles
@bagder

basic transfers by default
Transfers only do what is asked; basic by default
The API is largely protocol agnostic
Change behavior by setting options
@bagder

global init
An application should do a global libcurl init
And a global cleanup
@bagder
global.c
#include <stdio.h>
int main(void)
{
curl_global_init(CURL_GLOBAL_DEFAULT);
/* use libcurl, then before exiting... */
curl_global_cleanup();
return 0;
}

easy handle
CURL *handle - often referred to as an easy handle
A CURL * is an opaque handle
Compare to FILE *
A handle identiﬁes a single transfer
Can and should be reused
Create one with curl_easy_init()
Free it with curl_easy_cleanup()
@bagder
easy-init.c
#include <stdio.h>
int main(void)
{
CURL *easy = curl_easy_init();
curl_easy_setopt(easy, CURLOPT_URL,
“https://curl.se/”);
curl_easy_perform(easy);
curl_easy_cleanup(easy);
return 0;
}

easy options
An easy handle needs options set to know what to do
sticky
independent
order-independent
Options set copy data
CURLOPT_URL is the only mandatory option
Download is the default action
An application typically sets many options
Options control timeouts, name resolve alternatives, connectivity, protocol version selection,
TLS version selection, authentication, proxies, how to receive or receive data - and much much
more.
@bagder

There are 300+ options for curl_easy_setopt
ABSTRACT_UNIX_SOCKET, ACCEPT_ENCODING, ACCEPTTIMEOUT_MS, ADDRESS_SCOPE, ALTSVC, ALTSVC_CTRL, APPEND, AUTOREFERER, AWS_SIGV4, BUFFERSIZE, CA_CACHE_TIMEOUT,
CAINFO, CAINFO_BLOB, CAPATH, CERTINFO, CHUNK_BGN_FUNCTION, CHUNK_DATA, CHUNK_END_FUNCTION, CLOSESOCKETDATA, CLOSESOCKETFUNCTION, CONNECT_ONLY,
CONNECTTIMEOUT, CONNECTTIMEOUT_MS, CONNECT_TO, CONV_FROM_NETWORK_FUNCTION, CONV_FROM_UTF8_FUNCTION, CONV_TO_NETWORK_FUNCTION, COOKIE, COOKIEFILE,
COOKIEJAR, COOKIELIST, COOKIESESSION, COPYPOSTFIELDS, CRLF, CRLFILE, CURLU, CUSTOMREQUEST, DEBUGDATA, DEBUGFUNCTION, DEFAULT_PROTOCOL, DIRLISTONLY,
DISALLOW_USERNAME_IN_URL, DNS_CACHE_TIMEOUT, DNS_INTERFACE, DNS_LOCAL_IP4, DNS_LOCAL_IP6, DNS_SERVERS, DNS_SHUFFLE_ADDRESSES, DNS_USE_GLOBAL_CACHE,
DOH_SSL_VERIFYHOST, DOH_SSL_VERIFYPEER, DOH_SSL_VERIFYSTATUS, DOH_URL, EGDSOCKET, ERRORBUFFER, EXPECT_100_TIMEOUT_MS, FAILONERROR, FILETIME, FNMATCH_DATA,
FNMATCH_FUNCTION, FOLLOWLOCATION, FORBID_REUSE, FRESH_CONNECT, FTP_ACCOUNT, FTP_ALTERNATIVE_TO_USER, FTP_CREATE_MISSING_DIRS, FTP_FILEMETHOD, FTPPORT,
FTP_SKIP_PASV_IP, FTPSSLAUTH, FTP_SSL_CCC, FTP_USE_EPRT, FTP_USE_EPSV, FTP_USE_PRET, GSSAPI_DELEGATION, HAPPY_EYEBALLS_TIMEOUT_MS, HAPROXY_CLIENT_IP,
HAPROXYPROTOCOL, HEADER, HEADERDATA, HEADERFUNCTION, HEADEROPT, HSTS, HSTS_CTRL, HSTSREADDATA, HSTSREADFUNCTION, HSTSWRITEDATA, HSTSWRITEFUNCTION,
HTTP09_ALLOWED, HTTP200ALIASES, HTTPAUTH, HTTP_CONTENT_DECODING, HTTPGET, HTTPHEADER, HTTPPOST, HTTPPROXYTUNNEL, HTTP_TRANSFER_DECODING, HTTP_VERSION,
IGNORE_CONTENT_LENGTH, INFILESIZE, INFILESIZE_LARGE, INTERFACE, INTERLEAVEDATA, INTERLEAVEFUNCTION, IOCTLDATA, IOCTLFUNCTION, IPRESOLVE, ISSUERCERT,
ISSUERCERT_BLOB, KEEP_SENDING_ON_ERROR, KEYPASSWD, KRBLEVEL, LOCALPORT, LOCALPORTRANGE, LOGIN_OPTIONS, LOW_SPEED_LIMIT, LOW_SPEED_TIME, MAIL_AUTH, MAIL_FROM,
MAIL_RCPT, MAIL_RCPT_ALLOWFAILS, MAXAGE_CONN, MAXCONNECTS, MAXFILESIZE, MAXFILESIZE_LARGE, MAXLIFETIME_CONN, MAX_RECV_SPEED_LARGE, MAXREDIRS,
MAX_SEND_SPEED_LARGE, MIME_OPTIONS, MIMEPOST, NETRC, NETRC_FILE, NEW_DIRECTORY_PERMS, NEW_FILE_PERMS, NOBODY, NOPROGRESS, NOPROXY, NOSIGNAL,
OPENSOCKETDATA, OPENSOCKETFUNCTION, PASSWORD, PATH_AS_IS, PINNEDPUBLICKEY, PIPEWAIT, PORT, POST, POSTFIELDS, POSTFIELDSIZE, POSTFIELDSIZE_LARGE, POSTQUOTE,
POSTREDIR, PRE_PROXY, PREQUOTE, PREREQDATA, PREREQFUNCTION, PRIVATE, PROGRESSDATA, PROGRESSFUNCTION, PROTOCOLS, PROTOCOLS_STR, PROXY, PROXYAUTH,
PROXY_CAINFO, PROXY_CAINFO_BLOB, PROXY_CAPATH, PROXY_CRLFILE, PROXYHEADER, PROXY_ISSUERCERT, PROXY_ISSUERCERT_BLOB, PROXY_KEYPASSWD, PROXYPASSWORD,
PROXY_PINNEDPUBLICKEY, PROXYPORT, PROXY_SERVICE_NAME, PROXY_SSLCERT, PROXY_SSLCERT_BLOB, PROXY_SSLCERTTYPE, PROXY_SSL_CIPHER_LIST, PROXY_SSLKEY,
PROXY_SSLKEY_BLOB, PROXY_SSLKEYTYPE, PROXY_SSL_OPTIONS, PROXY_SSL_VERIFYHOST, PROXY_SSL_VERIFYPEER, PROXY_SSLVERSION, PROXY_TLS13_CIPHERS,
PROXY_TLSAUTH_PASSWORD, PROXY_TLSAUTH_TYPE, PROXY_TLSAUTH_USERNAME, PROXY_TRANSFER_MODE, PROXYTYPE, PROXYUSERNAME, PROXYUSERPWD, PUT, QUICK_EXIT, QUOTE,
RANDOM_FILE, RANGE, READDATA, READFUNCTION, REDIR_PROTOCOLS, REDIR_PROTOCOLS_STR, REFERER, REQUEST_TARGET, RESOLVE, RESOLVER_START_DATA,
RESOLVER_START_FUNCTION, RESUME_FROM, RESUME_FROM_LARGE, RTSP_CLIENT_CSEQ, RTSP_REQUEST, RTSP_SERVER_CSEQ, RTSP_SESSION_ID, RTSP_STREAM_URI, RTSP_TRANSPORT,
SASL_AUTHZID, SASL_IR, SEEKDATA, SEEKFUNCTION, SERVER_RESPONSE_TIMEOUT, SERVICE_NAME, SHARE, SOCKOPTDATA, SOCKOPTFUNCTION, SOCKS5_AUTH, SOCKS5_GSSAPI_NEC,
SOCKS5_GSSAPI_SERVICE, SSH_AUTH_TYPES, SSH_COMPRESSION, SSH_HOSTKEYDATA, SSH_HOSTKEYFUNCTION, SSH_HOST_PUBLIC_KEY_MD5, SSH_HOST_PUBLIC_KEY_SHA256,
SSH_KEYDATA, SSH_KEYFUNCTION, SSH_KNOWNHOSTS, SSH_PRIVATE_KEYFILE, SSH_PUBLIC_KEYFILE, SSLCERT, SSLCERT_BLOB, SSLCERTTYPE, SSL_CIPHER_LIST, SSL_CTX_DATA,
SSL_CTX_FUNCTION, SSL_EC_CURVES, SSL_ENABLE_ALPN, SSL_ENABLE_NPN, SSLENGINE, SSLENGINE_DEFAULT, SSL_FALSESTART, SSLKEY, SSLKEY_BLOB, SSLKEYTYPE, SSL_OPTIONS,
SSL_SESSIONID_CACHE, SSL_VERIFYHOST, SSL_VERIFYPEER, SSL_VERIFYSTATUS, SSLVERSION, STDERR, STREAM_DEPENDS, STREAM_DEPENDS_E, STREAM_WEIGHT,
SUPPRESS_CONNECT_HEADERS, TCP_FASTOPEN, TCP_KEEPALIVE, TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_NODELAY, TELNETOPTIONS, TFTP_BLKSIZE, TFTP_NO_OPTIONS, TIMECONDITION,
TIMEOUT, TIMEOUT_MS, TIMEVALUE, TIMEVALUE_LARGE, TLS13_CIPHERS, TLSAUTH_PASSWORD, TLSAUTH_TYPE, TLSAUTH_USERNAME, TRAILERDATA, TRAILERFUNCTION,
TRANSFER_ENCODING, TRANSFERTEXT, UNIX_SOCKET_PATH, UNRESTRICTED_AUTH, UPKEEP_INTERVAL_MS, UPLOAD, UPLOAD_BUFFERSIZE, URL, USERAGENT, USERNAME, USERPWD,
USE_SSL, VERBOSE, WILDCARDMATCH, WRITEDATA, WRITEFUNCTION, WS_OPTIONS, XFERINFODATA, XFERINFOFUNCTION, XOAUTH2_BEARER
@bagder

curl_easy_setopt
CURLcode curl_easy_setopt(CURL *handle, CURLoption option, parameter);
The parameter type depends on the option.
Beware: varargs makes the type checks lax
Might want extra typecasts
When setting strings, libcurl copies the data
Returns an error code
@bagder
easy-setopt.c
#include <stdio.h>
int main(void)
{
CURLcode result;
curl_easy_setopt(easy, CURLOPT_VERBOSE, 1L);
result = curl_easy_perform(easy);
return (int)result;
}

curl_easy_perform
For synchronous transfers
Performs to the end, successful or failure
Done as fast as possible
Might take a long - or short - time
You can limit how long time to allow
Returns success as an CURLcode
Stores cached info in the easy handle
@bagder
easy-perform.c
#include <stdio.h>
int main(void)
{
CURLcode result;
result = curl_easy_perform(easy);
return (int)result;
}

curl_easy_cleanup
Kills and frees all resources associated with an easy handle
You might want to call curl_easy_reset() and use it again
@bagder

callbacks
provide function pointer to curl (*FUNCTION)
provide a custom pointer to pass to the function (*DATA)
libcurl calls it when it needs things done
do not call libcurl functions from within callbacks
(unless you meet the exceptions)
@bagder

write callback
Data is provided to the application using the write callback: CURLOPT_WRITEFUNCTION
libcurl wants to write incoming data somewhere
size_t write_cb(char *ptr, size_t size, size_t nmemb, void *userdata);
ptr - points to the data that arrives
size - is always 1
nmemb - the size of the data in number of bytes
userdata - a custom pointer you set with CURLOPT_WRITEDATA
(prototype looks like fwrite)
@bagder

write callback example
@bagder
write-callback.c
struct memory {
char *ptr;
size_t size;
};
static size_t
write_cb(void *contents, size_t size, size_t nmemb, void *userp)
{
struct memory *mem = userp;
size_t realsize = size * nmemb;
char *ptr = realloc(mem->ptr, mem->size + realsize + 1);
if(!ptr) {
fprintf(stderr, "not enough memoryn");
return 0;
}
mem->ptr = ptr;
memcpy(&mem->ptr[mem->size], contents, realsize);
mem->size += realsize;
mem->ptr[mem->size] = 0;
return realsize;
}
int main(void)
{
CURL *easy;
CURLcode res;
struct memory chunk;
chunk.ptr = malloc(1); /* grown as needed */
chunk.size = 0;
curl_global_init(CURL_GLOBAL_ALL);
easy = curl_easy_init();
curl_easy_setopt(easy, CURLOPT_URL, "https://curl.se/");
curl_easy_setopt(easy, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(easy, CURLOPT_WRITEDATA, (void *)&chunk);
curl_easy_setopt(easy, CURLOPT_USERAGENT, "agent/1.22");
res = curl_easy_perform(easy);
if(res == CURLE_OK)
printf("%lu bytes retrievedn", (unsigned long)chunk.size);
free(chunk.ptr);
return 0;
}

multi handle
A “super handle” made for holding one or more easy handles.
Remember: each easy handle is a single transfer
With multi, all transfers are performed in parallel, in a non-blocking manner
Easy handles can be added and removed from the multi handle at any time
Create a multi handle with CURLM *handle = curl_multi_init()
Add an easy handle to a multi handle with curl_multi_add_handle(multi, easy)
Remove an easy handle again with curl_multi_remove_handle(multi, easy)
@bagder

curl_multi_perform
Do drive all transfers in the multi handle, call curl_multi_perform()
Does as much work it can without blocking, then returns
Your application waits until there is more work to do
Easiest done with curl_multi_poll()
Check if any of the transfers are done with curl_multi_info_read()
While there are unﬁnished transfers, continue performing them
@bagder

multi internals
Internally, the easy interface is using the multi interface
This ensures everything always work for either interface
All internals are done non-blocking (almost)
@bagder

curl_multi_info_read
The multi interface drives N parallel transfers
curl_multi_perform() does not return status about individual transfers
Returns messages when one or more of the transfers has ended
The message contains individual easy handle and CURLcode for the speciﬁc transfer
Call it again to get the next message
@bagder
struct CURLMsg {
CURLMSG msg; /* what this message means */
CURL *easy_handle; /* the handle it concerns */
union {
void *whatever; /* message-specific data */
CURLcode result; /* return code for transfer */
} data;
};
man curl_multi_info_read

curl_multi_cleanup
Closes a multi handle a frees all related resources
Does not close or free the individual easy handles
@bagder

multi.c
@bagder
CURL *easy2 = curl_easy_init();
curl_easy_setopt(easy, CURLOPT_URL, "https://example.com/");
curl_easy_setopt(easy2, CURLOPT_URL, "https://curl.se/");
CURLM *multi = curl_multi_init();
curl_multi_add_handle(multi, easy);
curl_multi_add_handle(multi, easy2);
int still = 1;
while(still) {
CURLMsg *msg;
CURLMcode mc = curl_multi_perform(multi, &still);
if(still) {
mc = curl_multi_poll(multi, NULL, 0, 1000, NULL);
if(mc)
break;
}
do {
int queued;
msg = curl_multi_info_read(multi, &queued);
if(msg) {
if(msg->msg == CURLMSG_DONE) {
printf("Completed: %dn", (int)msg->data.result);
}
}
} while(msg);
} /* loop while still is non-zero */
curl_multi_remove_handle(multi, easy);
curl_multi_remove_handle(multi, easy2);
curl_multi_cleanup(multi);
curl_easy_cleanup(easy2);
setup
tear down
transfer

caches
curl_easy_perform - caches in the easy handle
curl_multi_perform - caches in the multi handle, shared by all added easy
handles
@bagder
DNS
cache
connection
pool
TLS
session-ID
cache
CA store
cache
cookies
Alt-svc
data
HSTS data
per single handle even in multi handles

curl_multi_socket_action
The event-based API ﬂavor
Scales better when going above hundreds of transfers
Applications call libcurl and pass in the socket that has activity
And what speciﬁc activity it is (read, write etc)
libcurl tells application what sockets to monitor
libcurl tells application about when its timer expires
@bagder

Setting up
@bagder
slide 68 of 88
your app
your
transfers

Verbose
CURLOPT_ERRORBUFFER
CURLOPT_VERBOSE
@bagder
int main()
{
CURL *curl;
CURLcode res = CURLE_OK;
if(curl) {
char errbuf[CURL_ERROR_SIZE] = "";
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/");
if(res) {
if(errbuf[0])
fprintf(stderr, "Error: %s", errbuf);
else
fprintf(stderr, "Error: %sn", curl_easy_strerror(res));
}
}
return (int)res;
}
errorbuffer.c

Debug callback
CURLOPT_DEBUGFUNCTION
curl_global_trace
CURLINFO_XFER_ID
CURLINFO_CONN_ID
@bagder
struct data {
bool trace_ascii;
};
static int my_trace(CURL *handle, curl_infotype type,
char *data, size_t size,
void *userp)
{
struct data *config = userp;
}
int main()
{
CURL *curl;
CURLcode res;
struct data config;
config.trace_ascii = 1; /* enable ascii tracing */
if(curl) {
curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, my_trace);
curl_easy_setopt(curl, CURLOPT_DEBUGDATA, &config);
}
}
debug-callback.c

global trace conﬁg
@bagder
struct data {
bool trace_ascii;
};
void *userp)
{
}
int main()
{
CURL *curl;
CURLcode res;
struct data config;
curl_global_trace(“all”);
if(curl) {
}
}
global-trace.c

which transfer?
@bagder
struct data {
bool trace_ascii;
};
void *userp)
{
curl_off_t xfer_id, conn_id;
curl_easy_getinfo(handle, CURLINFO_XFER_ID, &xfer_id);
curl_easy_getinfo(handle, CURLINFO_CONN_ID, &conn_id);
}
int main()
{
CURL *curl;
CURLcode res;
struct data config;
curl_global_trace(“all”);
if(curl) {
}
}
xfer-id.c
CURLINFO_XFER_ID
CURLINFO_CONN_ID

curl_version
Figuring out libcurl version
… and what this libcurl supports
@bagder
#include <strings.h>
int main(int argc, char *argv[])
{
curl_version_info_data *v;
int i;
curl_global_init(CURL_GLOBAL_DEFAULT);
printf("curl_version: %sn", curl_version());
v = curl_version_info(CURLVERSION_NOW);
printf("version: %sn", v->version);
printf("version_num (%x) %d.%d.%dn", v->version_num,
v->version_num 16, (v->version_num 8) & 0 ff,
v->version_num & 0 ff);
printf("features: ");
for(i=0; v->feature_names[i]; i++) {
printf("%s, ", v->feature_names[i]);
}
printf("nprotocols: ");
for(i=0; v->feature_names[i]; i++) {
printf("%s, ", v->protocols[i]);
}
printf("n");
return 1;
}
version.c
$ gcc version.c -lcurl
$ ./a.out
curl_version: libcurl/8.4.0 OpenSSL/3.0.12 zlib/1.2.13 brotli/1.0.9
zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0
nghttp2/1.57.0 librtmp/2.3 OpenLDAP/2.5.13
v->version: 8.4.0
v->version_num (80400) 8.4.0
v->feature_names[]: alt-svc, AsynchDNS, brotli, GSS-API, HSTS, HTTP2,
HTTPS-proxy, IDN, IPv6, Kerberos, Largefile, libz, NTLM, PSL, SPNEGO,
SSL, threadsafe, TLS-SRP, UnixSockets, zstd,
v->protocols[]: dict, file, ftp, ftps, gopher, gophers, http, https,
imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp, rtmpe, rtmps, rtmpt,
rtmpte,
shell

symbols-in-versions
https://curl.se/libcurl/c/symbols-in-versions.html
@bagder

connecting
name resolve
happy eyeballs
IPv4 / IPv6
@bagder

persistent connections
@bagder
connection
pool
client
curl.se
example.com
1
2
3
1. HTTPS://example.com:443
2. HTTPS://curl.se:443
3

connection pool
connections stored if still alive
only stores up to N connections: CURLOPT_MAXCONNECTS
kept up to a maximum age: CURLOPT_MAXAGE_CONN
reuse is done per name, not IP address
works for all supported protocols
numerous extra conditions and requirements
opt-out reuse: CURLOPT_FRESH_CONNECT
make next transfer not reusable: CURLOPT_FORBID_REUSE
TCP keepalive: CURLOPT_TCP_KEEPALIVE
@bagder
connection
pool

Connections: name tricks
Pre-populate the DNS cache: CURLOPT_RESOLVE
@bagder
int main(void)
{
CURL *curl;
struct curl_slist *host;
host = curl_slist_append(NULL,
"example.com:443:127.0.0.1");
host = curl_slist_append(host,
" .example:443:127.0.0.1");
if(curl) {
curl_easy_setopt(curl, CURLOPT_RESOLVE, host);
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
}
curl_slist_free_all(host);
return (int)res;
}
resolve.c

Connections: name tricks
Host + port “redirect”: CURLOPT_CONNECT_TO
@bagder
connect-to.c
int main(void)
{
CURL *curl;
struct curl_slist *host;
host = curl_slist_append(NULL,
"curl.se:443:example.com:443");
if(curl) {
curl_easy_setopt(curl, CURLOPT_CONNECT_TO, host);
curl_easy_setopt(curl, CURLOPT_URL, "https://curl.se/");
/* unless we disable the check libcurl
returns CURLE_PEER_FAILED_VERIFICATION */
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
}
curl_slist_free_all(host);
return (int)res;
}

Connections: when you use c-ares
c-ares is a third party name resolver
CURLOPT_DNS_SERVERS
@bagder
dns-servers.c
int main(void)
{
CURL *curl;
if(curl) {
curl_easy_setopt(curl, CURLOPT_DNS_SERVERS,
“192.168.1.100:53,192.168.1.101”);
}
return (int)res;
}

Connections: network interface
Bind the socket to an interface or address: CURLOPT_INTERFACE
@bagder
interface.c
int main(void)
{
CURL *curl;
if(curl) {
curl_easy_setopt(curl, CURLOPT_INTERFACE, "enp3s0");
}
return (int)res;
}

Connections: local port number
Restrict local TCP port number: CURLOPT_LOCALPORT and
CURLOPT_LOCALPORTRANGE
@bagder
localport.c
int main(void)
{
CURL *curl;
if(curl) {
/* Try to use a local port number between 20000-20009 */
curl_easy_setopt(curl, CURLOPT_LOCALPORT, 20000L);
curl_easy_setopt(curl, CURLOPT_LOCALPORTRANGE, 10L);
}
return (int)res;
}

Connections: TCP KEEPALIVE
Prevent TCP connections from being
idle
By default, a connection can be
completely silent over long periods
or disconnected without it getting
noticed
@bagder
keepalive.c
int main(void)
{
CURL *curl;
if(curl) {
/* enable TCP keep-alive for this transfer */
curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
/* keep-alive idle time to 120 seconds */
curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 120L);
/* interval time between keep-alive probes: 60 seconds */
curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 60L);
}
return (int)res;
}

Connections: IP family
libcurl does IPv6 + IPv4 by default
Using happy eyeballs
You can change that
@bagder
ipv6only.c
int main(void)
{
CURL *curl;
if(curl) {
curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V6);
}
return (int)res;
}

Authentication
Most protocols offer authentication
Many more than one type
Set CURLOPT_USERNAME +
CURLOPT_PASSWORD
And optionally the type
For proxy, it has its own set of
credentials and auth type options:
CURLOPT_PROXYUSERNAME,
CURLOPT_PROXYPASSWORD etc
For HTTP, cookies is often used
@bagder
auth.c
int main(void)
{
CURL *curl;
if(curl) {
curl_easy_setopt(curl, CURLOPT_USERNAME, "clark");
curl_easy_setopt(curl, CURLOPT_PASSWORD, "kent");
}
return (int)res;
}

.netrc
old conﬁg ﬁle for ftp authentication
libcurl supports it for generic credentials per host
defaults to $HOME/.netrc
@bagder
netrc.c
int main(void)
{
CURL *curl;
if(curl) {
curl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
curl_easy_setopt(curl, CURLOPT_NETRC_FILE,
“/home/daniel/s3cr3ts.txt”);
}
return (int)res;
}

return codes
The easy interface returns CURLcode
The multi interface returns CURLMcode
The share interface returns CURLSHcode
The URL interface returns CURLUcode
0 is always success
non-zero is an error code
libcurl-errors.3 has all the details
@bagder

mastering libcurl part 1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to mastering libcurl part 1

Similar to mastering libcurl part 1 (20)

More from Daniel Stenberg

More from Daniel Stenberg (20)

Recently uploaded

Recently uploaded (20)

mastering libcurl part 1