SlideShare a Scribd company logo
May 7, 2020May 7, 2020
Common mistakes when
Common mistakes when
using libcurl - and how to fix them!
using libcurl - and how to fix them!
Daniel Stenberg
@bagder
Daniel Stenberg
@bagder
common libcurl mistakes
Documentation HTTP method CURLOPT_NOSIGNAL
Return codes Certificate checks -DCURL_STATICLIB
Verbose option Zero termination Set the URL
curl_global_init C++ strings callback invokes
Redirects Threading C++ methods
@bagder@bagder
Q&A in the end!Q&A in the end!
Why are these mistakes made?
Humans are lazy
Copy and pasted from questionable sources
Documentation is hard
Internet transfers are complicated
Maybe, just maybe, the curl way isn’t always the smartest...
@bagder@bagder
11
@bagder@bagder
Skipping the documentationSkipping the documentation
Lots of options have plain English names
Might trick you think you know what it does
Still might not work like you presume it does
Copy and paste from random web sites
There are also details
The devil is always in the details
@bagder@bagder
Lots of documentationLots of documentation
We offer man pages for every setopt option
We host over 100 stand-alone examples
Consider which docs you rely on (hello
stackoverflow.com)
@bagder@bagder
@bagder@bagder
22
Failure to check return codesFailure to check return codes
@bagder@bagder
Return codes areReturn codes are usefuluseful cluesclues
How to know if the call succeeded?
How to know why something doesn’t do what you expected?
What if the feature isn’t even built-in?
Our example source codes might be bad examples
@bagder@bagder
@bagder@bagder
33
Forgetting the verbose option
Strange, how come it doesn’t work?
Hm, why does it act like this?
Also:
/* please be verbose */
rc = curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
/* provide a buffer to store errors in */
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
@bagder@bagder
libcurl or content?
By using verbose, you’ll spot if this was libcurl that said it or if this
was actual content delivered from the server!
$ ./app
Error 505: HTTP Version Not Supported
Maybe even in production?
Consider it for debug options
Direct the output somewhere suitable with
CURLOPT_STDERR
Alternatively: CURLOPT_DEBUGFUNCTION
@bagder@bagder
44
@bagder@bagder
There's a global init function
It is called implicitly by curl_easy_perform() if not done
explicitly
Not calling it means relying on default, implicit behavior
It typically then implies not calling curl_global_cleanup()
This may result in not releasing all used memory (“Dear sirs,
why does valgrind report that...”)
@bagder@bagder
curl_global_init isn't thread-safe
curl_global_init needs to be called as a singleton
It is not thread-safe due to legacy and “reasons”
Will hopefully be rectified in a near future
@bagder@bagder
There's a global init function!
Call curl_global_init first
Alone!
Call curl_global_cleanup last
@bagder@bagder
55
@bagder@bagder
Consider the redirects!
HTTP/1.1 301 Moved Permanently
Server: M4gic server/3000
Retry-After: 0
Location: https://curl.haxx.se/
Content-Length: 0
Accept-Ranges: bytes
Date: Thu, 07 May 2020 08:59:56 GMT
Connection: close
@bagder@bagder
Consider the redirects!
Rethink if redirect-following is good
Limit what protocols to allow redirects
Do not set custom HTTP methods on requests that follow
redirects
@bagder@bagder
66
@bagder@bagder
Let users set (parts of) the URL
Scheme (maybe even use another protocol?)
Host name (maybe target a malicious server)
Extreme lengths (pass in 2GB of data?)
Also consider other inputs: user name, password etc risk
getting abused
@bagder@bagder
Limit scope!
Set CURLOPT_PROTOCOLS!
Whitelist/filter
Set only a limited part of the URL
@bagder@bagder
77
@bagder@bagder
Setting the HTTP method
CURLOPT_CUSTOMREQUEST is a footgun
will be used in follow-up requests as well in
redirects
Does not change libcurl's behavior
@bagder@bagder
88
@bagder@bagder
Disabled certificate checks
Widely abused and misunderstood
Only use while experimenting / developing
Never ship in production
This also goes for HTTPS proxies
SCP and SFTP is different
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
@bagder@bagder
Verify server certificates!
Avoid man-in-the-middle attacks
HTTPS is not secure without it!
May require regularly updating the CA store
Alternative: CURLOPT_PINNEDPUBLICKEY
@bagder@bagder
99
@bagder@bagder
Assume zero terminated data in callbacks
CURLOPT_WRITEFUNCTION and CURLOPT_HEADERFUNCTION set
callbacks
Libcurl provide data to the application using these callbacks
The data is provided as a pointer to the data and length of that data
When that data is primarily text oriented, many users wrongly assume
that this means the data comes as zero terminated “strings”.
size_t write_callback(char *dataptr, size_t size, size_t nmemb, void *userp);
@bagder@bagder
Typical mistake
size_t cb(char *dataptr, size_t size, size_t nmemb, void *userp)
{
printf(“Incoming data: %sn”, dataptr);
if(!strncmp(“Foo:”, dataptr, 4)) {
...
}
char *pos = strchr(dataptr, ‘n’);
}
@bagder@bagder
The callback data is binary
The data isn’t text or “string” based
printf(“%s”, ...), strcpy(), strlen() and similar will not work
on this pointer!
@bagder@bagder
1010
@bagder@bagder
C++ strings are not C strings
libcurl provides a C API
C and C++ are similar
C and C++ are also different!
C++ users like their std::string types
C++ Strings are not C strings
curl_easy_setopt() takes a vararg...
@bagder@bagder
C++ string bad code
// Keep the URL as a C++ string object
std::string url("https://example.com/");
// Pass it to curl
curl_easy_setopt(curl, CURLOPT_URL, url);
@bagder@bagder
C++ string good code
// Keep the URL as a C++ string object
std::string url("https://example.com/");
// Pass it to curl as a C string!
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
@bagder@bagder
1111
@bagder@bagder
Threading mistakes
libcurl is thread-safe but there are caveats:
1) No concurrent use of handles
2) OpenSSL < 1.1.0 need mutex callbacks setup
3) curl_global_init is not thread-safe
yet
@bagder@bagder
1212
@bagder@bagder
Understanding CURLOPT_NOSIGNAL
Signals is a unix-concept: “an asynchronous notification sent to a
process or to a specific thread within the same process in order to notify it of
an event that occurred”
Signals are complicated in a multi-threaded world and
when used by a library
@bagder@bagder
What does libcurl use signals for?
When using the synchronous name resolver, libcurl uses alarm()
to abort slow name resolves (if a timeout is set), which ultimately
sends a SIGALARM to the process and is caught by libcurl
libcurl installs its own sighandler while running, and restores the
original one again on return – for SIGALARM and SIGPIPE.
Closing TLS (with OpenSSL) can trigger a SIGPIPE if the connection
is dead.
Unless CURLOPT_NOSIGNAL is set!
@bagder@bagder
What does CURLOPT_NOSIGNAL do?
It stops libcurl from triggering signals
It prevents libcurl from installing its own sighandler
Generated signals must then be handled by the libcurl-
using application!
@bagder@bagder
1313
@bagder@bagder
Forgetting -DCURL_STATICLIB
Creating and using libcurl statically is easy and convenient
Seems especially popular on Windows
Requires the CURL_STATICLIB define to be set when building your
application!
Omission causes linker errors:
"unknown symbol __imp__curl_easy_init”
Because Windows need __declspec to be present or absent in the headers
depending on how it links!
@bagder@bagder
Static builds mean chasing deps
Libcurl can use many 3rd party dependencies
When linking statically, all those need to be provided to the linker
The curl build scripts (as well as your application linking) usually
need manual help to find them all
@bagder@bagder
1414
@bagder@bagder
@bagder@bagder
C++ methods
(Sibling to the C++ strings mistake)
C++ class methods look like functions
C++ class methods cannot be used as callbacks with
libcurl
… since they assume a ‘this’ pointer to the current object
Static member functions work!
@bagder@bagder
A C++ method that works
// f is the pointer to your object.
static size_t YourClass::func(void *buffer, size_t sz, size_t n, void *f)
{
// Call non-static member function.
static_cast<YourClass*>(f)->nonStaticFunction();
}
// This is how you pass pointer to the static function:
curl_easy_setopt(hcurl, CURLOPT_XFERINFOFUNCTION, YourClass::func);
curl_easy_setopt(hcurl, CURLOPT_XEFRINFODATA, this);
1515
@bagder@bagder
@bagder@bagder
Write callback invokes
Data is delivered by callback (CURLOPT_WRITEFUNCTION)
It might be called none, one, two or many times
Never assume you will get a certain amount of calls
Independently of the data amount
Because of network, server, kernel or other reasons
54
You can help!You can help!
@bagder@bagder
https://curl.haxx.se/book.html
@bagder@bagder
Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!Thank you!
Questions?Questions?
@bagder@bagder
License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder

More Related Content

What's hot

HTTP/3 for everyone
HTTP/3 for everyoneHTTP/3 for everyone
HTTP/3 for everyone
Daniel Stenberg
 
Curl with rust
Curl with rustCurl with rust
Curl with rust
Daniel Stenberg
 
curl - a hobby project that conquered the world
curl - a hobby project that conquered the worldcurl - a hobby project that conquered the world
curl - a hobby project that conquered the world
Daniel Stenberg
 
HTTP/3, QUIC and streaming
HTTP/3, QUIC and streamingHTTP/3, QUIC and streaming
HTTP/3, QUIC and streaming
Daniel Stenberg
 
Http3 fullstackfest-2019
Http3 fullstackfest-2019Http3 fullstackfest-2019
Http3 fullstackfest-2019
Daniel Stenberg
 
Just curl it!
Just curl it!Just curl it!
Just curl it!
Daniel Stenberg
 
HTTP/3
HTTP/3HTTP/3
DNS over HTTPS
DNS over HTTPSDNS over HTTPS
DNS over HTTPS
Daniel Stenberg
 
HTTP/3 an early overview
HTTP/3 an early overviewHTTP/3 an early overview
HTTP/3 an early overview
Praveen Kottarathil
 
HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!
Daniel Stenberg
 
gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!
Alex Borysov
 
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
Alex Borysov
 
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
Alex Borysov
 
gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!
Alex Borysov
 
"gRPC vs REST: let the battle begin!" OSCON 2018 edition
"gRPC vs REST: let the battle begin!" OSCON 2018 edition"gRPC vs REST: let the battle begin!" OSCON 2018 edition
"gRPC vs REST: let the battle begin!" OSCON 2018 edition
Alex Borysov
 
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchaginstackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
NETWAYS
 
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
Alex Borysov
 
Re-thinking Performance tuning with HTTP2
Re-thinking Performance tuning with HTTP2Re-thinking Performance tuning with HTTP2
Re-thinking Performance tuning with HTTP2
Vinci Rufus
 
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
Alex Borysov
 
Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...
Alex Borysov
 

What's hot (20)

HTTP/3 for everyone
HTTP/3 for everyoneHTTP/3 for everyone
HTTP/3 for everyone
 
Curl with rust
Curl with rustCurl with rust
Curl with rust
 
curl - a hobby project that conquered the world
curl - a hobby project that conquered the worldcurl - a hobby project that conquered the world
curl - a hobby project that conquered the world
 
HTTP/3, QUIC and streaming
HTTP/3, QUIC and streamingHTTP/3, QUIC and streaming
HTTP/3, QUIC and streaming
 
Http3 fullstackfest-2019
Http3 fullstackfest-2019Http3 fullstackfest-2019
Http3 fullstackfest-2019
 
Just curl it!
Just curl it!Just curl it!
Just curl it!
 
HTTP/3
HTTP/3HTTP/3
HTTP/3
 
DNS over HTTPS
DNS over HTTPSDNS over HTTPS
DNS over HTTPS
 
HTTP/3 an early overview
HTTP/3 an early overviewHTTP/3 an early overview
HTTP/3 an early overview
 
HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!HTTP/3 over QUIC. All is new but still the same!
HTTP/3 over QUIC. All is new but still the same!
 
gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!
 
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
"Enabling Googley microservices with gRPC" VoxxedDays Minsk edition
 
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
 
gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!
 
"gRPC vs REST: let the battle begin!" OSCON 2018 edition
"gRPC vs REST: let the battle begin!" OSCON 2018 edition"gRPC vs REST: let the battle begin!" OSCON 2018 edition
"gRPC vs REST: let the battle begin!" OSCON 2018 edition
 
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchaginstackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
 
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
 
Re-thinking Performance tuning with HTTP2
Re-thinking Performance tuning with HTTP2Re-thinking Performance tuning with HTTP2
Re-thinking Performance tuning with HTTP2
 
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
 
Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...
 

Similar to common mistakes when using libcurl

Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
Anne Nicolas
 
mastering the curl command line.pdf
mastering the curl command line.pdfmastering the curl command line.pdf
mastering the curl command line.pdf
DanielStenberg7
 
02 c++g3 d
02 c++g3 d02 c++g3 d
02 c++g3 d
mahago
 
mastering libcurl part 1
mastering libcurl part 1mastering libcurl part 1
mastering libcurl part 1
Daniel Stenberg
 
C++ boot camp part 1/2
C++ boot camp part 1/2C++ boot camp part 1/2
C++ boot camp part 1/2
Jesse Talavera-Greenberg
 
C++ Boot Camp Part 1
C++ Boot Camp Part 1C++ Boot Camp Part 1
C++ Boot Camp Part 1
Jesse Talavera-Greenberg
 
Secure Programming Practices in C++ (NDC Oslo 2018)
Secure Programming Practices in C++ (NDC Oslo 2018)Secure Programming Practices in C++ (NDC Oslo 2018)
Secure Programming Practices in C++ (NDC Oslo 2018)
Patricia Aas
 
The Naked Bundle - Tryout
The Naked Bundle - TryoutThe Naked Bundle - Tryout
The Naked Bundle - Tryout
Matthias Noback
 
Much ado about randomness. What is really a random number?
Much ado about randomness. What is really a random number?Much ado about randomness. What is really a random number?
Much ado about randomness. What is really a random number?
Aleksandr Yampolskiy
 
Good Coding Practices with JavaScript
Good Coding Practices with JavaScriptGood Coding Practices with JavaScript
Good Coding Practices with JavaScript
🏁 Pierre-Henry Soria 💡
 
Intermediate python
Intermediate pythonIntermediate python
Intermediate python
NaphtaliOchonogor1
 
Audit
AuditAudit
Strategy and best practice for modern RPG
Strategy and best practice for modern RPGStrategy and best practice for modern RPG
Strategy and best practice for modern RPG
Alemanalfredo
 
C++ Core Guidelines
C++ Core GuidelinesC++ Core Guidelines
C++ Core Guidelines
Thomas Pollak
 
mastering libcurl part 2
mastering libcurl part 2mastering libcurl part 2
mastering libcurl part 2
Daniel Stenberg
 
Writing Readable Code
Writing Readable CodeWriting Readable Code
Writing Readable Code
eddiehaber
 
designpatterns_blair_upe.ppt
designpatterns_blair_upe.pptdesignpatterns_blair_upe.ppt
designpatterns_blair_upe.ppt
banti43
 
Drupal 101: Tips and Tricks for Troubleshooting Drupal
Drupal 101: Tips and Tricks for Troubleshooting DrupalDrupal 101: Tips and Tricks for Troubleshooting Drupal
Drupal 101: Tips and Tricks for Troubleshooting Drupal
Acquia
 
Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]
RootedCON
 
02 c++g3 d (1)
02 c++g3 d (1)02 c++g3 d (1)
02 c++g3 d (1)
Mohammed Ali
 

Similar to common mistakes when using libcurl (20)

Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
Kernel Recipes 2014 - Writing Code: Keep It Short, Stupid!
 
mastering the curl command line.pdf
mastering the curl command line.pdfmastering the curl command line.pdf
mastering the curl command line.pdf
 
02 c++g3 d
02 c++g3 d02 c++g3 d
02 c++g3 d
 
mastering libcurl part 1
mastering libcurl part 1mastering libcurl part 1
mastering libcurl part 1
 
C++ boot camp part 1/2
C++ boot camp part 1/2C++ boot camp part 1/2
C++ boot camp part 1/2
 
C++ Boot Camp Part 1
C++ Boot Camp Part 1C++ Boot Camp Part 1
C++ Boot Camp Part 1
 
Secure Programming Practices in C++ (NDC Oslo 2018)
Secure Programming Practices in C++ (NDC Oslo 2018)Secure Programming Practices in C++ (NDC Oslo 2018)
Secure Programming Practices in C++ (NDC Oslo 2018)
 
The Naked Bundle - Tryout
The Naked Bundle - TryoutThe Naked Bundle - Tryout
The Naked Bundle - Tryout
 
Much ado about randomness. What is really a random number?
Much ado about randomness. What is really a random number?Much ado about randomness. What is really a random number?
Much ado about randomness. What is really a random number?
 
Good Coding Practices with JavaScript
Good Coding Practices with JavaScriptGood Coding Practices with JavaScript
Good Coding Practices with JavaScript
 
Intermediate python
Intermediate pythonIntermediate python
Intermediate python
 
Audit
AuditAudit
Audit
 
Strategy and best practice for modern RPG
Strategy and best practice for modern RPGStrategy and best practice for modern RPG
Strategy and best practice for modern RPG
 
C++ Core Guidelines
C++ Core GuidelinesC++ Core Guidelines
C++ Core Guidelines
 
mastering libcurl part 2
mastering libcurl part 2mastering libcurl part 2
mastering libcurl part 2
 
Writing Readable Code
Writing Readable CodeWriting Readable Code
Writing Readable Code
 
designpatterns_blair_upe.ppt
designpatterns_blair_upe.pptdesignpatterns_blair_upe.ppt
designpatterns_blair_upe.ppt
 
Drupal 101: Tips and Tricks for Troubleshooting Drupal
Drupal 101: Tips and Tricks for Troubleshooting DrupalDrupal 101: Tips and Tricks for Troubleshooting Drupal
Drupal 101: Tips and Tricks for Troubleshooting Drupal
 
Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]
 
02 c++g3 d (1)
02 c++g3 d (1)02 c++g3 d (1)
02 c++g3 d (1)
 

More from Daniel Stenberg

curl security by Daniel Stenberg from curl up 2024
curl security by Daniel Stenberg from curl up 2024curl security by Daniel Stenberg from curl up 2024
curl security by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
rust in curl by Daniel Stenberg from- curl up 2024
rust in curl by Daniel Stenberg from- curl up 2024rust in curl by Daniel Stenberg from- curl up 2024
rust in curl by Daniel Stenberg from- curl up 2024
Daniel Stenberg
 
trurl 2024 by Daniel Stenberg from curl up 2024
trurl 2024 by Daniel Stenberg from curl up 2024trurl 2024 by Daniel Stenberg from curl up 2024
trurl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
curl future 2024 by Daniel Stenberg from curl up 2024
curl future 2024 by Daniel Stenberg from curl up 2024curl future 2024 by Daniel Stenberg from curl up 2024
curl future 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
The state of curl 2024 by Daniel Stenberg from curl up 2024
The state of curl 2024 by Daniel Stenberg from curl up 2024The state of curl 2024 by Daniel Stenberg from curl up 2024
The state of curl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
curl - openfourm europe.pdf
curl - openfourm europe.pdfcurl - openfourm europe.pdf
curl - openfourm europe.pdf
Daniel Stenberg
 
curl experiments - curl up 2022
curl experiments - curl up 2022curl experiments - curl up 2022
curl experiments - curl up 2022
Daniel Stenberg
 
curl security - curl up 2022
curl security - curl up 2022curl security - curl up 2022
curl security - curl up 2022
Daniel Stenberg
 
HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022
Daniel Stenberg
 
The state of curl 2022
The state of curl 2022The state of curl 2022
The state of curl 2022
Daniel Stenberg
 
Let me tell you about curl
Let me tell you about curlLet me tell you about curl
Let me tell you about curl
Daniel Stenberg
 
The state of curl 2020
The state of curl 2020The state of curl 2020
The state of curl 2020
Daniel Stenberg
 
Writing safe and secure code
Writing safe and secure codeWriting safe and secure code
Writing safe and secure code
Daniel Stenberg
 

More from Daniel Stenberg (13)

curl security by Daniel Stenberg from curl up 2024
curl security by Daniel Stenberg from curl up 2024curl security by Daniel Stenberg from curl up 2024
curl security by Daniel Stenberg from curl up 2024
 
rust in curl by Daniel Stenberg from- curl up 2024
rust in curl by Daniel Stenberg from- curl up 2024rust in curl by Daniel Stenberg from- curl up 2024
rust in curl by Daniel Stenberg from- curl up 2024
 
trurl 2024 by Daniel Stenberg from curl up 2024
trurl 2024 by Daniel Stenberg from curl up 2024trurl 2024 by Daniel Stenberg from curl up 2024
trurl 2024 by Daniel Stenberg from curl up 2024
 
curl future 2024 by Daniel Stenberg from curl up 2024
curl future 2024 by Daniel Stenberg from curl up 2024curl future 2024 by Daniel Stenberg from curl up 2024
curl future 2024 by Daniel Stenberg from curl up 2024
 
The state of curl 2024 by Daniel Stenberg from curl up 2024
The state of curl 2024 by Daniel Stenberg from curl up 2024The state of curl 2024 by Daniel Stenberg from curl up 2024
The state of curl 2024 by Daniel Stenberg from curl up 2024
 
curl - openfourm europe.pdf
curl - openfourm europe.pdfcurl - openfourm europe.pdf
curl - openfourm europe.pdf
 
curl experiments - curl up 2022
curl experiments - curl up 2022curl experiments - curl up 2022
curl experiments - curl up 2022
 
curl security - curl up 2022
curl security - curl up 2022curl security - curl up 2022
curl security - curl up 2022
 
HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022HTTP/3 in curl - curl up 2022
HTTP/3 in curl - curl up 2022
 
The state of curl 2022
The state of curl 2022The state of curl 2022
The state of curl 2022
 
Let me tell you about curl
Let me tell you about curlLet me tell you about curl
Let me tell you about curl
 
The state of curl 2020
The state of curl 2020The state of curl 2020
The state of curl 2020
 
Writing safe and secure code
Writing safe and secure codeWriting safe and secure code
Writing safe and secure code
 

Recently uploaded

Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
ThousandEyes
 
Splunk_Remote_Work_Insights_Overview.pptx
Splunk_Remote_Work_Insights_Overview.pptxSplunk_Remote_Work_Insights_Overview.pptx
Splunk_Remote_Work_Insights_Overview.pptx
sudsdeep
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
Task Tracker
 
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptxWired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
SimonedeGijt
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
Break data silos with real-time connectivity using Confluent Cloud ConnectorsBreak data silos with real-time connectivity using Confluent Cloud Connectors
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
WEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service ProvidersWEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service Providers
Severalnines
 
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTIONBITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
ssuser2b426d1
 
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Softwares
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
Srinivas Dukka
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
908dutch
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
Philip Schwarz
 
Google ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learningGoogle ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learning
VishrutGoyani1
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
aslasdfmkhan4750
 
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
Hironori Washizaki
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
miso_uam
 
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docxComprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Aardwolf Security
 
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
neshakor5152
 
Safe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work PermitsSafe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work Permits
sheqnetworkmarketing
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdfAWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
karim wahed
 

Recently uploaded (20)

Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
 
Splunk_Remote_Work_Insights_Overview.pptx
Splunk_Remote_Work_Insights_Overview.pptxSplunk_Remote_Work_Insights_Overview.pptx
Splunk_Remote_Work_Insights_Overview.pptx
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
 
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptxWired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
Break data silos with real-time connectivity using Confluent Cloud ConnectorsBreak data silos with real-time connectivity using Confluent Cloud Connectors
Break data silos with real-time connectivity using Confluent Cloud Connectors
 
WEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service ProvidersWEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service Providers
 
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTIONBITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
 
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial Company
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
 
Google ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learningGoogle ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learning
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
 
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
 
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docxComprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
 
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
 
Safe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work PermitsSafe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work Permits
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdfAWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) .pdf
 

common mistakes when using libcurl

  • 1. May 7, 2020May 7, 2020 Common mistakes when Common mistakes when using libcurl - and how to fix them! using libcurl - and how to fix them!
  • 4. common libcurl mistakes Documentation HTTP method CURLOPT_NOSIGNAL Return codes Certificate checks -DCURL_STATICLIB Verbose option Zero termination Set the URL curl_global_init C++ strings callback invokes Redirects Threading C++ methods @bagder@bagder
  • 5. Q&A in the end!Q&A in the end!
  • 6. Why are these mistakes made? Humans are lazy Copy and pasted from questionable sources Documentation is hard Internet transfers are complicated Maybe, just maybe, the curl way isn’t always the smartest... @bagder@bagder
  • 8. Skipping the documentationSkipping the documentation Lots of options have plain English names Might trick you think you know what it does Still might not work like you presume it does Copy and paste from random web sites There are also details The devil is always in the details @bagder@bagder
  • 9. Lots of documentationLots of documentation We offer man pages for every setopt option We host over 100 stand-alone examples Consider which docs you rely on (hello stackoverflow.com) @bagder@bagder
  • 11. Failure to check return codesFailure to check return codes @bagder@bagder
  • 12. Return codes areReturn codes are usefuluseful cluesclues How to know if the call succeeded? How to know why something doesn’t do what you expected? What if the feature isn’t even built-in? Our example source codes might be bad examples @bagder@bagder
  • 14. Forgetting the verbose option Strange, how come it doesn’t work? Hm, why does it act like this? Also: /* please be verbose */ rc = curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L); /* provide a buffer to store errors in */ curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf); @bagder@bagder
  • 15. libcurl or content? By using verbose, you’ll spot if this was libcurl that said it or if this was actual content delivered from the server! $ ./app Error 505: HTTP Version Not Supported
  • 16. Maybe even in production? Consider it for debug options Direct the output somewhere suitable with CURLOPT_STDERR Alternatively: CURLOPT_DEBUGFUNCTION @bagder@bagder
  • 18. There's a global init function It is called implicitly by curl_easy_perform() if not done explicitly Not calling it means relying on default, implicit behavior It typically then implies not calling curl_global_cleanup() This may result in not releasing all used memory (“Dear sirs, why does valgrind report that...”) @bagder@bagder
  • 19. curl_global_init isn't thread-safe curl_global_init needs to be called as a singleton It is not thread-safe due to legacy and “reasons” Will hopefully be rectified in a near future @bagder@bagder
  • 20. There's a global init function! Call curl_global_init first Alone! Call curl_global_cleanup last @bagder@bagder
  • 22. Consider the redirects! HTTP/1.1 301 Moved Permanently Server: M4gic server/3000 Retry-After: 0 Location: https://curl.haxx.se/ Content-Length: 0 Accept-Ranges: bytes Date: Thu, 07 May 2020 08:59:56 GMT Connection: close @bagder@bagder
  • 23. Consider the redirects! Rethink if redirect-following is good Limit what protocols to allow redirects Do not set custom HTTP methods on requests that follow redirects @bagder@bagder
  • 25. Let users set (parts of) the URL Scheme (maybe even use another protocol?) Host name (maybe target a malicious server) Extreme lengths (pass in 2GB of data?) Also consider other inputs: user name, password etc risk getting abused @bagder@bagder
  • 26. Limit scope! Set CURLOPT_PROTOCOLS! Whitelist/filter Set only a limited part of the URL @bagder@bagder
  • 28. Setting the HTTP method CURLOPT_CUSTOMREQUEST is a footgun will be used in follow-up requests as well in redirects Does not change libcurl's behavior @bagder@bagder
  • 30. Disabled certificate checks Widely abused and misunderstood Only use while experimenting / developing Never ship in production This also goes for HTTPS proxies SCP and SFTP is different curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); @bagder@bagder
  • 31. Verify server certificates! Avoid man-in-the-middle attacks HTTPS is not secure without it! May require regularly updating the CA store Alternative: CURLOPT_PINNEDPUBLICKEY @bagder@bagder
  • 33. Assume zero terminated data in callbacks CURLOPT_WRITEFUNCTION and CURLOPT_HEADERFUNCTION set callbacks Libcurl provide data to the application using these callbacks The data is provided as a pointer to the data and length of that data When that data is primarily text oriented, many users wrongly assume that this means the data comes as zero terminated “strings”. size_t write_callback(char *dataptr, size_t size, size_t nmemb, void *userp); @bagder@bagder
  • 34. Typical mistake size_t cb(char *dataptr, size_t size, size_t nmemb, void *userp) { printf(“Incoming data: %sn”, dataptr); if(!strncmp(“Foo:”, dataptr, 4)) { ... } char *pos = strchr(dataptr, ‘n’); } @bagder@bagder
  • 35. The callback data is binary The data isn’t text or “string” based printf(“%s”, ...), strcpy(), strlen() and similar will not work on this pointer! @bagder@bagder
  • 37. C++ strings are not C strings libcurl provides a C API C and C++ are similar C and C++ are also different! C++ users like their std::string types C++ Strings are not C strings curl_easy_setopt() takes a vararg... @bagder@bagder
  • 38. C++ string bad code // Keep the URL as a C++ string object std::string url("https://example.com/"); // Pass it to curl curl_easy_setopt(curl, CURLOPT_URL, url); @bagder@bagder
  • 39. C++ string good code // Keep the URL as a C++ string object std::string url("https://example.com/"); // Pass it to curl as a C string! curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); @bagder@bagder
  • 41. Threading mistakes libcurl is thread-safe but there are caveats: 1) No concurrent use of handles 2) OpenSSL < 1.1.0 need mutex callbacks setup 3) curl_global_init is not thread-safe yet @bagder@bagder
  • 43. Understanding CURLOPT_NOSIGNAL Signals is a unix-concept: “an asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred” Signals are complicated in a multi-threaded world and when used by a library @bagder@bagder
  • 44. What does libcurl use signals for? When using the synchronous name resolver, libcurl uses alarm() to abort slow name resolves (if a timeout is set), which ultimately sends a SIGALARM to the process and is caught by libcurl libcurl installs its own sighandler while running, and restores the original one again on return – for SIGALARM and SIGPIPE. Closing TLS (with OpenSSL) can trigger a SIGPIPE if the connection is dead. Unless CURLOPT_NOSIGNAL is set! @bagder@bagder
  • 45. What does CURLOPT_NOSIGNAL do? It stops libcurl from triggering signals It prevents libcurl from installing its own sighandler Generated signals must then be handled by the libcurl- using application! @bagder@bagder
  • 47. Forgetting -DCURL_STATICLIB Creating and using libcurl statically is easy and convenient Seems especially popular on Windows Requires the CURL_STATICLIB define to be set when building your application! Omission causes linker errors: "unknown symbol __imp__curl_easy_init” Because Windows need __declspec to be present or absent in the headers depending on how it links! @bagder@bagder
  • 48. Static builds mean chasing deps Libcurl can use many 3rd party dependencies When linking statically, all those need to be provided to the linker The curl build scripts (as well as your application linking) usually need manual help to find them all @bagder@bagder
  • 50. @bagder@bagder C++ methods (Sibling to the C++ strings mistake) C++ class methods look like functions C++ class methods cannot be used as callbacks with libcurl … since they assume a ‘this’ pointer to the current object Static member functions work!
  • 51. @bagder@bagder A C++ method that works // f is the pointer to your object. static size_t YourClass::func(void *buffer, size_t sz, size_t n, void *f) { // Call non-static member function. static_cast<YourClass*>(f)->nonStaticFunction(); } // This is how you pass pointer to the static function: curl_easy_setopt(hcurl, CURLOPT_XFERINFOFUNCTION, YourClass::func); curl_easy_setopt(hcurl, CURLOPT_XEFRINFODATA, this);
  • 53. @bagder@bagder Write callback invokes Data is delivered by callback (CURLOPT_WRITEFUNCTION) It might be called none, one, two or many times Never assume you will get a certain amount of calls Independently of the data amount Because of network, server, kernel or other reasons
  • 54. 54 You can help!You can help! @bagder@bagder
  • 56. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
  • 57. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder