Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Protecting MySQL Network traffic

645 views

Published on

Using SSL/TLS to protect network traffic to and from a MySQL database

Published in: Software
  • Be the first to comment

Protecting MySQL Network traffic

  1. 1. Protecting MySQL network traffic Daniël van Eeden | 25 April 2017
  2. 2. Booking.com at a glance ● Started in 1996; still based in Amsterdam ● Member of the Priceline Group since 2005 (stock: PCLN) ● Amazing growth; continuous scaling challenges ● Online Hotel/Accommodation/Travel Agent (OTA): ● Over 1.2 million active properties in 227 countries ● Over 1.2 million room nights reserved daily ● 40+ languages (website and customer service) ● Over 13,000 people working in 187 offices in 70 countries ● We use a lot of MySQL and MariaDB: ● Thousands (1000s) of servers, ~90% replicating ● >150 masters: ~30 >50 slaves & ~10 >100 slaves 2
  3. 3. Why protect MySQL network traffic? ● Protect leaking of authentication data (passwords, etc) ● Protect leaking of sensitive data (PII, credit card numbers, medical records) ● Ensure data is not tampered with. ● Because of regulations ● Because why not? Are you still using telnet to manage servers?
  4. 4. How? ● Use SSL! ● Done!
  5. 5. SSL Support in MySQL ● MySQL doesn't have SSL support ● MySQL never had any SSL support ● MySQL has TLS support.. this is what is called SSL but isn't ● Supported since 4.0.0 (~2003) ● For now just assume SSL and TLS are the same
  6. 6. What is NOT protected by TLS ● Data-at-rest ○ InnoDB and MyISAM data files ○ Binlogs, redo logs, slow query logs ○ Backups ● Does not protect against a DoS ○ e.g. corrupting traffic ● Might not protect the query text ○ performance_schema etc. ● Does not hide the traffic pattern
  7. 7. First steps with TLS 1. Get a certificate 2. Restart MySQL 3. Enable TLS on the client 4. Check if the connection actually uses TLS
  8. 8. Generating the certificate ● With 5.7 and up: Might already be done by your installation ● If not use mysql_ssl_rsa_setup ● For older versions: https://github.com/dveeden/mysslgen ● Or use the openssl commandline utilities as described in the reference manual on https://dev.mysql.com/doc/mysql/en/creating-ssl-files-using-openssl.html ● Did you know MySQL Workbench has a SSL Wizard?
  9. 9. Configuration ● On 5.7+: Place the ca.pem, server-cert.pem and server-key.pem in your datadir. (already the case if you use mysql_ssl_rsa_setup) ● Or set ssl-ca, ssl-key, ssl-cert in your my.cnf ● Restart MySQL ● Enable SSL in your application. You probably want to copy your ca.pem file to your client
  10. 10. Checking your connection ● 'status' or s ● Look for 'Cipher in use' ● Or check the 'Ssl_cipher' session status.
  11. 11. What if it doesn't work? ● Check your mysqld.log ● Check the permissions on the pem files ○ Should be readable for the mysql user ● Try to connect with --ssl-mode=REQUIRED ● Use the OpenSSL commandline tools to see what's in the certificate. ● Use Wireshark. ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1) ERROR 2026 (HY000): SSL connection error: unknown error number ERROR 2026 (HY000): SSL connection error: SSL certificate validation failure ERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failed ERROR 2026 (HY000): SSL connection error: protocol version mismatch ERROR 2026 (HY000): --ssl-mode=REQUIRED option forbids non SSL connections ERROR 2026 (HY000): SSL connection error: Failed to set ciphers to use ERROR 2026 (HY000): SSL connection error: Unable to get certificate ERROR 2026 (HY000): SSL connection error: Unable to get private key
  12. 12. Now let's make it more secure ● Require the use of TLS on the server ● Require the use of TLS on the client ● Enable more security checks ● Security updates
  13. 13. Make TLS a requirement ● On a per user basis: ○ ALTER USER foo REQUIRE SSL ○ Undo: ALTER USER foo REQUIRE NONE ● But what happens if you accidentally create a user? ○ e.g. GRANT on a nonexistent user? ○ Set: sql_mode=NO_AUTO_CREATE_USER,… ● On a server level: ○ SET GLOBAL require_secure_transport=ON ○ This still allows UNIX socket connections w/o TLS
  14. 14. Issues with full-on TLS ● Is your monitoring capable of using TLS connections? ● What about load balancer health checks?
  15. 15. On the client ● Use --ssl-mode=REQUIRED ○ The default in 5.7 is PREFERRED ○ Older releases default to DISABLED ● This only makes a TLS connection a requirement ○ Does not check if issued by a trusted CA ○ Does not check if the hostname matches the cert ○ To do this use VERIFY_CA or VERIFY_IDENTITY ● On older versions: ○ Use --ssl-ca to allow TLS and enable CA checks ○ Use --ssl-verify-server-cert to do hostname checks. ○ Often not possible to force the use of TLS: this is the BACKRONYM vulnerability ● Use --ssl-ca=/path/to/ca.pem to specify which CA(s) are trusted.
  16. 16. Client checks ● The client could do these checks: ○ Is the certificate signed by a trusted CA? ○ Does the CommonName (CN) in the certificate match the hostname we are connecting to? ○ Is the certificate expired?
  17. 17. Certificate Authority validation ● Validates that the server certificate is signed by one of the CA's present in the specified CA file. ● Note that a CA file can have multiple CA's ● There is also a CA path option. ● The auto generated certificates from mysql_ssl_rsa_setup all have their own CA.
  18. 18. Hostname validation ● mysql_ssl_rsa_setup generates certificates with ○ CN=MySQL_Server_5.7.18_Auto_Generated_Server_Certificate ● So generate the certificates manually if want this to match your hostname ● A certificate can have a list of hostnames in SubjectAltName ○ MySQL doesn't check those... Bug #68052 ● So if you use a virtual-IP, cname, etc. it might be difficult to match this. ● What if your clients connect on a CNAME and your replicas connect on the hostname? You can't have both!
  19. 19. Security updates ● I reported a few issues to Oracle. ● CVE-2017-3590 for Connector/Python ● CVE-2017-3469 for MySQL Workbench ● CVE-2017-3467 for libmysqlclient ● Those are fixed. See the Oracle Critical Patch Update for details. ● But if you care about security you should follow the release notes and Critical Patch Update anyways...
  20. 20. What library does MySQL use? ● Community Edition: YaSSL ○ Because GPL and the OpenSSL license are not really compatible ○ This library is maintained by WolfSSL ○ This not CyaSSL/WolfSSL ○ WolfSSL made a patch to include WolfSSL in MySQL 5.6.30 (https://github.com/wolfSSL/mysql-patch) ● Enterprise Edition: OpenSSL ● If you build MySQL yourself: you can compile against either of them.
  21. 21. Why not TLS? ● Because it is SLOW! ● Because we trust our network! ● Because we encrypt with: ○ The application (store encrypted data) ○ SSH (Also works great with Workbench) ○ VPN ● Because we want to inspect our network traffic! ○ Wireshark can decrypt it if you hand over your private key. Some ciphers require you to somehow extract session keys.
  22. 22. How slow is slow? ● Overhead in milliseconds for setting up a TLS connection on localhost with TCP. ● Client: go ● 5.7 is faster than 5.6 ● OpenSSL is faster than YaSSL ● Using TLS tickets (OpenSSL only) helps ● Best case: 0.99ms (5.7 OpenSSL w/ tickets) vs. 0.60ms (no TLS) ● TLS does need more roundtrips, but this will change with TLS 1.3 ● OpenSSL performs better because it uses AVX2 and AES-NI
  23. 23. Bulk transfer performance ● Easy to test: mysqldump with and without TLS ● Different ciphers do make a difference. mysqldump performance with MySQL 5.6.35 (YaSSL) No TLS 4.5s TLS Default 10.4s RC4-MD5 7.1s DES-CBC3-SHA 23.2s
  24. 24. Monitoring ● Monitor the Expiry of certificates ● Not just the certificate on disk, also the one in memory. ● Use TLS for your monitoring on 5.6 and earlier, otherwise you might not see the status vars ● Performance schema can show you the ciphers and TLS versions in use by all connections ● Using SYS is even easier: ○ SELECT * FROM session_ssl_status
  25. 25. Client certificates ● This allows mutual authentication ● Often used together with a password ● You might want to use REQUIRE SUBJECT or REQUIRE ISSUER on accounts. ● At least use REQUIRE X509 instead of REQUIRE SSL
  26. 26. Replication ● Use CHANGE MASTER TO MASTER_SSL=1, etc ● Think about what happens if your certificate(s) expire ● Does the hostname match the certificate?
  27. 27. Changing certificates ● Needs restart ● Moving slaves around might not work until you restart.. ● Same for a switchover.
  28. 28. CRL and OCSP ● Only possible with OpenSSL ● Does not auto download the CRL from the distribution point ● Does not use OCSP ● Basically restart MySQL every time your CRL changes.. which is not practical
  29. 29. Where to get your certificate? ● Official CA? ● Internal CA? ● Self signed?
  30. 30. TLS handshake with MySQL ● server helo with ssl flag set ● 'empty' login packet with ssl flag set ● Start SSL handshake ● Basically STARTTLS-ish ○ SSL and non-SSL on the same port
  31. 31. Protection of authentication data ● native password with nonce ● sha256 password with RSA keys or TLS ● cleartext plugin
  32. 32. TLS ciphers ● Possible to set restriction on Server and Client ● How are you going manage and maintain that? ● 'REQUIRE cipher' also requires client certificates ● One practical use case would be to use a faster cipher for mysqldump ● Might help with compliance ● 5.7.10 already places more strict requirements on the list of ciphers
  33. 33. TLS versions ● Can be limited on the server and client ● Note that YaSSL only has TLS 1.0 and TLS 1.1 support ● Minimum is TLS 1.0
  34. 34. What about MariaDB? ● Doesn't use --ssl-mode ● Does have good TLS support ● MariaDB Connector/C has support for ○ fingerprint verification ○ password protected private keys ● 19 Open MDEV's tagged with SSL
  35. 35. Connector support ● Works for C, C++, Python (multiple), Perl, Java, ODBC, Go, etc ● The Go MySQL driver lets you specify a TLS Config, which is really flexible. ● Do update your Connector.. Many connectors did have security updates related to TLS.
  36. 36. Don't forget these ● MySQL Cluster (NDB) communication within the cluster ● Galera communication ● Sending backups to a central location (xbstream etc) ● Network traffic for iSCSI, FCP, NFS
  37. 37. Future ● TLSv1.3 with 0-RTT ● WolfSSL?
  38. 38. Oh, and Booking.com is hiring! ● Almost any role: ● MySQL Engineer / DBA ● System Administrator ● System Engineer ● Site Reliability Engineer ● Developer ● Designer ● Technical Team Lead ● Product Owner ● Data Scientist ● And many more… ● https://workingatbooking.com/39
  39. 39. Thank you! All references to “Booking.com", including any mention of “us”, “we” and “our” refer to Booking.com BV, the company behind Booking.com™

×