SlideShare a Scribd company logo
MySQL Security and
Standardization at PayPal
Stacy Yuan & Yashada Jadhav • 29th May 2019
Stacy Yuan
© 2019 PayPal Inc. Confidential and proprietary.
Over 15 years of experience on various flavors
of relational databases.
Focus on performance tuning, code reviews,
database deployment and infrastructure
management for MySQL
In her spare time, she enjoys reading books
and doing some volunteer work.
Yashada Jadhav
© 2019 PayPal Inc. Confidential and proprietary.
7+ years of being in a relationship with
databases
Presented previously at Percona Live and
Oracle Open World
Prior to PayPal, worked at Amazon Web
Services and Yahoo!
I share random database thoughts at -
https://mysql.dbgeekgirl.com
Agenda
© 2019 PayPal Inc. Confidential and proprietary.
MySQL Standardization at PayPal
MySQL Security at PayPal
• Account Management
• Encrypted connections
• Migration to Percona MySQL at PayPal
At PayPal, we put people at the
center of everything we do.
MySQL at PayPal
© 2019 PayPal Inc. Confidential and proprietary.
Internal Apps 3rd Party
OSS Backend databasesBuild, Test and Release Tools
Custom Monitoring
DBaaS
Database as a ServiceSite Facing Use Cases
Site
Replica
Scaling
Architecture
© 2019 PayPal Inc. Confidential and proprietary.
Application
Connection Cache
Master
AZ1
Local Read
Replica
Application
Connection Cache
Remote Read
Replica
AZ2
Replica Scaling
MySQL Standardization at PayPal
Challenges, Journey and What's Next
MySQL Standardization at PayPal
One DBA Team – Multiple Applications
Heterogenous Set ups
Migrating ”non-standard” setups to standard
© 2018 PayPal Inc. Confidential and proprietary.
Hardware
Bare
Metal
Servers
VMs
Cloud
VMs
Operating System
Ubuntu
14.x, 16.x,
18.x
RHEL
6.x, 7.x
OEL 7.x
MySQL Versions
MySQL
Community
Edition 5.6, 5.7
Percona MySQL
5.7
Challenges
MySQL Standardization at PayPal
© 2018 PayPal Inc. Confidential and proprietary.
Journey
• Ansible for more scalable deployment
• Deployment scripts for different OS versionsDeployment
• Common standards across databases
• Recommended MySQL parameter settings
• MySQL design standards
Best Practices
• In house monitoring and alerting
• TICK and PMM for OS and MySQL metricsMonitoring
• GitHub’s Orchestrator and VIPHigh Availability
• Performance Schema, PMM and Slow query logPerformance Tuning
• Percona Xtrabackup and mysqldump with onsite and offsite storageBackups
MySQL Standardization at PayPal
Change Automation
Connection Multiplexer & Router
Benchmarking MySQL 8.0
InnoDB Encryption at Rest
Binary Log and Relay Log Encryption at Rest
Dual Password Support & Password Rotation
Role Based Access Control
© 2018 PayPal Inc. Confidential and proprietary.
What’s next
MySQL Security at PayPal
First things first
Run MySQL on its own dedicated server
Keep the DB server as clean as possible
Limit SSH access to the DB server
Limit sudo access
Use specific hostnames while creating users instead of wildcards
OS and DB upgrades as necessary
© 2019 PayPal Inc. Confidential and proprietary.
Password Strength
Passwords can be your weakpoint.
MySQL’s Password Validation Plugin
• Activate the plugin in my.cnf file and restart MySQL server
plugin-load-add = validate_password.so
validate-password = FORCE_PLUS_PERMANENT
• OR without restart
mysql> install plugin validate_password soname 'validate_password.so';
Query OK, 0 rows affected (0.03 sec)
• Why validate-password setting?
mysql> uninstall plugin validate_password;
ERROR 1702 (HY000): Plugin 'validate_password' is force_plus_permanent and can
not be unloaded
© 2019 PayPal Inc. Confidential and proprietary.
Password Strength
mysql> show global variables like
'validate_password%';
+--------------------------------------+--------+
| Variable_name | Value |
+--------------------------------------+--------+
| validate_password_check_user_name | OFF |
| validate_password_dictionary_file | |
| validate_password_length | 8 |
| validate_password_mixed_case_count | 1 |
| validate_password_number_count | 1 |
| validate_password_policy | MEDIUM |
| validate_password_special_char_count | 1 |
+--------------------------------------+--------+
7 rows in set (0.00 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Plugin Variables
Policy Test Performed
0 or LOW Length
1 or MEDIUM Length; numeric,
lowercase/uppercase, and special
characters
2 or HIGH Length; numeric,
lowercase/uppercase, and
special characters; dictionary file
validate_password_policy
mysql> select password ("Abcdefg1");
ERROR 1819 (HY000): Your password does not satisfy
the current policy requirements
mysql> select password("Abcdef1@");
+-------------------------------------------+
| password("Abcdef1@") |
+-------------------------------------------+
| *035C6A33B68D295FDE7BAA22AB7DC5733E136BD8 |
+-------------------------------------------+
1 row in set, 1 warning (0.00 sec)
Password Expiration
Setting password to expire after N days
Setting password expiration on global level
SET GLOBAL default_password_lifetime = 180;
Setting password to never expire
SET GLOBAL default_password_lifetime = 0;
Setting password expiration for individual accounts
mysql> CREATE USER 'yashada'@'%' IDENTIFIED BY "Abcdef1@" PASSWORD EXPIRE INTERVAL 90 DAY;
Query OK, 0 rows affected (0.03 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Password Expiration
What happens when password expires?
• When password expires, the server restricts the client to a “sandbox” mode
mysql> SELECT 1;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this
statement.
mysql> ALTER USER USER() identified by 'Abcdef1@';
Query OK, 0 rows affected (0.04 sec)
mysql> SELECT 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.01 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Password Expiration
Forcing users to change password at login
mysql> CREATE USER 'yashada'@'%' IDENTIFIED BY "Abcdef1@”;
Query OK, 0 rows affected (0.03 sec)
mysql> alter user 'yashada'@'%' PASSWORD EXPIRE;
Query OK, 0 rows affected (0.03 sec)
• When user will try to login –
mysql> select 1;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this
statement.
© 2019 PayPal Inc. Confidential and proprietary.
Avoiding Password Exposure
mysql_config_editor
• mysql_config_editor enables you to store authentication credentials in a login path file named
.mylogin.cnf in the current user's home directory.
Installing mysql_config_editor
• To install mysql_config_editor, all you need is the MySQL client installed.
© 2019 PayPal Inc. Confidential and proprietary.
Avoiding Password Exposure
Set up log in paths using mysql_config_editor
~> mysql_config_editor set --login-path=monitor --user=monitor --password
Enter password:
Using login-path to log in to MySQL
• To use this path to log in to MySQL –
~> mysql --login-path=monitor
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 554
© 2019 PayPal Inc. Confidential and proprietary.
Avoiding Password Exposure
How mysql_config_editor works?
• mysql_config_editor creates an encrypted file in the user’s home director called .mylogin.cnf
~> pwd
/root
~> ls -lhtr .mylogin.cnf
-rw-------. 1 root root 324 Apr 13 16:03 .mylogin.cnf
~> cat .mylogin.cnf
t +~ !2 Z
Yf 8<A + Q { ; 7 _ U ] vf [ U◌֩ Y˟ e. # Lb Z ˫ x m 2 苐 ʈ% -
T |?G9 $Q t N le 5 G< 7 Bh ㎫| '@y e. # Lb Z ˫ x m 2 ~>
© 2019 PayPal Inc. Confidential and proprietary.
Avoiding Password Exposure
Using strong passwords without having to remember them
• The password for monitor can be B*kA2aBntGYdvJaf.
mysql --login-path=monitor
Using the login path in utilities without exposing the password
• MySQL Utilities
~> mysqladmin --login-path=mysqlconn ping
mysqld is alive
• Backup
innobackupex --login-path=mysqlconn /backups
xtrabackup: Transaction log of lsn (14148224090) to (14148224138) was copied.
190413 16:32:06 completed OK!
© 2019 PayPal Inc. Confidential and proprietary.
Avoiding Password Exposure
Using the login path in utilities without exposing the password
• Automation Scripts
[root@ ~]# mysql --login-path=mysqlconn -e "show slave status G" | grep -i "Slave_IO_Running:" |
awk -F':' '{print $2 }'|sed -e 's/^[ t]*//’
Yes
Caveat – It’s not THAT secure
• We can read the contents of the encrypted login file using the my_print_defaults utility.
[root@ ~]# my_print_defaults -s monitor
--user=monitor
--password=B*kA2aBntGYdvJaf
--host=XXXX
--port=3306
my_print_defaults is a part of standard MySQL install.
© 2019 PayPal Inc. Confidential and proprietary.
Storing and Retrieving Application Passwords
• Application passwords should not be
hardcoded
• Application passwords and certificates are
stored in a central key store
• Application passwords are retrieved via
API calls to the key store
• What if key store becomes unreachable?
© 2019 PayPal Inc. Confidential and proprietary.
Central
Keystore
DBA stores the password in
central keystore
Keymaker software, generates a key
with which password can be retrieved.
DBA shares the key with the developers
who store the key in the application
using a protected package
Application
Application fetches password from
keystore via keymaker API, and stores it
in local encrypted cache
LDAP Authentication
MySQL Server can be configured to use LDAP to authenticate users
Percona’s PAM Authentication Plugin
• Percona PAM Authentication Plugin acts as a mediator between the MySQL server, the MySQL client, and the
PAM stack.
© 2019 PayPal Inc. Confidential and proprietary.
LDAP Authentication
Installing the PAM plugin
• To install the PAM plugin –
mysql> INSTALL PLUGIN auth_pam SONAME 'auth_pam.so';
Query OK, 0 rows affected (0.08 sec)
mysql> INSTALL PLUGIN auth_pam_compat SONAME 'auth_pam_compat.so';
Query OK, 0 rows affected (0.05 sec)
Configure Percona PAM to authenticate to LDAP
• Configure Percona PAM to authenticate to LDAP by creating /etc/pam.d/mysqld with the following content –
auth required pam_ldap.so audit
account required pam_ldap.so audit
© 2019 PayPal Inc. Confidential and proprietary.
LDAP Authentication
Create a user with LDAP authentication
• Create a user that will use LDAP for authentication –
mysql> CREATE USER 'yashada'@'%' IDENTIFIED WITH auth_pam;
Query OK, 0 rows affected (0.04 sec)
mysql> GRANT SELECT ON ycsb.* TO 'yashada'@'%';
Query OK, 0 rows affected (0.02 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)
mysql -u yashada –p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or g.
© 2019 PayPal Inc. Confidential and proprietary.
LDAP Authentication
Proxy user support
• If belonging to a group has certain MySQL privileges, setup proxy users instead to map a user’s privilege to its
defined group.
mysql> CREATE USER ''@'' IDENTIFIED WITH auth_pam as 'mysqld,DBA=dba_users';
Query OK, 0 rows affected (0.05 sec)
mysql> CREATE USER dba_users@'localhost' IDENTIFIED BY 'somepassword';
Query OK, 0 rows affected (0.02 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO dba_users@'localhost';
Query OK, 0 rows affected, 1 warning (0.01 sec)
mysql> GRANT PROXY ON dba_users@'localhost' TO ''@'';
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.05 sec)
© 2019 PayPal Inc. Confidential and proprietary.
LDAP Authentication
Proxy user support
• When any user belonging to the group logs in
mysql -u yashada -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 8587
mysql> show grants;
+-------------------------------------------------+
| Grants for @ |
+-------------------------------------------------+
| GRANT USAGE ON *.* TO ''@'' |
| GRANT PROXY ON 'dba_users'@'localhost' TO ''@'' |
+-------------------------------------------------+
2 rows in set (0.00 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Audit Plugin
Install the plugin
mysql> INSTALL PLUGIN audit_log soname 'audit_log.so';
Query OK, 0 rows affected (0.06 sec)
Verify that the audit log plugin is installed
mysql> SELECT * FROM mysql.plugin WHERE name LIKE 'audit_log';
+-----------+--------------+
| name | dl |
+-----------+--------------+
| audit_log | audit_log.so |
+-----------+--------------+
1 row in set (0.00 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Installing the Audit Plugin
Audit Plugin
mysql> show variables like 'audit%';
+-----------------------------+---------------+
| Variable_name | Value |
+-----------------------------+---------------+
| audit_log_buffer_size | 1048576 |
| audit_log_file | audit.log |
| audit_log_flush | OFF |
| audit_log_format | OLD |
| audit_log_handler | FILE |
| audit_log_rotate_on_size | 0 |
| audit_log_rotations | 0 |
| audit_log_strategy | ASYNCHRONOUS |
| audit_log_syslog_facility | LOG_USER |
| audit_log_syslog_ident | percona-audit |
| audit_log_syslog_priority | LOG_INFO |
© 2019 PayPal Inc. Confidential and proprietary.
Plugin Variables
Variable Recommended Setting
audit_log_rotate_on_size 4M
audit_log_rotation 14
audit_log_file /path/audit.log
audit_log_handler FILE/SYSLOG
Audit Log file Management
Audit Plugin Variables
Audit Plugin
Commands available
mysql> SELECT name FROM
performance_schema.setup_instruments WHERE
name LIKE "statement/sql/%" ORDER BY name;
+---------------------------------------------+
| name |
+---------------------------------------------+
| statement/sql/alter_db |
| statement/sql/alter_db_upgrade |
| statement/sql/alter_event |
| statement/sql/alter_function |
| statement/sql/alter_instance |
| statement/sql/alter_procedure |
..
For example –
Set audit_log_include_commands to a list of DDLs
alter_db,alter_db_upgrade,alter_event,al
ter_function,alter_instance,alter_proced
ure,alter_server
© 2019 PayPal Inc. Confidential and proprietary.
Policy Recommended
audit_log_policy QUERIES (ALL,
LOGIN, QUERIES,
NONE)
audit_log_include_accounts If you want to track
specific accounts
audit_log_exclude_accounts Monitoring /
Tooling accounts
audit_log_include_databases
audit_log_exclude_databases Databases like
heartbeat
audit_log_include_commands Only include
commands of
interest
Audit Log Controls
Audit Plugin Variables
Audit Plugin
<AUDIT_RECORD
NAME="Query"
RECORD="2709669_2019-02-05T08:48:00"
TIMESTAMP="2019-03-07T08:49:13 UTC"
COMMAND_CLASS="alter_table"
CONNECTION_ID="1793745"
STATUS="0"
SQLTEXT="ALTER TABLE actors CHANGE `last_update` `last_modified` TIMESTAMP NOT NULL
DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP "
USER="yashada[yashada] @ [x.x.x.x]"
HOST=""
OS_USER=""
IP="x.x.x.x"
DB="sakila"
/>
© 2019 PayPal Inc. Confidential and proprietary.
Sample Audit Log
Encrypted Connections
Encrypted Connections
WHAT
• On-wire data encryption is the process of changing information that makes it unreadable except someone who
has the key
WHY
• The world is not perfect!
• PayPal uses it to protect PII, credit card, etc to guard against things like identity theft
How?
At PayPal, an unencrypted DB connection is unacceptable
© 2019 PayPal Inc. Confidential and proprietary.
Encrypted Connections + Authentication
• Public/private key pair - encryption
• CA Certificate - verify the identity of the
systems
• Username and password - DB
authentication
© 2019 PayPal Inc. Confidential and proprietary.
MySQL ClientEncrypted Connection
MITM
• Server private key
• Server public key
• CA cert
• AES256 ciphers
CA cert
Set up Encrypted Connections
Configure MySQL using encrypted connections
Generate SSL Certs
Create database user with SSL enabled
Connect to database with encrypted connections in various way
Verify connection is encrypted
Troubleshooting Tips
© 2019 PayPal Inc. Confidential and proprietary.
Configure MySQL using Encrypted Connections
ssl option in server
• ssl-ca : The path name of the Certificate Authority (CA) certificate file in pem format.
• ssl-cert: The path name of the server public key certificate file in pem format.
• ssl-key: The path name of the server private key file. Never give this file to anyone.
• ssl-cipher: The list of permitted strong ciphers (AES256) for connection encryption
• require_secure_transport: OFF (by default)
my.cnf
[mysqld]
ssl-ca=ca.pem
ssl-cert=cert.pem
ssl-key=key.pem
ssl_cipher=DH-DSS-AES256-GCM-SHA384,DH-DSS-AES256-SHA,…
© 2019 PayPal Inc. Confidential and proprietary.
Client Configure for Encrypted Connections
ssl option in client
• ssl-ca : The path name of the Certificate Authority (CA) certificate file
• ssl-cipher: The list of permitted ciphers for connection encryption
my.cnf
[client]
ssl-ca=client-ca.pem
ssl_cipher=DH-DSS-AES256-GCM-SHA384,DH-DSS-AES256-SHA,…
Client Options for ssl-mode
• PREFERRED
• REQUIRED
• VERIFY_CA
• VERIFY_IDENTITY
© 2019 PayPal Inc. Confidential and proprietary.
© 2019 PayPal Inc. Confidential and proprietary.
Generate Certificate Service Request
MySQL
Security team
Certificate
Authority
• Using openssl
creates CSR
• Private key remains
local Review CSR and order Certs Sign and issue CA certs
Send CA certs backVerification and send
CA certs back to KC
Using a vendor manages our intermediate CAs, which signs the CSRs of the MySQL
servers
Form CA Certs
© 2019 PayPal Inc. Confidential and proprietary.
Part1: user cert
Part2:
intermediate cert
Part3: root cert
ssl-cert = cert.pem ( user cert )
ssl-ca = ca.pem ( root cert + intermediate cert)
client.ca.pem = root cert
It will be distributed to the clients
ssl-key = key.pem
It is private and held securely in the DB server
Create User with TLS Options
Create user without REQUIRE-SSL
CREATE USER 'ssluser'@'ip-address' IDENTIFIED BY '**********';
GRANT ALL PRIVILEGES ON ssldb.* TO 'ssluser'@'ip-address';
Connect to database with this user
mysql –u ssluser –ppassword –h host –P port#
• The server performs certificate and key file autodiscovery.
• If the server discovers valid certificate and key files , it enables support for encrypted connections by clients.
mysql –u ssluser –ppassword –h host –P port# --ssl-mode=disabled
• If the server does not find valid certificate and key files, it will use non-encrypted connections.
TLS_OPTIONS
SSL, X509, CIPHER, ISSUER, SUBJECT
© 2019 PayPal Inc. Confidential and proprietary.
Alter user with SSL enabled
ALTER USER ‘ssluser’@’ip-address’ REQUIRE SSL;
Or
GRANT ALL PRIVILEGES ON test.* TO ‘ssluser’@’ip' REQUIRE SSL;
Verify
mysql> select user,host,ssl_type from mysql.user where user='ssluser';
+---------+------+----------+
| user | host | ssl_type |
+---------+------+----------+
| ssluser | *****| ANY |
+---------+------+----------+
1 row in set (0.00 sec)
Remove SSL constraint
ALTER USER ‘ssluser’@’ip-address’ REQUIRE NONE;
© 2019 PayPal Inc. Confidential and proprietary.
Create User with SSL Requirement
DB Encrypted Connections Support Verification
mysql> SHOW VARIABLES LIKE 'have_ssl';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_ssl | YES |
+---------------+-------+
mysql> SHOW SESSION STATUS LIKE 'Ssl%';
+---------------+-----------------------------+
| Variable_name | Value |
+---------------+-----------------------------+
| Ssl_version | TLSv1.2 |
| Ssl_cipher | ECDHE-RSA-AES256-GCM-SHA384 |
...
© 2019 PayPal Inc. Confidential and proprietary.
Common Errors
Launch DB connection with --ssl-mode=disabled
Case 1:
User "require SSL":
ERROR 1045 (28000): Access denied for user 'ssluser’@’***' (using password: YES)
Case 2:
DB level "require_secure_transport":
ERROR 3159 (HY000): Connections using insecure transport are prohibited while --
require_secure_transport=ON.
© 2019 PayPal Inc. Confidential and proprietary.
SSL Vulnerability
By default, clients attempt to connect using encryption connection
falling back to an unencrypted connection if an encrypted connection cannot be established.
mysql –u user –ppassword –h host –P port#
mysql> s
--------------
mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2
Connection id: 9051
Current database:
Current user: root@localhost
SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384
…..
If the client does not use --ssl-mode=REQUIRED, an attacker can downgrade the client->attacker connection to plain text, then make
an encrypted connection to the server.
If the client does use --ssl-mode=REQUIRED, an attacker can use their own self-signed certificate so the client->attacker is encrypted
with the attacker's certificate, decrypt the client's traffic, then make an encrypted connection to the server.
© 2019 PayPal Inc. Confidential and proprietary.
SSL Vulnerability
Clients require an encrypted connection, and also perform verification against the server CA certificate
Mysql –u user –ppassword –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_CA
mysql> s
--------------
mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2
Connection id: 9051
Current database:
Current user: root@localhost
SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384
…..
This will let the client verify that the server's certificate was signed by the CA they expect, which will prevent most MITM attacks.
The client would still be vulnerable to a MITM attack if the attacker has another certificate signed by the same CA (for example, a
different server within the same enterprise).
© 2019 PayPal Inc. Confidential and proprietary.
Prevent MITM Attacks
Clients require an encrypted connection, and also perform verification against the server CA certificate and
(with VERIFY_IDENTITY) against the server host name in its certificate.
Mysql –u user –p –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_IDENTITY
mysql> s
--------------
mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2
Connection id: 9051
Current database:
Current user: root@localhost
SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384
…..
This will let the client verify that the server's "Common Name" field matches the server's hostname the client uses.
This should prevent all MITM attacks.
But it is difficult to manage so many certs since we have so many database servers.
© 2019 PayPal Inc. Confidential and proprietary.
Practical Way
Clients require an encrypted connection, also perform verification against the server CA certificate
Create user using IP
GRANT USAGE ON *.* TO 'myuser'@’100.10.10.100’ IDENTIFIED BY ‘****’;
GRANT ALL PRIVILEGES ON `ssldb`.* TO 'myuser'@'100.10.10.100’;
Mysql –u myuser –p –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_CA
mysql> s
--------------
mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2
Current user: myuser@100.10.10.100
SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384
……
Connection: 100.10.10.100 via TCP/IP
This will let the client verify that the server's certificate was signed by the CA they expect, and from server side only accept db
connection request from client 100.10.10.100
© 2019 PayPal Inc. Confidential and proprietary.
Prevent MITM Attacks
© 2019 PayPal Inc. Confidential and proprietary.
MySQL
CA cert
Private key
Public key
CA cert
Require SSL
Certificate
Verification
DB account
user
+ password
+ individual host
Identity
Verification
Client
Verify Live DB Connections are Encrypted
Check live running database
mysql> SELECT sbt.variable_value AS tls_version, t2.variable_value AS cipher, processlist_user AS user,
processlist_host AS host
-> FROM performance_schema.status_by_thread AS sbt
-> JOIN performance_schema.threads AS t ON t.thread_id = sbt.thread_id
-> JOIN performance_schema.status_by_thread AS t2 ON t2.thread_id = t.thread_id
-> WHERE sbt.variable_name = 'Ssl_version' and t2.variable_name = 'Ssl_cipher' ORDER BY tls_version;
+-------------+---------------------------+----------+----------------+
| tls_version | cipher | user | host |
+-------------+---------------------------+----------+----------------+
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp1 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappa | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp1 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappb | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp3 | *.*.*.* |
| TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappb | *.*.*.* |
………….
© 2019 PayPal Inc. Confidential and proprietary.
Verify Encrypted Connections from User
Check table Information_schema.USER_STATISTICS
Use information_schema;
select USER,TOTAL_CONNECTIONS,TOTAL_SSL_CONNECTIONS from USER_STATISTICS;
+---------+-------------------+-----------------------+
| USER | TOTAL_CONNECTIONS | TOTAL_SSL_CONNECTIONS |
+---------+-------------------+-----------------------+
| root | 58 | 58 |
| monitor | 5 | 5 |
| ssluser | 3 | 3 |
| repl | 4 | 4 |
| user | 3 | 2 |
+---------+-------------------+-----------------------+
5 rows in set (0.00 sec)
© 2019 PayPal Inc. Confidential and proprietary.
Set up Replication with Secure Connections
Create replication user require SSL using ip
ip1 is master and ip2 and ip3 are slaves
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip1’ require SSL;
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip2’ require SSL;
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip3’ require SSL;
Enable replication
Change master to
MASTER_HOST=master-host,
MASTER_USER=‘repl’,
MASTER_PASSWORD=‘**’,
MASTER_SSL_CA =/path/client-ca.pem,
MASTER_SSL=1,
MASTER_AUTO_POSITION=1;
START SLAVE;
SHOW SLAVE STATUSG
© 2019 PayPal Inc. Confidential and proprietary.
Encrypted Connections
MySQL Client could be community version
• ERROR 2059 (HY000): Authentication plugin 'dialog' cannot be loaded: dlopen(/usr/local/mysql/lib/plugin/dialog.so, 2): image not found
CA certs file permission
• ERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failed
Errors Received With Cipher Mismatch
Please check ssl_ciphers
Errors Received Without A Complete Truststore
Please generate complete truststore file.
Add “-Djavax.net.debug=all” JVM arguments to debug TLS connections
• -jvm-args='-Djavax.net.debug=all -Djavax.net.ssl.trustStore=/tmp/truststore -Djavax.net.ssl.trustStorePassword=*** -
Djava.security.egd=file:///dev/urandom’
© 2019 PayPal Inc. Confidential and proprietary.
Troubleshooting Tips
Security-Enhanced Linux (SELINUX)
What is SELinux
• It is a Linux kernel security module for supporting access control security policies
• SELinux defines the access and transition rights of every user, application, process, and file on the system
• SELinux then governs the interactions of these entities using a security policy
© 2019 PayPal Inc. Confidential and proprietary.
Security-Enhanced Linux (SELINUX)
Two modes
• SELINUX=enforcing
• SELINUX=permissive
SELinux does not enforce its policy, but only logs what it would have blocked (or granted)
Change SELinux mode to permissive
• Edit the file /etc/selinux/config, reboot the server
• setenforce 0 #online change, it will be lost after the server reboots
• Recommended way: semanage permissive -a mysqld_t
• The best way:
Create policies
load policies, run for a while in permissive state
Change it to enforcing after no denied message in audit.log
© 2019 PayPal Inc. Confidential and proprietary.
Challenges while DB migration from Others to Percona
Audit log
• MySQL commercial version uses function/procedures to handle
• Remove audit related functions
• MySQL> delete from mysql.func where name like 'audit%’;
• MySQL> INSTALL PLUGIN audit_log SONAME 'audit_log.so’;
• Add audit related settings into my.cnf, restart mysql.
LDAP
• Drop all existing ldap accounts and re-create them
SSL
• MySQL community version is incompatible with Percona, need to remove all packages.
Missing library packages
• Need to install Perl related packages as well as other lib packages.
© 2019 PayPal Inc. Confidential and proprietary.
Questions?
MySQL Security and Standardization at PayPal - Percona Live 2019

More Related Content

What's hot

MySQL 8.0.22 - New Features Summary
MySQL 8.0.22 - New Features SummaryMySQL 8.0.22 - New Features Summary
MySQL 8.0.22 - New Features Summary
Olivier DASINI
 
MySQL Day Paris 2018 - Introduction & The State of the Dolphin
MySQL Day Paris 2018 - Introduction & The State of the DolphinMySQL Day Paris 2018 - Introduction & The State of the Dolphin
MySQL Day Paris 2018 - Introduction & The State of the Dolphin
Olivier DASINI
 
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
Olivier DASINI
 
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirementsMySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
Olivier DASINI
 
MySQL Performance Best Practices
MySQL Performance Best PracticesMySQL Performance Best Practices
MySQL Performance Best Practices
Olivier DASINI
 
MySQL Document Store for Modern Applications
MySQL Document Store for Modern ApplicationsMySQL Document Store for Modern Applications
MySQL Document Store for Modern Applications
Olivier DASINI
 
2014 OpenSuse Conf: Protect your MySQL Server
2014 OpenSuse Conf: Protect your MySQL Server2014 OpenSuse Conf: Protect your MySQL Server
2014 OpenSuse Conf: Protect your MySQL Server
Georgi Kodinov
 
MySQL Enterprise Monitor
MySQL Enterprise MonitorMySQL Enterprise Monitor
MySQL Enterprise Monitor
Mario Beck
 
MySQL Community and Commercial Edition
MySQL Community and Commercial EditionMySQL Community and Commercial Edition
MySQL Community and Commercial Edition
Mario Beck
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAs
Mario Beck
 
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
Olivier DASINI
 
Password Policies in Oracle Access Manager. How to improve user authenticatio...
Password Policies in Oracle Access Manager. How to improve user authenticatio...Password Policies in Oracle Access Manager. How to improve user authenticatio...
Password Policies in Oracle Access Manager. How to improve user authenticatio...
Andrejs Prokopjevs
 
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
Olivier DASINI
 
MySQL Manchester TT - Security
MySQL Manchester TT  - SecurityMySQL Manchester TT  - Security
MySQL Manchester TT - Security
Mark Swarbrick
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
Miguel Araújo
 
Percona Live 2019 - MySQL Security
Percona Live 2019 - MySQL SecurityPercona Live 2019 - MySQL Security
Percona Live 2019 - MySQL Security
Vinicius M Grippa
 
MySQL Tech Tour 2015 - 5.7 Security
MySQL Tech Tour 2015 - 5.7 SecurityMySQL Tech Tour 2015 - 5.7 Security
MySQL Tech Tour 2015 - 5.7 Security
Mark Swarbrick
 
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
Miguel Araújo
 
MySQL Security
MySQL SecurityMySQL Security
MySQL Security
Ted Wennmark
 
CTU June 2011 - Things that Every ASP.NET Developer Should Know
CTU June 2011 - Things that Every ASP.NET Developer Should KnowCTU June 2011 - Things that Every ASP.NET Developer Should Know
CTU June 2011 - Things that Every ASP.NET Developer Should Know
Spiffy
 

What's hot (20)

MySQL 8.0.22 - New Features Summary
MySQL 8.0.22 - New Features SummaryMySQL 8.0.22 - New Features Summary
MySQL 8.0.22 - New Features Summary
 
MySQL Day Paris 2018 - Introduction & The State of the Dolphin
MySQL Day Paris 2018 - Introduction & The State of the DolphinMySQL Day Paris 2018 - Introduction & The State of the Dolphin
MySQL Day Paris 2018 - Introduction & The State of the Dolphin
 
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
MySQL Day Paris 2018 - MySQL InnoDB Cluster; A complete High Availability sol...
 
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirementsMySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
MySQL Day Paris 2018 - MySQL & GDPR; Privacy and Security requirements
 
MySQL Performance Best Practices
MySQL Performance Best PracticesMySQL Performance Best Practices
MySQL Performance Best Practices
 
MySQL Document Store for Modern Applications
MySQL Document Store for Modern ApplicationsMySQL Document Store for Modern Applications
MySQL Document Store for Modern Applications
 
2014 OpenSuse Conf: Protect your MySQL Server
2014 OpenSuse Conf: Protect your MySQL Server2014 OpenSuse Conf: Protect your MySQL Server
2014 OpenSuse Conf: Protect your MySQL Server
 
MySQL Enterprise Monitor
MySQL Enterprise MonitorMySQL Enterprise Monitor
MySQL Enterprise Monitor
 
MySQL Community and Commercial Edition
MySQL Community and Commercial EditionMySQL Community and Commercial Edition
MySQL Community and Commercial Edition
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAs
 
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
MySQL Day Paris 2018 - What’s New in MySQL 8.0 ?
 
Password Policies in Oracle Access Manager. How to improve user authenticatio...
Password Policies in Oracle Access Manager. How to improve user authenticatio...Password Policies in Oracle Access Manager. How to improve user authenticatio...
Password Policies in Oracle Access Manager. How to improve user authenticatio...
 
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
MySQL Day Paris 2018 - Upgrade from MySQL 5.7 to MySQL 8.0
 
MySQL Manchester TT - Security
MySQL Manchester TT  - SecurityMySQL Manchester TT  - Security
MySQL Manchester TT - Security
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
 
Percona Live 2019 - MySQL Security
Percona Live 2019 - MySQL SecurityPercona Live 2019 - MySQL Security
Percona Live 2019 - MySQL Security
 
MySQL Tech Tour 2015 - 5.7 Security
MySQL Tech Tour 2015 - 5.7 SecurityMySQL Tech Tour 2015 - 5.7 Security
MySQL Tech Tour 2015 - 5.7 Security
 
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
MySQL Shell - A DevOps-engineer day with MySQL’s development and administrati...
 
MySQL Security
MySQL SecurityMySQL Security
MySQL Security
 
CTU June 2011 - Things that Every ASP.NET Developer Should Know
CTU June 2011 - Things that Every ASP.NET Developer Should KnowCTU June 2011 - Things that Every ASP.NET Developer Should Know
CTU June 2011 - Things that Every ASP.NET Developer Should Know
 

Similar to MySQL Security and Standardization at PayPal - Percona Live 2019

Guob - MySQL e LGPD
Guob - MySQL e LGPDGuob - MySQL e LGPD
Guob - MySQL e LGPD
Vinicius M Grippa
 
Using Vault to decouple MySQL Secrets
Using Vault to decouple MySQL SecretsUsing Vault to decouple MySQL Secrets
Using Vault to decouple MySQL Secrets
Derek Downey
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Curso de MySQL 5.7
Curso de MySQL 5.7Curso de MySQL 5.7
Curso de MySQL 5.7
Eduardo Legatti
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
MySQL 8.0 InnoDB Cluster demo
MySQL 8.0 InnoDB Cluster demoMySQL 8.0 InnoDB Cluster demo
MySQL 8.0 InnoDB Cluster demo
Keith Hollman
 
MySQL Shell - The Best MySQL DBA Tool
MySQL Shell - The Best MySQL DBA ToolMySQL Shell - The Best MySQL DBA Tool
MySQL Shell - The Best MySQL DBA Tool
Miguel Araújo
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Upgrading to my sql 8.0
Upgrading to my sql 8.0Upgrading to my sql 8.0
Upgrading to my sql 8.0
Ståle Deraas
 
Sql installation
Sql installationSql installation
Sql installation
Balakumaran Arunachalam
 
Security features In MySQL 8.0
Security features In MySQL 8.0Security features In MySQL 8.0
Security features In MySQL 8.0
Mydbops
 
Instrumenting plugins for Performance Schema
Instrumenting plugins for Performance SchemaInstrumenting plugins for Performance Schema
Instrumenting plugins for Performance Schema
Mark Leith
 
Securing your MySQL / MariaDB Server data
Securing your MySQL / MariaDB Server dataSecuring your MySQL / MariaDB Server data
Securing your MySQL / MariaDB Server data
Colin Charles
 
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
Cloud Native Day Tel Aviv
 
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
Amazon Web Services
 
Mysql 8 vs Mariadb 10.4 Webinar 2020 Feb
Mysql 8 vs Mariadb 10.4 Webinar 2020 FebMysql 8 vs Mariadb 10.4 Webinar 2020 Feb
Mysql 8 vs Mariadb 10.4 Webinar 2020 Feb
Alkin Tezuysal
 
20180605 sso with apex and adfs the weblogic way
20180605 sso with apex and adfs the weblogic way20180605 sso with apex and adfs the weblogic way
20180605 sso with apex and adfs the weblogic way
makker_nl
 
MySQL Manchester TT - Performance Tuning
MySQL Manchester TT  - Performance TuningMySQL Manchester TT  - Performance Tuning
MySQL Manchester TT - Performance Tuning
Mark Swarbrick
 
MySQL 8.0 - Security Features
MySQL 8.0 - Security FeaturesMySQL 8.0 - Security Features
MySQL 8.0 - Security Features
Harin Vadodaria
 
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
Amazon Web Services
 

Similar to MySQL Security and Standardization at PayPal - Percona Live 2019 (20)

Guob - MySQL e LGPD
Guob - MySQL e LGPDGuob - MySQL e LGPD
Guob - MySQL e LGPD
 
Using Vault to decouple MySQL Secrets
Using Vault to decouple MySQL SecretsUsing Vault to decouple MySQL Secrets
Using Vault to decouple MySQL Secrets
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
 
Curso de MySQL 5.7
Curso de MySQL 5.7Curso de MySQL 5.7
Curso de MySQL 5.7
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
 
MySQL 8.0 InnoDB Cluster demo
MySQL 8.0 InnoDB Cluster demoMySQL 8.0 InnoDB Cluster demo
MySQL 8.0 InnoDB Cluster demo
 
MySQL Shell - The Best MySQL DBA Tool
MySQL Shell - The Best MySQL DBA ToolMySQL Shell - The Best MySQL DBA Tool
MySQL Shell - The Best MySQL DBA Tool
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQLBest Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
 
Upgrading to my sql 8.0
Upgrading to my sql 8.0Upgrading to my sql 8.0
Upgrading to my sql 8.0
 
Sql installation
Sql installationSql installation
Sql installation
 
Security features In MySQL 8.0
Security features In MySQL 8.0Security features In MySQL 8.0
Security features In MySQL 8.0
 
Instrumenting plugins for Performance Schema
Instrumenting plugins for Performance SchemaInstrumenting plugins for Performance Schema
Instrumenting plugins for Performance Schema
 
Securing your MySQL / MariaDB Server data
Securing your MySQL / MariaDB Server dataSecuring your MySQL / MariaDB Server data
Securing your MySQL / MariaDB Server data
 
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.
 
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...
 
Mysql 8 vs Mariadb 10.4 Webinar 2020 Feb
Mysql 8 vs Mariadb 10.4 Webinar 2020 FebMysql 8 vs Mariadb 10.4 Webinar 2020 Feb
Mysql 8 vs Mariadb 10.4 Webinar 2020 Feb
 
20180605 sso with apex and adfs the weblogic way
20180605 sso with apex and adfs the weblogic way20180605 sso with apex and adfs the weblogic way
20180605 sso with apex and adfs the weblogic way
 
MySQL Manchester TT - Performance Tuning
MySQL Manchester TT  - Performance TuningMySQL Manchester TT  - Performance Tuning
MySQL Manchester TT - Performance Tuning
 
MySQL 8.0 - Security Features
MySQL 8.0 - Security FeaturesMySQL 8.0 - Security Features
MySQL 8.0 - Security Features
 
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
AWS Secrets Manager: Best Practices for Managing, Retrieving, and Rotating Se...
 

Recently uploaded

How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 

Recently uploaded (20)

How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 

MySQL Security and Standardization at PayPal - Percona Live 2019

  • 1. MySQL Security and Standardization at PayPal Stacy Yuan & Yashada Jadhav • 29th May 2019
  • 2. Stacy Yuan © 2019 PayPal Inc. Confidential and proprietary. Over 15 years of experience on various flavors of relational databases. Focus on performance tuning, code reviews, database deployment and infrastructure management for MySQL In her spare time, she enjoys reading books and doing some volunteer work.
  • 3. Yashada Jadhav © 2019 PayPal Inc. Confidential and proprietary. 7+ years of being in a relationship with databases Presented previously at Percona Live and Oracle Open World Prior to PayPal, worked at Amazon Web Services and Yahoo! I share random database thoughts at - https://mysql.dbgeekgirl.com
  • 4. Agenda © 2019 PayPal Inc. Confidential and proprietary. MySQL Standardization at PayPal MySQL Security at PayPal • Account Management • Encrypted connections • Migration to Percona MySQL at PayPal
  • 5. At PayPal, we put people at the center of everything we do.
  • 6. MySQL at PayPal © 2019 PayPal Inc. Confidential and proprietary. Internal Apps 3rd Party OSS Backend databasesBuild, Test and Release Tools Custom Monitoring DBaaS Database as a ServiceSite Facing Use Cases Site
  • 7. Replica Scaling Architecture © 2019 PayPal Inc. Confidential and proprietary. Application Connection Cache Master AZ1 Local Read Replica Application Connection Cache Remote Read Replica AZ2 Replica Scaling
  • 8. MySQL Standardization at PayPal Challenges, Journey and What's Next
  • 9. MySQL Standardization at PayPal One DBA Team – Multiple Applications Heterogenous Set ups Migrating ”non-standard” setups to standard © 2018 PayPal Inc. Confidential and proprietary. Hardware Bare Metal Servers VMs Cloud VMs Operating System Ubuntu 14.x, 16.x, 18.x RHEL 6.x, 7.x OEL 7.x MySQL Versions MySQL Community Edition 5.6, 5.7 Percona MySQL 5.7 Challenges
  • 10. MySQL Standardization at PayPal © 2018 PayPal Inc. Confidential and proprietary. Journey • Ansible for more scalable deployment • Deployment scripts for different OS versionsDeployment • Common standards across databases • Recommended MySQL parameter settings • MySQL design standards Best Practices • In house monitoring and alerting • TICK and PMM for OS and MySQL metricsMonitoring • GitHub’s Orchestrator and VIPHigh Availability • Performance Schema, PMM and Slow query logPerformance Tuning • Percona Xtrabackup and mysqldump with onsite and offsite storageBackups
  • 11. MySQL Standardization at PayPal Change Automation Connection Multiplexer & Router Benchmarking MySQL 8.0 InnoDB Encryption at Rest Binary Log and Relay Log Encryption at Rest Dual Password Support & Password Rotation Role Based Access Control © 2018 PayPal Inc. Confidential and proprietary. What’s next
  • 13. First things first Run MySQL on its own dedicated server Keep the DB server as clean as possible Limit SSH access to the DB server Limit sudo access Use specific hostnames while creating users instead of wildcards OS and DB upgrades as necessary © 2019 PayPal Inc. Confidential and proprietary.
  • 14. Password Strength Passwords can be your weakpoint. MySQL’s Password Validation Plugin • Activate the plugin in my.cnf file and restart MySQL server plugin-load-add = validate_password.so validate-password = FORCE_PLUS_PERMANENT • OR without restart mysql> install plugin validate_password soname 'validate_password.so'; Query OK, 0 rows affected (0.03 sec) • Why validate-password setting? mysql> uninstall plugin validate_password; ERROR 1702 (HY000): Plugin 'validate_password' is force_plus_permanent and can not be unloaded © 2019 PayPal Inc. Confidential and proprietary.
  • 15. Password Strength mysql> show global variables like 'validate_password%'; +--------------------------------------+--------+ | Variable_name | Value | +--------------------------------------+--------+ | validate_password_check_user_name | OFF | | validate_password_dictionary_file | | | validate_password_length | 8 | | validate_password_mixed_case_count | 1 | | validate_password_number_count | 1 | | validate_password_policy | MEDIUM | | validate_password_special_char_count | 1 | +--------------------------------------+--------+ 7 rows in set (0.00 sec) © 2019 PayPal Inc. Confidential and proprietary. Plugin Variables Policy Test Performed 0 or LOW Length 1 or MEDIUM Length; numeric, lowercase/uppercase, and special characters 2 or HIGH Length; numeric, lowercase/uppercase, and special characters; dictionary file validate_password_policy mysql> select password ("Abcdefg1"); ERROR 1819 (HY000): Your password does not satisfy the current policy requirements mysql> select password("Abcdef1@"); +-------------------------------------------+ | password("Abcdef1@") | +-------------------------------------------+ | *035C6A33B68D295FDE7BAA22AB7DC5733E136BD8 | +-------------------------------------------+ 1 row in set, 1 warning (0.00 sec)
  • 16. Password Expiration Setting password to expire after N days Setting password expiration on global level SET GLOBAL default_password_lifetime = 180; Setting password to never expire SET GLOBAL default_password_lifetime = 0; Setting password expiration for individual accounts mysql> CREATE USER 'yashada'@'%' IDENTIFIED BY "Abcdef1@" PASSWORD EXPIRE INTERVAL 90 DAY; Query OK, 0 rows affected (0.03 sec) © 2019 PayPal Inc. Confidential and proprietary.
  • 17. Password Expiration What happens when password expires? • When password expires, the server restricts the client to a “sandbox” mode mysql> SELECT 1; ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement. mysql> ALTER USER USER() identified by 'Abcdef1@'; Query OK, 0 rows affected (0.04 sec) mysql> SELECT 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.01 sec) © 2019 PayPal Inc. Confidential and proprietary.
  • 18. Password Expiration Forcing users to change password at login mysql> CREATE USER 'yashada'@'%' IDENTIFIED BY "Abcdef1@”; Query OK, 0 rows affected (0.03 sec) mysql> alter user 'yashada'@'%' PASSWORD EXPIRE; Query OK, 0 rows affected (0.03 sec) • When user will try to login – mysql> select 1; ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement. © 2019 PayPal Inc. Confidential and proprietary.
  • 19. Avoiding Password Exposure mysql_config_editor • mysql_config_editor enables you to store authentication credentials in a login path file named .mylogin.cnf in the current user's home directory. Installing mysql_config_editor • To install mysql_config_editor, all you need is the MySQL client installed. © 2019 PayPal Inc. Confidential and proprietary.
  • 20. Avoiding Password Exposure Set up log in paths using mysql_config_editor ~> mysql_config_editor set --login-path=monitor --user=monitor --password Enter password: Using login-path to log in to MySQL • To use this path to log in to MySQL – ~> mysql --login-path=monitor Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 554 © 2019 PayPal Inc. Confidential and proprietary.
  • 21. Avoiding Password Exposure How mysql_config_editor works? • mysql_config_editor creates an encrypted file in the user’s home director called .mylogin.cnf ~> pwd /root ~> ls -lhtr .mylogin.cnf -rw-------. 1 root root 324 Apr 13 16:03 .mylogin.cnf ~> cat .mylogin.cnf t +~ !2 Z Yf 8<A + Q { ; 7 _ U ] vf [ U◌֩ Y˟ e. # Lb Z ˫ x m 2 苐 ʈ% - T |?G9 $Q t N le 5 G< 7 Bh ㎫| '@y e. # Lb Z ˫ x m 2 ~> © 2019 PayPal Inc. Confidential and proprietary.
  • 22. Avoiding Password Exposure Using strong passwords without having to remember them • The password for monitor can be B*kA2aBntGYdvJaf. mysql --login-path=monitor Using the login path in utilities without exposing the password • MySQL Utilities ~> mysqladmin --login-path=mysqlconn ping mysqld is alive • Backup innobackupex --login-path=mysqlconn /backups xtrabackup: Transaction log of lsn (14148224090) to (14148224138) was copied. 190413 16:32:06 completed OK! © 2019 PayPal Inc. Confidential and proprietary.
  • 23. Avoiding Password Exposure Using the login path in utilities without exposing the password • Automation Scripts [root@ ~]# mysql --login-path=mysqlconn -e "show slave status G" | grep -i "Slave_IO_Running:" | awk -F':' '{print $2 }'|sed -e 's/^[ t]*//’ Yes Caveat – It’s not THAT secure • We can read the contents of the encrypted login file using the my_print_defaults utility. [root@ ~]# my_print_defaults -s monitor --user=monitor --password=B*kA2aBntGYdvJaf --host=XXXX --port=3306 my_print_defaults is a part of standard MySQL install. © 2019 PayPal Inc. Confidential and proprietary.
  • 24. Storing and Retrieving Application Passwords • Application passwords should not be hardcoded • Application passwords and certificates are stored in a central key store • Application passwords are retrieved via API calls to the key store • What if key store becomes unreachable? © 2019 PayPal Inc. Confidential and proprietary. Central Keystore DBA stores the password in central keystore Keymaker software, generates a key with which password can be retrieved. DBA shares the key with the developers who store the key in the application using a protected package Application Application fetches password from keystore via keymaker API, and stores it in local encrypted cache
  • 25. LDAP Authentication MySQL Server can be configured to use LDAP to authenticate users Percona’s PAM Authentication Plugin • Percona PAM Authentication Plugin acts as a mediator between the MySQL server, the MySQL client, and the PAM stack. © 2019 PayPal Inc. Confidential and proprietary.
  • 26. LDAP Authentication Installing the PAM plugin • To install the PAM plugin – mysql> INSTALL PLUGIN auth_pam SONAME 'auth_pam.so'; Query OK, 0 rows affected (0.08 sec) mysql> INSTALL PLUGIN auth_pam_compat SONAME 'auth_pam_compat.so'; Query OK, 0 rows affected (0.05 sec) Configure Percona PAM to authenticate to LDAP • Configure Percona PAM to authenticate to LDAP by creating /etc/pam.d/mysqld with the following content – auth required pam_ldap.so audit account required pam_ldap.so audit © 2019 PayPal Inc. Confidential and proprietary.
  • 27. LDAP Authentication Create a user with LDAP authentication • Create a user that will use LDAP for authentication – mysql> CREATE USER 'yashada'@'%' IDENTIFIED WITH auth_pam; Query OK, 0 rows affected (0.04 sec) mysql> GRANT SELECT ON ycsb.* TO 'yashada'@'%'; Query OK, 0 rows affected (0.02 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) mysql -u yashada –p Enter password: Welcome to the MySQL monitor. Commands end with ; or g. © 2019 PayPal Inc. Confidential and proprietary.
  • 28. LDAP Authentication Proxy user support • If belonging to a group has certain MySQL privileges, setup proxy users instead to map a user’s privilege to its defined group. mysql> CREATE USER ''@'' IDENTIFIED WITH auth_pam as 'mysqld,DBA=dba_users'; Query OK, 0 rows affected (0.05 sec) mysql> CREATE USER dba_users@'localhost' IDENTIFIED BY 'somepassword'; Query OK, 0 rows affected (0.02 sec) mysql> GRANT ALL PRIVILEGES ON *.* TO dba_users@'localhost'; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> GRANT PROXY ON dba_users@'localhost' TO ''@''; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.05 sec) © 2019 PayPal Inc. Confidential and proprietary.
  • 29. LDAP Authentication Proxy user support • When any user belonging to the group logs in mysql -u yashada -p Enter password: Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 8587 mysql> show grants; +-------------------------------------------------+ | Grants for @ | +-------------------------------------------------+ | GRANT USAGE ON *.* TO ''@'' | | GRANT PROXY ON 'dba_users'@'localhost' TO ''@'' | +-------------------------------------------------+ 2 rows in set (0.00 sec) © 2019 PayPal Inc. Confidential and proprietary.
  • 30. Audit Plugin Install the plugin mysql> INSTALL PLUGIN audit_log soname 'audit_log.so'; Query OK, 0 rows affected (0.06 sec) Verify that the audit log plugin is installed mysql> SELECT * FROM mysql.plugin WHERE name LIKE 'audit_log'; +-----------+--------------+ | name | dl | +-----------+--------------+ | audit_log | audit_log.so | +-----------+--------------+ 1 row in set (0.00 sec) © 2019 PayPal Inc. Confidential and proprietary. Installing the Audit Plugin
  • 31. Audit Plugin mysql> show variables like 'audit%'; +-----------------------------+---------------+ | Variable_name | Value | +-----------------------------+---------------+ | audit_log_buffer_size | 1048576 | | audit_log_file | audit.log | | audit_log_flush | OFF | | audit_log_format | OLD | | audit_log_handler | FILE | | audit_log_rotate_on_size | 0 | | audit_log_rotations | 0 | | audit_log_strategy | ASYNCHRONOUS | | audit_log_syslog_facility | LOG_USER | | audit_log_syslog_ident | percona-audit | | audit_log_syslog_priority | LOG_INFO | © 2019 PayPal Inc. Confidential and proprietary. Plugin Variables Variable Recommended Setting audit_log_rotate_on_size 4M audit_log_rotation 14 audit_log_file /path/audit.log audit_log_handler FILE/SYSLOG Audit Log file Management Audit Plugin Variables
  • 32. Audit Plugin Commands available mysql> SELECT name FROM performance_schema.setup_instruments WHERE name LIKE "statement/sql/%" ORDER BY name; +---------------------------------------------+ | name | +---------------------------------------------+ | statement/sql/alter_db | | statement/sql/alter_db_upgrade | | statement/sql/alter_event | | statement/sql/alter_function | | statement/sql/alter_instance | | statement/sql/alter_procedure | .. For example – Set audit_log_include_commands to a list of DDLs alter_db,alter_db_upgrade,alter_event,al ter_function,alter_instance,alter_proced ure,alter_server © 2019 PayPal Inc. Confidential and proprietary. Policy Recommended audit_log_policy QUERIES (ALL, LOGIN, QUERIES, NONE) audit_log_include_accounts If you want to track specific accounts audit_log_exclude_accounts Monitoring / Tooling accounts audit_log_include_databases audit_log_exclude_databases Databases like heartbeat audit_log_include_commands Only include commands of interest Audit Log Controls Audit Plugin Variables
  • 33. Audit Plugin <AUDIT_RECORD NAME="Query" RECORD="2709669_2019-02-05T08:48:00" TIMESTAMP="2019-03-07T08:49:13 UTC" COMMAND_CLASS="alter_table" CONNECTION_ID="1793745" STATUS="0" SQLTEXT="ALTER TABLE actors CHANGE `last_update` `last_modified` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP " USER="yashada[yashada] @ [x.x.x.x]" HOST="" OS_USER="" IP="x.x.x.x" DB="sakila" /> © 2019 PayPal Inc. Confidential and proprietary. Sample Audit Log
  • 35. Encrypted Connections WHAT • On-wire data encryption is the process of changing information that makes it unreadable except someone who has the key WHY • The world is not perfect! • PayPal uses it to protect PII, credit card, etc to guard against things like identity theft How? At PayPal, an unencrypted DB connection is unacceptable © 2019 PayPal Inc. Confidential and proprietary.
  • 36. Encrypted Connections + Authentication • Public/private key pair - encryption • CA Certificate - verify the identity of the systems • Username and password - DB authentication © 2019 PayPal Inc. Confidential and proprietary. MySQL ClientEncrypted Connection MITM • Server private key • Server public key • CA cert • AES256 ciphers CA cert
  • 37. Set up Encrypted Connections Configure MySQL using encrypted connections Generate SSL Certs Create database user with SSL enabled Connect to database with encrypted connections in various way Verify connection is encrypted Troubleshooting Tips © 2019 PayPal Inc. Confidential and proprietary.
  • 38. Configure MySQL using Encrypted Connections ssl option in server • ssl-ca : The path name of the Certificate Authority (CA) certificate file in pem format. • ssl-cert: The path name of the server public key certificate file in pem format. • ssl-key: The path name of the server private key file. Never give this file to anyone. • ssl-cipher: The list of permitted strong ciphers (AES256) for connection encryption • require_secure_transport: OFF (by default) my.cnf [mysqld] ssl-ca=ca.pem ssl-cert=cert.pem ssl-key=key.pem ssl_cipher=DH-DSS-AES256-GCM-SHA384,DH-DSS-AES256-SHA,… © 2019 PayPal Inc. Confidential and proprietary.
  • 39. Client Configure for Encrypted Connections ssl option in client • ssl-ca : The path name of the Certificate Authority (CA) certificate file • ssl-cipher: The list of permitted ciphers for connection encryption my.cnf [client] ssl-ca=client-ca.pem ssl_cipher=DH-DSS-AES256-GCM-SHA384,DH-DSS-AES256-SHA,… Client Options for ssl-mode • PREFERRED • REQUIRED • VERIFY_CA • VERIFY_IDENTITY © 2019 PayPal Inc. Confidential and proprietary.
  • 40. © 2019 PayPal Inc. Confidential and proprietary. Generate Certificate Service Request MySQL Security team Certificate Authority • Using openssl creates CSR • Private key remains local Review CSR and order Certs Sign and issue CA certs Send CA certs backVerification and send CA certs back to KC Using a vendor manages our intermediate CAs, which signs the CSRs of the MySQL servers
  • 41. Form CA Certs © 2019 PayPal Inc. Confidential and proprietary. Part1: user cert Part2: intermediate cert Part3: root cert ssl-cert = cert.pem ( user cert ) ssl-ca = ca.pem ( root cert + intermediate cert) client.ca.pem = root cert It will be distributed to the clients ssl-key = key.pem It is private and held securely in the DB server
  • 42. Create User with TLS Options Create user without REQUIRE-SSL CREATE USER 'ssluser'@'ip-address' IDENTIFIED BY '**********'; GRANT ALL PRIVILEGES ON ssldb.* TO 'ssluser'@'ip-address'; Connect to database with this user mysql –u ssluser –ppassword –h host –P port# • The server performs certificate and key file autodiscovery. • If the server discovers valid certificate and key files , it enables support for encrypted connections by clients. mysql –u ssluser –ppassword –h host –P port# --ssl-mode=disabled • If the server does not find valid certificate and key files, it will use non-encrypted connections. TLS_OPTIONS SSL, X509, CIPHER, ISSUER, SUBJECT © 2019 PayPal Inc. Confidential and proprietary.
  • 43. Alter user with SSL enabled ALTER USER ‘ssluser’@’ip-address’ REQUIRE SSL; Or GRANT ALL PRIVILEGES ON test.* TO ‘ssluser’@’ip' REQUIRE SSL; Verify mysql> select user,host,ssl_type from mysql.user where user='ssluser'; +---------+------+----------+ | user | host | ssl_type | +---------+------+----------+ | ssluser | *****| ANY | +---------+------+----------+ 1 row in set (0.00 sec) Remove SSL constraint ALTER USER ‘ssluser’@’ip-address’ REQUIRE NONE; © 2019 PayPal Inc. Confidential and proprietary. Create User with SSL Requirement
  • 44. DB Encrypted Connections Support Verification mysql> SHOW VARIABLES LIKE 'have_ssl'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | have_ssl | YES | +---------------+-------+ mysql> SHOW SESSION STATUS LIKE 'Ssl%'; +---------------+-----------------------------+ | Variable_name | Value | +---------------+-----------------------------+ | Ssl_version | TLSv1.2 | | Ssl_cipher | ECDHE-RSA-AES256-GCM-SHA384 | ... © 2019 PayPal Inc. Confidential and proprietary.
  • 45. Common Errors Launch DB connection with --ssl-mode=disabled Case 1: User "require SSL": ERROR 1045 (28000): Access denied for user 'ssluser’@’***' (using password: YES) Case 2: DB level "require_secure_transport": ERROR 3159 (HY000): Connections using insecure transport are prohibited while -- require_secure_transport=ON. © 2019 PayPal Inc. Confidential and proprietary.
  • 46. SSL Vulnerability By default, clients attempt to connect using encryption connection falling back to an unencrypted connection if an encrypted connection cannot be established. mysql –u user –ppassword –h host –P port# mysql> s -------------- mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2 Connection id: 9051 Current database: Current user: root@localhost SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384 ….. If the client does not use --ssl-mode=REQUIRED, an attacker can downgrade the client->attacker connection to plain text, then make an encrypted connection to the server. If the client does use --ssl-mode=REQUIRED, an attacker can use their own self-signed certificate so the client->attacker is encrypted with the attacker's certificate, decrypt the client's traffic, then make an encrypted connection to the server. © 2019 PayPal Inc. Confidential and proprietary.
  • 47. SSL Vulnerability Clients require an encrypted connection, and also perform verification against the server CA certificate Mysql –u user –ppassword –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_CA mysql> s -------------- mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2 Connection id: 9051 Current database: Current user: root@localhost SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384 ….. This will let the client verify that the server's certificate was signed by the CA they expect, which will prevent most MITM attacks. The client would still be vulnerable to a MITM attack if the attacker has another certificate signed by the same CA (for example, a different server within the same enterprise). © 2019 PayPal Inc. Confidential and proprietary.
  • 48. Prevent MITM Attacks Clients require an encrypted connection, and also perform verification against the server CA certificate and (with VERIFY_IDENTITY) against the server host name in its certificate. Mysql –u user –p –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_IDENTITY mysql> s -------------- mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2 Connection id: 9051 Current database: Current user: root@localhost SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384 ….. This will let the client verify that the server's "Common Name" field matches the server's hostname the client uses. This should prevent all MITM attacks. But it is difficult to manage so many certs since we have so many database servers. © 2019 PayPal Inc. Confidential and proprietary.
  • 49. Practical Way Clients require an encrypted connection, also perform verification against the server CA certificate Create user using IP GRANT USAGE ON *.* TO 'myuser'@’100.10.10.100’ IDENTIFIED BY ‘****’; GRANT ALL PRIVILEGES ON `ssldb`.* TO 'myuser'@'100.10.10.100’; Mysql –u myuser –p –h host –P port# –-ssl-ca=/path/client-ca.pem –ssl-mode=VERIFY_CA mysql> s -------------- mysql Ver 14.14 Distrib 5.7.23-23, for Linux (x86_64) using 6.2 Current user: myuser@100.10.10.100 SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384 …… Connection: 100.10.10.100 via TCP/IP This will let the client verify that the server's certificate was signed by the CA they expect, and from server side only accept db connection request from client 100.10.10.100 © 2019 PayPal Inc. Confidential and proprietary.
  • 50. Prevent MITM Attacks © 2019 PayPal Inc. Confidential and proprietary. MySQL CA cert Private key Public key CA cert Require SSL Certificate Verification DB account user + password + individual host Identity Verification Client
  • 51. Verify Live DB Connections are Encrypted Check live running database mysql> SELECT sbt.variable_value AS tls_version, t2.variable_value AS cipher, processlist_user AS user, processlist_host AS host -> FROM performance_schema.status_by_thread AS sbt -> JOIN performance_schema.threads AS t ON t.thread_id = sbt.thread_id -> JOIN performance_schema.status_by_thread AS t2 ON t2.thread_id = t.thread_id -> WHERE sbt.variable_name = 'Ssl_version' and t2.variable_name = 'Ssl_cipher' ORDER BY tls_version; +-------------+---------------------------+----------+----------------+ | tls_version | cipher | user | host | +-------------+---------------------------+----------+----------------+ | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp1 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappa | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp1 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappb | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp2 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userapp3 | *.*.*.* | | TLSv1.2 | DHE-RSA-AES256-GCM-SHA384 | userappb | *.*.*.* | …………. © 2019 PayPal Inc. Confidential and proprietary.
  • 52. Verify Encrypted Connections from User Check table Information_schema.USER_STATISTICS Use information_schema; select USER,TOTAL_CONNECTIONS,TOTAL_SSL_CONNECTIONS from USER_STATISTICS; +---------+-------------------+-----------------------+ | USER | TOTAL_CONNECTIONS | TOTAL_SSL_CONNECTIONS | +---------+-------------------+-----------------------+ | root | 58 | 58 | | monitor | 5 | 5 | | ssluser | 3 | 3 | | repl | 4 | 4 | | user | 3 | 2 | +---------+-------------------+-----------------------+ 5 rows in set (0.00 sec) © 2019 PayPal Inc. Confidential and proprietary.
  • 53. Set up Replication with Secure Connections Create replication user require SSL using ip ip1 is master and ip2 and ip3 are slaves GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip1’ require SSL; GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip2’ require SSL; GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl’@’ip3’ require SSL; Enable replication Change master to MASTER_HOST=master-host, MASTER_USER=‘repl’, MASTER_PASSWORD=‘**’, MASTER_SSL_CA =/path/client-ca.pem, MASTER_SSL=1, MASTER_AUTO_POSITION=1; START SLAVE; SHOW SLAVE STATUSG © 2019 PayPal Inc. Confidential and proprietary.
  • 54. Encrypted Connections MySQL Client could be community version • ERROR 2059 (HY000): Authentication plugin 'dialog' cannot be loaded: dlopen(/usr/local/mysql/lib/plugin/dialog.so, 2): image not found CA certs file permission • ERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failed Errors Received With Cipher Mismatch Please check ssl_ciphers Errors Received Without A Complete Truststore Please generate complete truststore file. Add “-Djavax.net.debug=all” JVM arguments to debug TLS connections • -jvm-args='-Djavax.net.debug=all -Djavax.net.ssl.trustStore=/tmp/truststore -Djavax.net.ssl.trustStorePassword=*** - Djava.security.egd=file:///dev/urandom’ © 2019 PayPal Inc. Confidential and proprietary. Troubleshooting Tips
  • 55. Security-Enhanced Linux (SELINUX) What is SELinux • It is a Linux kernel security module for supporting access control security policies • SELinux defines the access and transition rights of every user, application, process, and file on the system • SELinux then governs the interactions of these entities using a security policy © 2019 PayPal Inc. Confidential and proprietary.
  • 56. Security-Enhanced Linux (SELINUX) Two modes • SELINUX=enforcing • SELINUX=permissive SELinux does not enforce its policy, but only logs what it would have blocked (or granted) Change SELinux mode to permissive • Edit the file /etc/selinux/config, reboot the server • setenforce 0 #online change, it will be lost after the server reboots • Recommended way: semanage permissive -a mysqld_t • The best way: Create policies load policies, run for a while in permissive state Change it to enforcing after no denied message in audit.log © 2019 PayPal Inc. Confidential and proprietary.
  • 57. Challenges while DB migration from Others to Percona Audit log • MySQL commercial version uses function/procedures to handle • Remove audit related functions • MySQL> delete from mysql.func where name like 'audit%’; • MySQL> INSTALL PLUGIN audit_log SONAME 'audit_log.so’; • Add audit related settings into my.cnf, restart mysql. LDAP • Drop all existing ldap accounts and re-create them SSL • MySQL community version is incompatible with Percona, need to remove all packages. Missing library packages • Need to install Perl related packages as well as other lib packages. © 2019 PayPal Inc. Confidential and proprietary.