Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

M|18 How DBAs at TradingScreen Make Life Easier With Automation

73 views

Published on

M|18 How DBAs at TradingScreen Make Life Easier With Automation

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

M|18 How DBAs at TradingScreen Make Life Easier With Automation

  1. 1. MAKING LIFE EASIER WITH AUTOMATION M|18 PRESENTATION TRADINGSCREEN LTD AMY KETTLEWELL ROSALIND CHAN
  2. 2. WHO WE ARE TRADINGSCREEN IS A PROVIDER OF ELECTRONIC TRADING SOLUTIONS FOR THE GLOBAL FINANCIAL MARKETPLACE. WE ARE A US-BASED FINANCIAL TECHNOLOGY PROVIDER OF SAAS BASED TRADING SERVICES FOR HEDGE FUNDS, MUTUAL FUNDS, AND BROKERS. HEADQUARTERED IN NEW YORK CITY, AND WE HAVE OFFICES IN HONG KONG, TOKYO, LONDON, PARIS, GENEVA, SINGAPORE AND FRANKFURT. NEW YORK / TOKYO / LONDON 3 DBAs FOLLOW THE SUN
  3. 3. OUR SETUP Five environments with multiple masters and regional slaves 100 database server instances, mostly MariaDB. Currently running MariaDB 10.1 for around 200 unique non-system schemas Long migration from Sybase
  4. 4. WHY AUTOMATE? We can’t see everything all the time Overview of our databases Make regular tasks easier Standardise tasks across our servers Ensure replication integrity Reduce possibilities of human error
  5. 5. WHAT CAN YOU AUTOMATE? Backups Restoring from backups Slave consistency checks Deadlock checks Data archiving Key checks Slow log analysis User permission tracking Schema changes Maintenance (e.g. OPTIMIZE) Failover Almost anything you can think of!
  6. 6. WHAT TOOLS ARE AVAILABLE? - Crons - Percona Toolkit - pt-archiver - pt-table-checksum - pt-table-sync - XtraBackup / MariaBackup - Repositories (SVN, Git, etc) - Custom scripts (Bash, Ruby, etc.) - APIs (e.g. Confluence, JIRA, Graphite) - Other third party tools (e.g. Flyway)
  7. 7. HOW WE AUTOMATE AT TRADINGSCREEN - Replication overview maps
  8. 8. HOW WE AUTOMATE AT TRADINGSCREEN - Centralised database of information (DBABot) - Crons - Backup information - Server options - Schema changes - User permissions - Database objects (tables, SPs)
  9. 9. HOW WE AUTOMATE AT TRADINGSCREEN
  10. 10. HOW WE AUTOMATE AT TRADINGSCREEN - Graphite stat reporting - Disk usage - Replication lag
  11. 11. HOW WE AUTOMATE AT TRADINGSCREEN - Scripted alerting (E-mail / Opsview) - Down instances - Slow slaves - Long transactions - Backup errors - Deadlocks - Everything we want to know real time
  12. 12. HOW WE AUTOMATE AT TRADINGSCREEN - Slave Consistency Reports
  13. 13. HOW WE AUTOMATE AT TRADINGSCREEN - Processlist capturing - XtraBackup restores - Jira schema change deployments - Data archiving - Table optimization - Pulls from repository - Scripts - Database config files - Crons - Tools / Libraries - Rotate slow query logs
  14. 14. Examples
  15. 15. AUTOMATING XTRABACKUP-MULTIPLE SLAVES COPYING 150GB DATABASE FROM PRODUCTION TO UAT ENVIRONMENT CROSS REGION FAN-IN REPLICATION mysqldump IS SLOW WITH TOO MANY binlogs WINDOW PERIOD 1 DAY xtrabackup RESTORE DOESN’T REPLICATE ONLY 1 DBA
  16. 16. AUTOMATING XTRABACKUP-MULTIPLE SLAVES HOW IT WORKS TOOLS USED xtrabackup version 2.4.7 mysqldump Bash shell Git
  17. 17. AUTOMATING XTRABACKUP-MULTIPLE SLAVES FUTURE PLANS MariaBackup RESTORE TABLE/INDEX STATS FAILURE ALERTS GLOBAL MOUNT ON CLOUD CONFLUENCE REPORT FOR ONGOING BACKUP/RESTORE PROGRESS METHOD TO INCLUDE UNSUPPORTED ENGINES ( e.g. TokuDB)
  18. 18. AUTOMATING DAILY TASKS-RosBot lazyrosie WORKS WITH jira FOR SEMI-AUTO SCRIPT DEPLOYMENTS GET OPEN TICKETS FROM jiradb.jiraissue TABLE GET ATTACHMENTS FROM JIRA (GO-JIRA : https://github.com/netflix-skunkworks/go-jira) jira -u rchan attach list [jiraticket] #get attachment ids jira -u rchan attach get [att_id] #for every attachment id RosBot SSH TUNNEL INTO DB HOST FOR SCRIPT DEPLOYMENT ssh -f [service_user]@[jumphost] -l [local_port]:[db_host]:22 LOGIN AND ALLOW DBA TO RUN SCRIPTS ssh -t [db_host] -p[local_port] "cd [custom_dir] ; bash" </dev/tty
  19. 19. AUTOMATING DAILY TASKS-RosBot whosgone LOCKS LEAVERS ACROSS ALL DBS GET LIST OF DB INSTANCE HOST AND PORTS BASED ON LEAVER ID PROVIDED - DBABot SSH TUNNEL INTO DB HOST ssh -f [service_user]@[jumphost] -l [local_port]:[db_host]:22 LOCK USER ACCOUNTS GRANT USAGE ON *.* TO [leaver] WITH max_user_connections -1;
  20. 20. AUTOMATING DAILY TASKS-RosBot FUTURE PLANS RosBot SHOULD MARRY DBABot lazyrosie SHOULD GET LAZIER WITH DOING MORE whosgone WILL BE GONE AFTER DB LDAP AUTHENTICATION RosBot SHOULD GET ACQUAINTED WITH Flyway FOR BIG RELEASES
  21. 21. AUTOMATION CAVEATS - Interaction (e.g. locking processes) - Few “one size fits all” solutions - Automation is only as intelligent as the implementer - Engine-specific considerations - XtraBackup & TokuDB - Some things can’t be fully automated - Replication break recovery - Schema changes - Investigations and Resolutions
  22. 22. Automation should be about making a DBA’s life easier
  23. 23. Thank you for listening!

×