Hire or train: MySQL DBA onboarding and training at Box
•Download as PPTX, PDF•
3 likes•1,255 views
"We're hiring!" How many times have you heard this phrase at a conference? Every database-driven company is hiring and that makes for pretty stiff competition when trying to get a new DBA. Instead of searching for the perfect database administrator from a conference or Linkedin, why not look internally at your organization for system administrators or engineers who may be an equally good fit given the right training. In this talk, I'll explain how the DBAs at Box developed a knowledge-sharing culture around databases and disseminated important learnings to other members of the company. I'll also cover the mentorship process we established to train other members of our Operations team to become rock star DBAs and manage our MySQL and HBase infrastructure at Box.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Hire or train: MySQL DBA onboarding and training at Box
Similar to Hire or train: MySQL DBA onboarding and training at Box (20)
Recently uploaded
Recently uploaded (20)
Hire or train: MySQL DBA onboarding and training at Box
- 1. 1 To Hire, or to train, that is the question #PerconaLive 2014
- 2. 2 Geoffrey Anderson • DBA @ Box, Inc. • Tooling for MySQL and HBase • #DBHangOps
- 3. 3 Share, manage and access your content from any device, anywhere
- 5. 5 HEALTHCARE INDUSTRIAL HIGH TECH MEDIA RETAIL SERVICES
- 8. 8 What are you actually looking for?
- 9. 9 The Perfect MySQL Candidate • 5+ years with MySQL in a high volume production environment • “Strong hands-on experience” with MySQL ‒ SQL and Query optimization ‒ Replication ‒ Performance and Tuning ‒ Knowledge of InnoDB internals • “Strong systems administration” skills (e.g. “Knows Linux”) • Passionate about Devops and automation ‒ MySQL tools ‒ Monitoring • Self-Motivated • Known in the community ‒ Active blogger ‒ Active bug filer/fixer
- 10. 10
- 11. 11
- 13. 13 Benefits of External hiring • New Knowledge • Fresh perspectives ‒ Culture ‒ Performance and tuning ‒ Automating • Start on-call rotation in 1-2 months!
- 14. 14 Costs of External Hiring • Recruiting ‒ Sourcing ‒ Interviewing • New-hire package ‒ Benefits ‒ Salary/Equity • Still potentially a bad hire Time and money
- 15. 15 Benefits of Internal Hiring • Familiarity with environment • Familiarity with team • Known strengths and weaknesses • New organizational perspective • Retention
- 16. 16 Costs of Internal Hiring • Some training required • Ramp-up time • Starting in on-call rotation is probably 3-6 months out
- 18. 18
- 19. 19 Why’d YOU become a MySQL DBA? • You were the developer that knew the most MySQL • You were the Sys admin that knew the most MySQL • You knew Oracle/SQL Server/PostgreSQL/etc before • You studied it at school • Other reasons?
- 20. 20
- 21. 21 Let’s Talk About How Box Trains A MySQL DBA
- 22. 22 Necessary DBA Skills 1. Systems Administration 2. SQL 3. MySQL 4. MySQL at Box 5. MySQL Automation 6. Continued Learning 1. Systems Administration 2. Hadoop 3. HBase 4. HBase at Box 5. HBase Automation 6. Continued learning (HBase plan is nearly identical) That’s for another conference
- 23. 23 1. Systems Administration • Management ‒ Configuration Management ‒ Version Control ‒ Change Control • Triaging ‒ Disk – iostat, df ‒ CPU – top, ps ‒ Memory – free ‒ Network – ifconfig, ping ‒ What happened – sar, log files • Automation ‒ Scripts!
- 24. 24 2. SQL
- 25. 25 2. SQL – What • Used to talk to RDBMSs • Manipulate sets of data • For a DBA ‒ DDL vs. DML ‒ Query optimization
- 26. 26 2. SQL – Resources http://www.sqlcourse.com/
- 27. 27 2. SQL – Resources http://www.phlonx.com/resources/nf3/
- 28. 28 2. SQL – Resources http://databases.about.com/od/specificproducts/a/Should-I-Normalize-My-Database.htm
- 29. 29 3. MySQL
- 30. 30 3. MySQL – What • A Relational Database Management System ‒ Open source https://launchpad.net/mysql-server ‒ Active community • For DBAs ‒ Pluggable storage engines ‒ Cost-based optimizer
- 31. 31 3. MySQL – How to Read The (Freakin’) Manual • Version number is important! • Download PDF for your version! ‒ MySQL 5.5 – http://downloads.mysql.com/docs/refman-5.5-en.pdf ‒ MySQL 5.6 - http://downloads.mysql.com/docs/refman-5.6-en.pdf
- 32. 32 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html[ Optional ]
- 33. 33 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.htmluser_input
- 34. 34 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html defined_below:
- 35. 35 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html | (pipe) indicates full clauses
- 36. 36 3. MySQL – What all DBAs should do • Get the server - http://dev.mysql.com/downloads/mysql/ • Go through the tutorial - http://dev.mysql.com/doc/refman/5.5/en/tutorial.html
- 37. 37 3. MySQL – What all DBAs should do http://docs.linuxtone.org/ebooks/MySQL/Replication%20Tutorial%20Presentation.pdf
- 38. 38 3. MySQL – What all DBAs should do http://mysqlsandbox.net/
- 39. 39 3. MySQL – What all DBAs should do http://planet.mysql.com http://www.mysqlperformanceblog.com/ irc.freenode.net/#mysql
- 40. 40 3. MySQL – What all DBAs should do • http://education.oracle.com/ • http://www.percona.com/products/mysql-training • http://www.skysql.com/products/training
- 41. 41 3. MySQL – Backups (a.k.a. a DBA’s biggest responsibility) • Take them ‒ Seriously.Take them. ‒ No, Seriously.Take them. • Whole books have been written on this! • Whole podcasts have been done on this! ‒ http://bit.ly/oursql035 ‒ http://bit.ly/oursql159 • Backup tools ‒ Percona Xtrabackup ‒ MySQL Enterprise Backup ‒ mysqldump ‒ mydumper/myloader
- 42. 42 3. MySQL – Scalability and High Availability
- 43. 43 4. MySQL at Box
- 44. 44 4. MySQL at Box • Box Database infrastructure • Typical /etc/my.cnf • Tools ‒ Box Weather Station ‒ Querykiller • MySQL Backups and Recovery @ Box • Team responsibilities • Checklists ‒ Changing active server ‒ Granting access ‒ Applying schema changes
- 45. 45 4. MySQL at Box (Ramping up someone) Best Way To Ramp up
- 47. 47 5. MySQL Automation • DB Ops Script Framework ‒ Server tasks ‒ Clone replicas ‒ Promote slave to master ‒ Security tasks ‒ Grant management ‒ Auditing ‒ Sharding tasks ‒ Split shards ‒ Move shards
- 50. 50 ??? – http://dbareactions.tumblr.com When I call one of my former coworkers to see if they want the new DBA position
- 51. 51 ??? – http://dbareactions.tumblr.com When management asked me to fix the database scalability problem by adding an index
- 52. 52 In Closing • Don’t discount external hiring • Don’t discount internal hiring • Commit to knowledge sharing with others • Remember your learning curve – Help others through their own
- 53. 53 ?Email geoff@box.com Twitter @geodbz Tech Blog tech.blog.box.com Platform developers.box.com Github github.com/box
- 54. 54 References (seriously…check these out!) • http://www.jonathanlevin.co.uk/2010/07/where-have-all-mysql- dbas-gone.html • http://www.iheavy.com/2011/12/19/mythical-mysql-dba/ • http://www.iheavy.com/2012/02/26/top-mysql-dba-interview- questions-part-1/ • http://www.percona.com/live/mysql-conference- 2013/sites/default/files/slides/PLMCE13%20-%20Training%20- %20Cost%20or%20Investment%20%28final%29.pdf
- 55. 55 Image Credits • http://www.flickr.com/photos/atgeist/8663784742/sizes/o/ • http://www.flickr.com/photos/prodiffusion/4486297043/sizes/l/ • http://www.qwantz.com/fanart/dinocussing.png • http://www.flickr.com/photos/bparker321/7906708980/sizes/l/ • http://www.flickr.com/photos/96641479@N03/12309210613/sizes/o/ • http://farm4.staticflickr.com/3081/3202565643_c2d1a6f67b_o.jpg • https://www.flickr.com/photos/mhillyer/138766962/sizes/o/ • http://www.flickr.com/photos/tylermyeo/11618726115/sizes/o/ • http://www.flickr.com/photos/billkimandthecats/6338535597/ • http://www.flickr.com/photos/babykailan/2126901394 • https://www.flickr.com/photos/krupptastic/4988425044/sizes/o/ • http://danga.com/words/2007_yapc_asia/yapc-2007.pdf • http://albertolarripa.com/wordpress/wp-content/uploads/2012/06/mysql-replication-principle.png • http://www.cubrid.org/manual/91/en/_images/image38.png • https://www.flickr.com/photos/rfmcdonald/4301658325/sizes/o/ • https://www.flickr.com/photos/byte/120228210/sizes/o/ • http://www.nytimes.com/2013/07/16/business/last-car-plant-brings-detroit-hope-and-cash.html • https://www.flickr.com/photos/lens_envy/2212050922/sizes/o/ • https://www.flickr.com/photos/stvan4245/162066238/sizes/o/
Editor's Notes
- Lots of talks end with “We’re hiring”Keynotes do tooEveryone’s hiring
- What’s everyone actually looking for?
- The perfect candidate looks like..After searching for this stuff, my LinkedIn suggestions are pretty cluttered…5+ years expStrong hands-on mysqlYou instantly spot problems with queriesYou’ve read mysql and innodb source codeLinuxYou know and use strace/GDB like it’s your job (b/c it is!)DevOpsYou automate everything you doYou create tools to automatically resolve problems
- “The perfect candidate” is basically a unicorn riding a dragon that can fight dinosaurs…
- And win
- New knowledgeFresh perspectivesCulture – DBAs relationship with othersE.g. maybe more DB education for companyEngineering and Ops interactionsPerformance and TuningBig picture perf and tuningNew things to monitorE.g. throughputAutomatingTools for more thingsOn-call!
- RecruitingSourcing – dedicated recruiter for a weekInterviewing – 4, 1hr interviews in a week 4 hours of work time (times 5 people)New-hire packageBenefitsSalary/equityMight be a bad fitMight leave your company earlyMight not work out
- Knows your environmentKnows your teamYour team knows herKnown strengths/weaknessesProject completionfire-fightingPersonalityNew org perspectiveBring in engineering perspectiveBring in SA perspectiveEtc.Retention!Invested in the companyInvested in the vision for success
- Need to budget for training$7k-10k isn’t *too* bad when you consider cost of recruiting/sourcing/interviewing“$FUD” factor may not apply if new enough to teamRamp-up timeTakes your time to trainComfort with live issuesGet over “risk aversion” humpOn-call!When they go on-call, possibly more hand-holding at first
- You know thisJust nice to have it reiterated/said aloud
- PAUSEAudience participation time!
- Show of hands for each
- Back to our regularly scheduled program
- High-level list of (mostly) expected skillset for a DBAAll these are learnableIf anything here is new for you, spend time learning it! It’s helpful!List.: SA, SQL, MySQL, MySQL @ Box, Automation, More LearningSide note: a lot of this will be all-encompassingyou may have a very junior or very senior person!Also – same for HBaseSave for another conf
- Systems managementConfig management – Puppet, Chef, CFEngine, etc.Version control – manage your infrastructure code, historyChange control – commitment to process for change, validation, change checkpointsTriaging – is a system healthy? Look at key metrics (disk, cpu, memory, network)Familiarity with tools to gauge:DiskCPUMemoryNetworkIdentifying change – sar, reading logsAutomationDocument it onceAutomate it the second timeShell scripts are fine!Testable PHP, python, etc. better
- SQL skillz
- I could regurgitate wikipediaHigh-level: SQL is a language -- ANSI/ISO in 1986/87Used to talk to RDBMSUsed to manipulate sets of dataDBA:Care about Data Dictionary Language and Data Manipulate LangaugeQuery optimization for a given RDBMSRe-write query to get smaller sets of data and roll them up
- Some resources I give to folks new to SQLSqlcourseis great to learn DML and DDLThoroughexplanations
- Good breakdown of normalizationGoes up to 3NFUnderstand 1NF, 2NF, and 3NFGood refresherHelp engineers decompose problems into sane schemas
- Good high-level about denormalizationkey to performancehelp engineers decompose complex problems into something performant
- Now mysql
- it’s an RDBMS not a languageAble to get at source codeHelpful for debuggingActive and helpful communityFor DBAsUnderstanding basic RDBMS archMySQL gives pluggable storage engines – can leverage these in different situationsMyISAM,innodb, heap, blackhole, toku, etc.CBO – make intelligent decisions based on dataSQL parsing is done before getting dataOther functions done higher up
- Version # importantIf you get to the docs from a google search, click the right versionE.g. 5.5Or 5.6Download PDFIf you can’t get to docs, having a local is helpfulGet it off the main reference man web page
- Definitely read “Typographical and Syntax Conventions”Helps you understand how to read this (alter table syntax)E.g. Square brackets are optional
- Bolded values are either user defined input, or clauses defined later
- e.g. alter_specification is defined right after this
- Pipes delineate full individual clausesLike adding a columnOr an index
- Download the server on your laptopGo through tutorialWalks you through menagerie databaseLearn MySQLisms and SQL
- Mats Kindahl and Lars Thalman made awesome presentationGet a solid grasp of replicationStep-by-step for newbies to MySQL replicationHigh-level of replicationNeeded conf changesStatement changes to enable
- Have sandbox setup on your laptopGet comfortable with making and tearing down sandboxesGreat for testing codeE.g. Grant parsing code I wrote – used sandbox to test heavily before building unit testsValidating assumptionsCan setup custom replication topologies
- Keep up with Current eventsCheck out MySQL PlanetAlways new stuff here for MySQLMySQLPerfBlogNew performance or innodb things all the timeHop on #mysql on freenodeAsk questionHelp answer questionsGood way to keep up with esoteric things you don’t deal with anymore
- Check out training or suggest it for engineers and other DBAsOracle University has A LOT of choicesPercona does MySQL training (I took this, personally)SkySQL does training for MariaDB and MySQLSmart folks at all these companiesI recommend training for anyone who hasn’t done it
- Take themWhole books – like the Effective MySQL Backup and Recovery OurSQL did 2 pod casts on MySQL Backups, give ‘em a listen!Be familiar with backup toolsPhysical vs. LogicalHot vs. cold
- Replication – have slaves to support both customers and internal usersRead splitting through slaves (shed read load from master)Write splitting through partitioningVertical – move a table to a new serverHorizontal – shard table across multiple instances of mysqlGet HA with MySQL FabricAuto failoverAutomatically promote masters
- We introduce DBAs to our infrastructureMultiple clustersVertical and horizontal partition boundariesMy.cnf is great way to see how things work Great for mysql newbies – look up individual variablesToolsBox Weatherstation – Anemometer for query analysis and raingauge for capturing intermittent stalls – See Gavin Towey’s talk tomorrowQuerykiller (and other defensive tools) – protecting DBs from rogue requests and non-performant queriesBackups! – understand the whole backup infrastructureThis get complex with a sharded architectureTeam responsibilitiesWho’s responsible for whatOn-call responsibilitiesChecklists – defined processes
- To help ramp up someoneGET THEM INTO YOUR TICKET QUEUEGreat way to get feet wetIterate from easy to hard ticketsLet them fail, let them learnBudget double time to train (instead of doing it yourself)
- Pull a copy of mysql utilitiesGreat tools for: HA, management, etc.Easy to build new toolsPercona toolkit – out of the box, well-known DBA toolchainTriage server healthKill queriesEvaluate indexesEtc.Puppet – automate your setupUse MySQL module or check out mozilla’sWe have our own homegrown (similar to mozilla’s)At Boxnew 5.6 cluster auto-sets up in minutes with fabric using puppetNormal 5.5 server auto setup with monitoring et al. within 30min of a kickstart
- Additional tooling – We have our own frameworkCommon server tasks like:Clone replicasPromote slaves to masterSecurityManage grantsAudit users on serverShardingSplit shards Move shardsAll in PHP (some python)UNIT TESTEDTestable codeReusable code with other TechOps teams @ Box.
- Continued learning!If you haven’t, read:High Perf MySQL – amazing book. That’s how I learned high perfmysqlPro MySQLPHP and MySQL – Box uses a lot of PHP still, good to be familiar with itAnd any other books here
- Should be attending Percona LiveShould participate in free webinarsGo to MySQL meetups – Like #SFMySQL, or MySQLNYCAttend MySQLConnect
- Additional!DBAs need a way to maintain sanity!
- In closing…Don’t discount external hiring – if you luck out and find that awesome candidate, take ‘em!Don’t’ discount internal hiring – anyone can be a DBA if they’re driven. (even Sales, Customer Support, etc)Share knowledge@ Box we host DBA open hours to help folks with it@ Box we do periodic training on query optimization for engineering and OpsRemember your learning curve – you didn’t learn this overnightHelp new hires through their curve – challenge them
- Thanks!