1
To Hire,
or to train,
that is the
question
#PerconaLive 2014
2
Geoffrey Anderson
• DBA @ Box, Inc.
• Tooling for MySQL and HBase
• #DBHangOps
3
Share, manage and access your
content from any device, anywhere
4
25MM+
USERS
225K+
BUSINESSES
99%
FORTUNE 500
5
HEALTHCARE INDUSTRIAL HIGH TECH MEDIA RETAIL SERVICES
6
Why this talk?
7
We’re Hiring!
8
What are you actually looking for?
9
The Perfect MySQL Candidate
• 5+ years with MySQL in a high volume production environment
• “Strong hands-on experience”...
10
11
12
Let’s
backpedal
for a second
13
Benefits of External hiring
• New Knowledge
• Fresh perspectives
‒ Culture
‒ Performance and tuning
‒ Automating
• Star...
14
Costs of External Hiring
• Recruiting
‒ Sourcing
‒ Interviewing
• New-hire package
‒ Benefits
‒ Salary/Equity
• Still p...
15
Benefits of Internal Hiring
• Familiarity with environment
• Familiarity with team
• Known strengths and weaknesses
• N...
16
Costs of Internal Hiring
• Some training required
• Ramp-up time
• Starting in on-call rotation is probably 3-6 months ...
17
But You
Already
Know
All
This
18
19
Why’d YOU become a MySQL DBA?
• You were the developer that knew the most MySQL
• You were the Sys admin that knew the ...
20
21
Let’s Talk
About How
Box Trains
A MySQL
DBA
22
Necessary DBA Skills
1. Systems Administration
2. SQL
3. MySQL
4. MySQL at Box
5. MySQL Automation
6. Continued Learnin...
23
1. Systems Administration
• Management
‒ Configuration Management
‒ Version Control
‒ Change Control
• Triaging
‒ Disk ...
24
2. SQL
25
2. SQL – What
• Used to talk to RDBMSs
• Manipulate sets of data
• For a DBA
‒ DDL vs. DML
‒ Query optimization
26
2. SQL – Resources
http://www.sqlcourse.com/
27
2. SQL – Resources
http://www.phlonx.com/resources/nf3/
28
2. SQL – Resources
http://databases.about.com/od/specificproducts/a/Should-I-Normalize-My-Database.htm
29
3. MySQL
30
3. MySQL – What
• A Relational Database Management System
‒ Open source https://launchpad.net/mysql-server
‒ Active com...
31
3. MySQL – How to Read The (Freakin’) Manual
• Version number is important!
• Download PDF for your version!
‒ MySQL 5....
32
3. MySQL – How to Read The (Freakin’) Manual
• “Typographical and Syntax Conventions”
‒ http://dev.mysql.com/doc/refman...
33
3. MySQL – How to Read The (Freakin’) Manual
• “Typographical and Syntax Conventions”
‒ http://dev.mysql.com/doc/refman...
34
3. MySQL – How to Read The (Freakin’) Manual
• “Typographical and Syntax Conventions”
‒ http://dev.mysql.com/doc/refman...
35
3. MySQL – How to Read The (Freakin’) Manual
• “Typographical and Syntax Conventions”
‒ http://dev.mysql.com/doc/refman...
36
3. MySQL – What all DBAs should do
• Get the server - http://dev.mysql.com/downloads/mysql/
• Go through the tutorial -...
37
3. MySQL – What all DBAs should do
http://docs.linuxtone.org/ebooks/MySQL/Replication%20Tutorial%20Presentation.pdf
38
3. MySQL – What all DBAs should do
http://mysqlsandbox.net/
39
3. MySQL – What all DBAs should do
http://planet.mysql.com
http://www.mysqlperformanceblog.com/
irc.freenode.net/#mysql
40
3. MySQL – What all DBAs should do
• http://education.oracle.com/
• http://www.percona.com/products/mysql-training
• ht...
41
3. MySQL – Backups (a.k.a. a DBA’s biggest responsibility)
• Take them
‒ Seriously.Take them.
‒ No, Seriously.Take them...
42
3. MySQL – Scalability and High Availability
43
4. MySQL at Box
44
4. MySQL at Box
• Box Database infrastructure
• Typical /etc/my.cnf
• Tools
‒ Box Weather Station
‒ Querykiller
• MySQL...
45
4. MySQL at Box (Ramping up someone)
Best
Way
To
Ramp
up
46
5. MySQL Automation
https://code.launchpad.net/mysql-utilities
https://code.launchpad.net/percona-toolkit
https://forge...
47
5. MySQL Automation
• DB Ops Script Framework
‒ Server tasks
‒ Clone replicas
‒ Promote slave to master
‒ Security task...
48
6. Continued Learning
49
6. Continued Learning
50
??? – http://dbareactions.tumblr.com
When I call one of my former coworkers to see if they want the new DBA position
51
??? – http://dbareactions.tumblr.com
When management asked me to fix the database scalability problem by adding an index
52
In
Closing
• Don’t discount external hiring
• Don’t discount internal hiring
• Commit to knowledge sharing with others
...
53
?Email geoff@box.com
Twitter @geodbz
Tech Blog tech.blog.box.com
Platform developers.box.com
Github github.com/box
54
References (seriously…check these out!)
• http://www.jonathanlevin.co.uk/2010/07/where-have-all-mysql-
dbas-gone.html
•...
55
Image Credits
• http://www.flickr.com/photos/atgeist/8663784742/sizes/o/
• http://www.flickr.com/photos/prodiffusion/44...
Upcoming SlideShare
Loading in...5
×

To Hire, or to train, that is the question (Percona Live 2014)

327

Published on

"We're hiring!"
How many times have you heard this phrase at a conference? Every database-driven company is hiring and that makes for pretty stiff competition when trying to get a new DBA. Instead of searching for the perfect database administrator from a conference or Linkedin, why not look internally at your organization for system administrators or engineers who may be an equally good fit given the right training.

In this talk, I'll explain how the DBAs at Box developed a knowledge-sharing culture around databases and disseminated important learnings to other members of the company. I'll also cover the mentorship process we established to train other members of our Operations team to become rock star DBAs and manage our MySQL and HBase infrastructure at Box.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
327
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Lots of talks end with “We’re hiring”Keynotes do tooEveryone’s hiring
  • What’s everyone actually looking for?
  • The perfect candidate looks like..After searching for this stuff, my LinkedIn suggestions are pretty cluttered…5+ years expStrong hands-on mysqlYou instantly spot problems with queriesYou’ve read mysql and innodb source codeLinuxYou know and use strace/GDB like it’s your job (b/c it is!)DevOpsYou automate everything you doYou create tools to automatically resolve problems
  • “The perfect candidate” is basically a unicorn riding a dragon that can fight dinosaurs…
  • And win
  • New knowledgeFresh perspectivesCulture – DBAs relationship with othersE.g. maybe more DB education for companyEngineering and Ops interactionsPerformance and TuningBig picture perf and tuningNew things to monitorE.g. throughputAutomatingTools for more thingsOn-call!
  • RecruitingSourcing – dedicated recruiter for a weekInterviewing – 4, 1hr interviews in a week  4 hours of work time (times 5 people)New-hire packageBenefitsSalary/equityMight be a bad fitMight leave your company earlyMight not work out
  • Knows your environmentKnows your teamYour team knows herKnown strengths/weaknessesProject completionfire-fightingPersonalityNew org perspectiveBring in engineering perspectiveBring in SA perspectiveEtc.Retention!Invested in the companyInvested in the vision for success
  • Need to budget for training$7k-10k isn’t *too* bad when you consider cost of recruiting/sourcing/interviewing“$FUD” factor may not apply if new enough to teamRamp-up timeTakes your time to trainComfort with live issuesGet over “risk aversion” humpOn-call!When they go on-call, possibly more hand-holding at first
  • You know thisJust nice to have it reiterated/said aloud
  • PAUSEAudience participation time!
  • Show of hands for each
  • Back to our regularly scheduled program
  • High-level list of (mostly) expected skillset for a DBAAll these are learnableIf anything here is new for you, spend time learning it! It’s helpful!List.: SA, SQL, MySQL, MySQL @ Box, Automation, More LearningSide note: a lot of this will be all-encompassingyou may have a very junior or very senior person!Also – same for HBaseSave for another conf
  • Systems managementConfig management – Puppet, Chef, CFEngine, etc.Version control – manage your infrastructure code, historyChange control – commitment to process for change, validation, change checkpointsTriaging – is a system healthy? Look at key metrics (disk, cpu, memory, network)Familiarity with tools to gauge:DiskCPUMemoryNetworkIdentifying change – sar, reading logsAutomationDocument it onceAutomate it the second timeShell scripts are fine!Testable PHP, python, etc. better
  • SQL skillz
  • I could regurgitate wikipediaHigh-level: SQL is a language -- ANSI/ISO in 1986/87Used to talk to RDBMSUsed to manipulate sets of dataDBA:Care about Data Dictionary Language and Data Manipulate LangaugeQuery optimization for a given RDBMSRe-write query to get smaller sets of data and roll them up
  • Some resources I give to folks new to SQLSqlcourseis great to learn DML and DDLThoroughexplanations
  • Good breakdown of normalizationGoes up to 3NFUnderstand 1NF, 2NF, and 3NFGood refresherHelp engineers decompose problems into sane schemas
  • Good high-level about denormalizationkey to performancehelp engineers decompose complex problems into something performant
  • Now mysql
  • it’s an RDBMS not a languageAble to get at source codeHelpful for debuggingActive and helpful communityFor DBAsUnderstanding basic RDBMS archMySQL gives pluggable storage engines – can leverage these in different situationsMyISAM,innodb, heap, blackhole, toku, etc.CBO – make intelligent decisions based on dataSQL parsing is done before getting dataOther functions done higher up
  • Version # importantIf you get to the docs from a google search, click the right versionE.g. 5.5Or 5.6Download PDFIf you can’t get to docs, having a local is helpfulGet it off the main reference man web page
  • Definitely read “Typographical and Syntax Conventions”Helps you understand how to read this (alter table syntax)E.g. Square brackets are optional
  • Bolded values are either user defined input, or clauses defined later
  • e.g. alter_specification is defined right after this
  • Pipes delineate full individual clausesLike adding a columnOr an index
  • Download the server on your laptopGo through tutorialWalks you through menagerie databaseLearn MySQLisms and SQL
  • Mats Kindahl and Lars Thalman made awesome presentationGet a solid grasp of replicationStep-by-step for newbies to MySQL replicationHigh-level of replicationNeeded conf changesStatement changes to enable
  • Have sandbox setup on your laptopGet comfortable with making and tearing down sandboxesGreat for testing codeE.g. Grant parsing code I wrote – used sandbox to test heavily before building unit testsValidating assumptionsCan setup custom replication topologies
  • Keep up with Current eventsCheck out MySQL PlanetAlways new stuff here for MySQLMySQLPerfBlogNew performance or innodb things all the timeHop on #mysql on freenodeAsk questionHelp answer questionsGood way to keep up with esoteric things you don’t deal with anymore
  • Check out training or suggest it for engineers and other DBAsOracle University has A LOT of choicesPercona does MySQL training (I took this, personally)SkySQL does training for MariaDB and MySQLSmart folks at all these companiesI recommend training for anyone who hasn’t done it
  • Take themWhole books – like the Effective MySQL Backup and Recovery OurSQL did 2 pod casts on MySQL Backups, give ‘em a listen!Be familiar with backup toolsPhysical vs. LogicalHot vs. cold
  • Replication – have slaves to support both customers and internal usersRead splitting through slaves (shed read load from master)Write splitting through partitioningVertical – move a table to a new serverHorizontal – shard table across multiple instances of mysqlGet HA with MySQL FabricAuto failoverAutomatically promote masters
  • We introduce DBAs to our infrastructureMultiple clustersVertical and horizontal partition boundariesMy.cnf is great way to see how things work Great for mysql newbies – look up individual variablesToolsBox Weatherstation – Anemometer for query analysis and raingauge for capturing intermittent stalls – See Gavin Towey’s talk tomorrowQuerykiller (and other defensive tools) – protecting DBs from rogue requests and non-performant queriesBackups! – understand the whole backup infrastructureThis get complex with a sharded architectureTeam responsibilitiesWho’s responsible for whatOn-call responsibilitiesChecklists – defined processes
  • To help ramp up someoneGET THEM INTO YOUR TICKET QUEUEGreat way to get feet wetIterate from easy to hard ticketsLet them fail, let them learnBudget double time to train (instead of doing it yourself)
  • Pull a copy of mysql utilitiesGreat tools for: HA, management, etc.Easy to build new toolsPercona toolkit – out of the box, well-known DBA toolchainTriage server healthKill queriesEvaluate indexesEtc.Puppet – automate your setupUse MySQL module or check out mozilla’sWe have our own homegrown (similar to mozilla’s)At Boxnew 5.6 cluster auto-sets up in minutes with fabric using puppetNormal 5.5 server auto setup with monitoring et al. within 30min of a kickstart
  • Additional tooling – We have our own frameworkCommon server tasks like:Clone replicasPromote slaves to masterSecurityManage grantsAudit users on serverShardingSplit shards Move shardsAll in PHP (some python)UNIT TESTEDTestable codeReusable code with other TechOps teams @ Box.
  • Continued learning!If you haven’t, read:High Perf MySQL – amazing book. That’s how I learned high perfmysqlPro MySQLPHP and MySQL – Box uses a lot of PHP still, good to be familiar with itAnd any other books here
  • Should be attending Percona LiveShould participate in free webinarsGo to MySQL meetups – Like #SFMySQL, or MySQLNYCAttend MySQLConnect
  • Additional!DBAs need a way to maintain sanity!
  • In closing…Don’t discount external hiring – if you luck out and find that awesome candidate, take ‘em!Don’t’ discount internal hiring – anyone can be a DBA if they’re driven. (even Sales, Customer Support, etc)Share knowledge@ Box we host DBA open hours to help folks with it@ Box we do periodic training on query optimization for engineering and OpsRemember your learning curve – you didn’t learn this overnightHelp new hires through their curve – challenge them
  • Thanks!
  • To Hire, or to train, that is the question (Percona Live 2014)

    1. 1. 1 To Hire, or to train, that is the question #PerconaLive 2014
    2. 2. 2 Geoffrey Anderson • DBA @ Box, Inc. • Tooling for MySQL and HBase • #DBHangOps
    3. 3. 3 Share, manage and access your content from any device, anywhere
    4. 4. 4 25MM+ USERS 225K+ BUSINESSES 99% FORTUNE 500
    5. 5. 5 HEALTHCARE INDUSTRIAL HIGH TECH MEDIA RETAIL SERVICES
    6. 6. 6 Why this talk?
    7. 7. 7 We’re Hiring!
    8. 8. 8 What are you actually looking for?
    9. 9. 9 The Perfect MySQL Candidate • 5+ years with MySQL in a high volume production environment • “Strong hands-on experience” with MySQL ‒ SQL and Query optimization ‒ Replication ‒ Performance and Tuning ‒ Knowledge of InnoDB internals • “Strong systems administration” skills (e.g. “Knows Linux”) • Passionate about Devops and automation ‒ MySQL tools ‒ Monitoring • Self-Motivated • Known in the community ‒ Active blogger ‒ Active bug filer/fixer
    10. 10. 10
    11. 11. 11
    12. 12. 12 Let’s backpedal for a second
    13. 13. 13 Benefits of External hiring • New Knowledge • Fresh perspectives ‒ Culture ‒ Performance and tuning ‒ Automating • Start on-call rotation in 1-2 months!
    14. 14. 14 Costs of External Hiring • Recruiting ‒ Sourcing ‒ Interviewing • New-hire package ‒ Benefits ‒ Salary/Equity • Still potentially a bad hire  Time and money
    15. 15. 15 Benefits of Internal Hiring • Familiarity with environment • Familiarity with team • Known strengths and weaknesses • New organizational perspective • Retention
    16. 16. 16 Costs of Internal Hiring • Some training required • Ramp-up time • Starting in on-call rotation is probably 3-6 months out
    17. 17. 17 But You Already Know All This
    18. 18. 18
    19. 19. 19 Why’d YOU become a MySQL DBA? • You were the developer that knew the most MySQL • You were the Sys admin that knew the most MySQL • You knew Oracle/SQL Server/PostgreSQL/etc before • You studied it at school • Other reasons?
    20. 20. 20
    21. 21. 21 Let’s Talk About How Box Trains A MySQL DBA
    22. 22. 22 Necessary DBA Skills 1. Systems Administration 2. SQL 3. MySQL 4. MySQL at Box 5. MySQL Automation 6. Continued Learning 1. Systems Administration 2. Hadoop 3. HBase 4. HBase at Box 5. HBase Automation 6. Continued learning (HBase plan is nearly identical) That’s for another conference
    23. 23. 23 1. Systems Administration • Management ‒ Configuration Management ‒ Version Control ‒ Change Control • Triaging ‒ Disk – iostat, df ‒ CPU – top, ps ‒ Memory – free ‒ Network – ifconfig, ping ‒ What happened – sar, log files • Automation ‒ Scripts!
    24. 24. 24 2. SQL
    25. 25. 25 2. SQL – What • Used to talk to RDBMSs • Manipulate sets of data • For a DBA ‒ DDL vs. DML ‒ Query optimization
    26. 26. 26 2. SQL – Resources http://www.sqlcourse.com/
    27. 27. 27 2. SQL – Resources http://www.phlonx.com/resources/nf3/
    28. 28. 28 2. SQL – Resources http://databases.about.com/od/specificproducts/a/Should-I-Normalize-My-Database.htm
    29. 29. 29 3. MySQL
    30. 30. 30 3. MySQL – What • A Relational Database Management System ‒ Open source https://launchpad.net/mysql-server ‒ Active community • For DBAs ‒ Pluggable storage engines ‒ Cost-based optimizer
    31. 31. 31 3. MySQL – How to Read The (Freakin’) Manual • Version number is important! • Download PDF for your version! ‒ MySQL 5.5 – http://downloads.mysql.com/docs/refman-5.5-en.pdf ‒ MySQL 5.6 - http://downloads.mysql.com/docs/refman-5.6-en.pdf
    32. 32. 32 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html[ Optional ]
    33. 33. 33 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.htmluser_input
    34. 34. 34 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html defined_below:
    35. 35. 35 3. MySQL – How to Read The (Freakin’) Manual • “Typographical and Syntax Conventions” ‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html | (pipe) indicates full clauses
    36. 36. 36 3. MySQL – What all DBAs should do • Get the server - http://dev.mysql.com/downloads/mysql/ • Go through the tutorial - http://dev.mysql.com/doc/refman/5.5/en/tutorial.html
    37. 37. 37 3. MySQL – What all DBAs should do http://docs.linuxtone.org/ebooks/MySQL/Replication%20Tutorial%20Presentation.pdf
    38. 38. 38 3. MySQL – What all DBAs should do http://mysqlsandbox.net/
    39. 39. 39 3. MySQL – What all DBAs should do http://planet.mysql.com http://www.mysqlperformanceblog.com/ irc.freenode.net/#mysql
    40. 40. 40 3. MySQL – What all DBAs should do • http://education.oracle.com/ • http://www.percona.com/products/mysql-training • http://www.skysql.com/products/training
    41. 41. 41 3. MySQL – Backups (a.k.a. a DBA’s biggest responsibility) • Take them ‒ Seriously.Take them. ‒ No, Seriously.Take them. • Whole books have been written on this! • Whole podcasts have been done on this! ‒ http://bit.ly/oursql035 ‒ http://bit.ly/oursql159 • Backup tools ‒ Percona Xtrabackup ‒ MySQL Enterprise Backup ‒ mysqldump ‒ mydumper/myloader
    42. 42. 42 3. MySQL – Scalability and High Availability
    43. 43. 43 4. MySQL at Box
    44. 44. 44 4. MySQL at Box • Box Database infrastructure • Typical /etc/my.cnf • Tools ‒ Box Weather Station ‒ Querykiller • MySQL Backups and Recovery @ Box • Team responsibilities • Checklists ‒ Changing active server ‒ Granting access ‒ Applying schema changes
    45. 45. 45 4. MySQL at Box (Ramping up someone) Best Way To Ramp up
    46. 46. 46 5. MySQL Automation https://code.launchpad.net/mysql-utilities https://code.launchpad.net/percona-toolkit https://forge.puppetlabs.com/puppetlabs/mysql https://github.com/mozilla-it/puppet-mysql
    47. 47. 47 5. MySQL Automation • DB Ops Script Framework ‒ Server tasks ‒ Clone replicas ‒ Promote slave to master ‒ Security tasks ‒ Grant management ‒ Auditing ‒ Sharding tasks ‒ Split shards ‒ Move shards
    48. 48. 48 6. Continued Learning
    49. 49. 49 6. Continued Learning
    50. 50. 50 ??? – http://dbareactions.tumblr.com When I call one of my former coworkers to see if they want the new DBA position
    51. 51. 51 ??? – http://dbareactions.tumblr.com When management asked me to fix the database scalability problem by adding an index
    52. 52. 52 In Closing • Don’t discount external hiring • Don’t discount internal hiring • Commit to knowledge sharing with others • Remember your learning curve – Help others through their own
    53. 53. 53 ?Email geoff@box.com Twitter @geodbz Tech Blog tech.blog.box.com Platform developers.box.com Github github.com/box
    54. 54. 54 References (seriously…check these out!) • http://www.jonathanlevin.co.uk/2010/07/where-have-all-mysql- dbas-gone.html • http://www.iheavy.com/2011/12/19/mythical-mysql-dba/ • http://www.iheavy.com/2012/02/26/top-mysql-dba-interview- questions-part-1/ • http://www.percona.com/live/mysql-conference- 2013/sites/default/files/slides/PLMCE13%20-%20Training%20- %20Cost%20or%20Investment%20%28final%29.pdf
    55. 55. 55 Image Credits • http://www.flickr.com/photos/atgeist/8663784742/sizes/o/ • http://www.flickr.com/photos/prodiffusion/4486297043/sizes/l/ • http://www.qwantz.com/fanart/dinocussing.png • http://www.flickr.com/photos/bparker321/7906708980/sizes/l/ • http://www.flickr.com/photos/96641479@N03/12309210613/sizes/o/ • http://farm4.staticflickr.com/3081/3202565643_c2d1a6f67b_o.jpg • https://www.flickr.com/photos/mhillyer/138766962/sizes/o/ • http://www.flickr.com/photos/tylermyeo/11618726115/sizes/o/ • http://www.flickr.com/photos/billkimandthecats/6338535597/ • http://www.flickr.com/photos/babykailan/2126901394 • https://www.flickr.com/photos/krupptastic/4988425044/sizes/o/ • http://danga.com/words/2007_yapc_asia/yapc-2007.pdf • http://albertolarripa.com/wordpress/wp-content/uploads/2012/06/mysql-replication-principle.png • http://www.cubrid.org/manual/91/en/_images/image38.png • https://www.flickr.com/photos/rfmcdonald/4301658325/sizes/o/ • https://www.flickr.com/photos/byte/120228210/sizes/o/ • http://www.nytimes.com/2013/07/16/business/last-car-plant-brings-detroit-hope-and-cash.html • https://www.flickr.com/photos/lens_envy/2212050922/sizes/o/ • https://www.flickr.com/photos/stvan4245/162066238/sizes/o/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×