Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TIAD - Is Automation Worth My Time?

5,549 views

Published on

Closing talk at TIAD Paris

Published in: Technology
  • Be the first to comment

  • Be the first to like this

TIAD - Is Automation Worth My Time?

  1. 1. IS IT WORTH MY TIME? AUTOMATION
  2. 2. @JRHUNT J. RANDALL HUNT
  3. 3. @JRHUNT J. RANDALL HUNT
  4. 4. AGENDA ▸Why automate? ▸When to automate? ▸How to automate? ▸Stories from the field
  5. 5. WHY THE HELL AM I DOING THIS?
  6. 6. EXAMPLE! sorry, I like big fonts!
  7. 7. WHY? FIND LARGEST FILE IN A DIRECTORY
  8. 8. WHY? FIND LARGEST FILE IN A DIRECTORY ▸ ls -S | head -1 ▸ takes about 4 seconds to type ▸ takes about 45 seconds to google
  9. 9. WHY? FIND LARGEST FILE IN A DIRECTORY ▸ ls -S | head -1 ▸ takes about 4 seconds to type ▸ takes about 45 seconds to google ▸ largest () { ls -S | head -1 } ▸ takes about 15 seconds to type... ▸ and then 2 seconds forever after
  10. 10. WHY? FIND LARGEST FILE IN A DIRECTORY ▸ ls -S | head -1 ▸ takes about 4 seconds to type ▸ takes about 45 seconds to google ▸ largest () { ls -S | head -1 } ▸ takes about 15 seconds to type... ▸ and then 2 seconds forever after
  11. 11. WHY? FIND LARGEST FILE IN A DIRECTORY
  12. 12. WHY? FIND LARGEST FILE IN A DIRECTORY ▸ find . -type f -print0 | xargs -0 du -h | sort -hr | head -1
  13. 13. WHY? FIND LARGEST FILE IN A DIRECTORY ▸ find . -type f -print0 | xargs -0 du -h | sort -hr | head -1
  14. 14. WHY? 5 HOURS OF FASCINATED CLICKING LATER
  15. 15. https://xkcd.com/1319/
  16. 16. https://xkcd.com/1319/
  17. 17. https://xkcd.com/974/
  18. 18. AWS SIMPLE SYSTEMS MANAGEMENT
  19. 19. WHEN SHOULD I DO THIS?
  20. 20. https://xkcd.com/1205/
  21. 21. https://xkcd.com/1205/
  22. 22. https://xkcd.com/1205/ PERFORMANCE QUALITY
  23. 23. WHEN? CHESS GAME WINNERS ▸ ~14 million games recorded in ASCII ▸ Read in files, aggregate winners, report stats ▸ Sounds like a job for:
  24. 24. AMAZON EMR
  25. 25. AMAZING PERFORMANCE -- ENTIRE DATASET COMPUTED IN JUST 20 MINUTES FOR <$10!!
  26. 26. HMM....
  27. 27. WHEN? CHESS GAME WINNERS ▸ ~14 million games == ~10 gigs ▸ Read in files, aggregate winners, report stats == grep / awk ▸ Sounds like a job for: THE COMMAND LINE!
  28. 28. find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/ Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw += $4; } END { print games, white, black, draw }' Thanks to Adam Drake for figuring this out! http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
  29. 29. 30 SECONDS
  30. 30. OK, TELL ME HOW!
  31. 31. HOW? I HAVE NO IDEA WHAT I'M DOING, BUT THIS IS WHAT I DO 1. Identify constraints ‣ Money, Business Goals, Time, Performance, Beauty 2. Pursue mitigating factors within constraints 3. Relentlessly pursue constraints until the laws of physics tell you to stop 4. Iterate
  32. 32. WHAT ARE THE ACTUAL CONSTRAINTS?
  33. 33. CONSTRAINTS HARDWARE ▸ AWS EBS gp2 volumes have max ~160MiB/s* ▸ AWS EBS io2 volumes have max ~320 MiB/s* ▸ Realistic max network performance of ~800 MiB/s* ▸ CPU performance varies on instance type but up to 8 terraflops per instance
  34. 34. CONSTRAINTS BUSINESS CONSTRAINTS (MONEY) ▸ How much money can we spend on this problem? ▸ How much time can we spend on this problem? ▸ Will a partially automated solution allow us to kick this problem down the road? ▸ Is it cheaper/easier/more accurate to hire humans?
  35. 35. CONSTRAINTS QUALITY CONSTRAINTS (AI VS HUMANS) ▸ How much better are humans at solving this?
  36. 36. CRAZY STUFF WE'VE AUTOMATED
  37. 37. BRINGING IT BACK HOME
  38. 38. CLOSING THOUGHTS INFRASTRUCTURE AS CODE == AUTOMATION AS CODE ▸ Chef, Puppet, Ansible or something similar is vital (you already know this) ▸ AWS CloudFormation lets you launch entire FLEETS and NETWORKS in moments. This system is free... ▸ Heterogeneity is the death of automation (cattle not pets)
  39. 39. CLOSING THOUGHTS STOP DOING THE THINGS YOU ALREADY KNOW HOW TO DO ▸ You already know how to run a reverse proxy -- write the code once and be done with it (Amazon API Gateway). ▸ You already know how to maintain your database (AWS RDS) set up protocols and be done with it ▸ You already know how to manage security and permissions, don't set it up yet again
  40. 40. CLOSING THOUGHTS ▸ Identify the biggest automation wins ▸ Pursue your automation by pursuing constraints ▸ Add humanity to the process ▸ Don't overthink it, but definitely think about it
  41. 41. AWS User Groups Lille Paris Rennes Nantes Bordeaux Lyon Montpellier Toulouse facebook.com/groups/AWSFrance/ @aws_actus
  42. 42. AWS Enterprise Summit – 27/10/2016, Paris http://amzn.to/1X2yp0i

×