Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dev Ops without the Ops

2,772 views

Published on

Do you need Ops in your new startup? If not now, then when? And...what is Ops?

Learn how to scale ruby-based distributed software infrastructure in the cloud to serve 4,000 requests per second, handle 400 updates per second, and achieve 99.97% uptime – all while building the product at the speed of light.

Unimpressed? Now try doing the above altogether without the Ops team, while growing your traffic 100x in 6 months and deploying 5-6 times a day!

It could be a dream, but luckily it's a reality that could be yours.

Published in: Internet

Dev Ops without the Ops

  1. 1. Konstantin Gredeskoul
 CTO, wanelo.com DevOps without the “Ops” A fallacy? A dream? A ________? @kig @kigster How Wanelo handles thousands of writes per second with 99.97% uptime without an operations team @kig
  2. 2. Proprietary and Wanelo is the digital mall of the future, and a place to find the most amazing products.
  3. 3. What are you running on? No really, what’s your stack? Are you on Mongo? No!?!?!?? or… You running ruby? WTF? It’s slow! You are running Erlang? WTF? It’s in Swedish! etc. People often ask…
  4. 4. Backend Stack & Key Vendors ■ MRI Ruby, jRuby, Sinatra, Ruby on Rails ■ PostgreSQL, Solr, redis, twemproxy
 memcached, nginx, haproxy, pgbouncer,
 elastic search ■ Joyent Cloud, SmartOS, Manta Object Store
 ZFS, ARC Cache, superb IO, SMF, Zones, dTrace, humans ■ DNSMadeEasy, MessageBus, Chef, SiftScience ■ LeanPlum, MixPanel, Graphite analytics ■ AWS S3 + Fastly CDN for user / product images ■ Circonus, NewRelic, statsd, Boundary, 
 PagerDuty, nagios, SumoLogic 
 monitoring, alerting, error reporting
  5. 5. Proprietary and How much traffic does your app get? • If you are building an internal web-site in Rails you’d be lucky to get 100 RPMs – your users are only a limited set of employees • Semi-Popular sites with up to a few hundreds of concurrent users can expect about 1K-2K RPM • When you cross 100K RPM mark, you joined the “small big boys” :) • When you are Pinterest, Facebook or Twitter… You are probably doing 1-10M RPMs
  6. 6. So what is this talk about? • Review Operations, DevOps, and the Cloud, and how the new technologies are changing the landscape • Learn some key points and patterns that dramatically reduce stress and pain associated with running a site, particularly ruby and/or rails • Discuss if modern startups really need a dedicated operations team, and if so – at what point?
  7. 7. Let’s start with the basics DevOps
  8. 8. Proprietary and What the heck is DevOps? • “Today, many organizations are confused on what DevOps means for them..”[2] 1. WikiPedia article on DevOps
 2. FORRESTER: “Eliminate DevOps Myths With Situational-Awareness-Based Performance”. John Rakowski, October 10, 2014 • DevOps is a software development method that stresses communication, collaboration, integration, automation and measurement cooperation between software developers and other information-technology (IT) professionals. [1]
  9. 9. Proprietary and “…Efficient teams are deploying code 30 times more frequently with 50 percent fewer failures in 2014…” [3] “…DevOps practices correlate strongly with high organizational performance” [3] 3. Source: PuppetLabs “State of DevOps Report”, 2014 DevOps however, works…
  10. 10. Traditional “Heavy” Agile • Traditional Ops responsibilities were often in conflict with product development: stability versus change. Product Dev QA OperationsProduct Dev QA Operations
  11. 11. Traditional Operations • Uptime, stability and reliability • On-call, fixing site at night • Backups and disaster recovery • Security, patching, OpenSSL :) • Hardware • Networking • Colocation / DC
  12. 12. “The Cloud” changed things • Uptime, stability and reliability • On-call, fixing site at night • Backups and disaster recovery • Security, patching, OpenSSL :) • Hardware • Networking • Colocation / DC
  13. 13. So the Cloud is a big part of what makes DevOps possible
  14. 14. Let’s talk about a simpler and more friendly way to build and deploy software.
  15. 15. Early Company Goals (based on Wanelo) • Maximize iteration speed • Practice “aggro-agile”™ • Scale up as we go, keep the app fast • Break things, learn, move on • Enable, empower and inspire our team • Remain in control of our infrastructure
  16. 16. And while moving really fast… We just never hired Ops But we did hire several brilliant engineers who actually enjoyed infrastructure / platform work. Except they approach it like … code.
  17. 17. Not having Ops meant • We had to deploy our app to the cloud, and learn how to provision the nodes we needed, as well as: • How to provision load balancers and app servers • How to configure new Solr masters and replicas • How to install and tune PostgreSQL databases • memcaches, redis shards, twemproxy, haproxy
  18. 18. Fast forward to today • 100% cloud hosted (Joyent Cloud) • 100% automated (Chef) • 10,000% traffic growth in 6 months and survived • 99.97% uptime (without trying very hard) • on call engineers get 1-2 pages per week • 80% of engineers are on call rotation, including iOS & Android developers
  19. 19. Still no “Ops” team, but plenty of Ops work
  20. 20. How?
  21. 21. 1. Automation and Deployment • Infrastructure is a first class citizen • Pairs deliver user stories which include automation • Did I mention we pair program? It rocks! • We run Chef continuously in production • I want to trust my tools, and if they break, fix them • Partition staging and production environments
  22. 22. Incremental Deployment • Roll code out everywhere, restart 2% of servers • Watch errors, latency, other anomalies • When satisfied continue rolling all servers • Ensure old and new code can co-exist • Ensure no “drop/rename” migrations happen on live tables • Ensure no exclusive locking migrations (eg. create index concurrently)
  23. 23. 2. Fault tolerant infrastructure • Ensure aggressive client timeouts • Achieving fault tolerance today is much cheaper than ever before! It’s a crime not to do it :) • Put haproxy in front of everything, literally • Stateless services only • Put makara, twemproxy, Dalli in front of database, redis and memcached
  24. 24. Let’s look at a couple of recipes for resilience Resilience keeps you sleeping at night
  25. 25. Where is everything? HAProxy + Chef Search + Stateless App talks to
 http://127.0.0.1:8000
 http://127.0.0.1:8001 App HAProxy Backend 1 Backend 2 Solr Web Service Backend 2 ElasticSearch Virtual Zone / Server
  26. 26. This pattern allows us to have one place that knows about everything else, in Chef
  27. 27. What the hell Makara? • Makara is a simple database routing tool for ActiveRecord that has been in production on Wanelo and TaskRabbit for years • https://github.com/wanelo/makara (PostgreSQL) • https://github.com/taskrabbit/makara (MySQL)
  28. 28. Proprietary and • Was the simplest library to understand, and port to • Worked in the multi-threaded environment of Sidekiq Background Workers • automatically retries if replica goes down • load balances with weights • Was running in production
  29. 29. Replicate everything that replicates App HAProxy Backend 1 Backend 2 Solr Replica Backend 2 Solr Replica Solr Replica Solr Master Web / API Requests Background WorkerQueue reads writes
  30. 30. App HAProxy Backend 1 Backend 2 Solr Replica Backend 2 Solr Replica Solr Replica Solr Master Web / API Requests Background WorkerQueue Degraded State, but still up! Many replicas can be down reads writes
  31. 31. Replicas are great because they are easy to add and often ok to ignore when they die/reboot/etc.
  32. 32. Don’t buy an expensive load balancer Load Balancer haproxy nginx Load Balancer haproxy nginx 200.200.234.145 200.200.234.146 example.com App Server App Server App Server App Server App Server App Server
  33. 33. You can build a decent one with DNS App Server App Server App Server Load Balancer haproxy nginx App Server App Server App Server Load Balancer haproxy nginx DNS Provider pingping 200.200.234.145 200.200.234.146 DNS auto-failover is offered with some enterprise DNS services, e.g. from DNSMadeEasy
  34. 34. When LB goes down, it is removed from the DNS pool App Server App Server App Server Load Balancer haproxy nginx App Server App Server App Server Load Balancer haproxy nginx DNS Provider pingping 200.200.234.145 200.200.234.146 It works pretty well
  35. 35. It works pretty well Load Balancer haproxy nginx Dead Load Balancer 200.200.234.145 200.200.234.146 DNS Provider ping App Server App Server App Server App Server App Server App Server example.com This works best with a short TTL Configure LBs in pairs, as the others failover, to account for network partitioning When LB goes down, it is removed from the DNS pool
  36. 36. This pattern allows us to tolerate reboots and maintenance with minimal effect on our users
  37. 37. Failover to the overflow pattern Two queues: large primary, small secondary The primary distributes jobs to a large set of specialized workers, assigned to specific queues App HAProxy Primary Backend 1 Failover Backend 2 Primary Background Workers Redis Primary Queue Redis Failover "Overflow" Workers The failover queue has only a small number of overflow workers, but they will accept any work
  38. 38. During spikes in traffic, this pattern allows our application to continue enqueuing jobs when the primary is overwhelmed This is useful in situations when you can’t easily round robin between multiple shards. 
 
 Example: Sidekiq with a “Unique Job” extension.
  39. 39. • Some tools allow alerting on the first derivative of an observed metric. • This is what we want: rapid drop (or increase) in a key metric to generate an alert. 3. Alert only on what’s important • Nagios is great for visibility • Not great for knowing when to drop everything because the site is on fire
  40. 40. • We never page on “host down”
 
 Because, who cares?
 The host is likely redundant, and will be back. 
 
 …Probably. Alerting examples • We only page for things like “sudden drop in product saves per second”, or a spike in error rate, etc. • Monitoring / alerting tool Circonus supports this
  41. 41. 4. Obsessive monitoring • Modern tools offer unprecedented visibility • Real time application monitoring • Real time business stats monitoring • Real time network monitoring • Dashboards, TV Monitor, alerts • Real time, real time, real time.
  42. 42. Systems Status: Dashboard Monitoring & Graphing with Circonus, NewRelic, statsd, nagios
  43. 43. 5. Cloud vendor is your partner • We get phenomenal customer support from Joyent • Our Cloud Partner, in a way, is our Ops • Joyent is innovative in that they develop and run their own cloud stack: from the OS layer (SmartOS) to the data center management software • They offer a unique option to take our “cloud” in- house when that time comes
  44. 44. 6. DevOps, really, is just code • Hire folks who write code, so that they don’t have to repeat the same task twice • Everyone will be happier that way.
  45. 45. So here is how to reduce stress! 1. Insist on 100% automation 2. Deploy fault tolerant patterns wherever possible 3. Page only on what’s important to the business 4. Monitor everything else obsessively 5. Choose a cloud provider that can be your partner 6. Infrastructure work is software engineering
  46. 46. Thanks! slideshare.net/kigster
 github.com/kigster
 github.com/wanelo github.com/wanelo-chef 
 wanelo technical blog building.wanelo.com Proprietary and @kig @kig @kigster

×