Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Zero Downtime App Deployment using Hadoop

398 views

Published on

Zero Downtime App Deployment using Hadoop

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Zero Downtime App Deployment using Hadoop

  1. 1. ZERO DOWNTIME APP DEPLOYMENT USING HADOOP Hadoop Summit 2016 – San Jose Heman Duraiswamy Solutions Engineer Wei Wang Data Scientist Jun 30, 2016
  2. 2. Agenda  Introduction  Our Story that ends with “Happily ever after” “Zero downtime app deployment”  Reference Architecture  Demo
  3. 3. Introduction: Why?  Expedia revenue in 2015 == $6.67 B  >18.2MM/day (or) >$750,000/hr  Cost of 15 min deployment window and having one deployment every other week??  ~5MM $$  Same figure for Amazon?? - $$80MM  And offcourse, customer trust & confidence!
  4. 4. Introduction: How? Innovate (Tools, technology and architecture) Monitor (In-real time) React (Actionable Intelligence)
  5. 5. Zero downtime app deployment using Hadoop
  6. 6. Our Story: Once upon a time…
  7. 7. Our Story: Once upon a time…
  8. 8. Our Story: Once upon a time…
  9. 9. Our Story: Once upon a time…
  10. 10. Our Story: then they evolved Rolling deployment  Near Zero downtime deployment  Hampers Innovation  Operational over-head
  11. 11. End game… Using HADOOP
  12. 12. Innovate -- Micro services Header module Search module localDest module topDeal module Loyalty module
  13. 13. Deploy-at-will -- Continuous Delivery Continuous Integration Jenkins Build Glu Deployment
  14. 14. Learn & Succeed (or fail) Canary Deployment a/b testing Operational monitoring 24*7 Instant feedback
  15. 15. Reference Infrastructure layout Header module Search module localDest module topDeal module Loyalty module Server001 Server002 Server003 Server004 Server005 Server006 Server007 Server012 Server013 Server014 Server008 Server009 Server010 Server011 Use case 1: server in bad state – meaning serves higher proportion of 404 &503 pages Use case 2: server is slow – takes longer to process requests Use case 3: Deploy bad app version – serve high propotion of application CRIT and ERROR messages
  16. 16. Reference Architecture HDF (nifi) Kafka Storm Topology Application log files Access log files Server s Server s Solr Hive HDFS Banana view Ansible script
  17. 17. DEMO: Log prep
  18. 18. DEMO: HDF flow
  19. 19. DEMO: Storm Topology Storm Kafka Bolt Aggregation Bolt Timer Bolt Calculatio n Bolt Solr Bolt Hive Bolt
  20. 20. DEMO: Storm Topology
  21. 21. DEMO: Storm Topology
  22. 22. DEMO: Analytics Results in Banana
  23. 23. DEMO…
  24. 24. THANK YOU! hduraiswamy@hortonworks.com @hduraiswamy https://www.linkedin.com/in/hemananthan wwang@hortonworks.com https://www.linkedin.com/in/wei-wang-0957902 https://github.com/heman-duraiswamy/ZeroDowntimeDeployment https://github.com/ww2265columbia/hadoopsummit2016/ Heman Duraiswamy Solutions Engineer Wei Wang Data Scientist

×