Thinking DevOps in the Era of the Cloud - Demi Ben-Ari
•
0 likes•67 views
The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides. In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry. Of course how can we leave out the real enabler of the whole deal, "The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Thinking DevOps in the Era of the Cloud - Demi Ben-Ari
Similar to Thinking DevOps in the Era of the Cloud - Demi Ben-Ari (20)
More from Demi Ben-Ari
More from Demi Ben-Ari (20)
Recently uploaded
Recently uploaded (20)
Thinking DevOps in the Era of the Cloud - Demi Ben-Ari
- 1. 1 Demi Ben-Ari Co-Founder & CTO at Panorays
- 2. Demi Ben-Ari, Co-Founder & CTO @ Panorays ● Google Developer Expert ● Co-Founder of Communities: ○ “Big Things” - Big Data, Data Science, DevOps ○ Google Developer Group Cloud ○ Ofek Alumni Association In the Past: ● Sr. Data Engineer - Windward ● Team Leader & Sr. Java Software Engineer, Missile defense and Alert System - “Ofek” – IAF 2
- 3. ● A lot of (NOT) funny Jokes ● DevOps? WTF? ● Covering the world of cloud ● Process and Environment ● Monitoring pipeline solutions ■ Metrics, Datastores, Dashboards, Alerting ● Conclusions 4
- 5. 6 A breach to even the smallest supplier may cause a cyber typhoon in the industry.
- 6. 77 ~300 Employees PNI Data Breach – Photo Services Affected – By Thomas George BREACHED - Supplier - Evaluator
- 7. DevOps
- 8. Development Software Engineering Front End Backend QA UX Operations System Admin Database Admin Support
- 9. DevOps Cross-functional teams Operators automating systems Developers operating systems
- 11. Running environments ● —Development – Testing – Production ○ Don’t you need more? Be as flexible as you can ○ Ease of creation and Deployment ○ Identical or Not? ● Virtual Machines vs Bare Metal ● Configuration
- 13. ● ● ● ● ●
- 15. ● ● ●
- 16. ● ■ What is your application like? ■ What are the distinct pieces? ■ What data and / or logic must remain together? ■ Can some parts be asynchronous? ●
- 17. ● ○ ○ ○ ●
- 18. ● ● ● ● ●
- 20. Docker Environments ● Docker? ● Orchestration? VS ● Wait, What about local mode? ○ Minikube vs Docker Engine
- 22. 35 ● ○ ○ ● ●
- 23. ● ● ●
- 24. ● ● ●
- 25. 38
- 26. 39 ● Linux ○ Bash scripts ○ Crontab ● Automation via Jenkins ● —Continuous Deployment – with every GIT push
- 27. 43 ● ● ● ● ● ● ●
- 28. 44
- 29. 46
- 31. OS CPU Memory Disk Processes Java Application Server Database Web Server Load Balancer Users - Other Applications Monitorin g System UI Many times...all of this was on a single physical server! 48
- 32. Service A Queue DB Service B DBCache Cache DBService C Web Server DB Analytics Cluster Master Slave Slave Slave Monitoring System??? 49
- 33. 50
- 34. Basic Concepts ● Monitoring ● White-box ○ internals ● Black-box ○ behavior ● Dashboard ● Alert ● Root cause ● Node and machine ● Deploy ○ Any change to a service’s running software or its configuration. ● KPI - Key Performance Indicator
- 35. Monitoring Stack - Let’s fill in the blanks Alerting Metrics Collection Datastore Dashboard Data Monitoring Log Monitoring
- 36. Situations & Problems https://imgflip.com/i/1ap5kr http://kingofwallpapers.com/otter/otter-004.jp
- 37. MongoDB + Spark Worker 1 Worker 2 …. …. … … Worker N Spark Cluster Master Write Read MasterSharded MongoDB Replica Set
- 38. Cassandra + Spark Worker 1 Worker 2 …. …. … … Worker N Cassandra Cluster Spark Cluster Write Read
- 39. Cassandra + Serving Cassandra Cluster Write Read UI Client UI Client UI Client UI Client Web ServiceWeb ServiceWeb ServiceWeb Service
- 42. Monitoring Operation System Metrics ● What to measure: ○ CPU ○ Memory ○ Disk Space ● How to measure: ○ ○ ■ Nice and easy UI ■ Even the free account gives great tool ■ Alerting of thresholds
- 43. Some help from “the Cloud”
- 44. AWS’s CloudWatch / GCP StackDriver
- 45. Report to Where? ● We chose: ● ● Can correlate System and Application metrics in one place :)
- 47. Drivers to Datastores ● Actions they usually do: ○ Open connection ○ Apply actions ■ Select, Insert, Update, Delete ○ Close connection ● Do you monitor each? ○ Hint: ● Creating a wrapper in any programming language and reporting the metrics ○ Count, execution times, errors… ○ Infrastructure code that will give great visibility
- 49. ELK - Elasticsearch + Logstash + Kibana http://www.digitalgov.gov/2014/05/07/analyzing-search-data-in-real-time-to-drive-decisions/
- 51. 79 ● http://redash.io/ ● Open Source: https://github.com/getredash/redash ● Came out as one of many Open source tool by Everything.me ● Created and Maintained by Arik Fraimovich (You rock!) ● Written in Python ● Has an on-premise and hosted solution
- 52. 80
- 53. 81
- 54. 82 ● Syren - Open source ● Reporting to: ○ Email, Flowdock, HipChat, HTTP, Hubot, IRCcat, PagerDuty, Pushover, SLF4J, Slack, SNMP, Twilio
- 55. Summary - Monitoring Stack Alerting Metrics Collection Datastore Dashboard Data Monitoring Log Monitoring
- 58. 86 ● Correlating Application and System metrics!!!! ● Ask the right monitoring questions -> answer with Dashboards ● KISS - simple is key, what’s hard, we tend not to do at all ● Alert about what you can actually react to ○ (And to the relevant person) ● Measure whatever you can ○ only way to know if you’re improving ● Monitor your business KPIs too
- 59. 87 ● DevOps requires Culture also ● Take responsibility ○ Production is everybody's responsibility ● Leverage the Cloud and the possibilities of being Flexible