Empex - Monitoring Elixir Applications
•Download as PPTX, PDF•
2 likes•297 views
1) Monitoring availability is important for Elixir applications. Having applications in multiple AWS regions can provide 99.99998% availability but failover between regions is not instant due to DNS propagation delays. 2) 70% of outages are caused by changes to live systems so monitoring release processes is key. Logging requests with request IDs and aggregating data helps monitor availability and failures. 3) Measuring business and system metrics along with VM metrics provides full visibility. Bleacher Report's monitoring includes logging to Logz, metrics to Datadog, release tracking with Jenkins, and alerting with Opsgenie.
Recommended
More Related Content
Similar to Empex - Monitoring Elixir Applications
Similar to Empex - Monitoring Elixir Applications (20)
Recently uploaded
Recently uploaded (20)
Empex - Monitoring Elixir Applications
- 1. Monitoring Elixir Applications John Kelly, Sr. Engineer, Bleacher Report
- 2. Availability
- 3. Time Based Availability2 = uptime uptime + downtime
- 5. “Evidence for the long-term operational stability of the system had also not been collected in any systematic way. For the Ericsson AXD301 the only information on the long-term stability of the system came from a power-point presentation showing some figures claiming that a major customer had run an 11 node system with a 99.9999999% reliability, though how these figure had been obtained was not documented.”4
- 6. The Cloud
- 8. AWS EC2 SLA 99.95%5
- 9. What IF you were in multiple regions?
- 11. 2 Region AWS Availability 99.95% = .9995 Errors = .0005 .0005 * .0005 = 0.00000025 1 - 0.00000025 = 0.99999975 ~99.99998% (More than 6 nines almost 7!) ~ 6s downtime in a year7,8, but…)
- 12. Failover between Regions isn’t free DNS TTL + (Interval * Threshold)6 60s + (5 * 10s) = 110s Now we’re in the minutes range Let’s say 1 region fails 3 times in a year10 That would be 5.5 minutes
- 13. Uh oh, 5.26 minutes is 5 9s and we have more problems...
- 14. Some ISPs don’t honor DNS TTLs11 (so some long tail will hit the region with the outage)
- 15. I haven’t even touched the issue of data replication across geographically distant data centers (regions)
- 16. Auto Scaling from increased traffic takes around 5 minutes...
- 17. Sharks! Really12
- 18. Does any of this matter? It depends*
- 19. “Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability!”9
- 20. So where do we go from here?
- 21. Aggregate Availability3 = Successful Requests Total Requests
- 22. HTTP API Availability = 2XX + 3XX + 4XX 2XX + 3XX + 4XX + 5XX
- 24. B/R Elixir Services at Origin LB 99.26% Average of 9 April 15 - May 15
- 25. B/R Ruby Services at Origin LB 99.91% Average of 6 April 15 - May 15
- 26. Wait, that’s better than Elixir!
- 27. 70% of outages due to changes in a live system1
- 28. Step 1: If 70% of outages are caused by change, the best place to start monitoring is your change/release process
- 30. Step 2: Setup centralized request logging with X-Request-id Headers
- 32. Step 3: Measure aggregate availability
- 34. Step 4: Alerting
- 35. B/R’s Alerting & On Call Process
- 39. Step 6: Measure Latency - End to End and at the service level
- 42. Step 7: Monitor Business Metrics
- 44. Step 8: Monitor System Metrics
- 46. Step 9: Measure VM Metrics
- 49. Application: Use a logging system Use an aggregate data system Log Releases Use an alerting system
- 50. B/R’s Application of Monitoring Logz Datadog Jenkins Opsgenie
- 51. Remember: The application / writing the code for monitoring is the easy part
- 52. References 1 Site Reliability Engineering - p.10 2 Site Reliability Engineering - Equation 3-1 3 Site Reliability Engineering - Equation 3-2 4 http://erlang.org/download/armstrong_thesis_2003.pdf p.191 5 https://aws.amazon.com/ec2/sla/ 6https://aws.amazon.com/blogs/aws/route-53-health-check-improvements-faster-interval-and- configurable-failover/
- 53. References 7 https://en.wikipedia.org/wiki/High_availability 8 https://aws.amazon.com/blogs/developer/working-with-different-aws-regions/ 9 Site Reliability Engineering - p.25 10 (fairly conservative based on SLA it would be 4.38 hours for a region in a year) 11 https://www.godaddy.com/help/what-factors-affect-dns-propagation-time-1746 12 http://www.wired.co.uk/article/shark-cables
Editor's Notes
- Mention B/R and consumer team Lead on monitoring Mention the initial focus on availability Mention the shift then to how to monitor with examples from BR
- The first metric to discuss What is it? Is it a useful metric? Deep dive on availability but this same process should be applied to all metrics you rely on
- Conventional definition
- Erlang’s nine nines Where does that figure come from What does it mean I looked for the source
- This quote comes from the 2003 thesis that I believe introduced the nine nines Having seen the source how reputable is this claim?
- The name cloud invokes this nice simple visualization, in reality tc/ip in a virtualized environment is very complex Virtualized environments add a lot more complexity Talk about our AWS setup Multiple availability zones Load balancers
- When you enable an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone. If you register instances in an Availability Zone but do not enable the Availability Zone, these registered instances do not receive traffic. Note that your load balancer is most effective if you ensure that each enabled Availability Zone has at least one registered instance.
- Say over 3 nines This is a contrived worse case. Financially incentivised to not go lower. Of course it could drop lower and it could be much higher “Region Unavailable” and “Region Unavailability” mean that more than one Availability Zone in which you are running an instance, within the same Region, is “Unavailable” to you. Our elixir apps therefore are limited to 99.95% availability Can we do better
- We’re in a single region, but let’s explore going multi-region What kind of complexity does this introduce
- Route 53 Failover Routing Policy (active-passive failover) There are others Each AWS region is a completely independent stack of services, totally isolated from other regions. https://aws.amazon.com/blogs/developer/working-with-different-aws-regions/
- This assumes regions are independent which is what AWS states This downtime is the downtime where both systems are down simultaneously.
- Walk though this slide What does the interval mean Ttl is how long for the caching name servers to cache from the authoritative name server What does the threshold mean Why not threshold 1
- Quick slide
- Depending on persistence layer (kafka, cassandra) vs mysql, postgres
- Possibe mitigation vectors for this Burstable instances, container orchestration, serverless What else
- Possibe mitigation vectors for this Burstable instances, container orchestration, serverless
- Quick slide
- From Google, how reliable is 1 out of 100 or 1 out of 1000 cellular network fail
- Quick slide
- Google introduces Aggregate Availability in their SRE book Contrast with traditional uptime version Can you accurately measure time in a cloud system? VMs preempted for tens of milliseconds
- What we use at BR is a modified version
- Let’s revisit the architecture of 1 of B/R’s many elixir services We collect metrics from the LB nodes and send them to Datadog via cloudwatch WE CHEAT USAGE OF CDN AND THIS IS ORIGIN
- We didn’t spend 2 years switching to get worse! Quick slide
- Ruby Apps were under less active development and being sunsetted in favor Elixir apps Ruby apps aren’t changed except for exceptional cases This is a comparison of microservices The elixir apps see deploys daily or even multiple times a day
- Version Control - Software Version Control - Infrastructure Record releases with data time and releaser Planning for rollback We don’t deploy after 3pm*
- What is happening in my system?
- ELK Logz plug_logger_json Ecto_logger_json Bug me on the docs
- Is my system working correctly?
- Is my system working correctly?
- When something goes wrong? How do I know? The danger of false positives. Start conservative Alerting and avoiding mid night wake up calls
- Opsgenie 1 week shifts Different severities / different response times Anyone in the company can submit an issue not just automation We have a slack channel for these alerts An alert is then split out into a dedicated channel with stakeholders Goal is to triage and try to solve yourself but have permission to drag someone not on call in if necessary. Management also gets paged. Phone calls if not answered with 2 minutes
- When something goes wrong? How do I know? The danger of false positives. Start conservative
- Needed for capacity planning and figuring out user behaviors
- Needed for capacity planning and figuring out user behaviors
- There have been many studies on the importance of latency with regards to customer behavior. 100ms difference can have a measurable impact on revenue or satisfaction Percentiles not averages!!!! Clocks - how accurate can you actually measure latency?
- We tend to shove business metrics into a seperate analytics bucket? But what is analytics? Is that not business logic monitoring? How many user’s are using this piece of software, etc
- How many user’s are using this piece of software, etc
- CPU, open file descriptors, memory, disk space, network
- CPU, open file descriptors, memory, disk space, network These can creep up over time
- Check time Run Queue
- Run Queue 100% CPU - after a few minutes, run queue would increase to 20ish Infinite loop by missed pattern match case
- Run Queue
- You need to deep dive like we did with availability for all the metrics you monitor