C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016
•Download as PPTX, PDF•
3 likes•1,010 views
Netflix stores 98 percent of data related with streaming services: right from bookmarks, viewing history to billing and payment information. These services / applications simply desire highly available and scalable persistence solution to keep themselves running efficiently in a normal and disastrous situation. How does Netflix plan for capacity for it's new as well as existing services? In this talk, Arun Agrawal, Senior Software Engineer and Ajay Upadhyay, Cloud Data Architect @Netflix will talk about the capacity planning and capacity forecasting in cassandra world. We will take you through the science behind forecasting the short and long term usage and auto-scaling adequate capacity well before C* clusters reach their limit. This guarantees highly scalable and available persistence solution meeting our SLAs @ Netflix. About the Speakers ajay upadhyay Senior Database Engineer, Netflix Responsible for persistent layer at Netflix, part of CDE [Cloud Database Engineering] team. Working with application team, suggesting and guiding them with the best practices for various persistent layers provided by CDE team. Arun Agrawal Senior Software Engineer, Netflix Arun Agrawal is part of Cloud Database Engineering where they provide CAAS (Cassandra as a service). Ensuring smooth operations of service and finding innovative ways to reduce the management overheads of having CAAS.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016
Similar to C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016 (20)
More from DataStax
More from DataStax (20)
Recently uploaded
Recently uploaded (20)
C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016
- 1. Capacity Forecast @ Scale CDE, Cloud Database Engineering Netflix.
- 2. ● CDE, Cloud Database Engineering ● Providing data stores as a service ○Cassandra, ○ Dynomite, ○ Elasticsearch and RDS Ajay Upadhyay Cloud Data Architect @ Netflix Arun Agrawal Sr. Software Engineer @ Netflix Who are we?
- 3. ●Cassandra @ Netflix ●Cassandra footprint ●Capacity planning lifecycle ●Forecasting the capacity ●Q and A Agenda
- 4. • 98% of streaming data is stored in Cassandra • Data ranges from customer details to Viewing history / streaming bookmarks to billing and payment Cassandra @ Netflix
- 7. Capacity Planning • Able to predict – Current usage and available capacity – Resources needing upgrade – Life cycle of current configuration – Appropriate configuration for new and existing App/Service • Optimize – Under or over utilized resource – Increased business productivity
- 8. Capacity Planning Avoid: • Impact on Business • No service or SLA disruption • Un-planned maintenance • Firefighting
- 9. Life Cycle Capture Requirement Requirement Analysis/feasibility Proxy or Simulate Requirement Monitoring / Trending New / Increased traffic Optimization
- 10. Capture Requirement – IOPs and SLA – Maintenance overhead – Failover – Access pattern
- 11. IOPs and SLA Questions Response Read OPS/sec [avg, peak] 5k - 10k Read Latency requirement 95th - 20ms 99th - 100ms Write OPS/sec [avg, peak] 1k - 2k Write Latency requirement 95th - 20ms 99th - 100ms Num Columns / Row 100 Avg col size / or avg row size 64k Num of rows 100 Mil TTL [life Cycle of data] 365 Days Data store C* Gutenberg publisher service Gutenberg publisher serviceRead Write
- 12. Maintenance Overhead Repairs / Compactions Y/N Node replacement Y Backup - Full / Incrementals Y/N Type Response
- 13. Failover Region Failover Y/N SLA in case of region failover Y/N Questions Response
- 14. Access Pattern Questions Response Read Point read All row readers Column slices Write Part existing row New rows
- 15. Proxy/Simulate Traffic – Proxy existing traffic – Simulate traffic –NDBench – Generate actual / synthetic traffic before final deployment using app
- 16. Optimization • Cache - Application level - Fronting cache engine before C* - Stagger R - W operations if possible
- 17. Cluster Sharding
- 18. Trend Analysis Continuous monitoring / trending on usage pattern
- 19. New / Increased Traffic Capacity planning cycle begins Capture Requirement Requirement Analysis/feasibility Proxy or Simulate Requirement Monitoring / Trending New / Increased traffic Optimization
- 21. Arun Agrawal Sr. Software Engineer
- 22. Demo
- 26. Pain Points • No support for complex relationships • Hardware failure could fail leading to false positives
- 27. Winston • Bridge between atlas and oncall • Complex relationship modeling between metrics • Reduce false positives • Auto remediation platform
- 28. Lesson Learnt • It might be already too late to fix the system. • Reactive than proactive
- 29. Requirements • Show us trend for the clusters. • Warn us of what is coming if trend continues. • Give us time to scale their cluster
- 32. Aggregation • Daily • Instance Level • Cluster Level •Instance Failures •Adding capacity over days
- 33. Growth Criteria f(x) of – Subscriber – Netflix content – # Viewing Sessions
- 36. ARIMA – AR •Regression on prior values –I •Data values are replaced with (x(i) - x(i-1)) –MA •Linear combination of error terms
- 43. Q & A
- 44. You may not control all the events that happen to you, but you CAN decide not to be reduced by them. - Maya Angelou
Editor's Notes
- For business to delivery - quality service to meet and exceed customers expectations - need right capacity and resources
- Work with app team -
- Cluster / ring size 9 nodes 300 nodes 10k instances - right from ms to i2 to d2 instances
- Cluster / ring size 9 nodes 300 nodes 10k instances - right from ms to i2 to d2 instances
- Current usage and available capacity Resources needing upgrade Cost-effective configuration - just vertical upgrade - no need to add nodes or increase ring size Life cycle of current configuration - when cluster will run out of resources Appropriate configuration for new and existing App/Service
- Analysis – In the analysis phase data collected in the Monitoring phase and analyze them to find problems and evaluate the quality of the deployment. Optimization - stagger R - W
- Repair overheads - amount of writes and data size - Entropy in the system No repair - quorum R and W - aggressive ttl data Compactions - implicit - compaction-threshold - 2 - GC grace period more aggressive Node replacement - replace early if node is still healthy - bootstrap from neighboring nodes Backup overheads - throttle if creates a big bottleneck on network
- Read - full row or column slices Write - full row or few columns at a time STCS - size tiered LCS - Leveled compaction straregy Aggressive TTLs - few hours to few days Variable Payloads - 1k - 1m range
- Model/Simulate traffic using NDBench for new requirement
- Cache for aggressive latencies Cluster sharding for high and low latency required data
- Continuous monitoring to keep track of usage pattern Useful for predicting it’s clusters life For proxing for traffic similar to one captured here
- New requirement or change in existing traffic capacity planning cycle begins
- Let’s see how we really manage CAAS at netflix. A short video where we get notified on slack about the cluster which may reach its capacity and then we do some investigation, talk with app teams, warn them, take proper steps (if required) or increase the capacity of the cluster.
- Pretty cool and neat stuff. Right. So let’s see how we do this? Its actually some human doing the analysis and posting on slack. No, let’s see what is the science behind this but before we get there we need to understand netflix ecosystem.
- Pretty cool and neat stuff. Right. So let’s see how we do this? Its actually some human doing the analysis and posting on slack. No, let’s see what is the science behind this but before we get there we need to understand netflix ecosystem.
- Pretty cool and neat stuff. Right. So let’s see how we do this? Its actually some human doing the analysis and posting on slack. No, let’s see what is the science behind this but before we get there we need to understand netflix ecosystem.
- Every instance in netflix uploads all the telemetry information to Atlas. Atlas is very useful tool as it combines all the raw inputs from multiple instances based on availability zones, region, application etc. Really handy tool to find performance issues, debug, triage and have aggregated view of app. One of the multiple features of atlas is the ability to set a threshold and duration for a metric which when tripped can page on-call.
- But let’s face it, if you paged a person based on single metric being tripped, is it right? It is NEVER a single metric which can tell you about the cluster. There are hooks in atlas where you can define basic relationship between metrics but again, it is always complex relationships which we are after. In addition to that, there could be false positives being reported because let’s face it, we are hosted in AWS and failure of machines is not a exception but norm. When machines fail they don’t always report metrics leading us to believe we are in false positive zone.
- So to reduce oncall pain, we needed a middle layer logic which could sit between atlas and oncall, where we could provide complex relationship between multiple metrics, add context, do basic triage and remove those false positives. We brought “Winston” which is based on stackstorm which does all this and provides a great UI to work with. Winston has native integration with Atlas and thus you can write some python code which will be triggered when Atlas fires the event. This combination of Atlas and Winston greatly reduces false positives for oncall.
- But wait, now we were getting paged which is accurate but how can we save the cluster? It might be already too late. It is more reactive than proactive.
- How can we build the system which tells us that system might get under pressure if the trend continues. If we have such system, then we are better prepared for what is about to come. Chances of us getting paged at middle of night for degraded performance or latencies alert can be reduced drastically if not avoided completely. This is where we started to think about a system which could predict the future of a cluster.
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!
- In netflix, atlas metrics are pushed to big data platform which is netflix’s data warehouse. Here all the metrics are stored and all analysis can run here.
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!
- RSS - RESIDUAL SUM OF SQUARES RMSE - ROOT MEAN SQUARED ERROR
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!
- With such a system, we can take a break and have more confidence on our system that it will be able to handle what is about to come. Again this is to say if the “trend” continues, if you try to do something which is not expected we still might have issues where we would be increasing the capacity of the cluster at the very last moment. So we set the expectations that this is not magic ball which will solve all the problems but it will surely help you find the problem areas before they happen. Don’t expect the clusters to auto-scale when you suddenly add another 100m subscribers!