BigDataInOperationsV8
•Download as PPTX, PDF•
0 likes•1,024 views
This document discusses how TripAdvisor uses big data to handle its large-scale operations. Some key points: - TripAdvisor operates the world's largest travel site, with over 280 million monthly visitors reviewing over 1 million restaurants and hotels. It handles up to 1.5 million hits per minute. - Big data plays a key role, with TripAdvisor storing 10 TB of site data, 2.5 PB of data in Hadoop, and 280 TB of logs each month. This data is analyzed to monitor site performance. - Traditional monitoring tools did not scale sufficiently for TripAdvisor's traffic levels. The solution was to implement an analytics system capturing 700,000 metrics per second, detecting anomalies
Recommended
More Related Content
What's hot
What's hot (20)
Similar to BigDataInOperationsV8
Similar to BigDataInOperationsV8 (20)
BigDataInOperationsV8
- 1. Benefits of Big Data: Handling Operations at Scale
- 2. Where it all began in 2000
- 3. More than 190 million reviews and opinions from travellers around the world 250,000+ attractions 680,000+ hotels 1,000,000+ restaurants Operates in 45 countries 315 million unique visitors per month
- 4. Where we are now: World’s largest travel site NORTH AMERICA 22% EUROPE 44% MIDDLE EAST & AFRICA 4% LATAM 8% ASIA PACIFIC 22% Source: comScore Media Metrix for TripAdvisor Sites, worldwide, August 2014 With nearly 280 million unique monthly visitors
- 5. Traffic and Infrastructure o From 500K to 1.5 million hits per minute o > 1000 Production Servers (Real, not Virtual) o Split across multiple data centres in the US o >2 TB of Compressed Log data per day Managed by a team of just 12 engineers
- 6. So where does Big Data fit In?
- 7. Big Data @ TripAdvisor o 10 TB of Postgres Site Data o 2.5 PB of Data in Hadoop o ~160 Large Hadoop Nodes o ~280 TB of Logs (last 7 months) on site o Redshift and Tableau for Ad hoc exploration of data o SSAS Data Cubes for static models
- 8. So what about Operations?
- 9. Challenges in Operations o Traditional Ops tools don’t scale well o A human can’t review 30K Graphs (30 metrics x 1000 Servers) o Aggregate Chart data is a hack and not very helpful o Tools record and present the data only, a human has to interpret it
- 10. Pray Example: Release Day Monitor 30K Cacti Charts Start Release Imagine 9+ releases a week with Cacti and Nagios… The tools are not designed for this!
- 11. So what’s the Solution?
- 12. Solution: Better Analytics o Change to more flexible technology o Measure everything! (~700K per second) o Interpret the data within 10 to 15 minutes o Tune! Remove as many false positives as possible o Alert – Page someone on unexpected changes If its worth doing, its worth measuring
- 13. What we built Web Servers Log Central Servers File Servers
- 14. Why Postgres? Analysis 2–3 TB of metric’s data Anomaly Detection 90+ days aggregate data Holding ~2TB of data
- 15. Anomaly Detection Capture what is happening now Compare it against data 1 day ago Compare it against what other Pools are doing Look at historical variance over 9 days Track statistically significant changes
- 17. Rollout Test for anomaly Release Day v2 Rollback Monitor key metrics Deploy new code to two pools No praying needed!
- 18. Results Roll back 1 release per week 15 minutes vs 3 hours Dramatic reduction in user facing issues Increasing number of releases Strict process on rolling back
- 20. What’s Next? Reduce False negatives Alert On performance or behaviour changes Visualise Data through dashboards Expand Set of metrics we’re alerting on Correlate system and application metrics
Editor's Notes
- Believe it or not, this is where it all began for TripAdvisor back in 2000 – in a tiny office over a pizza place in Needham, Massachusetts.
- World’s Largest Travel Site ~315 million unique visitors a month > 190 million reviews and opinions > 4.4 million accommodations, restaurants, and attractions Operates in 45 countries including China as DaoDao.com
- TripAdvisor is now the world’s largest travel site, with over 315 million unique visitors every month. We operate in 45 countries around the globe, and over 75% of our traffic comes from outside the U.S.
- From 500K to 1.5 million hits per minute Not including our CDN Traffic (many Gbps) > 1000 Production Servers (Real, not Virtual) Split across multiple data centers in the US >2 TB of Compressed Log data per day Managed by a team of 10 Software Engineers!
- Traditional Ops Tools don’t scale well Data “Trapped” in tools like Cacti A human can’t review 30K Graphs (30 metrics x 1000 Servers) 30 metrics per server is not enough! Automating the provisioning of a new Server with tools like Cacti is challenging (Magic numbers etc.) Aggregate Chart data is a hack and not very helpful The tools record and present the data only, a human has to interpret it
- Imagine 9+ releases a week with Cacti and Nagios Start Release Manually Monitor 30K Cacti Charts Pray! Ok the site is stable but is it still making money? High Stress / Time Consuming process Release Engineer is full time monitoring the progress of a release Why? Because our tools are not designed for this!
- Change to more flexible technology that can: Scale out, not up Be provisioned using automation (Puppet/Chef) Extensible via plugins. Measure everything! From 30 metrics to 530 per server Adding more all the time. Interpret the Data within 10 to 15 mins Anomaly Detection, whats changed? Tune! Remove as many false positives as possible Alert – Page someone on unexpected changes
- Collectd to gather key metrics from our servers Scribe (because it was already there) to stream the logs to a collector Rsync to ship logs to our Boston area Data center ETL the Data into Graphite and Postgres Graphite provides all the graphing/Dashboards Postgres is more flexible and allows us to do our anomaly detection and custom reporting Graphite was faster than Postgres for Drawing Graphs (Postgres was doing full table scans)
- Pushed a lot of analysis work to Postgres Way better than a time series database for the anomaly detection With SSD’s our Server can easily handle 2 to 3 TB of Metric’s data Holds a day of raw metrics data (~1TB) Over 90 days of aggregate data at an hourly resolution Currently holding ~2TB of data
- Capture what’s happening now Compare it against Data from 1 day ago Compare it against what the other Pools (collections of web servers)are doing now Look at the Historical Variance over the past 9 days If the change is greater by some statistically significant amount (x2 maybe x3 depending on the metric) Alert!
- Deploy new code to two small pools first Allow the code to soak for ~30 mins Monitor key metrics such as: Revenue per session Rate of 500’s 50 additional System stats (CPU, Memory, JVM Metrics) Internal Error logging (More than before?) Red/Green Status on Jenkins Green – Complete Rollout Red – Rollback Alert (Page) on any major deviations
- Roll back 1 Release per week Typically because of Higher Error Rates or Commerce Impact Typical Releases require 15 mins of effort vs 3 hours Dramatic reduction in user facing issues Increasing the Number of releases so we can push more features out on a daily basis Very strict on Rolling back No more heroic effort to “make it work” Dev will fix it for the next release
- Latency! Can’t wait for the Jobs to run to get the results Watching Projects like Apache Phoenix We didn’t need it YET As we grow will have to move to Hadoop or something like it For now Postgres is working well for us Aggregates will be archived in Hadoop As we expand our capacity planning more historical data is critical Did I mention we LOVE Postgres?
- Expand the set of metrics we’re alerting on Add Postgres specific metrics Add more Network Equipment Reduce the level of false positives or noise by using more sophisticated algorithms Move to Apache Kafka instead of Scrib Pub/Sub Messaging Framework Low Latency, HA, and Scalable Alert on performance/behavior changes on key site functionality More Dashboards for visualizing the data Correlate system and application metrics E.g. Response Time drops while IOWait increases on system X.