Uploaded byHarish Ganesan
Cloud Connect 2013- Lock Stock and x Smoking EC2's
The presentation details a case study on using AWS cloud services for analyzing large volumes of unstructured data from mobile advertising company logs, emphasizing the reduction of costs through innovative data transfer, storage, and analysis methods. It highlights the successful implementation of Amazon EMR and spot instances for significant cost savings and improved efficiency in processing big data. Key learnings include the importance of automation, file management, and leveraging elasticity for scalable operations.
![[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-themodernstackbuildingwebaiapplicationswithserverless-251124030844-388cf04f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-agenticaiarchitectureredefiningsystemcommunication-251124030838-e6c70cc2-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels](https://cdn.slidesharecdn.com/ss_thumbnails/md-exploringappleson-devicefoundationmodels-251124030840-d690542c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
Cloud Connect 2013- Lock Stock and x Smoking EC2's
- 1.
- 2. P1) This Presentationis P2) Strongly Inspired by “Guy Ritchie” Movies P3) Disclaimer : All images are downloaded from internet. If you find any of the content / images violating copyright, please let me know and I will act upon it immediately
- 3. AGENDA • Case • Challenge •Solution • Learning's • About us
- 4. Case Cigarette smoking isinjurious to health • Mobile Advertising company, USA • Forbes 1000 clientele • TB’s of unstructured data -> Big Data Problem
- 5. Lock • Hourly ~1TB • CDN Logs • Text Files • XML Files • Geo data files • Server logs • DB records
- 6. STOCK • Reduce thecost leakage • How to Save $$$ ?
- 7. Challenges • Daily (wasOK), Monthly (Pain) and Historical analysis ( almost dead ) • How do we Transfer, Store, Analyze and Share ? • How to optimize costs at this scale ?
- 8. Solution Cigarette smoking isinjurious to health • Use AWS Cloud for hosting Analytics module • Amazon EMR for unstructured Log Analysis • Automation using Scripts, Java code and other tools
- 9. Social / 3rd Party Feeds/Cloud Logs Stage1: Data Transfer • Tsunami UDP • ~1TB un compressed logs every hour • High bandwidth EC2’s for Tsunami UDP • Other Popular Options : • Aspera • AWS Import/Export • WAN optimization • AWS Direct Connect
- 10. Amazon S3 Logs Stage 2:Storage • Amazon Web Services Building Block – S3 • Scalable Object Store • Inherently Fault Tolerant • ~2 TB of compressed logs every day • S3 RR option for intermediate outputs • Amazon Glacier for archivalSocial / 3rd Party Feeds/Cloud
- 11. Amazon S3 Elastic MapReduce Logs Stage 3:Analyze • Elastic MapReduce Service of Amazon • Minimal Setup time • Log Analysis • ~2000 mappers / 750 reducers @ peak • ~250 m1.xlarge task nodes (1000 cores, 3750 GB RAM) @ peakSocial / 3rd Party Feeds/Cloud
- 12. • Amazon EMRis great • But adding Spot EC2 is super cool Wait !!!
- 13. What is AmazonSpot ? 13 • Time-flexible, interruption-tolerant tasks • Bid Price & Spot Price • M1.xlarge Price Comparison • $0.480 per Hour – On Demand • $0.052 per Hour - Spot • You will never pay more than your maximum bid price per hour •Spot Instance may be interrupted • If interrupted you will not be charged for any partial hour of usage. (*Free)
- 14. Spot Bidding Strategies 14 •Justabove Spot Price •Between Spot Price & On Demand Price •On Demand Price •Above On Demand Price
- 15.
- 16. Amazon EMR withSpot Instance Project Master Instance Group Core Instance Group Task Instance Group Long-running clusters on-demand on-demand Spot Cost-driven workloads spot spot Spot Data-critical workloads on-demand on-demand Spot Application testing spot Spot Spot
- 17. Amazon S3 Elastic MapReduce Social / 3rdParty Feeds Logs Stage 4: Custom EMR Manager • We created a Custom EMR Manager • Choose spot based on: • Past price trend intelligence • Choose AZ based on Current Market Prices • Choose between Large vs Extra Large • Spot Pricing Strategy : • Set Spot Price = On Demand Price • Over board <20% of On Demand Price at times • Dynamic Sizing the Core / Task nodes • Dynamic EMR Cluster creationCustom EMR Manager
- 18. Some Spot UseCases 18 • Analytics & Big Data • Scientific computing • Web crawling • Financial model and Analysis • Testing • Image & Media Encoding 66 % savings 50 % savings 57 % savings
- 19. Learning • Spot +On demand EC2 is a deadly combination for cost savings • Every millisecond matters in MR – Tune your code • Merge Files – Bigger ones are better for processing
- 20. More Learning … •Custom Job Manager was designed by us • 1 File Per Mapper was better for our case in AWS • Understand the performance constraints of AWS and work with it • Compress data : Both storage and transit(.LZO & Snappy)
- 21. Continues… • Keep configurationdata in local memory or Amazon DynamoDB • Reducers split files suitable for next job mappers • Elasticity – Increase/Decrease Task nodes • Elasticity – Create new EMR Clusters matching the Logs (Core + Task)
- 22. Value • ~56% costsavings from pure On-Demand model for Core+ Task Nodes • Automation vastly reduced Labor cost ( initial + on going) • Customer CXO’s were happy
- 23. • AWS PremiumPartner • Solution Experts in • Cloud Computing • Big Data • Identity Management About US
- 24.