AWS re:Invent 2016: Fanatics: Deploying Scalable, Self-Service Business Intelligence on AWS (BDA207)

206 views

Published on

Data is growing at a quantum scale and one of challenges you face is to enable your users to analyze all this data, extract timely insights from it, and visualize it. In this session, you learn about business intelligence solutions available on AWS. We discuss best practices for deploying a scalable and self-serve BI platform capable of churning through large datasets. Fanatics, the nation’s largest online seller of licensed sports apparel, talks about their experience building a globally distributed BI platform on AWS, that delivers massive volumes of reports, dashboards, and charts on a daily basis to an ever growing user base. Fanatics shares the architecture of their data platform, built using Amazon Redshift, Amazon S3, and open source frameworks like Presto and Spark. They talk in detail about their BI platform including Tableau, Microstrategy, and other tools on AWS to make it easy for their analysts to perform ad-hoc analysis and get real-time updates, alerts, and visualizations. You also learn about the experimentation-based approach that Fanatics adopted to fully engage their business intelligence community and make optimal use of their BI platform resources on AWS.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
206
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
31
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

AWS re:Invent 2016: Fanatics: Deploying Scalable, Self-Service Business Intelligence on AWS (BDA207)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Rahul Bhartia, Amazon Web Service, Principal Solutions Architect Amit Jain, Fanatics, Sr. Manager - BI Platform and Reporting December 1, 2016 Deploying Scalable, Self-Service Business Intelligence on AWS featuring
  2. 2. What to Expect from the Session • Learn about various Business Intelligence (BI) solutions on AWS • Hear from Fanatics about their scalable & elastic BI stack on AWS • What this session is not about • Various BI solutions and their features
  3. 3. BI on AWS Amazon QuickSight AWS Big Data Competency partners
  4. 4. Get Started Quickly • Amazon QuickSight – Get started today!!! • Managed on AWS • Tableau Online, Microstrategy Cloud, ChartIO, WingArc1st • AWS Marketplace • Tableau Server, Tibco Jaspersoft, Microstrategy, Looker
  5. 5. BI workloads on AWS Cloud Self-service Scalable
  6. 6. BI workloads on AWS Cloud Self-service Scalable
  7. 7. Making BI self-service on AWS Get started today! https://quicksight.aws/ Managed offerings vs. self-managed Amazon QuickSight AWS Big Data Competency partners
  8. 8. Managing yourself • Custom integration • AWS CloudFormation • Automate Tableau - https://github.com/tableau/server-install- script-samples • AWS Marketplace 1-click • Tibco Jaspersoft • Looker
  9. 9. Self-service Scalable BI workloads on AWS Cloud
  10. 10. Scale – Infrastructure • Scale-out • Cluster or Distributed • Scale-up • Leverage bigger or more specific instances • Scale-with • Scale the underlying data-store
  11. 11. Tableau Online – Scaling on AWS Tableau online on AWS
  12. 12. Scale – Data Create an In-Memory aggregation - Extracts or Cubes or Cache Leverage the underlying cluster - In-database, Live connection, Live Connect Amazon QuickSight AWS Big Data Competency partners
  13. 13. Also, remember to 1. Leverage the right AWS Services 1. Amazon RDS 2. Amazon Redshift 3. Amazon EMR 4. Amazon S3 2. Leverage integrations with AWS Services 1. Amazon QuickSight – Direct ingestion from Amazon S3 2. Microstrategy –VLDB Properties for Amazon Redshift 3. Looker & ChartIO – Amazon EMR (Spark-SQL/Presto)
  14. 14. Largest retailer of officially licensed sports merchandise
  15. 15. All Major US Leagues If you are a sports fan, you’ve likely had a Fanatics Experience
  16. 16. 26,000,000 Minutes of customer contact 250,000,000 Visitors across Fanatics’ platform of sites 31,000,000 Units shipped annually 6,000 Peak season employees (1,700 non-peak) Major Scale, Advantage $1B in sales through eCommerce and sport venues
  17. 17. Business Centric Technology Centric Financials Inventory Customer Support Marketing Experimentation S I T E S E R V I C E S Engineering Hardware Site Performance Click Stream Personalization 2016 - Data and Analytics everywhere
  18. 18. 18 Infrastructure Analytical Content Developers Scaling our BI environment
  19. 19. Current Fanatics Data Architecture SSIS Stone Branch Spark Data Integration Qubole PIGAttunity Data Platform 400 TB Data Warehouse FanHouse EDW (Redshift) 100 TB Relational Data Legacy Storage Football (SQL Server) 500 TB Unstructured Data Pattern Detection Deep Storage HADOOP CLUSTERS Analyze & Report Discover & Explore MS Excel Tableau Data Access SOA/DAL SQL Custom AppsSSRS MicroStrategy Business Centric Technology Centric R
  20. 20. Evolution timeline on AWS Microstrategy ‘’06 ’08 ‘18’End ’14 ‘15 Nov ‘16 ‘17 Access DB & MS Excel Reports (3 MB) SQL Server SSIS & SSRS (500 GB) Redshift S3 Spark Presto Storm/Kafka/Scala Real Time Reporting R Integration Machine Learning Tableau Hadoop
  21. 21. Why we chose AWS Scalability & Agility Elasticity and Cost Automation & Self-service Availability & Disaster Recovery
  22. 22. Why we chose Microstrategy Tableau Enterprise Business Intelligence Data Discovery and Prototyping
  23. 23. Our Journey with Microstrategy 02-2015 TECH ASSESSMENT 10 LICENSES {T2.XLARGE} (WIN / ACCESS MD) 03-2015 IN PRODUCTION DISTRIBUTION SERVICES REPORTS {T2.XLARGE , RDS) 06-2015 WEB USERS ALPHA {M4.4XLARGE (WIN),RDS) 07-2015 WEB USERS PRODUCTION {R3.4X LARGE (LINUX), M4.4X LARGE (WIN), RDS) 09-2016 500 WEB USERS 7 CUBES (AVG 100 MILL ROWS) 3-10 SEC CUBE RESPONSE (AWS X1 INSTANCE) 2017 Goal : 1500+ Users All adhoc users on Microstrategy DELIVER FAST, GATHER FEEDBACK, IMPROVE Just took 1 month to be in Production
  24. 24. Current Microstrategy Architecture 500 WEB USERS AWS X1 INSTANCE* 1 TB OF RAM, 8 TB OF SSD CAN BE CLUSTERED
  25. 25. Our Journey with Tableau 02-2015 TECH ASSESSMENT TABLEAU ONLINE (2 DESKTOP , 5 USERS) 03-2015 IN PRODUCTION TABLEAU ONLINE (3 DESKTOP, 30 USERS) 06-2015 OWN TABLEAU ENVIRONMENT ON AWS (5 DESKTOP USERS, 75 WEB USERS) 02-2016 HARDWARE UPGRADE ON AWS (12 DESKTOP USERS , 125 WEB USERS)
  26. 26. Why we chose AWS Scalability & Agility Elasticity and Cost Automation & Self-service Availability & Disaster Recovery
  27. 27. Tableau Capacity Management Regular Capacity (Single Server) Peak Capacity (Distributed Workers)
  28. 28. Cost Control (CloudWatch and Tags) MLB WORLD SERIES FINALS DR TESTING TABLEAU 10 UPGRADE TEST AWS X1 INSTANCE
  29. 29. Why we chose AWS Scalability & Agility Elasticity and Cost Automation & Self-service Availability & Disaster Recovery
  30. 30. Self-service for the users Web Based Command Manager and tabadmin • Web-service based event triggering & control mechanism • Triggers both Microstrategy and Tableau events • No need for client installation • Has offset (or delay) mechanism • Saves Significant resources and complexity for ETL and Database http://bitechapi.fanatics.corp:8080/FanBiAutomation/WBCM?triggername=testAmitemail&cmd=tableau&cmddelay=0&projname =
  31. 31. Microstrategy Systems Manager for Cluster Capacity • Launch a New Governed AWS Instance • Automatically Start I-server • Add to existing Microstrategy Cluster
  32. 32. Why we chose AWS Scalability & Agility Elasticity and Cost Automation & Self-service Availability & Disaster Recovery
  33. 33. Run Hot/Cold Stand By Machines • Disaster Recovery • Redundant deployment in different Availability Zones • Cold Stand By with a higher RPO/RTO • Availability • During Critical Business events/seasons • Hot Stand By with instant failover capability
  34. 34. Best Practices for BI on AWS • Automate • Use CloudFormation Templates • AMIs (and Maintain them) • Distribute the workload • Managed shared storage (EFS) • Flexible infrastructure • Microstrategy and Tableau • Monitor your cost and budget • CloudWatch Metrics and Tags
  35. 35. 36 3-10 second Data Exploration time for Business Users Real time Reporting (consume elastic search web services) Distribute 100s of PDF Reports Daily from the same Metadata and Infrastructure, Run 1000s of Jobs per hour Site Data services based on the same Metadata Some Business Use Cases Solved
  36. 36. 37 Custom BI Portal and Real time analytics using AWS
  37. 37. 38 Hybrid Ownership Model –Hardware on AWS / Owned Software Always buy “User Based Licenses” – Never CPU Core The BI Platform should be scalable and should have enough Automation APIs to mimic cloud functionality Experiment with small number of user licenses to prototype (start with 1 or 2 user license) Try not to get locked in : If your vendor is only subscription based then you are locked in Cloud BI Vs On Premises BI (Get the best of both)
  38. 38. Thank you!
  39. 39. Remember to complete your evaluations!
  40. 40. Related Sessions

×