Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Alluxio, Inc., profile picture
Uploaded byAlluxio, Inc.
101 views

Bursting on-premise analytic workloads to Amazon EMR using Alluxio

The document discusses the benefits of transitioning on-premise analytics workloads to Amazon EMR, highlighting its cost-effectiveness, ease of management, and advanced performance for big data applications. It outlines various modernization strategies, emphasizing the importance of a cloud-based lake house architecture that enhances analytics capabilities. Overall, Amazon EMR is presented as a fully managed solution that accelerates data processing and facilitates efficient workload scaling.

Software◦
Download to read offline
Bursting on-premise analytics workloads to Amazon EMR using Roy Hasson Principal Analytics Specialist LinkedIn: /in/royhasson Twitter: royhasson
© 2020, Amazon Web Services, Inc. or its Affiliates. Customers want more value from their data Growing exponentially From new sources Increasingly diverse Used by many people Analyzed by many applications
© 2020, Amazon Web Services, Inc. or its Affiliates. On-premise Hadoop is rigid and costly Difficult to integrate with latest tech Costly to maintain and scale Difficult to manage and upgrade Inhibits rapid experimentation
© 2020, Amazon Web Services, Inc. or its Affiliates. Devices Web Sensors Social Hadoop Silo Business Intelligence Machine learning BI + analyticsData warehousing Data lakes Open file formats Central catalog/governance Modernization is a journey On-premise sources
© 2020, Amazon Web Services, Inc. or its Affiliates. Amazon EMR Easily Run Spark, Hadoop, Hive, Presto, HBase, and more big data apps on AWS Low cost 50–80% reduction in costs with EC2 Spot and Reserved Instances Per-second billing for flexibility Use S3 storage Process data in S3 securely with high performance using the EMRFS connector Latest versions Updated with latest open source frameworks within 30 days Fully managed no cluster setup, node provisioning, cluster tuning Easy
© 2020, Amazon Web Services, Inc. or its Affiliates. Optimized Runtime Runtime built on a optimized version of Spark Best performance • 2.6x faster than Spark on EMR without runtime • 1.6x faster than 3rd party Managed Spark (with their runtime) Lowest price • 1/10th the cost of 3rd party Managed Spark (with their runtime) 100% compliant with Spark API’s *Based on TPC-DS 3TB Benchmarking running 6 node C4x8 extra large clusters and EMR 5.28, Spark 2.4 10,164 16,478 26,478 0 10,000 20,000 30,000 Spark with EMR (with runtime) 3rd party Managed Spark (with their runtime) Spark with EMR (without runtime) Runtime total on 104 queries (seconds - lower is better)
© 2020, Amazon Web Services, Inc. or its Affiliates. Managed scaling improves ease of use Automatically scale cluster to meet workload demand in < 10sec and save up to 60% on cost Requested Resize
© 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates. How to approach modernization
© 2020, Amazon Web Services, Inc. or its Affiliates. 3 key modernization approaches Lift & Shift Rearchitect Hybrid Less time and effort Gain maximum value from cloud Burst the new, rearchitect the old
© 2020, Amazon Web Services, Inc. or its Affiliates. Burst workloads to Amazon EMR Hive Metastore Hadoop compute & HDFS storage On-premise cluster CatalogServiceUnifiedFS Amazon S3 AWS Glue Data Catalog Automatic sync Move on-demand Amazon EMR
© 2020, Amazon Web Services, Inc. or its Affiliates. Amazon EMR on AWS Outpost • Ideal for • Highly sensitive data and workloads • Edge computing of high volume data (AV) • Same user experience as on the cloud • Simple to manage and stay on latest version Launch data applications on-premise using EMR for AWS Outpost
© 2020, Amazon Web Services, Inc. or its Affiliates.© 2020, Amazon Web Services, Inc. or its Affiliates. The future state
© 2020, Amazon Web Services, Inc. or its Affiliates. The Lake House – Integrated and simple to use Key Benefits • Unified analytics experience • Managed and governed • Scalable & Elastic • Flexible & Agile • Cost effective • Easy to use Amazon S3 AWS Glue Data Catalog AWS Lake Formation Secure access layer Amazon Redshift Amazon EMRAmazon AthenaAmazon SageMaker Single source of truth for metadata Single source of truth for data Central governance and authorization Amazon QuickSight BI & Visualization SageMaker Studio Unified Data & ML Experience AWS Glue Studio / DataBrew Visual ETL / Data Prep Federation & caching AWS Data Exchange
© 2020, Amazon Web Services, Inc. or its Affiliates. What did we learn • Modernization is a journey – Burst to get value quicker • Amazon EMR – Fully managed service to run big data workloads • Amazon EMR + Alluxio – Makes bursting big data workloads easier • Lake House – Future state architecture combining pace of innovation, separation of concerns, elasticity, portability and cost.
© 2020, Amazon Web Services, Inc. or its Affiliates. Thank you and Q&A Roy Hasson Principal Analytics Specialist LinkedIn: /in/royhasson Twitter: royhasson https://www.alluxio.io/products/aws/