"Customers are migrating their analytics, data processing (ETL), and data science workloads running on Apache Hadoop/Spark to AWS in order to save costs, increase availability, and improve performance. In this session, AWS customers Airbnb and Guardian Life discuss how they migrated their workload to Amazon EMR. This session focuses on key motivations to move to the cloud. It details key architectural changes and the benefits of migrating Hadoop/Spark workloads to the cloud.  "

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate Your Hadoop/Spark Workload to
Amazon EMR and Architect It for Security
and Governance on AWS
Abhishek Sinha
Principal Product Manger
Amazon Web Services
A N T 3 1 2
Jian Chen & Guang Yang
Software Engineer
Airbnb
Wang Cheug
Director Data Platform Architecture
Guardian Life Insurance

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. What are the major reasons customers migrate their on-
premises Hadoop/Spark environment to Amazon EMR
2. Airbnb’s story
3. Guardian Life’s story

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hadoop/Spark deployments on-premises
Tightly coupled

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Compute and storage grow together
Storage grows along with compute
Compute requirements vary
Tightly coupled

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Underutilized or scarce resources
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Underutilized or scarce resources
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Re-processingWeekly peaks
Steady state

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Contention for same resources
Compute
bound
Memory
bound

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Separation of resources creates data silos
Team A

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication adds to cost
Single data center
3x

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application point of view
Large scale transformation: Map/Reduce, Hive, Pig, Spark
Interactive queries: Impala, Spark SQL, Presto
Machine learning: Spark ML, MxNet, TensorFlow
Interactive notebooks: Jupyter, Zeppelin
NoSQL: HBase

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Swim lane of jobs
Type of Job /Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
EDR
CDR
CDR JavaMR
LSR Load Load Load Load Load Load Load Load Load Load Load Load
Aggregation
LSR Rotation
Performance Performance
RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC
BQR BQR
Others
10EDR DailyAggregations10EDR DailyAggregations
58Hive Extracts- CDR
Others
Over utilized
Under utilized

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Decouple storage and compute

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Simple Storage Service (Amazon S3) is your
persistent data store
11 nines of durability
Low cost
Life cycle policies
Versioning
Distributed by default
EMR FS
Amazon S3

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefit 1: Switch off clusters
Amazon S3Amazon S3
Amazon S3

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Separate them to run on persistent or long-running clusters
Type of Job /Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
EDR
CDR
CDR JavaMR
LSR Load Load Load Load Load Load Load Load Load Load Load Load
Aggregation
LSR Rotation
Performance Performance
RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC
BQR BQR
Others
10EDR DailyAggregations10EDR DailyAggregations
58Hive Extracts- CDR
Others

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Run a long-running cluster
Amazon EMR Cluster
Benefit 2: Autoscale persistent clusters to save costs

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefit 3: Logical separation of jobs
Hive, Pig,
Cascading
Prod
Presto Ad-Hoc
Amazon S3

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefit 4: Disaster recovery built in
Cluster 1 Cluster 2
Cluster 3 Cluster 4
Amazon S3
Availability Zone
Availability Zone

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating from Amazon Elastic Compute
Cloud (Amazon EC2) to Amazon EMR &
Amazon S3
Jian Chen, Software Engineer
Guang Yang, Software Engineer
Airbnb

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Snapshot of data infrastructure at Airbnb
• Challenges
• Why Amazon EMR/Amazon S3
• Migration and lessons learned

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Airbnb is a data driven company
1. Recommendations
2. Search relevance
3. Smart pricing
4. Company metrics
5. Financial reports
6. Experimentation
framework
7. And many more …

23. MySQL
Dumps
Gold Cluster
HDFS
Hive
Kafka Silver Cluster
ReAir
Airflow scheduling
YARN HDFS
Hive
YARN
]
23
Event
Logs
Sqoop
Data infrastructure (year 2014 - 2015)

24. MySQL
Dumps
Gold Cluster
HDFS
Hive
Kafka Silver Cluster
ReAir
Airflow Scheduling
Amazon S3
Presto Clusters
YARN HDFS
Hive
YARN
]
24
Event
Logs
Sqoop
Compute Clusters
YARN
Spark
Data infrastructure (year 2015 -)
HBASE HBASE
Flink
Spark Streaming

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data infrastructure—Trending
• More Spark workloads
• Hive to Spark, better performance, testability, and maintainability
• Machine learning
• More streaming workloads
• Airstream, in-house streaming framework on top of Spark streaming
• Apache Flink

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Growth
• Business
• Employees (2x YoY)
• Datasets (3x YoY)
• Number of jobs (2x YoY)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Compute and storage are tightly coupled

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scalability issues
• 1000 x d2.8xlarge instances
• YARN does not schedule jobs even there are available resources
• HDFS: 150 MM blocks
• Long GC pauses causing fail-over
• Takes more than 10 hours to rolling restart the cluster
• Several major outages

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lack of elasticity
Wasted resources
Contention at peak

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Old Hadoop hardware stack—Hard to upgrade
• Hadoop 2.5
• Hive 0.13
• OS upgrade for security patches

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High CapEx and OpEx
• High CapEx
• Provision for peak load
• Hard to maintain
• Hard to add instances, even harder to remove
• Very hard to allocate costs across the organization with multi-tenanted
clusters

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Amazon EMR/Amazon S3?
decouple compute and storage
• Amazon S3 as the data lake
• Stateless compute infrastructure using Amazon EMR
• Easy to set up, rotate, upgrade, scale out/in
• Better AWS integration
• EMRFs, spot instances, connectors with various AWS services
• EMR clusters for each business unit
• Better isolation
• Cost attribution
• Customized software/hardware

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration path
Sounds great ... but how do we get there?
• User only writes to HDFS, we archive the data to Amazon S3 later
• Started Amazon EMR migration conservatively
• Non-critical jobs as pilot use case
• Long-running clusters with auto-scaling
• Spark first with Hive-0.13 client, to learn/tune the system
• Four production long running clusters for Spark (with auto-scaling)
• Jobs run 3x faster
• Hive migration
• Amazon EMR job server
• Transient clusters

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New architecture
Transient clustersManaged by Airflow
Managed by
Terraform
Long-running clusters
Amazon EMR Job Server
Payment staging Payment prod Ad-hoc test/dev Streaming clusters
Amazon S3 as data lake

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lesson learned
• Get it working first
• r/w to existing HDFS, Hive Metastore, setup gateways
• Start with less hardware variations
• r4.8xl with different size of EBS volume
• Pick representative use cases, make them happy
• Default configuration may not work well
• Heap size, yarn conf
• Auto-scaling is awesome
• Long-running cluster with auto-scaling

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Guardian Life’s migration to
Amazon EMR
Wang Cheung
Director, Data Platform Architecture

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About Guardian
37
9000 Employees
Over2,750 financial
representatives and
more than 55 agencies
Annuities
Investment
Life insurance
Dental insurance
Employee benefits
Disability income
Insurance
158-year-old mutual company
Fortune 239 ranking For more information, visit Guardian’s
website: www.GuardianLife.com

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
History lesson—No enterprise data
38
Group
Individual
life
Individual
disability
Annuities
PAS

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Guardian big data journey
39
• AWS Cloud Migration Program
• Claim predictability and claims
termination advanced analytics
• Client profiling & segmentation
• Launched big data platform
• Guardian Data Council established
• Built Enterprise Data Marketplace
• Ingested group data
• LOB Data Summits
• Initiated advanced analytics primers
• Self-service analytics
• Customer 360 lifecycle
management
• Artificial intelligence
• Machine learning
2015
2016
2017
2018
2019
• Amazon EMR upgrades
• Business COE sponsorship
• Enterprise search
• Analytics across all digital
properties
• Prototyping artificial intelligence
• Prototyping machine learning
• Amazon EMR migration
• Customer hub (master data)

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why did we migrate to Amazon EMR?
40
Presentation title | Business Unit

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On-premises Hadoop architecture
Pivotal HAWQ
Pivotal HAWQ
Pivotal HAWQ
Ops
Data science
DR

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On-prem Hadoop challenges
• Combined storage and compute on shared servers
• Fixed storage
• Disk failures
• Inability to quickly scale
• Costly TCO – multiple clusters
• Costly DR – third-party software
• Unused capacity during off-peak periods
• Team of dedicated operators to maintain hardware
• Slow adoption to address changing business needs

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR migration strategy
Q1 Q2
POC
Cloud assessment & POC Security certification & environment
readiness
App dev and regression testing Data migration & cutover
DECISION POINT
Managed Service Provider
Go / No-Go Migration Decision
Migration
POC &
Evaluation
Future State
Architecture
Key Milestones
Security
Certification
Prod
Parallel
Automation Development Decomm on-
prem hw
Cutover to
AWS jobs
AWS Snowball
migration of
historical data
Infrastructure
buildout
Q3 Q4

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS EMR Migration Strategy
44
Category EMR
Functional capabilities Meets data processing and analytics requirements
Infrastructure and software costs Medium
Ease of AD and Kerberos Integration Achieved using custom solutions – not available out of the box
Amazon EC2 Image used
Amazon Linux – new minimum baseline security image required to meet Guardian
security & compliance standards
DNS
Internal DNS – Amazon EMR is very sensitive to DNS and does not work with
Guardian’s DNS
Familiarity to the team Medium
Ease of complete product installation High
Ease of deployment automation High
Ease of DR setup High
Risks for unknown factors Medium
Q1 Q2
Cloud assessment & POC Security certification & environment
readiness
App dev and regression testing Data migration & cutover
Q3 Q4

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR migration strategy
45
• Partner with IT security team
• Amazon Linux - require new minimum baseline security standard
• Obtain security exceptions – Kerberos, CIS benchmarks
• Third-party software changes required for integration
• Edge node setup - security hardening by shutting off SSH access on the cluster
• Custom DNS
• Data protection and controls – Amazon S3 encryption; SSL / HTTPS
• Multi-region DR requirements – Amazon S3; automation
Q2
Security certification & environment
readiness
App dev and regression testing Data migration & cutover
Q3 Q4Q1
Cloud assessment & POC

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR migration strategy
46
• Automation – Terraform, Puppet
• CI/CD integration – Bitbucket, Jenkins
• Refactor and test 300+ workloads
• Scope of code changes – Syncsort, Pig, Python, R, Shell Scripts
• Code migration plan
• Accommodate in-flight AppDev projects (Dev & UAT)
• Migrate to AWS dev and promote up
• Set code freeze for on-prem AppDev
App dev and regression testing Data migration & cutover
Q3 Q4Q1
Cloud assessment & POC
Q2
Security certification & environment
readiness

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR migration strategy
47
• Conduct parallel production testing between on-prem and AWS
• Determine up front the data set to snapshot for parallel production runs
• Historical data reserved for Snowball
• Utilize multiple Snowball Edge – migrate 350 TB
• Archive operations – migrate data to lower tier Amazon S3 (for example, S3-IA, Amazon
Glacier)
• Shutdown on-prem workloads and repurpose hardware
Data migration & cutover
Q4Q1
Cloud assessment & POC
Q2
Security certification & environment
readiness
App dev and regression testing
Q3

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR migration
48
corporate data center
Pivotal HAWQ
AWS Snowball
Amazon
S3
MS SQL
Hive Metastore
Amazon
EC2
Amazon
EMR
Presto
EMR
Spark/Hive
EMR

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Overall architectural design pattern
49

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lake architectural patterns
50
Interactive
and batch
Process
Store

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lake architectural patterns
51
Real-time
Interactive
and batch
Data
Outputs
Process
Store

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lake architectural patterns
52
Real-time
Interactive
and batch
Dataconsumers
Data outputs
Process
Store

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lake architectural patterns
53
S3
Data
Sources
Transactions
Real-time
Interactive
and batch
Dataconsumers
Amazon EMR
Presto
Hive
Pig
Spark
MySQL
SQL Server
RDS
Data outputsNoSQL
Files
Change Data
Capture
Process
Store
Reporting
Edge
node

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lake architectural patterns
54
S3
Data
Sources
Transactions
Stream
Real-time
Interactive
and batch
Spark
Streaming
Storm/Flink
AWS
Lambda
Dataconsumers
Amazon EMR
Presto
Hive
Pig
Spark
MySQL
SQL Server
RDS
Data outputsNoSQL
Files
Change Data
Capture
Process
Store
Reporting
Kafka
Amazon Kinesis
Streams
Amazon
Kinesis Data
Firehose
Edge
node

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS EMR & Amazon S3
Key takeaways for adoption
• Cost-effective
• Decoupling storage from compute
• Security
• Automation
• Open source integrations
• Resilient and scalability
55

56. Thank you.
Park Avenue Securities LLC (PAS) is an indirect, wholly-
owned subsidiary of The Guardian Life Insurance Company
of America (Guardian). PAS is a registered broker-dealer
offering investment products, as well as a registered
investment advisor offering financial planning and
investment advisory services. PAS is a member of FINRA
and SIPC.
Individual disability income products underwritten and
issued by Berkshire Life Insurance Company of America
(BLICOA), Pittsfield, MA. BLICOA is a wholly owned stock
subsidiary of and administrator for The Guardian Life
Insurance Company of America (Guardian), New York, NY
or provided by Guardian. Product provisions and
availability may vary by state.
2018-66009 Exp. 09/2020

57. Please complete the session
survey in the mobile app.
!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.