Modernizing your data warehouse can unlock new insights while substantially improving query and data load performance, increasing scalability, and saving costs. In this chalk talk, we discuss how to migrate your Netezza data warehouse to Amazon Redshift and achieve agility and faster time to insight while reducing costs. Financial Engines joins us to share their migration journey, after which they were able to improve performance by 10x and reduce cost by 1/3 for a saving of ~$100K annually.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate from Netezza to Amazon Redshift:
Best Practices with Edelman Financial Engines
Matt Scaer
Principal DW Specialist SA
Amazon Web Services
A N T 3 7 7
Shrikant Kathane
Director of Data Engineering
Edelman Financial Engines Inc.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Overview and why move to
Amazon Redshift
Before and after
Migration process
Lesson learned
Current status

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Overview
• About Edelman Financial Engines
• Largest independent RIA providing personalized financial guidance to all
• Merger with Edelman Financial Services
• Over 10 million people across over 700 companies including 140+ of the Fortune 500
• About me
• Lead the data engineering team at Edelman Financial Engines
• Enterprise Data Warehouse is the centralized backbone
• Powers daily decisions through 200+ Tableau dashboards for decision support teams
• Fuels Salesforce CRM with critical campaign performance metrics for marketing and
distribution teams
• Supports call activity tracking for National Advisor Center and financial reporting for FP&A
teams
• Drives product recommendations for over one million clients through user experience

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why move to Amazon Redshift
• Growth
• Daily data for over 10 million clients and financial data of approximately 12 TB
• Challenges with data growth—physically shard data across two Netezza instances
• Primary instance for frequently accessed queries with daily SLA
• Secondary instance for infrequent workloads like SOX compliance queries, historical
snapshots
• Costs
• Costs of two Netezza appliances
• Tipping point reached with move from data center to AWS Cloud and contract expiration
• Reinvent
• Modernize the data analytics platform for company’s growth
• Reduce operational overhead—single instance with easy to scale cluster configuration
• Future proof for adopting technologies like data lake, Amazon Athena, Amazon Redshift
Spectrum and machine learning

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
After
Route 53

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration process
• Early start
• Overall effort took ~10 months from start in mid-Q2 2017 to go live in Q1/Q2 2018
• Established roadmap with monthly and quarterly milestones and tracking
• Engaged external team stakeholders from security, compliance, and IT teams
• Staffing
• Cross trained existing team members with weekly AWS Friday sessions, couple of team
members completed AWS certification
• 50% team on migration efforts + hired one AWS DevOps resource for infrastructure as code
• Big technology decisions
• Amazon Redshift over Amazon EMR and Snowflake
• Airflow with python based ETL solution over Luigi and commercial alternatives
• Adopted AWS best practices for security—defense in depth using network isolation and
encryption
• Adopted infrastructure as code using Terraform, Ansible, AWS CodePipeline, and AWS
CodeDeploy

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices
• Choose regions wisely
• We decided to migrate to us-west-1 due to direct connectivity between our data center
• In hind sight, we could have engaged IT earlier for us-west-2 region – That would have made
Amazon Athena and Amazon Redshift Spectrum available for us earlier!
• Costs
• We decided to go live with on-demand Amazon Redshift billing due to lack of experience
• In hind sight, we would have benefited with additional cost savings with reserved instances!
• Data migration
• We built custom tools to migrate data from Netezza to Amazon Redshift, lack of Netezza
support in AWS Data Migration Service (AWS DMS)
• We recommend use of AWS DMS now that the support is available

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current status
• In production for over 6 months!
• Soft launch in March and go live at end of May 2018
• Reduced hard costs by 1/3—we expect over 100K savings annually in our hard costs, more in
TCO!
• We are seeing up to 10X performance improvements with columnar technology on certain
aggregate queries over Netezza on tables with over ~1 billion records
• What’s next
• Data lake rolled out
• Amazon Athena and Amazon Redshift Spectrum rollout in progress
• Data science workloads planned ahead

11. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Matt Scaer
Principal DW Specialist SA
Amazon Web Services
mscaer@amazon.com
Shrikant Kathane
Director of Data Engineering
Edelman Financial Engines Inc.
www.linkedin.com/in/skathane

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.