Full 360 is a cloud consulting firm that provides big data, API/UX, and cloud operations services. They helped a customer migrate their data from Netezza to Redshift, building a structured data lake and optimizing queries for equivalent or better performance. Lessons from the project included data standardization, tuning techniques like encoding and sort keys, and creating reusable ingestion processes. The migration reduced license costs and improved operational flexibility.

1. Netezza to Redshift
Data Migration and Reference Architecture
NYC May 2018

2. Our Experts, not Bodies℠, have been building Fast, Wide, and
Big Data systems in the cloud since the earliest days.

3. Enabling Fast, Wide and Big Data Systems in the Cloud
Full 360 is a multinational cloud enabler providing strategy,
consulting, and managed services in three practices — Big Data,
CloudOps, and Custom API/UX Delivery.

4. We Have Been Doing
This for Longer than
Anyone…

5. 2007
Oracle EPM on AWS using the only
available services at the time - S3 and EC2
2008
Inaugural AWS SI Partner
2010
Talend + Jaspersoft + Vertica
elasticBI End to End Data Warehousing Solution on AWS

6. 2013
AWS Premier Partner
2015-16
AWS Big Data Competency
Subsidiaries in AsiaPAC and LATAM
2017
AWS Redshift Solution Competency

7. Our Practices
• Big Data and Data Warehousing
• Custom API/UX Delivery
• Cloud Strategy and Operations

8. Full 360 Migration Engagement
• Customer required Netezza to Redshift Lift & Shift
• Structured as Fixed Price + Contingency
• AWS Accelerator
• Onsite Analysis & Design, Offsite Build & Test
• Development Priorities Agreed
• Key Activities, Deliverables, Acceptance Criteria Agreed
• Some Assumptions Changed During Project

9. Migration Project Deliverables
• One Time data migration
• Structured Data Lake
• CSV to JSON conversion
• Data Reconciliation
• Ingestion
• Migration of multiple environments
• Upstream + Downstream Interfaces
• Equivalent or better performance
• Process Optimizations

10. Target Environment
• Users: 20-50
• Front End: Qlik, SAS
• Data Frequency: Daily, Weekly, Monthly data refresh
• Target Redshift Cluster: 2x DC1.8XL (Decided based on
right-sizing)
• Number of Database / Schemas: 4
• Number of Tables: 150

11. Producer Ingestion Transformation
• Data Sources
• Producers
• Structured Data Lake
• Ingestors
• Transformers
• Flexibility
• Cost Effectiveness

12. Simple PITBull Architecture

13. Sample Architecture

14. Lessons & Best Practices
• String Type Standardization
• Time Zone Standardization
• Trim Standardization
• NULL Partitions
• Schema vs. Database

15. Upshift-Preparation
• Collected thousands of queries and normalized them to
300-500 common queries
• Identified top 200 queries from the set
• Added queries important to users
• Generated test suites

16. Upshift-Analysis & Tuning
• Iteratively optimized using FULL 360 Upshift utility
– Encoding
– Sort Keys
– Distribution Key
– Denormalization
– Materialization

17. Upshift-Performance Improvements
• Observed 3%-30% improvement in identified queries
after Upshift exercise
• Some queries degraded, but these queries were
infrequently used and customer was okay with the minor
degradation
• Created production deployment scripts for upshifted
tables
• Use the repeatable ingestion process to reload the data
in the newly created upshifted schemas

18. Cost Savings
• Saving on license costs
• Savings due to flexibility on right sizing of development
environments
• Savings due to disposable development environments

19. Operational Efficiency Improvements
• DR related efficiency
• Flexibility to upsize or downsize
• Flexibility to create additional environments
• Better and cheaper support
• Alignment with future state
• Highly available and reliable data lake
• Potential to build on the data lake

20. The Full 360 Difference
• DDL Conversion utility
• Extractor utility
• SneaQL (open source available on github)
– https://github.com/full360/sneaql
• Upshift utility

21. Multi-Platform Offerings
• Available as a packaged offering
• Multiple source warehouses supported including:
– Oracle
– SQL server
– Vertica
– Teradata
– Others that support JDBC

22. Big Data and Data Warehousing
Technical Solutions
• Data Warehouse Managed Services
• Data Warehouse Tuning
• Big Data Systems Design
• Data Lake Design
• Data Ingestion, ETL
• BI and Reporting
Business Solutions (Examples)
• Airline Aircraft, Crew, Fuel
Management
• Loyalty Membership and
Marketing
• Pharmaceutical Sales and
Compensation Models
• Gaming Event Data Ingestion
and Processing
Amazon
RDS
Amazon
Redshift
AWS Data
Migration
Service

23. Custom API/UX Delivery
Technical Solutions
• API Design
• Microservices
• Containerization of Legacy
Applications
• UX and Mobile Design and Delivery
• Streaming and High Speed Ingestion
and Routing
• Real-time Messaging
Business Solutions (Examples)
• Microservices based API for
Delivering Airline and Loyalty
Data
• Fuel Management
Application for Pilots
• Real-time campaign and offer
delivery via Mobile
Amazon ECS

24. Cloud Strategy and Operations
Technical Solutions
• Cloud Adoption Review and Strategy
• Best Practice Cloud Infrastructure
Design
• Container Orchestration and
Scheduling
• Devops Process Design and
Implementations
• Serverless Implementations
• Service Discovery and Secrets
Management
• Infrastructure as Code
Business Solutions (Examples)
• Systems Analysis and Deliver
Cloud Adoption and
Migration Blueprint
• Microservices Architecture
and Implementation for
Loyalty
Amazon ECS
Amazon
EC2
AWS
Lambda
AWS
CloudFormation

25. Q&A