To gain insight from data you must be able to interact with various types, volumes and velocity. One size does not fit all. In this session you will learn how to use the AWS portfolio to address your big data needs, apply analytics for real time & reporting use cases, then visualize these insights with an automated cloud-native approach.

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Macey
AWS Specialist Solutions Architect
Big data and Insights
3rd July 2019
Multi Source, Multi Speed
Data Consumption & Analytics on AWS

2. Agenda
How did it come to this?
Unintended data pipelines
Well architected data pipelines
Accelerated Data Lake
Multisource, multispeed patterns
Wrap up

3. How did it come to this?

4. Unintended data pipelines
Geospatial Girl
The Oracle
The Data Scientist
External request
Sales
Analyst
Operations
Security Guy
The Boss

5. Well architected data pipelines
Operational
Excellence Security Reliability
Performance
Efficiency
Cost
Optimization

6. Security
Day 0
Data governance
& metadata
Data centralised
& scalable
SQL & BI
ready
Analytical &
Data Science
foundation
Repeatable &
Extensible
Accelerated Data Lake

7. Customer example
Health Organisation
In New Zealand
Challenge :
Needed to bring disparate datasets together
Too many external files
Current State:
SQL Server 2008
Large text files
ETL (SSIS) https://www.linkedin.com/pulse/how-we-built-data-lake-less-than-4-weeks-alex-poor/
Solution :
Uses the ADL solution as a base
Extended the solution to access FTP server data
Entirely serverless

8. Multisource, multispeed
data and analytics patterns

9. Streaming

10. Streaming Pattern
Smart City
Camera
camera_stream
datetime
sensorId
locationId
currentTemperature
battery
objectDetected
status
camera_stream
Create SQL Schema Write SQL to query stream
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Analytics

11. API Pattern - Inbound
WeatherTraffic
Amazon
CloudWatch
AWS Lambda Amazon S3

12. API Pattern - Outbound
Camera, weather, and traffic Insights
AWS Lambda Amazon S3Amazon
API Gateway

13. SFTP Pattern
+ No end-user disruption
+ Fully managed servers
+ Simple to use
+ Pay as you use
+ Native cloud integrations
AWS SFTP
Amazon S3
AWS Transfer
for SFTP

14. Data Stores
Data stores
CDC
AWS Database
Migration Service
* Target only
Amazon
DynamoDB*
Amazon
Kinesis
Amazon S3
Amazon RDS

15. Data Stores

16. Who is saying what about DMS and SCT?
"We migrated hundreds of our clients from our in-house data-center to Amazon RDS
Oracle 12c using the AWS Data Migration Service (DMS). Due to this service, we
could live-replicate the databases between our data-center and RDS before the
migration. That kept the migration down-time to the very minimum. We are very
happy with DMS and are planning to use it for Oracle to MySQL migration next”.
”The SCT Assessment Report was the key enabler to allow us to understand the
scope of effort required to complete an Oracle to PostgreSQL migration. What
was originally thought to be a largely manual task that no one was particularly
excited about having to do became a very straight-forward quick and easy
process."
“We are in the process of migrating some databases to Amazon Aurora. The ease
by which we can do this using the AWS Database Migration Service has
simplified this process for us and enabled us to accelerate our migration
efforts. The ability to closely monitor the process, the detailed logging feature, and
the support we received from AWS have given us a great deal of confidence in a
successful migration.”

17. Shared
file systems
NFS TLS
AWS storage resources
AWS DataSync
agent
AWS DataSync
Amazon S3
Amazon Elastic
File System
Region
Batch Processing Pattern
Corporate data center
Transfer data at speeds up to 10 times faster than open-source tools

18. AWS has services and patterns to create
YOUR organisations data goldmine
csv
json
xls
Batch
FTPStreaming
IoT
Geospatial Girl
The Oracle
Operations
External
Requests
The Data Scientist
Analyst
Sales

19. References
AWS Accelerated Data Lake (Git)
https://github.com/aws-samples/accelerated-data-lake
AWS Accelerated Data Lake Blog (part 1 & 2)
https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-
together
https://aws.amazon.com/blogs/publicsector/securing-your-data-by-knowing-your-data
Our data lake story: How Woot.com built a serverless data lake on AWS
https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-
on-aws
Kinesis Data Generator
https://awslabs.github.io/amazon-kinesis-data-generator/

20. References
IoT analytics
https://aws.amazon.com/iot-
analytics/
AWS Database Migration Service
https://aws.amazon.com/dms/
AWS Transfer for SFTP
https://aws.amazon.com/sftp/
AWS Datasync
https://aws.amazon.com/datasync/
Amazon Kinesis
https://aws.amazon.com/kinesis/
Amazon Kinesis Analytics
https://aws.amazon.com/kinesis/data-
analytics/

21. Available today @ GitHub
https://github.com/aws-samples/accelerated-data-lake
Includes
Data lake pipeline (CloudFormation)
Instructions
Data configuration, security and metadata templates
Delivery
Professional services
AWS partners
Accelerated Data Lake