This document discusses serverless big data architectures using AWS services. It describes how serverless computing eliminates the need to provision and manage servers. AWS services like S3, DynamoDB, Kinesis, Lambda, Glue and Athena allow ingesting, storing, processing and analyzing data without managing infrastructure. These services scale automatically based on usage and provide built-in availability and fault tolerance.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Francesco Rinaudo,
Sr. Manager, Solutions Architecture, AWS
June 07, 2017
Serverless Big Data Architectures
Serverless Big Data Architectures

2. Cloud Architecture Evolution
Virtualized Managed Serverless
Virtualized
Servers
Managed
Platforms
Serverless
Analytics

3. No servers to provision
or manage
Scales with usage
Never pay for idle Availability and fault
tolerance built in
Serverless characteristics

4. Data and Analytics Flow
Ingest/
Collect Store
Analyze/
Process
Visualization/
Consume
Orchestrate/Transform

5. What Is the Temperature of Your Data / Access ?

6. Orchestration/Transform
AWS Big Data Services
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Batch
ETL/ELT
Realtime
ETL/ELT
Transactional
/ CDC
B.I. Tools
Data Science
Notebooks
Bulk Transport
File/Object Upload
Streaming Ingest
Commits Transactional
NoSQL
Data Lake
Streaming Storage
Dashboards
Batch Analytics
Interactive
Querying
Machine Learning/
Deep Learning
Realtime Analytics
…

7. Orchestration/Transform
AWS Big Data Services
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
= Serverless
Serverless
Managed
Virtualized
Batch
ETL/ELT
Realtime
ETL/ELT
Transactional
/ CDC
B.I. Tools
Data Science
Notebooks
Bulk Transport
File/Object Upload
Streaming Ingest
Commits Transactional
NoSQL
Data Lake
Streaming Storage
Dashboards
Batch Analytics
Interactive
Querying
Machine Learning/
Deep Learning
Realtime Analytics

8. Orchestration/Transform
AWS Big Data Services
EMR EC2
S3
RedshiftDynamoDB
AWS DMS (CDC)
AWS Lambda
Kinesis Analytics Amazon Athena
Amazon
QuickSight
Aurora
AWS Glue AWS Step
Functions
Kinesis
Streams
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
AWS
Snowball
ISV
Connectors
Kinesis
Firehose
S3 Transfer
Acceleration
= Serverless
Amazon
ElasticSearch

9. Anti-Pattern
DataTier

10. Key Services Overview

11. Big Data Storage for Virtually All AWS Services
Amazon S3
• Store anything
• Object storage
• Scalable
• 99.999999999% durability
• Extremely low cost

12. Amazon
DynamoDB
Fast & Flexible NoSQL Database Service
• NoSQL Database
• Seamless scalability
• Zero admin
• Single digit millisecond latency

13. Amazon
Kinesis
Real-time Streaming Platform
• Streams, Firehose, Analytics
• Real-time processing
• High throughput; elastic
• Easy to use
• Integration with S3, EMR,
Redshift, DynamoDB

14. Amazon Kinesis
Streams
• For Technical Developers
• Build your own custom
applications that process
or analyze streaming
data
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into S3, Amazon Redshift
and Amazon Elasticsearch
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver and process streams on AWS

15. AWS Lambda
• Run your code in the cloud - fully
managed and highly-available
• Triggered through API or state
changes in your setup
• Scales automatically to match the
incoming event rate
• Node.js (JavaScript), Python, Java,
and C#
• Charged per 100ms execution time
Serverless Compute

16. Amazon
Athena
Interactive Query Service
• Query directly from Amazon S3
• Use ANSI SQL
• Serverless
• Multiple Data Formats
• Pay per query

17. AWS Glue
Fully Managed ETL Service
• Catalog data sources
• Identify data formats & data types
• Error Handling
• Manage and scale resources
• Generate ETL code
• Schedules, executes ETL jobs

18. AWS Glue: services
Data Catalog
§ Hive metastore compatible metadata repository of data sources.
§ Crawls data source to infer table, data type, partition format.
Job Execution
§ Runs jobs in Spark containers – automatic scaling based on SLA.
§ Glue is serverless - only pay for the resources you consume.
Generate and
Edit Transformations
§ Generates Python code to move data from source to destination.
§ Edit with your favorite IDE; share code snippets using Git.

19. •Fast and cloud-powered
•Easy to use, no infrastructure to
manage
•Scales to 100s of thousands of
users
•Quick calculations with SPICE
•1/10th the cost of legacy BI software
Business Intelligence
Amazon
QuickSight

20. AWS Summit Milan – 8 June 2017
Marco Viganò - Head of Digital Development
AWS Summit Milan
8 June 2017

21. AWS Summit Milan – 8 June 2017
Who is Condé Nast?
1 M 10 M
Month
Daily average
220 M22 M
Page ViewsUU
Social 55%
SEO 30%
Mobile 75%
Desktop 19%

22. AWS Summit Milan – 8 June 2017
The Challenge: Can we use a DMP?
1. Behaviors analysis on authenticated & not authenticated users
2. Users clustering and profiling
3. Better advertiser performance: targeting ads, newsletter and
ecommerce
4. Prediction: success of a content based on user navigation and
social network reaction

23. AWS Summit Milan – 8 June 2017
Offline: Subscriber print DB + events
Amazon
S3 Bucket
Old Legacy User Profiles
RDS
(MS SQL)
Condé Nast user data sources
Amazon DynamoDB
Registration + profile Enrichment
RDS
(MySQL)
Commenting platform
Analytics

24. AWS Summit Milan – 8 June 2017
*Foreign Data Wrapper:
- MySQL (mysql_fdw)
- Tabular Data Stream - MS SQL Server (tds_fdw)
- PostegreSQL (postgres_fdw)
- S3 (s3_fwd)
- Unicorn Python
Data sources
DMP
DWH – Layer 2
Amazon EMR Amazon
Redshift
ETLETL
Data collection
FDW*
Amazon
S3 Bucket
DWH – Layer 1
Amazon
Redshift
How we collect data

25. AWS Summit Milan – 8 June 2017
We need more data
• User clickstream and navigation path
• Website interactions:
- What/When are you reading?
- Which authors, topics, tags have you followed?
- How you make interaction between the website and the social
networks?
• Realtime:
- What is the most readed content on an website?
- And the most shared?
- And the most commented?

26. AWS Summit Milan – 8 June 2017
Genius Data Lake
Amazon S3
Amazon ES
Amazon Kinesis
Analytics
Amazon
Kinesis Firehose
Amazon
Kinesis Stream
Amazon
Lambda Function
Amazon
Kinesis Firehose
Amazon
Cognito (datalake)

27. AWS Summit Milan – 8 June 2017
The Benefits with AWS
1. Quicker provisioning of resources: from days to hours
2. No scaling problem due to data collection growth
3. Saving: time & money
4. You don’t need to reinvent the wheel

28. AWS Summit Milan – 8 June 2017
Next AWS development
1. Moving Genius Datalake EC2 with node.js to Lambda functions
2. Cost saving on S3: lifecycle policy with Glacier
3. Genius ML: making prediction of success on piece of content
based on users cluster and behaviors
4. Improve Genius with Rekognition
5. Add voice with Lex - be ready when Echo will be in italian

29. AWS Summit Milan – 8 June 2017
Thank You
Marco Viganò
@Sasha0423

30. Serverless Design Patterns

31. Real-time Analytics
Producer
Apache
Kafka
KCL
AWS
Lambda
Spark
Streaming
Apache
Storm
Amazon
SNS
Notifications
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
Alert
Analytics
Output KPI
Serverless
Managed
DynamoDB
Streams
Kinesis
Streams
Virtualized
Kinesis
Analytics
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Apache
FlinkSQS

32. Interactive Queries
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Producer Amazon S3
Amazon
Redshift
Amazon EMR
Presto
Impala
Spark
Interactive
Amazon
Athena
Serverless
Managed
Virtualized
QuickSight

33. Catalog' &'Search
Access%and%search%metadata
Access'&'User'Interface
Give%your%users%easy%and%secure%access
DynamoDB Elasticsearch API'Gateway Identity'&'Access'
Management
Cognito
QuickSight Amazon' AI EMR Redshift
Athena Kinesis RDS
Central'Storage
Secure,%cost5effective
Storage%in%Amazon%S3
S3
Snowball Database' Migration'
Service
Kinesis' Firehose Direct'Connect
Data'Ingestion
Get%your%data%into%S3
Quickly%and%securely
Protect'and'Secure
Use%entitlements% to%ensure%data% is%secure%and% users’% identities% are% verified
Processing' &'Analytics
Use%of%predictive%and%prescriptive%
analytics%to%gain%better%understanding
Security'Token'
Service
CloudWatch CloudTrail Key'Management'
Service
Data Lake Reference Architecture
= Serverless

34. Amazon S3
Data Lake
Amazon Kinesis
Streams & Firehose
Hadoop / Spark
Streaming Analytics Tools
Amazon Redshift
Data Warehouse
Amazon DynamoDB
NoSQL Database
AWS Lambda
Spark Streaming
on EMR
Amazon
Elasticsearch Service
Relational Database
Amazon EMR
Amazon Aurora
Amazon Machine Learning
Predictive Analytics
Any Open Source Tool
of Choice on EC2
Data Science Sandbox
Visualization /
Reporting
Apache Storm
on EMR
Apache Flink
on EMR
Amazon Kinesis
Analytics
Serving Tier
Clusterless SQL Query
Amazon Athena
DataSourcesTransactionalData
Amazon Glue
Clusterless ETL
Amazon ElastiCache
Redis
Data Lake and
Real-time
Analytics

35. Serverless nicely fits into big data platforms
• AWS Serverless Big Data Services
• Compliments existing big data flows
• Focus on the analytics and not on infrastructure or servers
• Don’t focus on the scaling, availability, and undifferentiated
heavy lifting
• Pay only for what you use
• Innovation: easily try out different tools, analytics, and
solutions

36. Thank you!