This is your chance to learn directly from top CTOs and Cloud Architects from some of the most innovative AWS customers. In this lightning round session, we'll have an action-packed hour, jumping straight to the architecture and technical detail for some of the most innovative data storage solutions of 2017. Hear how Insitu collects and analyzes data from drone flights in the field with AWS Snowball Edge. See how iRobot collects and analyzes IoT data from their robotic vacuums, mops, and pool cleaners. Learn how Viber maintains a petabyte-scale data lake on Amazon S3. Understand how Alert Logic scales their massive SaaS cloud security solution on Amazon S3 & Amazon Glacier.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
This Is My Architecture:
Most I nnovative Storag e Solu tions
L I G H T N I N G R O U N D
N o v e m b e r 2 8 , 2 0 1 7
AWS re:INVENT

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s in store…
7- to 10 minute highly-technical presentations
• Alert Logic
• Viber
• iRobot
• Insitu
• Celgene

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scalable Ingestion, Storage & Analytics
Paul Fisher
Technical Fellow
A l e r t L o g i c

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
About Alert Logic
Alert Logic provides fully managed
security monitoring and protection for
cloud-deployed applications and
workloads
• Intrusion detection
• Web attack detection and blocking
• Vulnerability scanning
• Asset analysis and configuration
assessment

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The problem we faced
Doing this produced a large
data-processing problem
• 4,100+ customers
• 1.2M messages/second
• 2 petabytes per month
• 3 months to 7 years retention
• Adding 110 percent data volume/year
0
5
10
15
20
25
30
35
40
Jan-12
Aug-12
Mar-13
Oct-13
May-14
Dec-14
Jul-15
Feb-16
Sep-16
Apr-17
Nov-17
Jun-18
Jan-19
Aug-19
Mar-20
Oct-20
May-21
Petabytes

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key architecture challenges
• Achieving performance, durability, and availability, all at
once, is hard
• Providing multi-region availability with Recovery Time
Objective (RTO) of 15 minutes
• Designing for scale up 150x to 10PB per day for 100k
customers
• Doing this all for less than $0.02 per GB, with a team of 20
engineers

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alert Logic system architecture
Foundational Components
Partners EnvironmentsCustomer Environments
AWS
Cloud Collection
appliance
host
agent
host
agent
Azure
appliance
host
agent
Traditional
appliance
host
agent
host
agent
Security Subsystems
AWS
Partner Account
log
ids
Ingestion,
Storage &
Access
Assets &
Config
Correlation &
Analytics
Incident
Analysis &
Workflow
Vulnerability
Assessment
Support
Customers
Reporting
Analysts
Partners
External APIs
Internal UI/UX & APIs
Cloud Collection
Partner DC
Ticketing
Monitoring

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alert Logic system architecture
Ingestion, Storage & Access
appliance
host
agent
host
agent
Ingestion
Data Access
Search
Collection
Control Flow
Data Flow
AWS
Partner Account
log
ids
Foundational Components
Partners EnvironmentsCustomer Environments
AWS
Cloud Collection
appliance
host
agent
host
agent
Azure
appliance
host
agent
Traditional
appliance
host
agent
host
agent
Security Subsystems
AWS
Partner Account
log
ids
Ingestion,
Storage &
Access
Assets &
Config
Correlation &
Analytics
Incident
Analysis &
Workflow
Vulnerability
Assessment
Support
Customers
Reporting
Analysts
Partners
External APIs
Internal UI/UX & APIs
Cloud Collection
Partner DC
Ticketing
Monitoring

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data access architecture

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inventory for bundling optimization

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hear more from Paul…

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolving the Data Store Supporting IoT
Historical State Changes
Jim Teal
Senior Cloud Solution Architect – Product IT
i R o b o t C o r p o r a t i o n

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The problem…
• We needed a facility to support ad-hoc queries, through an API to
report historical states for AWS IoT Thing Shadow attributes
• Our initial design pattern was based on Amazon Elastic
MapReduce, Spark Streaming, Amazon DynamoDB, Amazon API
Gateway, and AWS Lambda
• DynamoDB became costly for this use case, and we needed to find
an alternative solution to meet cost and performance expectations

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key architecture considerations
Balance
performance & cost
Minimize
DevOps impact
Focused on AWS
Managed Services

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
iRobot analytics architecture
Accepted
Rule
Processed
Data
Robot
DataAPI
Amazon
Athena
1 7 9
5
6
AWS ElastiCache
Accepted
Firehose
Accepted
Lambda
Kinesis
Firehose
8
Last Hour of
Data
Data
Pipeline
Landing
Data
3 4
Raw
Data
Aspen 2.x
Platform
Aspen 2.x Analytics
RS Data
Pipeline
Redshift
HDH
Tables
Accepted
Bucket
2
13 14
10 11 12

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more about IoT & analytics

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Linux Migration
Lance Smith
Associate Director
Celgene
C e l g e n e

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2014
2015
2016
2017
2018+
AWS WAF
Amazon SQS
AWS Direct Connect
AWS CloudFront
AWS
IoT
AWS Storage Gateway
AWS Lambda
Amazon S3
Amazon EC2
Amazon RDS
Amazon Aurora
Amazon EFS
Amazon EMR
Amazon WorkSpaces
Linux migration
• Migrate COTS-HPC applications to AWS
• On-premise clusters no longer sized for data intensive projects
• Desire to move to elastic computational environment
• Data protections in place
First Foray First Migrations Cloud First

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key architecture challenges
• Application not
made for cloud
• No app/code
changes
• Need to refactor
for best practices
GENOMICS HPC
COMPUTATIONAL
CHEMISTRY

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HPC architecture
VPC subnet
Auto
Scaling
Amazon
EFS
VPC subnet
Auto
Scaling
Auto
Scaling
Amazon
EC2
Clients
Amazon
EC2
Clients
Amazon
EC2
Clients
HPC Cluster
GPFS workstation
Celgene Sites
AWS File
Gateway
Amazon
S3
AWS
CloudWatch
Amazon
SQS

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hear more from Celgene…

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Viber—Data Lake Challenges
Amir Ish-Shalom
Chief Architect
R a k u t e n V i b e r

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Messaging (including group)
• Secure end-to-end encryption
• Rich media and chat extensions
• Full multiple-device support
• HD video and phone calls
• Viber Out and Viber In
• Public chats and accounts
• Chatbots

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Big data @ Viber
• Close to 1 billion users worldwide
• Globally used in 230 countries
• 10-15 billion events daily (2TB)
• 300,000 events per second (peak hours)
• 5PB of data stored on Amazon S3
• NoSQL DB (Couchbase) performing
2 million TPS on 20TB of data
with 35 billion keys

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Viber—big data architecture
RT data pipeline
Kinesis
Viber backend
servers
RT data processor
Apache Storm
Data storage
S3
Raw data backup
Kinesis Firehose
ETL jobs
Spark, Presto, Pig,
Lambda functions
Query engines
Presto, Athena,
Spark SQL, Pig
Reporting tools
Tableau, Redash,
Zeppelin, others
Databases &
data warehouses
Redshift, Aurora,
MySQL
Events
NoSQL profile database
Couchbase

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenge
• Over 300 different event types with large throughput variance
• Storm data processor created many small files, especially for lower
throughput events
• Events were stored in Hive-partitioned folders, which are not
optimal for Amazon S3
• Running a query over these events using Presto could generate up
to 15K tps on a single S3 bucket, resulting in 5xx errors and
throttling the whole bucket for other processes

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Current solution
• Concatenate small files into large files, optimally 100MB+
• Convert files into columnar file format such as Parquet or ORC
S3DistCp Spark
Concatenate
files
Convert to
Parquet
S3 S3 S3

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Future solution
• Concatenate and convert files in a single process (Glue?)
• Use better partitioned Hive directory format (H/D/M/Y instead of
Y/M/D/H)
• Use even larger files for high throughput events
Spark/Glue
Concatenate files &
convert to Parquet
S3 S3

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more about data lakes

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
INEXA Cloud—Answers at the Edge
Rahul C. Thakkar, Director, Commercial Cloud
Steven Hoffert, Lead, Special Projects
I N S I T U I n c . , a B o e i n g C o m p a n y

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Yes Problemo!
• We are in the middle of nowhere – no Internet, think Australian Outback, an open-pit mine, a collection of
gas well heads, linear infrastructure like pipelines
• We want to build an accurate (under 10 cm/pixel) 3D (or what we also call 2.5D) model - Why? Because our
clients want our spatio-temporal algorithms to find what, if anything is broken
• We get 30+ TB of data for that location, we may have many many such locations operational globally
Problem
• We want to verify that the data we collected on that day was decent on the
ground at the “edge”
• We want that data to reach our cloud for further and continuous future
processing

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key architecture challenges
1
2
• Middle of nowhere
• Risk for people
• Less area covered
• Lot of data (TB)
• Time to result in days
• $$$$$
• Use unmanned
• Minimize risk to people
• 10x or more area
• Data to cloud
• Time to result in hours
• $$

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key architecture challenges
A boat load
of images
TB++
3
Days
Finally, we can
start analysis

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The solution

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Q&A
Don’t forget about…
• Speaker Meet & Greet in the Park (behind the Expo, by
the LINQ) —1 to 2 p.m. Thursday
• STG312 & STG313
• Demo in the booth
• Stop anyone with this logo to talk about Amazon S3

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!