Calling all serverless developers! Wild Rydes (www.wildrydes.com), the world’s leading unicorn transportation startup, needs your help! The company's rydesharing network of unicorns has grown to thousands, fulfilling hundreds of thousands of passenger rydes each day. Your mission is to collect, store, process, and analyze data to track the real-time location and health of our unicorns. In this workshop, learn how to build infrastructure to process data streams in real time using Amazon Kinesis. Build serverless applications using Amazon Kinesis Analytics to aggregate and summarize data, and use AWS Lambda to store aggregated data in Amazon DynamoDB. Finally, use Amazon Kinesis Firehose to build a data lake in Amazon S3, and use Amazon Athena to run ad-hoc queries against it. Requirements: laptop, text editor, AWS account, and AWS Command Line Interface (CLI) installed and configured.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building Serverless Real-Time Data
Processing (Now with Unicorns!)
W o r k s h o p
J o h n P i g n a t a , R o b C a m b r a , R y a n N i t z
AWS re:INVENT
S R V 3 3 2
N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect
This workshop is an interactive exercise that builds infrastructure
to collect, process, and persist data without using servers
Agenda
• Overview of workshop scenario, modules, and the services we’ll
utilize
• Review of workshop prerequisites and tools
• Execution of four modules—each about thirty minutes in
length

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hello!
I’m John. That’s Ryan and Rob.
We’re here to work with you to explore serverless data
processing. Please flag us down any time to help or if
there’s anything we can do.
Work through this workshop with your fellow participants!
Let’s take five minutes to go around the table and
introduce ourselves.

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workshop scenario

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wild Rydes is an innovative
transportation service that
helps people get to their
destination faster and hassle-
free via unicorns
www.wildrydes.com

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenge
Your operations team needs a way to monitor the status
and health of thousands of unicorns in real-time
Where are they?
How many magic points do they have left?
How fast are they traveling?

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Modules

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module one: Real-time streaming data
Shadowfax Kinesisstream Consumer
Build and demonstrate a stream
for real-time data from the
unicorn fleet
• Create an Amazon Kinesis stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream every second
• Use the Kinesis command-line
consumer to read the sensor data
from the stream
• Use the Unicorn Dashboard to view
unicorns in real-time on a map

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module two: Streaming aggregation
Shadowfax Kinesisstream
Consumer
Create a serverless application to
summarize sensor data every
minute
• Create an Amazon Kinesis Analytics
application to output one row per
minute per unicorn to a new stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream
• Use the Kinesis command-line
consumer to read the summarized
sensor data from the stream
KinesisAnalytics
application
Kinesisstream

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module three: Stream processing
Shadowfax Kinesisstream
Build an AWS Lambda function
to persist sensor data to an
Amazon DynamoDB table
• Create a new DynamoDB table
• Build a Lambda function and
configure it to trigger when data is in
the stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream
• Use the DynamoDB console to verify
that data is being persisted
Lambda
DynamoDB

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module four: Data lake
Shadowfax Kinesisstream
Write raw sensor data to Amazon
S3 via Amazon Kinesis Firehose
and run queries using Amazon
Athena
• Create an Amazon S3 bucket to store
raw sensor data from our unicorns in
JSON format
• Create a Kinesis Firehose delivery
stream to deliver batches from the
Kinesis stream to Amazon S3
• Use Amazon Athena to run queries
against the raw data stored in
Amazon S3
KinesisFirehose
S3 Bucket
Athena

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tools and prerequisites

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Web browser
This workshop has been tested
in the latest versions of Mozilla
Firefox and Google Chrome
Both browsers feature a
Developer Console which you
may need to use to see log
information from the Unicorn
Dashboard

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab guide
Includes setup
information, instructions
for each module, and
step-by-step instructions
if you get stuck
Expand ▶ Step-by-step
Instructions for a
detailed walkthrough in
each section

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Unicorn Dashboard

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS account
You’ll need an AWS account and access to
administer resources for:
• AWS Identity and Access Management (IAM)
• Amazon Simple Storage Service (Amazon
S3)
• Amazon DynamoDB
• AWS Lambda
• Amazon Kinesis Streams
• Amazon Kinesis Firehose
• Amazon Kinesis Analytics
• Amazon Athena
The workshop instructions assume only
one participant is using a given AWS
account at a time
It’s best to use a personal account or
register a new account rather than using
your organization’s AWS account
You will receive $25 of AWS credit to
cover any expenses incurred in this
workshop

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS region
• Build within a single region that supports all necessary services
• The Region Table shows what services are supported within each region
• https://aws.amazon.com/about-aws/global-infrastructure/regional-
product-services/ (search “aws region table”)
• Regions for this workshop:
• US East (N. Virginia)
• US West (Oregon)
• EU (Ireland)

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis command-line clients
CLI client to simulate and display sensor data from the unicorn fleet
Producer
• Simulates unicorn sensor data
• Unicorn name
• Timestamp
• Current position (latitude, longitude)
• Distance traveled in the last second in meters
• Magic points
• Health points
• Data encoded as a JSON object
Consumer
• Reads from the end of the stream and outputs data

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis command-line clients
Written in the Go Programming Language and provided as binaries for Windows, Linux, and macOS or build
from source yourself with the Go tools

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
http://reinvent2017.wildrydes.com

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions?

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
http://reinvent2017.wildrydes.com

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!