Knowing what users are doing on your websites in real time provides insights you can act on without waiting for delayed batch processing of clickstream data. Watching the immediate impact on user behavior after new releases, detecting and responding to anomalies, situational awareness, and evaluating trends are all benefits of real-time website analytics. In this workshop, we build a cost-optimized platform to capture web beacon traffic, analyze it for interesting metrics, and display it on a customized dashboard. We start by deploying the Web Analytics Solution Accelerator, then once the core is complete, we extend their solution to capture new and interesting metrics, process those with Amazon Kinesis Analytics, and display new graphs on their custom dashboard. Participants come away with a fully functional system for capturing, analyzing, and displaying valuable website metrics in real time.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-Time Web Analytics with
Kinesis Data Analytics
Luke Youngblood
Principal Solutions Architect
Amazon Web Services
A D T 4 0 1
Chris Marshall
Senior Solutions Architect
Amazon Web Services
Mark LaRosa
Principal Solutions Architect
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction
Module 1—Capturing real-time clickstream events from web servers
Module 2—Performing real-time analytics with Amazon Kinesis Data
Analytics
Module 3—Visualizing metrics using Amazon CloudWatch dashboards
Module 4—Adding custom metrics and extending the solution

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why are we here?
Every company is a digital
engagement company
With IoT even physical is digital
If you’re not using data to drive
decisions, you’re doing it wrong

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why real-time?
With the AWS Cloud, infrastructure is not a constraint
Time is the new constraint—For both operations and
decision making

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Timely decisions require new data in minutes
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
Real-time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Most common uses of streaming
Industrial
Automation
Smart home
Smart city
Data
lakes
AWS IoT
Analytics
Log
analytics

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis:Act on information as it happens
Capture, store, and analyze streaming data
Build custom applications that process
or analyze streaming data such as
real-time content recommendations
Quickly load TBs per hour of
streaming data into the cloud for
applications such as social media
analysis or IoT
Hearst Corporation sends clickstream data containing content and audience
information from over 250 digital properties with Amazon Kinesis to make
real-time content recommendations to maximize audience engagement

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New ad technologies: In image
New advertising formats IN-IMAGE
 400M daily visitors across 2,000 premium
publishers
 1 billion events (approximately 6 TB of
data) every day
“With AWS, we can smoothly process billions of
events daily with 24/7 reliability, and can scale
quickly to meet spikes in demand ...”
Ken Weiner, chief technology officer

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What you will build today …
VPC
Availability zone
Subnet
AWS Auto
Scaling group
Availability zone
Subnet
Web server
Web server
Application
Load
Balancer
Dashboard
users

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze data and video streams in real-time
Capture, process, and
store video streams
Amazon Kinesis Video
Streams
Load data streams into
AWS data stores
Analyze data streams with
SQL
Capture, process, and
store data streams
Amazon Kinesis Data
Streams
Amazon Kinesis Data
Firehose
Amazon Kinesis Data
Analytics

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams overview
• Easy administration and low cost
• Build real-time applications with framework of choice
• Secure, durable storage

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams producers and consumers
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer Library
Get* APIs
Kinesis Client Library
+ Connector Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module 1—What you will build …
VPC
Availability zone
Subnet
AWS Auto
Scaling group
Availability zone
Subnet
Web server
Web server
Application
Load
Balancer

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accessing the workshop
Start here: bit.ly/2CLtFYY
Create a AWS Cloud9
environment
Use EU (Ireland) as your region
Apply AWS credits to your account
Be sure to clean up resources at
the end to avoid unexpected costs

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Work in progress—Module 1—20 minutes

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Analytics—How it works
• Powerful real-time applications
• Easy to use, fully managed
• Automatic elasticity

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Analytics applications
Easily write SQL code to process streaming data
Connect to streaming source
Continuously deliver SQL results
1011101
1011010
0101010
1011101
1011010
0101010

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Connect to streaming data sources
• Easily connect to Kinesis Data Streams and
Kinesis Data Firehose delivery streams
• Automatic schema discovery which works
for CSV and JSON data
• Supports multiple event types, arbitrary
object nesting, single level of array nesting
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose
1011101
1011010
0101010

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easily write SQL code to process data streams
• Sub-second end to end processing latencies
• SQL steps can be chained together in serial or
parallel steps
• Build applications with one or hundreds of queries
• Pre-built functions include everything from sum and
count distinct to machine learning algorithms
• Aggregations run continuously using Window operators
SQL

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Window types
Sliding, tumbling, and custom windows
Tumbling windows are fixed size and grouped keys do not overlap
Source
Time
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Writing streaming SQL
Streams (in memory tables)
CREATE STREAM calls_per_ip_stream(
eventTimeStamp TIMESTAMP,
callCount BIGINT,
ipAddress VARCHAR(256),
);

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Writing streaming SQL
Pumps (continuous query)
CREATE OR REPLACE PUMP calls_per_ip_pump AS
INSERT INTO calls_per_ip_stream
SELECT STREAM "eventTimestamp",
COUNT(*),
"sourceIPAddress"
FROM source_sql_stream_001 ctrail
GROUP BY "sourceIPAddress",
STEP(ctrail.ROWTIME BY INTERVAL '1' MINUTE);

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automated machine learning capabilities
Anomaly detection
Anomaly detection with explanations
Hotspot detection
Unsupervised
AUTO
Online Real-timeAdaptive

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module 2—What you will build …

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Work in progress—Module 2—20 minutes

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB
Fully managed nonrelational database for any scale
Fast, consistent performance
Virtually unlimited throughput
Virtually unlimited storage
Encryption at rest and transit
Fine-grained access control
PCI, HIPAA, FIPS140-2 eligible
Service level agreement
Maintenance-free
Serverless
Auto scaling
Backup and restore
Global tables

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
$$$ Savings
Automated
scaling policies
Scales up when
you need it
Scales down when
you don’t
Scheduled
auto scaling
Fully managed
Auto scaling

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High performance
Millions of requests
per second
Trillions of items
Petabytes of storage
Single-digit
millisecond
read and write
latencies

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB Streams
• DynamoDB Streams captures a time-ordered sequence of
item-level modifications (stored up to 24 hours)
• A DynamoDB stream is an ordered flow of information about
changes to items in an Amazon DynamoDB table
DynamoDB Streams guarantees the following:
• Each stream record appears exactly once in the stream
• For each item that is modified in a DynamoDB table, the
stream records appear in the same sequence as the actual
modifications to the item

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event source Function Services (anything)
Changes in
data state
Requests to
endpoints
Change in
resource
• Java
• Python
• Node.js
• .NET Core
• Golang
• PowerShell
• Morecoming …
Triggers & AWS Lambda: Short intro

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module 3—What you will build …

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Work in progress—Module 3—20 minutes

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Work in progress—Module 4—15 minutes

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Extra credit—Ideas to extend the solution …
• Create a AWS Glue Crawler on the Amazon Simple Storage Service
(Amazon S3) data, use Amazon Athena to query it
 Connect Amazon QuickSight to Athena and create some dashboards
• Create a notification email when the event anomaly score exceeds a
threshold. Generate anomalies and test real-time alerting.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related events
Tuesday, November 27th
Leadership Session: Digital Advertising—Customer Learning & the Road Ahead
9:15 – 10:15 a.m. | Venetian, Level 5, Palazzo O
Wednesday, November 28th
Consumer Verticals Happy Hour
5:30 – 7:30 p.m. | Buddy V’s Ristorante, Venetian Hotel

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

48. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Luke Youngblood
Email: lukey@amazon.com
Twitter: @LukeYoungblood
Chris Marshall
chmars@amazon.com
Mark LaRosa
mlarosa@amazon.com