Amazon Kinesis makes it easy to speed up the time it takes for you to get valuable, real-time insights from your streaming data. In this session, we walk through the most popular applications that customers implement using Amazon Kinesis, including streaming extract-transform-load, continuous metric generation, and responsive analytics. Our customer Autodesk joins us to describe how they created real-time metrics generation and analytics using Amazon Kinesis and Amazon Elasticsearch Service. They walk us through their architecture and the best practices they learned in building and deploying their real-time analytics solution.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecting for Real-Time Insights
with Amazon Kinesis
Vinayak Lokhande
Principal DevOps Engineer
Autodesk
A N T 3 1 0
Ryan Nienhuis
Senior Technical Product Manager
AWS, Amazon Kinesis

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is streaming data?
Typical characteristics
Low-latencyContinuous Ordered,
incremental
High volume

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why streaming data?
Get actionable insights quickly
Source: Perishable insights, Mike Gualtieri, Forrester
Real time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams
Load data streams
into AWS data
stores
Analyze data
streams in real time
Capture, process,
and store data
streams

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams overview

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data ingestion from a variety of sources
Kinesis Data
Streams
Transactions
ERP
Web logs/
cookies
Connected
devices
AWS SDKs
• Publish directly from application code via APIs
• AWS Mobile SDK
• Managed AWS sources: CloudWatch Logs, AWS IoT, Kinesis Data
Analytics and more
• RDS Aurora via Lambda
Kinesis Agent
• Monitors log files and forwards lines as messages to Kinesis Data Streams
Kinesis Producer Library (KPL)
• Background process aggregates and batches messages
3rd party and open source
• Log4j appender
• Apache Kafka
• Flume, fluentd, and more …

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data processing from a variety of consumers
Fully managed service for real-time processing of streaming data
Cost-effective: $0.014 per 1,000,000 PUT Payload Units
Millions of sources
producing 100’s of
terabytes per hour
Amazon Web Services
Front
End
AZ AZ AZAuthentic
authorization
Durable, highly consistent storage replicas data
across three data centers (availability zones)
Ordered stream of
events supports
multiple readers
Amazon Kinesis
Client Library
on EC2
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Analytics
AWS Lambda

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Firehose—How it works
Ingest Transform Deliver
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
AWS IoT
Amazon Kinesis Agent
Amazon Kinesis Streams
Amazon CloudWatch Logs
Amazon CloudWatch Events
Apache Kafka

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch with Kibana
Derive insights from logs

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticsearch Service
Fully-managed service that is easy to deploy, manage, and scale
+
Fully managed

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Firehose delivery architecture
Amazon
Elasticsearch Service
Transformed
records
Data transformation
function
• For public access domains
• Easily transform data
• Serverless with built-in batching,
index rollover, error handling
Data source
Source records
Firehose
delivery stream
Source records
Delivery failure
Transformation failure
Source records

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Analytics
Continuously read, process, and deliver streaming data in real-time

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL on streaming data?
Aggregations (count, sum, min, … ) take granular real-time data and
turn it into insights
Data is continuously processed so you need to tell the application when
you want results
Windows!

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Window types
Sliding, tumbling, and stagger
Tumbling windows are fixed size and grouped keys do not overlap
Source
Time
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Writing streaming SQL
Pump (continuous query) using stagger window
CREATE OR REPLACE PUMP calls_per_ip_pump AS
INSERT INTO calls_per_ip_stream
SELECT STREAM source_ip_address,
COUNT(*)
FROM source_sql_stream_001
WINDOWED BY STAGGER(
PARTITION BY source_ip_address
RANGE INTERVAL '1' MINUTE);

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. Why instrumentation
2. Design principles
3. Architecture
4. Self service
5. Best practices
6. Summary

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What we do
Autodesk gives you the power to make anything!

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How we are doing

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Autodesk Forge platform
Forge is a Connected Developer Platform that powers the future of making things

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer workflow

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support scenario
Customer experience really matters
“Hey Autodesk support, I am not sure why I cannot see the model I
uploaded in the viewer yet, I need to share this for review with my
contractors and this is holding up the project.”
“Why can't I see my model yet?”

24. “Failures in todays complex, distributed and
interconnected systems are not the
exception. They are normal cases, not
predictable, and not avoidable.”
Uwe Friedrichsen
https://www.slideshare.net/ufried/patterns-of-resilience

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why instrumentation?
• Trace transactions across interconnected Forge services
• Reduce MTTD, real time problem detection and notification
• Provide forensic and log analysis capabilities
• Provide analytics—derive insights to drive features, resiliency
A consistent way to collect and measure metrics of Forge services

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Design principles
Separation of
concerns
Minimize
maintenance cost
Highly scalable
system
Fault tolerant
system
Easy to use Extensible system

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unified logging
Log data in different
formats
Cross-service tracing almost
impossible
Forensics, monitoring, and
analytics on log data hard
SLO/SLI metrics from log data
impossible
Unified logging via
standardized log data
model
Annotate log records with
distributed tracing states
Based on open tracing
specification:
http://opentracing.io
SDK supporting multiple
languages

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample UL log record
{
"ul-ctx-caller-span-id": "d62f04051cc5f312",
"ul-ctx-head-span-id": "1-5af39328-7cf81c9c9f3ac30af7693a65",
"ul-operation": "Operation_name",
"ul-span-duration": 160,
"ul-span-id": "d62f04051cc5f312",
"ul-span-start": 1525912360890,
"ul-tag-status": "INFO",
"ul-tag-ul-version": "2.1.0",
"ul-timestamp-epoch": 1525912361050
"ul-log-data": {
"cur_service_id": “xxxxxxxxxxxxxxxxxxx",
"cur_service_name": "xxxxxxx",
"method_name": "xxxx",
"module_name": "xxxxx",
"provider_name": "oxygen",
"scope": "C",
"sdk_version": "java_unified-logging-2.1.0",
"server_id": "xxxxxxx",
"significance": "INFO",
"src_service_id": "xxxxxx",
}
}

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
SQL
Processor
S P
O G

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Trace transactions across services
5+ days of log search and analysis
Log search and analysis up to five days
Real-time analytics to feed CloudWatch and SPOG
Alerting on real time metrics

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Onboarding
SQL
Processor
S P
O G

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Onboarding
SQL
Processor
1
2

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Config.yaml
predefined_templates:
- UL_p99_p95_p90_max_count200_count400_count500:
#Define variables for the template below
ul_operation_field: 'COL_uloperation'
ul_operation_duration_field: 'COL_ulspanduration'
ul_http_status_field: 'http_response_code'
ul_timestamp_field: 'COL_ultimestampepoch'
operations:
#List of operations on which you have to apply the above template
- api.method.v1
- api.method.v2

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kibana widely adopted

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example of x-ray

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: SPOG

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SPOG cont’d

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Performance testing—Lessons learned
• Identify and test your solutions limits
• We tested:
• 30 data producers
• 3 TB ingested and process per day
• 700 shards
• 200 queries per second

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon ElasticSearch Service—Lessons learned
• Monitor additional metrics, including:
• per node CPU
• per node JVM heap
• _bulk queue
• Per index field count
• Per index field limit of 1000
• Shards to index to CPU ratio
• CPU utilization below 75%
• JVM memory below 75%

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Firehose—Lessons learned
• Using AWS Lambda with your deliver stream to transform your data
• Rate limiting mechanism
• Filtering data prior to delivery
• Setting up a Firehose per data producer
• Buffer sizing
• Kinesis Agent buffers on the client side
• AWS Lambda buffer
• Delivery stream configured buffer size

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Analytics—Lessons learned
• Monitor milliseconds behind latest
• Abstract SQL code using templates
• Size your KDA app correctly

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
Separation of
concerns
Minimize
maintenance cost
Highly scalable
system
Fault tolerant
system
Easy to use
Extensible system
Sharding per data producer
AWS Managed Services
Rate limiting, shardingOnboarding, Self service

43. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.