Stream processing facilitates the collection, processing, and analysis of real-time data and enables the continuous generation of insights and quick reactions to emerging situations. Yet, despite these advantages compared to traditional batch-oriented analytics applications, streaming applications are much more challenging to operate. Some of these challenges include the ability to provide and maintain low end-to-end latency, to seamlessly recover from failure, and to deal with a varying amount of throughput. We all know and love Flink to take on those challenges with grace. In this session, we explore an end to end example that shows how you can use Apache Flink and Amazon Kinesis Data Analytics for Java Applications to build a reliable, scalable, and highly available streaming applications. We discuss how you can leverage managed services to quickly build Flink based streaming applications and show managed services can help to substantially reduce the operational overhead that is required to run the application. We also review best practices for running streaming applications with Apache Flink on AWS. So you will not only see how to actually build streaming applications with Apache Flink on AWS, you will also learn how leveraging managed services can help to reduce the overhead that is usually required to build and operate streaming applications to a bare minimum.

1. © 2019, Amazon Web Services, Inc. or its Affiliates.
Dr. Steffen Hausmann (@sthmmm)
Specialist Solutions Architect Analytics, EMEA
Amazon Web Services
Build and run streaming applications with Apache
Flink and Amazon Kinesis Data Analytics for Java
Applications

2. © 2019, Amazon Web Services, Inc. or its Affiliates.
Architecture for Streaming Analytics with Apache Flink
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics for Java
Applications
Amazon Elasticsearch
Service
Ingestion layer Processing layer Presentation layer

3. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates.
Let’s go build!

4. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates.
Kinesis Data Analytics under the
hood and best practices

5. © 2019, Amazon Web Services, Inc. or its Affiliates.
Amazon Kinesis Data Analytics under the hood
Kinesis Data Analytics manages the underlying infrastructure
• Operates Zookeeper for high availability
• Configures checkpoints and savepoints
• Automatic failover to healthy nodes
Kinesis Processing Units (or KPUs)
• Basic scaling unit
• Determines parallelism of the application
• Can be further adapted for I/O bound applications with
ParallelismPerKPU

6. © 2019, Amazon Web Services, Inc. or its Affiliates.
Scaling and Autoscaling
A scaling operation causes downtime of the application
• Scaling triggers a savepoint and
• starts a new application with adapted parallelism
Autoscaling
• Scale up triggered in minutes after a constant spike in CPU usage
• Scale down triggered a few hours after a drop in CPU usage
• Disable autoscaling if you are not CPU bound

7. © 2019, Amazon Web Services, Inc. or its Affiliates.
Monitoring and Metrics
Application metrics are exposed through Amazon CloudWatch
• IncomingRecords/ Bytes
• Write/ ReadProvisionedThroughputExceeded
• millisBehindLatest

8. © 2019, Amazon Web Services, Inc. or its Affiliates.
Logging
Application logs are exposed through CloudWatch Logs
• Send custom messages with Log4J & SLF4J
• Avoid extensive logging on the data plane path
• Search and analyze logs with CloudWatch Logs Insights

9. © 2019, Amazon Web Services, Inc. or its Affiliates.
Data protection and security
Data Protection
• You can encrypt data on the incoming Kinesis data stream
• All data stored in running application storage is encrypted at rest
• You can encrypt data in transit and at rest in Elasticsearch service
Avoid baking credentials into your code
• Use temporary credentials for integrating with AWS services
• Use AWS Secrets Manager to retrieve and rotate passwords
kinesisConsumerConfig.setProperty(
AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO"
);

10. © 2019, Amazon Web Services, Inc. or its Affiliates.
Further Readings
The AWS Big Data blog is a great resource to learn more about Apache Flink and
stream processing on AWS in general.
• https://aws.amazon.com/blogs/big-data/build-and-run-streaming-applications-
with-apache-flink-and-amazon-kinesis-data-analytics-for-java-applications/
• https://aws.amazon.com/blogs/big-data/build-a-real-time-stream-processing-
pipeline-with-apache-flink-on-aws/
• https://github.com/aws-samples/amazon-kinesis-analytics-taxi-consumer

11. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates.
Thank you!