Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sas 2015 event_driven

569 views

Published on

Shows how to build a distributed sy,stem using Vertx, Camel, AWS Lambda, Kinesis and Kafka

Published in: Technology
  • Be the first to comment

Sas 2015 event_driven

  1. 1. Sascha Möllering | zanox AG Event-driven architecture with Java
  2. 2. Sascha Möllering sascha@autoscaling.io Lead Engineer at zanox AG http://autoscaling.io @sascha242 About me
  3. 3. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  4. 4. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  5. 5. Basics • Today: building event driven system • Using: – Apache Kafka/Amazon Kinesis – Docker – Vert.x – Apache Camel/AWS Lamdba – Google’s Protobuf
  6. 6. Basics
  7. 7. Basics • Producers putting data into messaging system • Messages in Google’s Protobuf format • Consumers pulling data from messaging system
  8. 8. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  9. 9. Infrastructure • Publish-subscribe messaging • Implemented as distributed commit log • Fast: 100s of MB (reads and writes) per s from thousands of clients • Scalable: elastically and transparently expanded without downtime • Durable: Messages persisted on disk
  10. 10. Infrastructure
  11. 11. Infrastructure
  12. 12. Infrastructure • Depends on Apache Zookeeper • Zookeeper for: – Cluster management – Offset storage
  13. 13. Infrastructure • Managed service for real-time data processing • Decoupling services • Data stored for 24 hours • 1MB messages max
  14. 14. Infrastructure • Stream: – Ordered sequence of data records – Data records are distributed into shards
  15. 15. Infrastructure • Shard: – Group of data records in a stream – 1MB write per second – 2MB read per second – 1000 puts per second
  16. 16. Infrastructure • Package application with dependencies • Standardized unit for software development • Layered filesystem, share common files • Isolate applications from each other
  17. 17. Infrastructure • Docker container: stripped-to-basics version of a Linux operating system • Docker image: software you load into a container
  18. 18. Infrastructure • Docker image built with a Dockerfile • Docker images built using “inheritance” • Custom image based on “base image”
  19. 19. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  20. 20. Software & Frameworks • Toolkit for reactive applications • Based on the JVM • Event driven and non-blocking • Polyglot (Java, JS, Groovy, Ruby) • Lightweight and modular
  21. 21. Software & Frameworks
  22. 22. Software & Frameworks • Producer application built using Vert.x • Sends a message every 5s • Kafka or Kinesis depending on deployment target
  23. 23. Software & Frameworks But … how can we detect where the application is running?
  24. 24. Software & Frameworks http://169.254.169.254/latest/meta-data
  25. 25. Software & Frameworks • Framework based on EI Patterns • Small library with minimal dependencies • Define routing and mediation rules
  26. 26. Software & Frameworks
  27. 27. Software & Frameworks • Language-neutral • Platform-neutral • Extensible mechanism for serializing structured data • Support for Java, Python, and C++
  28. 28. Software & Frameworks
  29. 29. Software & Frameworks
  30. 30. Software & Frameworks
  31. 31. Software & Frameworks
  32. 32. Software & Frameworks • Compute service • Runs your code in response to events • Manages underlying compute resources
  33. 33. Software & Frameworks • Triggered by: – Modifications in S3 buckets – Notifications by SNS – Messages in Kinesis – Table updates in DynamoDB
  34. 34. Software & Frameworks • Code run in Lambda: “Lambda function” • Don’t confuse with Java 8 Lambda • Lambda functions support – Java 8 – JavaScript
  35. 35. Software & Frameworks • Two different event models – Pull event model – Push event model
  36. 36. Software & Frameworks
  37. 37. Software & Frameworks
  38. 38. Software & Frameworks
  39. 39. Software & Frameworks • Building the applications – ingestion-service – kafka-consumer-camel – kinesis-consumer-lambda
  40. 40. Software & Frameworks • ingestion-service 1. git clone https://github.com/SaschaMoelle ring/ingestion-service.git 2. mvn -Dmaven.test.skip=true package 3. docker build -t autoscaling/ingestion-service .
  41. 41. Software & Frameworks • kafka-consumer-camel 1. git clone https://github.com/SaschaMoelle ring/kafka-consumer-camel.git 2. mvn -Dmaven.test.skip=true package 3. docker build -t autoscaling/kafka-consumer .
  42. 42. Software & Frameworks • kinesis-consumer-lambda 1. git clone https://github.com/SaschaMoelle ring/kinesis-consumer- lambda.git 2. mvn -Dmaven.test.skip=true package
  43. 43. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  44. 44. Deployment • Locally 1. Start the Spotify Kafka Docker Container
  45. 45. Deployment • Why Spotify Kafka Docker Image? – Kafka depends on Zookeeper – Spotify Kafka runs Kafka and Zookeeper – No dependency to external Zookeeper – Runs out of the box
  46. 46. Deployment • Locally 2. Start the Apache Camel Kafka consumer
  47. 47. Deployment • Kafka-Consumer Docker Container – Based on phusion/baseimage – Installs Oracle Java 8 – Add consumer Fat-JAR – Starts the Fat-JAR
  48. 48. Deployment • Locally 3. Start the Vert.x Kafka Producer
  49. 49. Deployment • Vert.x Producer Docker Container – Based on phusion/baseimage – Installs Oracle Java 8 – Add producer Fat-JAR – Starts the Fat-JAR
  50. 50. Deployment • Requirements for AWS: – VPC – User Role for Kinesis access from EC2 – User Role for Kinesis access from Lambda – EC2 instance – Kinesis stream – Lambda package
  51. 51. Deployment • In AWS – Create Kinesis stream (in our case SUMMIT_STREAM)
  52. 52. Deployment • In AWS – Create Lambda function • Upload JAR to S3 bucket • Specify function • Add event source (SUMMIT_STREAM)
  53. 53. Deployment • In AWS – Start an EC2 instance • t2.small is sufficient • Install Docker and run container using EC2 user data • Important: select correct IAM role
  54. 54. Deployment • EC2 User Data #!/bin/bash -ex yum -y update yum install docker -y service docker start docker run autoscaling/ingestion-service
  55. 55. Agenda • Basics • Infrastructure • Software & Frameworks • Deployment • Putting it all together • Q&A
  56. 56. Putting it all together • Integration of Kinesis and Kafka – Kinesis consumer that processes records – Record processing -> sending to Kafka – AWS Lambda perfect choice – Problem: Lambda and VPN (VPC) not working*
  57. 57. Putting it all together • Integration testing Kinesis and Kafka – AWS API: • Create Kinesis stream in @BeforeClass • Produce data and write into stream • Delete stream in @AfterClass
  58. 58. Putting it all together
  59. 59. Putting it all together
  60. 60. Putting it all together • Integration testing Kinesis and Kafka – Spotify Docker Client • Run Spotify Kafka container in @BeforeClass • Produce data and write into stream • Stop Spotify Kafka container in @AfterClass
  61. 61. Putting it all together
  62. 62. Putting it all together • Integrationtests Kinesis and Kafka – Put messages into Kinesis – Consumer messages in application – Put messages in Kafka – Consume messages from Kafka – Compare messages
  63. 63. Putting it all together • Integrationtests Kinesis and Kafka – After tests: clean up infrastructure – Very cost effective – Real world tests without mocking – Quite fast
  64. 64. Recap • What have we achieved today? – We created a distributed, message driven system – Based on JVM and Docker – Running locally and AWS
  65. 65. Q&A
  66. 66. Resources • ingestion-service – https://github.com/SaschaMoellering/ingest ion-service.git – https://hub.docker.com/r/autoscaling/ingest ion-service/
  67. 67. Resources • kafka-consumer – https://github.com/SaschaMoellering/kafka- consumer-camel – https://hub.docker.com/r/autoscaling/kafka- consumer/
  68. 68. Resources • EC2 User Data – https://gist.github.com/SaschaMoellering/c 6ee24ec999325c43e90 • EC2 User Role – https://gist.github.com/SaschaMoellering/ a971fb73626f41ad80f4
  69. 69. Resources • Lambda User Role – https://gist.github.com/SaschaMoellering/b 14540b144263e5fea4b
  70. 70. Resources • kinesis-consumer-lamdba • https://github.com/SaschaMoellering/kinesis- consumer-lambda
  71. 71. Resources • Spotify Kafka Docker Image – https://github.com/spotify/docker-kafka.git – https://hub.docker.com/r/spotify/kafka/ • Spotify Docker Client – https://github.com/spotify/docker-client.git

×