Downloaded 31 times
Uploaded byJeremy Daly
Building Event-Driven Applications with Serverless and AWS - AWS Summit New York
The document outlines a presentation on building event-driven applications using serverless technologies on AWS, led by Jeremy Daly, CTO at alertme.news. It covers key concepts such as event-driven architecture, communication types (synchronous vs asynchronous), and the use of services like AWS Lambda, SNS, and EventBridge for creating scalable, serverless workflows. The emphasis is on maximizing service autonomy and minimizing coupling through effective event management and orchestration.
More Related Content
Similar to Building Event-Driven Applications with Serverless and AWS - AWS Summit New York
Building Event-Driven Applications with Serverless and AWS - AWS Summit New York
- 1. S U MM I T NEW YORK
- 2. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Building event-driven applications with serverless and AWS Jeremy Daly Chief Technology Officer, AlertMe.news @jeremy_daly D E V 1 1
- 3. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Agenda How Serverless Works Event-Driven Applications Multicasting Events Serverless Webhook Patterns
- 4. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T About Me • CTO at AlertMe.news • Consult with companies building in the cloud • 20+ year veteran of technology startups • Started working with AWS in 2009 and started using Lambda in 2015 • Blogger (jeremydaly.com), open-source contributor, speaker • Publish the Off-by-none serverless newsletter • Host of the Serverless Chats podcast
- 5. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 6. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Basic Serverless Workflow LAMBDA EVENT SOURCES … CLOUD RESOURCES Event Trigger Unique business logicEVENT Output
- 7. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple Web Service Client API Gateway Lambda DynamoDB “Synchronous” Request
- 8. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple S3 Trigger Client Lambda DynamoDBS3 Bucket “Asynchronous” Request
- 9. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 10. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T What are Events? • Events are FACTS • Events are IMMUTABLE • Events are OBSERVABLE • Events are TEMPORAL Timestamp Event 2019-07-11 16:35:20 OrderCreated 2019-07-11 16:35:43 PaymentProcessed 2019-07-11 16:36:01 InventoryUpdated 2019-07-11 16:36:07 ConfirmationSent … …
- 11. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Event-Driven versus Event-Sourcing Event-Driven • Domain events, i.e. something happened within a bounded context • High-value semantics (user registered, order placed, payment processed) • Relevant within a bounded context and across bounded contexts Event-Sourcing • A stored sequence of events that describes the current state • Events are specific to a bounded context, i.e. domain logic is required to reconstruct state (provide email, verify email) • These events aren’t particularly interesting outside of the bounded context
- 12. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Communication Types Synchronous Communication Services can be invoked by other services and must wait for a reply. This is considered a blocking request, because the invoking service cannot finish executing until a response is received. Asynchronous Communication This is a non-blocking request. A service can invoke (or trigger) another service directly or it can use another type of communication channel to queue information. The service typically only needs to wait for confirmation (ack) that the request was received.
- 13. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Orchestration versus Choreography 💃🕺 Orchestration • Use a controller to coordinate the behavior of multiple components or services • Uas a way to maintain state between multiple steps in a transaction Choreography • Messages are sent to an event bus or queue and are picked up by other services • We are not waiting for downstream services to complete before continuing “Each service is smart enough to understand its role in the whole dance.” ~ Sam Newman AWS Step Functions
- 14. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 15. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Using SNS to Route Events Client SNSTopic “Asynchronous” Request ack Serverless Microservice HTTP SMS Lambda SQS Email Key Points: • SNS has a “well-defined API” • Decouples downstream processes • Allows multiple subscribers with message filters
- 16. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Event Forking with SNS “Asynchronous” Request ack SQS Queue LambdaSNS Topic DynamoDBClient API Gateway Lambda Event processing pipeline SQS Queue Lambda Kinesis Firehose Event Search & Analytics pipeline Elasticsearch S3 Dead Letter Bucket SQS Queue Lambda Kinesis Firehose Event Storage & Backup pipeline S3 Backup Bucket amount > $100 Key Points: • SNS to SQS is “guaranteed” (100,010 retries) • Filters allow you to control pipeline data • Common patterns that can be reused Event Fork Pipelines Available in the Serverless Application Repository Event Fork Pipelines Available in the Serverless Application Repository
- 17. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distribute & Throttle Stripe API ack SQS Queue Lambda (concurrency 25) SNS Topic Client API Gateway Lambda Order Service total > $0 RDS SQS Queue Lambda (concurrency 10) SMS Alerting Service Twilio API SQS Queue Lambda (concurrency 5) Billing Service status == ”order_complete” Event Service Key Points: • Filter events to selectively trigger services • Manage throttling/quotas per service • Reuses the same SNS topic to consolidate events
- 18. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudWatch Events https://aws.amazon.com/blogs/aws/ building-serverless-pipelines-with- amazon-cloudwatch-events/
- 19. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 20. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 21. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T EventBridge has 95+ Event Sources and 17 Target Types! • Batch job queue • CloudWatch log group • CodeBuild project • CodePipeline • EC2 (CreateSnapshot, RebootInstances, StopInstances, TerminateInstances APIs) • ECS task • Event bus in another AWS account • Firehose delivery stream • Inspector assessment template • Kinesis stream • Lambda function • SNS topic • SQS queue • SSM Automation • SSM OpsItem • SSM Run Command • Step Functions state machine NO SMS, HTTP, MOBILE or EMAIL
- 22. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T SNS versus CloudWatch Events versus Kinesis • SNS subscriptions can FILTER events • Advanced pattern matching through filter policies • String and Numeric Value Matching (Whitelisting, Blacklisting, and Prefix Matching and Range Matching) – ONLY ON MESSAGE ATTRIBUTES • CloudWatch Event rules can MATCH events • Matching is exact (character-by-character) using event patterns, without case- folding or any other string normalization – ON ALL EVENT FIELDS • Kinesis Streams can do NEITHER • Event filtering must be done by the consumer
- 23. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multicasting with EventBridge Asynchronous “PutEvents” Request ack w/ event id Key Points: • Create up to 100 event buses per account • Allows multiple subscribers with RULES and EVENT PATTERNS • Forward events to other accounts Amazon EventBridge Lambda SQS Client Step Function Event Bus +13 others
- 24. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Creating Rules with Targets
- 25. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create custom event buses
- 26. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 27. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple Serverless Webhook Client API Gateway Lambda DynamoDB Highly Scalable Highly Scalable Highly Scalable
- 28. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple Serverless Webhook Client API Gateway Lambda Highly Scalable Highly Scalable ^ not so RDS NotThat Scalable 😳 RDBMS and FaaS don’t play nicely together: • Concurrency model doesn’t allow connection pooling • Limited number of DB connections available • Recycled containers create zombies
- 29. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalable Serverless Webhook RDSClient API Gateway SQS Queue SQS (DLQ) Lambda Lambda (throttled) ack “Asynchronous” Request Synchronous Request
- 30. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. High Throughput Serverless Webhook RDSClient SQS Queue SQS (DLQ) Lambda Lambda (throttled) ack “Asynchronous” Request Synchronous Request ALB Lambda TargetType Use Application Load Balancer as your HTTP endpoint
- 31. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Direct Serverless Webhook RDSClient API Gateway SQS Queue SQS (DLQ) Lambda (throttled) ack “Asynchronous” Request Synchronous Request Use service integrations with VTL to bypass Lambda
- 32. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 33. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Integrating with Partners (no more webhooks) • Provide AWS Account ID to an approved Partner • Partner submits request to publish events to AWS Account • Select Partner Event Sources, approve access, and create event bus • Partner publishes events (e.g. standard webhook events) using the PutPartnerEvents API • Account creates rules and event patterns to subscribe to events and route them to targets • Partner is responsible for retries to EventBridge
- 34. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Partners
- 35. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge -Add Partners in 3 Steps
- 36. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. EventBridge “Webhook” RDSSaaS Provider SQS Queue SQS (DLQ) Lambda (throttled) SaaS Provider sends events directly to an Partner specific Event Bus in your AWS account Amazon EventBridge Use rules and event patterns to subscribe consumers
- 37. S U MM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 38. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Takeaways • Events are immutable facts that can be multicast • Events minimize service coupling and increase autonomy • Think asynchronously and embrace eventual consistency • Event-driven applications should broadcast events with high-value semantics that are relevant across bounded contexts • Use the right tool for the job, and don’t reinvent the wheel (SNS, CloudWatch Events, Partner Event Buses, SQS, Event Fork Pipelines) • Use Step Functions for distributed transactions
- 39. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I T Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Jeremy Daly @jeremy_daly jeremydaly.com
- 40. © 2019, AmazonWeb Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.