Download to read offline
![52
Kafka Streams
● Data driven (dataflow)
● Partition aware
● Scale
● Resilient
● Delivery guarantees
● Replay events and time-travel
[Java, Scala]](https://image.slidesharecdn.com/cloudnative2019-faascompositionusingkafkaandcloud-events-190925160736/85/Cloud-Native-London-2019-Faas-composition-using-Kafka-and-cloud-events-52-320.jpg)
![Streaming events to FaaS: Fan-out Fan-in
{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using a ‘unit-of-work’ pattern
{
digest.merge(digest)
}
Stream processor with
a session window to
join related events by
key (k, v[1-10])
Stream
processor
fan out is natural (data-driven)
OR
use a Stream processor flatMap()](https://image.slidesharecdn.com/cloudnative2019-faascompositionusingkafkaandcloud-events-190925160736/85/Cloud-Native-London-2019-Faas-composition-using-Kafka-and-cloud-events-57-320.jpg)
Uploaded byNeil Avery
Cloud Native London 2019 Faas composition using Kafka and cloud-events
The document discusses the transformation of technology from being merely innovative for growth to being essential for survival, emphasizing the need for modern, real-time data infrastructures. It covers the evolution of microservices, the move towards asynchronous event-driven architecture, and the significance of cloud-native tools such as Kafka and FaaS. Key takeaways include the importance of event-driven applications, composition styles, and the adoption of CloudEvents for interoperability across services.
More Related Content
Similar to Cloud Native London 2019 Faas composition using Kafka and cloud-events
Cloud Native London 2019 Faas composition using Kafka and cloud-events
- 1. 1 FaaS Composition usingKafka and CloudEvents Neil Avery, Office of the CTO, @avery_neil
- 2. 22 The world ischanging.
- 3. 33 The New BusinessReality Past Technology was Innovation required for growth Running the business on yesterday’s data was “good enough” Today Technology Innovation required for survival Yesterday’s data = failure. Modern, real-time data infrastructure is required.
- 4.
- 5. 55 Transportation ETA Real-time sensor diagnostics Driver-rider match Banking Frauddetection Trading and risk systems Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-app purchases
- 6.
- 7. 77 Cloud Machine Learning Mobile Event Streaming Rethink DecisionMaking Rethink User Experience Rethink Data Rethink Data Centers
- 8. 8 is a distributedevent streaming platform Publish & Subscribe to Events Store Events Process & Analyze Events
- 9.
- 10. 1010 Out of thetar pit, 2006
- 11. 1111 “We believe thatthe major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system.” Out of the tar pit, 2006
- 12. 1212 We like ‘allthe things’
- 13.
- 14.
- 15. 1515 What are microservices? Microservicesare a software development technique - a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. https://en.wikipedia.org/wiki/Microservices
- 16. 1616 structures an applicationas a collection of loosely coupled services. https://en.wikipedia.org/wiki/Microservices this is new!
- 17. 1717 So what wentwrong?
- 18.
- 19. 19 REST anyone? Makingchanges is risky
- 20. 20 Handling state ishard Cache? Embedded? Route to right instance?
- 21. 21 Shifts in responsibility,redundancy
- 22. 22 ● Scaling ishard ● Handling state is hard ● Sharing, coordinating is hard ● Run a database in each microservice - is hard What have we learned about microservices?
- 23. 2323 Microservices We had itall wrong (again) FIX: Make them asynchronous and use streams of events We didn’t really know what they were anyway
- 24.
- 25. 25 Typical event →command interaction
- 26. 26 Event → firstinteraction
- 27.
- 28. 2828 Stop! Isn’t FaaS likea fine-grained, event driven microservice?
- 29.
- 30. What is FaaS? ● ● ● ● ● ● ● DRIVENBY EVENTS {...}
- 31. Stream processors FaaS(serverless fns) {...}
- 32.
- 33.
- 34. 34 Time travel userexperience? how many users affected?has it happened before? Ask many questions of the same data, again and again time
- 35. 35 Evolvability user experience? how manyusers affected? has it happened before? new old supports data change, logic change, logic extension, schema evolution, loose coupling, add processors, A/B path
- 36. { user: 100 type: bid item:389 cat: bikes/mtb region: dc-east } Kafka partitions give you horizontal scale /bikes/ by item-id key# Key space {...} {...} {...} Topic Partition Partition assignment Stream processor FaaS function
- 37. 37 Stream processors handlestate Stream processor Stream processor Stream processor Topic: click-stream Interactive query CDC events from KTable CDC Stream partition partition partition CQRS Elastic
- 38. 3838 Rate of adoption ●Serverless is the top-growing extended cloud service for the second year in a row, with a 50 percent growth over 2018 24 to 36 percent adoption. ● Stream processing is tied for first, increasing from 20 to 30 percent adoption. Rightscale cloud report 2019
- 39.
- 40.
- 41. 4141 Composition styles 1. Manualwiring 2. Chaining 3. Choreography 4. Orchestration / Workflow (Step functions, Durable Fns, Zeebee, Apache Airflow)
- 42. 4242 Composition styles 1. Manualwiring 2. Chaining 3. Choreography 4. Orchestration / Workflow (Step functions, Durable Fns, Zeebee, Apache Airflow)
- 43.
- 44.
- 45.
- 46. 46 payment.incoming.pre 2. Payment Inflight payment.incoming 1. {...} Fraudulent Check “FaaS Connector” a bridgebetween Kafka topic(s) and serverless function(s)
- 47.
- 48. 48 Bad stream? As wesay in IT...
- 49.
- 50. 50 FaaS Connector modes 1.Sync: ● fire and queue response ● preserve order 2. Async: ● fire and forget ● order agnostic https://www.confluent.io/hub → AWS Lambda Connector (preview) → Azure Functions Connector (preview) → GCP Functions Connector (coming) <confluent-platform> ../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties
- 51.
- 52. 52 Kafka Streams ● Datadriven (dataflow) ● Partition aware ● Scale ● Resilient ● Delivery guarantees ● Replay events and time-travel [Java, Scala]
- 53. 53 Stream processing ‘FaaS’Patterns ● Filter (gate) ● Transform ● Fan-out (one to many) ● Fan-in (many to one - join) ● Branch/Choose (if ‘n’ then) ● Aggregate ● ...more...
- 54. 54 Kafka Streams: Filter/Gate- payment.isComplete() { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.filter( (key, payment) -> payment.isComplete()).to(TASK_COMPLETE); … Topology gateTopology = builder.build(); streams = new KafkaStreams(gateTopology, streamsConfig); streams.start(); }
- 55. 55 Kafka Streams: Fanout { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.flatMapValues(payment -> { ArrayList<String> fanTasks = new ArrayList<>(); for (int i = 0; i < payment.getAmount(); i++) { fanTasks.add("fan-" + i); } return fanTasks; } ).to(TASK_FANOUT); flatMapValues: Many V to the same K Or flatMap: Many K,V to one K,V pair
- 56. 56 Kafka Streams: FanIn (join) builder.stream(TASK_FANOUT) .groupByKey(Grouped.with(Serdes.String(), Serdes.String())) .windowedBy(SessionWindows.with(Duration.ofMinutes(1))) .reduce(new Reducer<String>() { public String apply(String agg1, String newValue) { return agg1 + newValue; } }) .toStream() .map((key, value) -> new KeyValue<>(key.key() + "@" + key.window().start() + "->" + key.window().end(), value)) .to(TASK_REDUCE); or: groupBy(KeyValueMapper)
- 57. Streaming events toFaaS: Fan-out Fan-in {...} {...} {...} { bid:100; sold:1000;} /bid-history { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } Calculating percentiles using a ‘unit-of-work’ pattern { digest.merge(digest) } Stream processor with a session window to join related events by key (k, v[1-10]) Stream processor fan out is natural (data-driven) OR use a Stream processor flatMap()
- 58. 58 Multiple Bounded contexts -Chaining - Layering (Events as APIs) 2. Logistics payment.incoming 1. Payments payment.complete
- 59. {faas} Central nervous system appappappapp PaymentsDepartment 2 {faas}appappappapp Department 3 Department 4 Pattern: Central nervous system Choreography Choreography Orchestration Hierarchical topic naming
- 60. {faas} What is themessage format James? appappappapp Payments Department 2 Cloud Events: bringing order to chaos I’m so confused!
- 61.
- 62. { "specversion" : "0.4-wip", "type": "com.github.pull.create", "source" : "https://github.com/cloudevents/spec/pull", "subject" : "123", "id" : "A234-1234-1234", "time" : "2018-04-05T17:31:00Z", "comexampleextension1" : "value", "comexampleextension2" : { "othervalue": 5 }, "datacontenttype" : "text/xml", "data" : "<much wow="xml"/>" }
- 63. ● Specification (anEvent envelope) ● Transport Bindings (Kafka, AMQP, HTTP others) ● SDKs: Java, .NET, Go-lang etc ● Still early - nearly version 1.0 ● Useful for exposing to external apps (Events as APIs) ● Support is coming for Kafka SerDes and CE.clients (sdk) CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems. You will adopt CloudEvents
- 64. 6464 What about thatsoftware crisis that started in 1968? “We believe that the major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system.” Out of the tar pit, 2006
- 65. Key takeaways ● Event-Streamingapps ● Patterns ● Composition ● Scale
- 66. 66 Questions? @avery_neil “Journey to eventdriven” blog 1. Event-first thinking 2. Programming models 3. Serverless 4. Pillars of event-streaming ms’s https://bit.ly/2tFfU84 or @avery_neil twitter profile
- 67.