Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Serverless London 2019 FaaS composition using Kafka and CloudEvents

212 views

Published on

FaaS composition using Kafka and Cloud-Events
LOCATION: Burton & Redgrave, DATE: November 7, 2019, TIME: 2:30 pm - 3:15 pm
https://serverlesscomputing.london/sessions/faas-composition-using-kafka-and-cloud-events/
Serverless functions or FaaS are all the rage. By leveraging well established event-driven microservice design principles and applying them to serverless functions we can build a homogenous ecosystem to run FaaS applications.
Kafka’s natural ability to store and replay events means serverless functions can not only be replayed, but they can also be used to choreograph call chains or driven using orchestration. Kafka also means we can democratize and organize FaaS environments in a way that scales across the enterprise.
Underpinning this mantra is the use of Cloud Events by the CNCF serverless working group (of which Confluent is an active member).
Objective of the talk
You will leave the talk with an understanding of what the future of cloud holds, a methodology for embracing serverless functions and how they become part of your journey to a cloud-native, event-driven architecture.

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Serverless London 2019 FaaS composition using Kafka and CloudEvents

  1. 1. 1 FaaS Composition using Kafka and CloudEvents Neil Avery, Office of the CTO, @avery_neil {...}
  2. 2. What is FaaS? ● ● ● ● ● ● ● DRIVEN BY EVENTS {...}
  3. 3. Origins of FaaS ● ● ● ● “It’s easy to run without managing infrastructure. It will take care of managing, scaling, monitoring and logging for you. Your applications run milliseconds after the event has been triggered and thousands of functions can work in parallel. You run the code only when needed.”
  4. 4. 44
  5. 5. 55
  6. 6. 66 Microservice/FaaS/Stream-Processor Single atomic unit does one thing like
  7. 7. 77 Composition <events don’t exist in isolation>
  8. 8. 88 Composition
  9. 9. 99 Composition styles 1. Wiring 2. Chaining 3. Orchestration / Workflow (Step functions, Durable Fns, Zeebe, Apache Airflow) 4. Choreography
  10. 10. 1010 1. Wiring: Infrastructure events Triggered by S3 bucket, DynamoDB, SNS or CloudWatch
  11. 11. 1111 1. Wiring: REST Endpoint Http Request payload
  12. 12. 1212 2. Call chaining Send event from one Fn to other(s) Sync or Async(s) What could go wrong? < debug, logs, testing, portability>
  13. 13. 1313 3. Orchestration / Workflow Vendor: AWS Step functions, Azure Durable Fns OpenSource: Zeebe, Apache Airflow, CNCF Serverless workflow spec
  14. 14. 14
  15. 15. 15
  16. 16. 1616 3. Orchestration using a DataFlow
  17. 17. 1717
  18. 18. 1818 Rate of adoption ● Serverless is the top-growing extended cloud service for the second year in a row, with a 50 percent growth over 2018 24 to 36 percent adoption. ● Stream processing is tied for first, increasing from 20 to 30 percent adoption. Rightscale cloud report 2019
  19. 19. 1919 What is stream processing? Event stream processing, or ESP, is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing (CEP). SELECT DataStream Orders.TimeStamp, Orders.orderId, Orders.ticker,Orders.amount, Trade.amount FROM Orders JOIN Trades OVER (RANGE INTERVAL '1' SECOND FOLLOWING) ON Orders.orderId = Trades.orderId; https://en.wikipedia.org/wiki/Event_stream_processing
  20. 20. 20 Streaming processing as SQL Stream processor STREAM /user-reg FILTER SELECT users > 18 PROJECT SELECT user.name JOIN SELECT u.name, a.country from user-reg u JOIN address a WHERE u.id = a.id GROUP BY (TABLE) SELECT u.country, count(u.name) FROM user-reg u GROUP BY u.country WINDOW (TABLE) SELECT u.country, count(u.name) FROM user-reg u WINDOW TUMBLING (SIZE 1 MIN) group by u.country STREAM /address
  21. 21. 21 1. Stream processor runs ‘future’ on local-state of item-bids auction items 3. Rejoined removes item 4. Stream triggers FaaS complete item processor bid-history Stream table join 5. FaaS Notify all bidders bid-notifications 2. Rewrite ‘completed’ item status to retrigger join item status
  22. 22. 22 Kafka Streams ● Data driven (dataflow) ● Partition aware ● Scale ● Resilient ● Delivery guarantees ● Replay events and time-travel [Java, Scala]
  23. 23. 23 { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.filter( (key, payment) -> payment.isComplete()).to(TASK_COMPLETE); … Topology gateTopology = builder.build(); streams = new KafkaStreams(gateTopology, streamsConfig); streams.start(); }
  24. 24. 24 { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.flatMapValues(payment -> { ArrayList<String> fanTasks = new ArrayList<>(); for (int i = 0; i < payment.getAmount(); i++) { fanTasks.add("fan-" + i); } return fanTasks; } ).to(TASK_FANOUT); flatMapValues: Many V to the same K Or flatMap: Many K,V to one K,V pair
  25. 25. 25 builder.stream(TASK_FANOUT) .groupByKey(Grouped.with(Serdes.String(), Serdes.String())) .windowedBy(SessionWindows.with(Duration.ofMinutes(1))) .reduce(new Reducer<String>() { public String apply(String agg1, String newValue) { return agg1 + newValue; } }) .toStream() .map((key, value) -> new KeyValue<>(key.key() + "@" + key.window().start() + "->" + key.window().end(), value)) .to(TASK_REDUCE); or: groupBy(KeyValueMapper)
  26. 26. {...} {...} {...} { bid:100; sold:1000;} /bid-history { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } Calculating percentiles using a ‘unit-of-work’ pattern { digest.merge(digest) } Stream processor with a session window to join related events by key (k, v[1-10]) Stream processor fan out is natural (data-driven) OR use a Stream processor flatMap()
  27. 27. 27 Stream processing ‘FaaS’ Patterns ● Filter (gate) ● Transform ● Fan-out (one to many) ● Fan-in (many to one - join) ● Branch/Choose (if ‘n’ then) ● Aggregate ● ...more...
  28. 28. 2828 A Stream processing DataFlow is both workflow and choreography (EDA, SEDA, Event-driven, streaming)
  29. 29. 2929 Choreography 1. Fire and forget / Async events 2. Producers emit state (not commands) 3. Consumers react to events 4. There is no master plan, no central brain LOOSE COUPLING
  30. 30. 30 Choreography
  31. 31. 31 Choreography and Evolvability
  32. 32. 32 An Event records the fact that something happened 32 A good was sold An invoice was issued A payment was made A new customer registered
  33. 33. 33 Capture behavior
  34. 34. 34 Evolvability user experience? how many users affected? has it happened before? new old supports data change, logic change, logic extension, schema evolution, loose coupling, add processors, A/B path
  35. 35. 35 is a distributed event streaming platform Publish & Subscribe to Events Store Events Process & Analyze Events
  36. 36. 3636 Choreography with Kafka
  37. 37. 37 payment.incoming.pre 2. Payment Inflight payment.incoming 1. {...} Fraudulent Check “FaaS Connector” a bridge between Kafka topic(s) and serverless function(s)
  38. 38. 3838 Confluent Hub confluent.io/hub
  39. 39. 39 FaaS Connector modes 1. Sync: ● fire and queue response ● preserve order 2. Async: ● fire and forget ● order agnostic https://www.confluent.io/hub → AWS Lambda Connector (preview) → Azure Functions Connector (preview) → GCP Functions Connector (preview) <confluent-platform> ../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties
  40. 40. 40 payment.incoming.pre 2. Payment Inflight payment.incoming 1. {...} Fraudulent Check Connector Bridge between Kaka topic and Fn invocation Producer Write back to Kafka?
  41. 41. {...} {...} {...} {...} bad stream
  42. 42. 42 Bad stream? As we say in IT...
  43. 43. 4343
  44. 44. 44 Explain FaaS Connector modes 1. Sync: use case (ordered) - state change analytics: add then remove values; i.e. CDC - by topic partition 2. Async: use case (not ordered) - compute intensive analytics; what-if scenario analysis - machine learning training - ETL <confluent-platform> ../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties
  45. 45. 45 Kafka Streams is a Java app…. It’s not very ‘Cloud native’ - without 45 Kubeless Cloud Foundry Knative GraalVm Quarkus
  46. 46. 4646 Composition at scale?
  47. 47. {faas} Central nervous system appappappapp Payments Department 2 {faas}appappappapp Department 3 Department 4 Choreography Choreography Orchestration Hierarchical topic naming
  48. 48. 4848 At Scale: CloudEvents
  49. 49. {faas} What is the message format James? appappappapp Payments Department 2 I’m so confused!
  50. 50. 50 https://cloudevents.io/
  51. 51. 51
  52. 52. 52 ● CNCF ● Version 1.0 October 2019 ● Specification (an Event envelope) ● Transport Bindings (Kafka, AMQP, HTTP others) ● SDKs: Java, .NET, Go-lang, Javascript etc ● Useful for exposing to external apps (Events as APIs) CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems.
  53. 53. 5353 CloudEvent semantics ● Self describing / self contained / stand alone ● Suppressed / non-intruisive ● Language, transport, data format independent ● Propagation ● Extensions ● Batteries included
  54. 54. 54 { "specversion" : "0.4-wip", "type" : "com.github.pull.create", "source" : "https://github.com/cloudevents/spec/pull", "subject" : "123", "id" : "A234-1234-1234", "time" : "2018-04-05T17:31:00Z", "comexampleextension1" : "value", "comexampleextension2" : { "othervalue": 5 }, "datacontenttype" : "text/xml", "data" : "<much wow="xml"/>" }
  55. 55. 55 CloudEvents Producer Consumer SerDes Kafka Streams and KSQL SerDes Kafka Connect Convertor Cloud Function Connectors AWS, GCP, Azure and the Kafka Ecosystem
  56. 56. 56 KStream<String, CloudEvent<AttributesImpl, Payment>> payments = builder.stream(paymentTopic, Consumed.with(StringSerde, new CeBinarySerDes<>(Payment.class))); payments.filter((k,v) -> { return v.getData().get().getOrigin().contains("UK"); }) CloudEvents with Kafka Streams
  57. 57. {faas} Central nervous system appappappapp Payments Department 2 {faas}appappappapp Department 3 Department 4 Choreography Orchestration Choreography Orchestration Zeebe Hierarchical topic naming Kafka Streams Quarkus GraalVM KNative CloudEvents Confluent Cloud
  58. 58. 5858 Where to go from here for more details on event-driven architectures with Kafka @avery_neil
  59. 59. 59 Questions? @avery_neil “Journey to event driven” blog 1. Event-first thinking 2. Programming models 3. Serverless 4. Pillars of event-streaming ms’s https://bit.ly/2tFfU84 or @avery_neil twitter profile
  60. 60. 60 @avery_neil
  61. 61. 61 Cloud Native Kafka Streams (wow!) 61 1. StreamsTopology.java → docker-compose up --build. → topology.q-native-container 2. Create a Knative service descriptor topology-java-quarkus.yaml 3. Run locally: ./mvnw compile quarkus:dev 4. Publish docker push {username}/topology-java-quarkus 5. Deploy kubectl apply --filename service.yaml 6. Observe: kubectl get ksvc topology-java-quarkus NAME URL : topology-java-quarkus http://topology-java-quarkus.default.1.2.3.4.xip.io https://quarkus.io/guides/kafka-streams-guide https://knative.dev/community/samples/serving/helloworld-java-quarkus/

×