Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Serverless Eventings
with Apache® Kafka
and Cloud Run
Viktor Gamov
Jason (Jay) Smith
By 2025, more than
a quarter of data
created in the global
datasphere will be real
time in nature.
What’s driving
the growth in
real time data?
Digital communications
Social media posts, photos, email, collaboration
platf...
I want to create a data driven organization.
I want to create a data driven organization.
event
What are the events that go into a transaction?
Need
Recognition
Information
Search
Evaluation of
alternatives
Purchase
De...
Event Streaming Enables New Outcomes
• Call for driver availability
• No knowledge of driver arrival
• No data on feature ...
Kafka Stores Events in a Log
An immutable sequence of Events
Older
Messages
Append New
Messages
Time
The Stream is spread over many machines
An immutable sequence of Events
Operations: It’s hard
It doesn’t have to be that way...
Confluent Operator It can be hard to deploy a complete event streaming
platform.
And eve...
It doesn’t have to be that way...
● Kubernetes* scales workers elastically
● Confluent Operator scales brokers elastically...
Kubernetes isn’t the endgame
Do developers want to use Kubernetes directly?
Have to do
Write code
Build docker image
Upload image to registry
Deploy se...
Serverless usage models
developers
perspective
No
Servers
Pay only
for usage
Event-driven
Portable
18
Knative
Building blocks for serverless
workloads on Kubernetes
19
What Knative is
● An open source project
● Set of building blocks to construct your own FaaS/PaaS
○ abstracts common ta...
20
What Knative is not
● It's not a Google product.
● It's not FaaS.
21
Knative Serving
Benefits
● Seamlessly scale up and down
● Built-in traffic splitting between
revisions
● Integrates net...
22
Knative eventing
Benefits
● Declaratively bind between event producers and deployed
services
● Scales from just few eve...
23
Knative Eventing
24
Knative Eventing
25
Cloud Run
Google’s Managed Version of
Knative Serving
26
27
DEMO
28
Knative Eventing
Anthos is our
platform for
application
modernization
Managed Operations
Service Management
Marketplace
Other CloudsOn-prem...
Anthos gives
you freedom to
modernize without
being locked in
Developer
Knative Cloud Run
Networking
Istio Anthos Service ...
Anthos
lets you…
Confluent lets you deploy on any k8s Platform
including Anthos
Kubernetes Engine
Build-your-own
Kubernetes
Enterprise
Dist...
Events streams connect different environments
together
Private Cloud Hybrid Cloud Public Cloud /
Multi-Cloud
• Deploy on p...
Two Ways to Deploy
Confluent
Self-Managed Software Fully-Managed Service
Confluent Platform
The Enterprise Distribution
of...
Viktor Gamov
Jason (Jay) Smith
Thank You
ORIGIN
AL
SLIDES
38
Serverless Eventings
with Kafka and Cloud
Run
39
Jason (Jay) Smith
@thejaysmith
Viktor Gamov
@gamussa
Hosts
By 2025, more than a
quarter of data created in
the global datasphere will
be real time in nature.
*IDC
What’s driving
the growth in
real time data?
Digital communications
Social media posts, photos, email, collaboration
platf...
I want to create a data driven organization.
event
What are the events that go into a transaction?
Need
Recognition
Information
Search
Evaluation of
alternatives
Purchase
De...
44
Kafka Represents Events as a Stream
An immutable sequence of Events
44
Older
Messages
Append New
Messages
45
The Stream is spread over many machines
45
4646
Auto / Transport
Without Event Streaming With Event Streaming
Event Streaming Enables New Outcomes
Call for driver av...
47
Serverless usage models
Portable
No servers Pay only for usage
Event-driven
developer perspective
48
I want to create a data driven organization.
event
49
Knative serving
Benefits
● Seamlessly scale up and down
● Built-in traffic splitting between
revisions
● Integrates net...
Anthos is our
platform for
application
modernization
Google Cloud On-prem Other Clouds
Anthos gives you
freedom to
modernize without
being locked in
Infrastructure Operator
Kubernetes Anthos GKE
Service Operat...
5252
Anthos lets you….
5353
Confluent lets you deploy on any k8s Platform
including Anthos
Build-your-own
Kubernetes
Enterprise
Distributions
Clou...
5454
Event streams connect different environments
together
Private Cloud
● Deploy on premises with Confluent
Platform
● Dep...
5555
Two Ways to Deploy Confluent
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Confluent Cloud
Apache Kafka...
Serverless Eventings with Apache Kafka® and Cloud Run
Upcoming SlideShare
Loading in …5
×

of

Serverless Eventings with Apache Kafka® and Cloud Run Slide 1 Serverless Eventings with Apache Kafka® and Cloud Run Slide 2 Serverless Eventings with Apache Kafka® and Cloud Run Slide 3 Serverless Eventings with Apache Kafka® and Cloud Run Slide 4 Serverless Eventings with Apache Kafka® and Cloud Run Slide 5 Serverless Eventings with Apache Kafka® and Cloud Run Slide 6 Serverless Eventings with Apache Kafka® and Cloud Run Slide 7 Serverless Eventings with Apache Kafka® and Cloud Run Slide 8 Serverless Eventings with Apache Kafka® and Cloud Run Slide 9 Serverless Eventings with Apache Kafka® and Cloud Run Slide 10 Serverless Eventings with Apache Kafka® and Cloud Run Slide 11 Serverless Eventings with Apache Kafka® and Cloud Run Slide 12 Serverless Eventings with Apache Kafka® and Cloud Run Slide 13 Serverless Eventings with Apache Kafka® and Cloud Run Slide 14 Serverless Eventings with Apache Kafka® and Cloud Run Slide 15 Serverless Eventings with Apache Kafka® and Cloud Run Slide 16 Serverless Eventings with Apache Kafka® and Cloud Run Slide 17 Serverless Eventings with Apache Kafka® and Cloud Run Slide 18 Serverless Eventings with Apache Kafka® and Cloud Run Slide 19 Serverless Eventings with Apache Kafka® and Cloud Run Slide 20 Serverless Eventings with Apache Kafka® and Cloud Run Slide 21 Serverless Eventings with Apache Kafka® and Cloud Run Slide 22 Serverless Eventings with Apache Kafka® and Cloud Run Slide 23 Serverless Eventings with Apache Kafka® and Cloud Run Slide 24 Serverless Eventings with Apache Kafka® and Cloud Run Slide 25 Serverless Eventings with Apache Kafka® and Cloud Run Slide 26 Serverless Eventings with Apache Kafka® and Cloud Run Slide 27 Serverless Eventings with Apache Kafka® and Cloud Run Slide 28 Serverless Eventings with Apache Kafka® and Cloud Run Slide 29 Serverless Eventings with Apache Kafka® and Cloud Run Slide 30 Serverless Eventings with Apache Kafka® and Cloud Run Slide 31 Serverless Eventings with Apache Kafka® and Cloud Run Slide 32 Serverless Eventings with Apache Kafka® and Cloud Run Slide 33 Serverless Eventings with Apache Kafka® and Cloud Run Slide 34 Serverless Eventings with Apache Kafka® and Cloud Run Slide 35 Serverless Eventings with Apache Kafka® and Cloud Run Slide 36 Serverless Eventings with Apache Kafka® and Cloud Run Slide 37 Serverless Eventings with Apache Kafka® and Cloud Run Slide 38 Serverless Eventings with Apache Kafka® and Cloud Run Slide 39 Serverless Eventings with Apache Kafka® and Cloud Run Slide 40 Serverless Eventings with Apache Kafka® and Cloud Run Slide 41 Serverless Eventings with Apache Kafka® and Cloud Run Slide 42 Serverless Eventings with Apache Kafka® and Cloud Run Slide 43 Serverless Eventings with Apache Kafka® and Cloud Run Slide 44 Serverless Eventings with Apache Kafka® and Cloud Run Slide 45 Serverless Eventings with Apache Kafka® and Cloud Run Slide 46 Serverless Eventings with Apache Kafka® and Cloud Run Slide 47 Serverless Eventings with Apache Kafka® and Cloud Run Slide 48 Serverless Eventings with Apache Kafka® and Cloud Run Slide 49 Serverless Eventings with Apache Kafka® and Cloud Run Slide 50 Serverless Eventings with Apache Kafka® and Cloud Run Slide 51 Serverless Eventings with Apache Kafka® and Cloud Run Slide 52 Serverless Eventings with Apache Kafka® and Cloud Run Slide 53 Serverless Eventings with Apache Kafka® and Cloud Run Slide 54 Serverless Eventings with Apache Kafka® and Cloud Run Slide 55
Upcoming SlideShare
What to Upload to SlideShare
Next

1 Like

Share

Serverless Eventings with Apache Kafka® and Cloud Run

Viktor Gamov, Confluent, Developer Advocate + Jason "Jay" Smith, Google Cloud, Customer Engineer

When it comes time to choose a distributed messaging system, the answer is increasingly - Apache Kafka®.

What if you need more than publish and subscribe messaging, but also long-term storage (spoiler: Core Kafka), a flexible integration framework (another spoiler: Kafka Connect), and a means of deploying real-time stream processing applications (another one: Kafka Streams) at scale?

The answer is still Apache Kafka - a next-generation event streaming platform.

This session introduces the foundations of Apache Kafka® and walks through some code examples. Jay and Viktor will demonstrate how to begin using Kafka on Google Cloud with Confluent Operator on GKE and Serverless Kafka via Confluent Cloud. We will also demonstrate streaming applications using Google Cloud Run and Knative Eventing with Kafka.

https://www.meetup.com/KafkaBayArea/events/270303178/

Related Books

Free with a 30 day trial from Scribd

See all

Serverless Eventings with Apache Kafka® and Cloud Run

  1. 1. Serverless Eventings with Apache® Kafka and Cloud Run
  2. 2. Viktor Gamov Jason (Jay) Smith
  3. 3. By 2025, more than a quarter of data created in the global datasphere will be real time in nature.
  4. 4. What’s driving the growth in real time data? Digital communications Social media posts, photos, email, collaboration platforms, chat boards, B2B/B2C outreach Digital devices/connected industry Next-gen manufacturing, industrial sensors, smartphones, wearables, cars/navigation Digital commerce Online interactions, purchase histories, clickstreams, advertising, inventories, ledgers Digital media consumption Newspapers, magazines, videos, games, television, movies
  5. 5. I want to create a data driven organization.
  6. 6. I want to create a data driven organization. event
  7. 7. What are the events that go into a transaction? Need Recognition Information Search Evaluation of alternatives Purchase Decision Post-purchase behavior Views digital ad Talks to friend Views influencers with product Views peer with product Online category Online views of substitutes/ alternatives Visits product pages Social community outreach Views price across websites Visits consumer reports Visits product pages Visits online forums Best price determined Quality/cost trade offs made Vendor / distributor selected Social posts Written reviews Customers service calls Usage Patterns
  8. 8. Event Streaming Enables New Outcomes • Call for driver availability • No knowledge of driver arrival • No data on feature usage • Real-time driver-rider match • Real-time ETA • Real-time sensor diagnostics • Nightly updated account balance • Batch fraud checks • Batch regulatory reporting • Real-time account updates • Real-time credit card fraud alerts • Real-time regulatory reporting • Post-order “out of stock” emails • No upsell through personalization • Batch point-of sale reports • Real-time inventory • Real-time recommendations • Real-time sales reporting Without Event Streaming With Event Streaming Banking Auto/ Transport Retail
  9. 9. Kafka Stores Events in a Log An immutable sequence of Events Older Messages Append New Messages Time
  10. 10. The Stream is spread over many machines An immutable sequence of Events
  11. 11. Operations: It’s hard
  12. 12. It doesn’t have to be that way... Confluent Operator It can be hard to deploy a complete event streaming platform. And even harder to secure and evolve the platform over time. Confluent wants to make it easy. Confluent Operator takes our operational knowledge and leverages Kubernetes to automate it all for you. Every cluster in Confluent Cloud is powered by the Confluent Operator.
  13. 13. It doesn’t have to be that way... ● Kubernetes* scales workers elastically ● Confluent Operator scales brokers elastically ● Self-Balancing rebalances data automatically ● Tiered Storage makes it much faster $ git diff -- values.yaml ... kafka: - brokers: 5 + brokers: 10 ... $ helm upgrade kafka confluent-operator --set kafka.enabled=true -f values.yml
  14. 14. Kubernetes isn’t the endgame
  15. 15. Do developers want to use Kubernetes directly? Have to do Write code Build docker image Upload image to registry Deploy service Expose to the internet Set up monitoring Set up autoscaling Want to do Write code
  16. 16. Serverless usage models developers perspective No Servers Pay only for usage Event-driven Portable
  17. 17. 18 Knative Building blocks for serverless workloads on Kubernetes
  18. 18. 19 What Knative is ● An open source project ● Set of building blocks to construct your own FaaS/PaaS ○ abstracts common tasks through custom Kubernetes API objects ● An abstraction on top of Kubernetes ○ It's still Kubernetes: Runs containers at the end of the day.
  19. 19. 20 What Knative is not ● It's not a Google product. ● It's not FaaS.
  20. 20. 21 Knative Serving Benefits ● Seamlessly scale up and down ● Built-in traffic splitting between revisions ● Integrates networking and service mesh automatically ● Easy to reason about object model Pluggable ● Connect to your own logging and monitoring platform, or use the built-in system ● Auto-scaler can be tuned or swapped out for custom code
  21. 21. 22 Knative eventing Benefits ● Declaratively bind between event producers and deployed services ● Scales from just few events to live streams ● Custom event pipelines to connect with your own existing systems
  22. 22. 23 Knative Eventing
  23. 23. 24 Knative Eventing
  24. 24. 25 Cloud Run Google’s Managed Version of Knative Serving
  25. 25. 26
  26. 26. 27 DEMO
  27. 27. 28 Knative Eventing
  28. 28. Anthos is our platform for application modernization Managed Operations Service Management Marketplace Other CloudsOn-premGoogle Cloud
  29. 29. Anthos gives you freedom to modernize without being locked in Developer Knative Cloud Run Networking Istio Anthos Service Mesh Infrastructure Kubernetes Anthos GKE
  30. 30. Anthos lets you…
  31. 31. Confluent lets you deploy on any k8s Platform including Anthos Kubernetes Engine Build-your-own Kubernetes Enterprise Distributions Cloud Services
  32. 32. Events streams connect different environments together Private Cloud Hybrid Cloud Public Cloud / Multi-Cloud • Deploy on premises with Confluent Platform • Deploy on Kubernetes with Operator • Deploy a consistent platform across on-prem and cloud • Build a persistent bridge between datacenter and cloud with Replicator • Run self-managed with Confluent Platform • Leverage a fully managed service with Confluent Cloud
  33. 33. Two Ways to Deploy Confluent Self-Managed Software Fully-Managed Service Confluent Platform The Enterprise Distribution of Apache Kafka Confluent Cloud Apache Kafka Re-engineered for the Cloud VM
  34. 34. Viktor Gamov Jason (Jay) Smith
  35. 35. Thank You
  36. 36. ORIGIN AL SLIDES
  37. 37. 38 Serverless Eventings with Kafka and Cloud Run
  38. 38. 39 Jason (Jay) Smith @thejaysmith Viktor Gamov @gamussa Hosts
  39. 39. By 2025, more than a quarter of data created in the global datasphere will be real time in nature. *IDC
  40. 40. What’s driving the growth in real time data? Digital communications Social media posts, photos, email, collaboration platforms, chat boards, B2B/B2C outreach Digital devices/connected industry Next-gen manufacturing, industrial sensors, smartphones, wearables, cars/navigation Digital commerce Online interactions, purchase histories, clickstreams, advertising, inventories, ledgers Digital media consumption Newspapers, magazines, videos, games, television, movies
  41. 41. I want to create a data driven organization. event
  42. 42. What are the events that go into a transaction? Need Recognition Information Search Evaluation of alternatives Purchase Decision Post-purchase behavior Views digital ad Talks to friend Views influencers with product Views peer with product Online category Online views of substitutes/alternatives Visits product pages Social community outreach Views price across websites Visits consumer reports Visits product pages Visits online forums Best price determined Quality/cost trade offs made Vendor / distributor selected Social posts Written reviews Customers service calls Usage Patterns
  43. 43. 44 Kafka Represents Events as a Stream An immutable sequence of Events 44 Older Messages Append New Messages
  44. 44. 45 The Stream is spread over many machines 45
  45. 45. 4646 Auto / Transport Without Event Streaming With Event Streaming Event Streaming Enables New Outcomes Call for driver availability No knowledge of driver arrival No data on feature usage Real-time driver-rider match Real-time ETA Real-time sensor diagnostics Banking Nightly updated account balance Batch fraud checks Batch regulatory reporting Real-time account updates Real-time credit card fraud alerts Real-time regulatory reporting Retail Post-order “out of stock” emails No upsell through personalization Batch point-of sale reports Real-time inventory Real-time recommendations Real-time sales reporting
  46. 46. 47 Serverless usage models Portable No servers Pay only for usage Event-driven developer perspective
  47. 47. 48 I want to create a data driven organization. event
  48. 48. 49 Knative serving Benefits ● Seamlessly scale up and down ● Built-in traffic splitting between revisions ● Integrates networking and service mesh automatically ● Easy to reason about object model Pluggable ● Connect to your own logging and monitoring platform, or use the built-in system ● Auto-scaler can be tuned or swapped out for custom code
  49. 49. Anthos is our platform for application modernization Google Cloud On-prem Other Clouds
  50. 50. Anthos gives you freedom to modernize without being locked in Infrastructure Operator Kubernetes Anthos GKE Service Operator / SRE Istio Anthos Service Mesh Developer Knative Cloud Run
  51. 51. 5252 Anthos lets you….
  52. 52. 5353 Confluent lets you deploy on any k8s Platform including Anthos Build-your-own Kubernetes Enterprise Distributions Cloud Services Kubernetes Engine
  53. 53. 5454 Event streams connect different environments together Private Cloud ● Deploy on premises with Confluent Platform ● Deploy on Kubernetes with Operator Public Cloud / Multi- Cloud ● Run self-managed with Confluent Platform ● Leverage a fully managed service with Confluent Cloud Hybrid Cloud ● Deploy a consistent platform across on-prem and cloud ● Build a persistent bridge between datacenter and cloud with Replicator
  54. 54. 5555 Two Ways to Deploy Confluent Confluent Platform The Enterprise Distribution of Apache Kafka Confluent Cloud Apache Kafka Re-engineered for the Cloud Self-Managed Software Fully-Managed Service VM Deploy on any platform, on-prem or cloud Available on GCP Marketplace
  • patrickchassany

    Jun. 21, 2020

Viktor Gamov, Confluent, Developer Advocate + Jason "Jay" Smith, Google Cloud, Customer Engineer When it comes time to choose a distributed messaging system, the answer is increasingly - Apache Kafka®. What if you need more than publish and subscribe messaging, but also long-term storage (spoiler: Core Kafka), a flexible integration framework (another spoiler: Kafka Connect), and a means of deploying real-time stream processing applications (another one: Kafka Streams) at scale? The answer is still Apache Kafka - a next-generation event streaming platform. This session introduces the foundations of Apache Kafka® and walks through some code examples. Jay and Viktor will demonstrate how to begin using Kafka on Google Cloud with Confluent Operator on GKE and Serverless Kafka via Confluent Cloud. We will also demonstrate streaming applications using Google Cloud Run and Knative Eventing with Kafka. https://www.meetup.com/KafkaBayArea/events/270303178/

Views

Total views

390

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

0

Shares

0

Comments

0

Likes

1

×