SlideShare a Scribd company logo
CloudandOn-premisesApplications
Integration
UsingEvent-driven
Rich Lee
Taipei, November 2020
About Me
Rich Lee
• Integration Architect at Cathay Financial Holdings
• Microservices
• Event-driven
• Cloud Native App Development
• Kafka & Event-driven consultant
rich04230@gmail.com RICH0423 @rich04230
Agenda
• FaaS
– Google Cloud Functions
• Kafka Connect
• Demo: Cloud and on-premises integration with Event-driven
– Github repo
FaaS
Backend as a service approaches
Application
Runtime
Containers
OS
Hardware
Functions
IaaS
Data
CaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
PaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
FaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
Developer
Cloud
Provider
Function-as-a-Service (FaaS)
• FaaS is focused on the event-driven computing paradigm wherein
application code, or containers, only run in response to events or
requests.
Serverless Event-driven
computing
FaaS
Function-as-a-Service (FaaS)
container-based
AWS Lambda GCP Cloud Functions Azure Functions
Azure Fargate GCP Cloud Run
Google Cloud Functions
Cloud Functions is a serverless execution environment for building and
connecting cloud services.
• Function types:
– Http functions
– Background functions
• Triggers:
– HTTP
– Cloud Sotrage
– Cloud Pub/Sub
Kafka Connect
Why Kafka Connect
• Simplify and improve getting data in and out of Kafka
– Fault Tolerance
– Exactly Once
– Distribution
– Ordering
Kafka Connect
• Kafka Connect is a tool for scalable and reliable streaming data
between Apache Kafka and other systems.
Kafka Connect
• The Kafka Connect API is an interface that simplifies and automates the
integration of a new data source or sink to your Kafka cluster.
Confluent HUB Discover and share Connectors
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks
• Tasks are executed by Kafka Connect Workers (servers)
– A worker is a single java process
– A worker can be standalone or in a cluster (distributed)
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
• Kafka Connect Workers
– Standalone
– Distributed
Standalone Distributed
Parallelism A single process runs your connectors
and tasks
Multiple workers run your connectors
and tasks
Configuration bundled with your process submitted using a REST API
Environment for development and testing for production deployment of
connectors
Reliability /
Scalability
Not fault tolerant, no scalability Easy to scale, and fault tolerant
Kafka Connectors
• REST API
– connector
– connector-plugins
Resource
Type
Endpoint Method Description
connector /connectors GET Get a list of active connectors
/connectors POST Create a new connector
/connectors/{name} GET Get information about the connector
/connectors/{name}/config GET Get the configuration for the connector
/connectors/{name}/config PUT Create or update the configuration for an
existing connector
/connectors/{name}/status GET Get current status of the connector
/connectors/{name}/restart POST Restart the connector and its tasks
/connectors/{name}/ DELETE Delete a connector
push event
poll event
Kafka Connect
Kafka Connectors
• GCP CloudPubSubConnector
– sink connector
– source connector
GCP CloudPubSubConnector
• Configure sink connector
{
"name": "cps-sink-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
"tasks.max": "1",
"topics": "iot-topic",
"cps.topic": "cps-iot-input",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs
GCP CloudPubSubConnector
• Configure source connector
{
"name": "cps-source-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.source.CloudPubSubSourceConnector",
"tasks.max": "1",
"kafka.topic": "iot-processed-topic",
"cps.subscription": "subscription2",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.converters.ByteArrayConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs
Reference
• GCP Pubsub Connector
• Kafka Connect
• Creating Apache Kafka Connectors with the Kafka Connect API
• Source and Sink for Kafka and PubSub

More Related Content

What's hot

Microservices & API Gateways
Microservices & API Gateways Microservices & API Gateways
Microservices & API Gateways
Kong Inc.
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
Amazon Web Services
 
Convert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS LambdaConvert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS Lambda
Amazon Web Services
 
Amazon API Gateway
Amazon API GatewayAmazon API Gateway
Amazon API Gateway
Mark Bate
 
Clean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NETClean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NET
Marcin Tyborowski
 
Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201
Amazon Web Services
 
OpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-outOpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-out
Antons Kranga
 
2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda
devopsdaysaustin
 
03 spring cloud eureka service discovery
03 spring cloud eureka   service discovery03 spring cloud eureka   service discovery
03 spring cloud eureka service discovery
Janani Velmurugan
 
Netflix0SS Services on Docker
Netflix0SS Services on DockerNetflix0SS Services on Docker
Netflix0SS Services on Docker
Docker, Inc.
 
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
Amazon Web Services
 
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops FrameworkToward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
LibbySchulze
 
Aws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API GatewayAws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API Gateway
aws-marketing-il
 
MicroServices sur AWS
MicroServices sur AWSMicroServices sur AWS
MicroServices sur AWS
Amazon Web Services
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the Trenches
Yan Cui
 
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays
 
Serverless Apps with Open Whisk
Serverless Apps with Open Whisk Serverless Apps with Open Whisk
Serverless Apps with Open Whisk
Dev_Events
 
Microservices pros and cons dark
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons dark
Andrew Siemer
 
Scalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the TradeScalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the Trade
C4Media
 
Manage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API GatewayManage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API Gateway
Thibault Charbonnier
 

What's hot (20)

Microservices & API Gateways
Microservices & API Gateways Microservices & API Gateways
Microservices & API Gateways
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
 
Convert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS LambdaConvert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS Lambda
 
Amazon API Gateway
Amazon API GatewayAmazon API Gateway
Amazon API Gateway
 
Clean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NETClean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NET
 
Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201
 
OpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-outOpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-out
 
2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda
 
03 spring cloud eureka service discovery
03 spring cloud eureka   service discovery03 spring cloud eureka   service discovery
03 spring cloud eureka service discovery
 
Netflix0SS Services on Docker
Netflix0SS Services on DockerNetflix0SS Services on Docker
Netflix0SS Services on Docker
 
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
 
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops FrameworkToward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
 
Aws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API GatewayAws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API Gateway
 
MicroServices sur AWS
MicroServices sur AWSMicroServices sur AWS
MicroServices sur AWS
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the Trenches
 
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
 
Serverless Apps with Open Whisk
Serverless Apps with Open Whisk Serverless Apps with Open Whisk
Serverless Apps with Open Whisk
 
Microservices pros and cons dark
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons dark
 
Scalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the TradeScalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the Trade
 
Manage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API GatewayManage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API Gateway
 

Similar to GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven

Implementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using KubelessImplementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using Kubeless
Ahmed Misbah
 
Serverless Architecture
Serverless ArchitectureServerless Architecture
Serverless Architecture
july mon
 
DevOps and BigData Analytics
DevOps and BigData Analytics DevOps and BigData Analytics
DevOps and BigData Analytics
sbbabu
 
Function as a Service
Function as a ServiceFunction as a Service
Function as a Service
rich fernandez
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
HostedbyConfluent
 
Service fabric and azure service fabric mesh
Service fabric and azure service fabric meshService fabric and azure service fabric mesh
Service fabric and azure service fabric mesh
Mikkel Mørk Hegnhøj
 
The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017
Amazon Web Services
 
Cloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment WorkshopCloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment Workshop
Manuel Garcia
 
Kafka connect 101
Kafka connect 101Kafka connect 101
Kafka connect 101
Whiteklay
 
Sas 2015 event_driven
Sas 2015 event_drivenSas 2015 event_driven
Sas 2015 event_driven
Sascha Möllering
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
Amazon Web Services
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Amazon Web Services
 
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
Pahud Hsieh
 
Setup kubernetes federation between clusters
Setup kubernetes federation between clustersSetup kubernetes federation between clusters
Setup kubernetes federation between clusters
ssuser75c76a2
 
Managing Your Cloud Assets
Managing Your Cloud AssetsManaging Your Cloud Assets
Managing Your Cloud Assets
Amazon Web Services
 
What's New in AWS Serverless and Containers
What's New in AWS Serverless and ContainersWhat's New in AWS Serverless and Containers
What's New in AWS Serverless and Containers
Amazon Web Services
 
The Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid ConnectivityThe Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid Connectivity
Daniel Toomey
 
Continuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWSContinuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWS
Amazon Web Services
 
Kubernetes 101 Workshop
Kubernetes 101 WorkshopKubernetes 101 Workshop
Kubernetes 101 Workshop
Vishal Biyani
 
Introduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdfIntroduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdf
Amazon Web Services
 

Similar to GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven (20)

Implementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using KubelessImplementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using Kubeless
 
Serverless Architecture
Serverless ArchitectureServerless Architecture
Serverless Architecture
 
DevOps and BigData Analytics
DevOps and BigData Analytics DevOps and BigData Analytics
DevOps and BigData Analytics
 
Function as a Service
Function as a ServiceFunction as a Service
Function as a Service
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
 
Service fabric and azure service fabric mesh
Service fabric and azure service fabric meshService fabric and azure service fabric mesh
Service fabric and azure service fabric mesh
 
The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017
 
Cloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment WorkshopCloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment Workshop
 
Kafka connect 101
Kafka connect 101Kafka connect 101
Kafka connect 101
 
Sas 2015 event_driven
Sas 2015 event_drivenSas 2015 event_driven
Sas 2015 event_driven
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
 
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
 
Setup kubernetes federation between clusters
Setup kubernetes federation between clustersSetup kubernetes federation between clusters
Setup kubernetes federation between clusters
 
Managing Your Cloud Assets
Managing Your Cloud AssetsManaging Your Cloud Assets
Managing Your Cloud Assets
 
What's New in AWS Serverless and Containers
What's New in AWS Serverless and ContainersWhat's New in AWS Serverless and Containers
What's New in AWS Serverless and Containers
 
The Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid ConnectivityThe Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid Connectivity
 
Continuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWSContinuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWS
 
Kubernetes 101 Workshop
Kubernetes 101 WorkshopKubernetes 101 Workshop
Kubernetes 101 Workshop
 
Introduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdfIntroduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdf
 

More from Rich Lee

COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google CloudCOSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
Rich Lee
 
2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發
Rich Lee
 
Redis Cache design
Redis Cache designRedis Cache design
Redis Cache design
Rich Lee
 
Couchbase & FTS
Couchbase & FTSCouchbase & FTS
Couchbase & FTS
Rich Lee
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
Rich Lee
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice Architecture
Rich Lee
 
Apache Spark Introduction
Apache Spark IntroductionApache Spark Introduction
Apache Spark Introduction
Rich Lee
 
AWS IoT in action
AWS IoT in actionAWS IoT in action
AWS IoT in action
Rich Lee
 
Realtime web development
Realtime web developmentRealtime web development
Realtime web development
Rich Lee
 
Event sourcing
Event sourcingEvent sourcing
Event sourcing
Rich Lee
 

More from Rich Lee (10)

COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google CloudCOSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
 
2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發
 
Redis Cache design
Redis Cache designRedis Cache design
Redis Cache design
 
Couchbase & FTS
Couchbase & FTSCouchbase & FTS
Couchbase & FTS
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice Architecture
 
Apache Spark Introduction
Apache Spark IntroductionApache Spark Introduction
Apache Spark Introduction
 
AWS IoT in action
AWS IoT in actionAWS IoT in action
AWS IoT in action
 
Realtime web development
Realtime web developmentRealtime web development
Realtime web development
 
Event sourcing
Event sourcingEvent sourcing
Event sourcing
 

Recently uploaded

The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
kalichargn70th171
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
Tier1 app
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
Reetu63
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
chandangoswami40933
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
Paul Brebner
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Peter Caitens
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio, Inc.
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
Alberto Brandolini
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Jhone kinadey
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
sandeepmenon62
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
michniczscribd
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
Streamlining End-to-End Testing Automation
Streamlining End-to-End Testing AutomationStreamlining End-to-End Testing Automation
Streamlining End-to-End Testing Automation
Anand Bagmar
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies
 
Orca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container OrchestrationOrca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container Orchestration
Pedro J. Molina
 

Recently uploaded (20)

The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
Streamlining End-to-End Testing Automation
Streamlining End-to-End Testing AutomationStreamlining End-to-End Testing Automation
Streamlining End-to-End Testing Automation
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
 
Orca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container OrchestrationOrca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container Orchestration
 

GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven

  • 2. About Me Rich Lee • Integration Architect at Cathay Financial Holdings • Microservices • Event-driven • Cloud Native App Development • Kafka & Event-driven consultant rich04230@gmail.com RICH0423 @rich04230
  • 3. Agenda • FaaS – Google Cloud Functions • Kafka Connect • Demo: Cloud and on-premises integration with Event-driven – Github repo
  • 5. Backend as a service approaches Application Runtime Containers OS Hardware Functions IaaS Data CaaS Application Runtime Containers OS Hardware Functions Data PaaS Application Runtime Containers OS Hardware Functions Data FaaS Application Runtime Containers OS Hardware Functions Data Developer Cloud Provider
  • 6. Function-as-a-Service (FaaS) • FaaS is focused on the event-driven computing paradigm wherein application code, or containers, only run in response to events or requests. Serverless Event-driven computing FaaS
  • 7. Function-as-a-Service (FaaS) container-based AWS Lambda GCP Cloud Functions Azure Functions Azure Fargate GCP Cloud Run
  • 8. Google Cloud Functions Cloud Functions is a serverless execution environment for building and connecting cloud services. • Function types: – Http functions – Background functions • Triggers: – HTTP – Cloud Sotrage – Cloud Pub/Sub
  • 9.
  • 11. Why Kafka Connect • Simplify and improve getting data in and out of Kafka – Fault Tolerance – Exactly Once – Distribution – Ordering
  • 12. Kafka Connect • Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other systems.
  • 13. Kafka Connect • The Kafka Connect API is an interface that simplifies and automates the integration of a new data source or sink to your Kafka cluster.
  • 14. Confluent HUB Discover and share Connectors
  • 15. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them!
  • 16. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them! • Connectors + Configurations --> Tasks – A task is linked to a connector configuration – A job configuration may spawn multiple tasks
  • 17. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them! • Connectors + Configurations --> Tasks – A task is linked to a connector configuration – A job configuration may spawn multiple tasks • Tasks are executed by Kafka Connect Workers (servers) – A worker is a single java process – A worker can be standalone or in a cluster (distributed)
  • 21. Kafka Connect Concepts • Kafka Connect Workers – Standalone – Distributed Standalone Distributed Parallelism A single process runs your connectors and tasks Multiple workers run your connectors and tasks Configuration bundled with your process submitted using a REST API Environment for development and testing for production deployment of connectors Reliability / Scalability Not fault tolerant, no scalability Easy to scale, and fault tolerant
  • 22. Kafka Connectors • REST API – connector – connector-plugins Resource Type Endpoint Method Description connector /connectors GET Get a list of active connectors /connectors POST Create a new connector /connectors/{name} GET Get information about the connector /connectors/{name}/config GET Get the configuration for the connector /connectors/{name}/config PUT Create or update the configuration for an existing connector /connectors/{name}/status GET Get current status of the connector /connectors/{name}/restart POST Restart the connector and its tasks /connectors/{name}/ DELETE Delete a connector
  • 23.
  • 25. Kafka Connectors • GCP CloudPubSubConnector – sink connector – source connector
  • 26. GCP CloudPubSubConnector • Configure sink connector { "name": "cps-sink-connector1", "config": { "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector", "tasks.max": "1", "topics": "iot-topic", "cps.topic": "cps-iot-input", "cps.project": "app-project-294009", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "key.converter.schemas.enable": false, "value.converter.schemas.enable": false } } Cloud Pub/Sub configs
  • 27. GCP CloudPubSubConnector • Configure source connector { "name": "cps-source-connector1", "config": { "connector.class": "com.google.pubsub.kafka.source.CloudPubSubSourceConnector", "tasks.max": "1", "kafka.topic": "iot-processed-topic", "cps.subscription": "subscription2", "cps.project": "app-project-294009", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "key.converter.schemas.enable": false, "value.converter.schemas.enable": false } } Cloud Pub/Sub configs
  • 28. Reference • GCP Pubsub Connector • Kafka Connect • Creating Apache Kafka Connectors with the Kafka Connect API • Source and Sink for Kafka and PubSub