SlideShare a Scribd company logo
CloudandOn-premisesApplications
Integration
UsingEvent-driven
Rich Lee
Taipei, November 2020
About Me
Rich Lee
• Integration Architect at Cathay Financial Holdings
• Microservices
• Event-driven
• Cloud Native App Development
• Kafka & Event-driven consultant
rich04230@gmail.com RICH0423 @rich04230
Agenda
• FaaS
– Google Cloud Functions
• Kafka Connect
• Demo: Cloud and on-premises integration with Event-driven
– Github repo
FaaS
Backend as a service approaches
Application
Runtime
Containers
OS
Hardware
Functions
IaaS
Data
CaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
PaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
FaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
Developer
Cloud
Provider
Function-as-a-Service (FaaS)
• FaaS is focused on the event-driven computing paradigm wherein
application code, or containers, only run in response to events or
requests.
Serverless Event-driven
computing
FaaS
Function-as-a-Service (FaaS)
container-based
AWS Lambda GCP Cloud Functions Azure Functions
Azure Fargate GCP Cloud Run
Google Cloud Functions
Cloud Functions is a serverless execution environment for building and
connecting cloud services.
• Function types:
– Http functions
– Background functions
• Triggers:
– HTTP
– Cloud Sotrage
– Cloud Pub/Sub
Kafka Connect
Why Kafka Connect
• Simplify and improve getting data in and out of Kafka
– Fault Tolerance
– Exactly Once
– Distribution
– Ordering
Kafka Connect
• Kafka Connect is a tool for scalable and reliable streaming data
between Apache Kafka and other systems.
Kafka Connect
• The Kafka Connect API is an interface that simplifies and automates the
integration of a new data source or sink to your Kafka cluster.
Confluent HUB Discover and share Connectors
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks
Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks
• Tasks are executed by Kafka Connect Workers (servers)
– A worker is a single java process
– A worker can be standalone or in a cluster (distributed)
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
Distributed Mode
Kafka Connect Concepts
• Kafka Connect Workers
– Standalone
– Distributed
Standalone Distributed
Parallelism A single process runs your connectors
and tasks
Multiple workers run your connectors
and tasks
Configuration bundled with your process submitted using a REST API
Environment for development and testing for production deployment of
connectors
Reliability /
Scalability
Not fault tolerant, no scalability Easy to scale, and fault tolerant
Kafka Connectors
• REST API
– connector
– connector-plugins
Resource
Type
Endpoint Method Description
connector /connectors GET Get a list of active connectors
/connectors POST Create a new connector
/connectors/{name} GET Get information about the connector
/connectors/{name}/config GET Get the configuration for the connector
/connectors/{name}/config PUT Create or update the configuration for an
existing connector
/connectors/{name}/status GET Get current status of the connector
/connectors/{name}/restart POST Restart the connector and its tasks
/connectors/{name}/ DELETE Delete a connector
push event
poll event
Kafka Connect
Kafka Connectors
• GCP CloudPubSubConnector
– sink connector
– source connector
GCP CloudPubSubConnector
• Configure sink connector
{
"name": "cps-sink-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
"tasks.max": "1",
"topics": "iot-topic",
"cps.topic": "cps-iot-input",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs
GCP CloudPubSubConnector
• Configure source connector
{
"name": "cps-source-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.source.CloudPubSubSourceConnector",
"tasks.max": "1",
"kafka.topic": "iot-processed-topic",
"cps.subscription": "subscription2",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.converters.ByteArrayConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs
Reference
• GCP Pubsub Connector
• Kafka Connect
• Creating Apache Kafka Connectors with the Kafka Connect API
• Source and Sink for Kafka and PubSub

More Related Content

What's hot

Microservices & API Gateways
Microservices & API Gateways Microservices & API Gateways
Microservices & API Gateways Kong Inc.
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesAmazon Web Services
 
Convert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS LambdaConvert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS LambdaAmazon Web Services
 
Amazon API Gateway
Amazon API GatewayAmazon API Gateway
Amazon API GatewayMark Bate
 
Clean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NETClean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NETMarcin Tyborowski
 
Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201Amazon Web Services
 
OpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-outOpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-outAntons Kranga
 
2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambdadevopsdaysaustin
 
03 spring cloud eureka service discovery
03 spring cloud eureka   service discovery03 spring cloud eureka   service discovery
03 spring cloud eureka service discoveryJanani Velmurugan
 
Netflix0SS Services on Docker
Netflix0SS Services on DockerNetflix0SS Services on Docker
Netflix0SS Services on DockerDocker, Inc.
 
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...Amazon Web Services
 
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops FrameworkToward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops FrameworkLibbySchulze
 
Aws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API GatewayAws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API Gatewayaws-marketing-il
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the TrenchesYan Cui
 
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...apidays
 
Serverless Apps with Open Whisk
Serverless Apps with Open Whisk Serverless Apps with Open Whisk
Serverless Apps with Open Whisk Dev_Events
 
Microservices pros and cons dark
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons darkAndrew Siemer
 
Scalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the TradeScalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the TradeC4Media
 
Manage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API GatewayManage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API GatewayThibault Charbonnier
 

What's hot (20)

Microservices & API Gateways
Microservices & API Gateways Microservices & API Gateways
Microservices & API Gateways
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
 
Convert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS LambdaConvert Your Code into a Microservice using AWS Lambda
Convert Your Code into a Microservice using AWS Lambda
 
Amazon API Gateway
Amazon API GatewayAmazon API Gateway
Amazon API Gateway
 
Clean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NETClean up this mess - API Gateway & Service Discovery in .NET
Clean up this mess - API Gateway & Service Discovery in .NET
 
Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201Building Scalable Services with Amazon API Gateway - Technical 201
Building Scalable Services with Amazon API Gateway - Technical 201
 
OpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-outOpenSlava 2014 - CloudFoundry inside-out
OpenSlava 2014 - CloudFoundry inside-out
 
2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda2016 - Serverless Microservices on AWS with API Gateway and Lambda
2016 - Serverless Microservices on AWS with API Gateway and Lambda
 
03 spring cloud eureka service discovery
03 spring cloud eureka   service discovery03 spring cloud eureka   service discovery
03 spring cloud eureka service discovery
 
Netflix0SS Services on Docker
Netflix0SS Services on DockerNetflix0SS Services on Docker
Netflix0SS Services on Docker
 
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
AWS August Webinar Series - Building Serverless Backends with AWS Lambda and ...
 
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops FrameworkToward Hybrid Cloud Serverless Transparency with Lithops Framework
Toward Hybrid Cloud Serverless Transparency with Lithops Framework
 
Aws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API GatewayAws Technical Day 2015 - Amazon API Gateway
Aws Technical Day 2015 - Amazon API Gateway
 
MicroServices sur AWS
MicroServices sur AWSMicroServices sur AWS
MicroServices sur AWS
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the Trenches
 
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...apidays LIVE New York 2021 - Service reliability through autoscaling workload...
apidays LIVE New York 2021 - Service reliability through autoscaling workload...
 
Serverless Apps with Open Whisk
Serverless Apps with Open Whisk Serverless Apps with Open Whisk
Serverless Apps with Open Whisk
 
Microservices pros and cons dark
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons dark
 
Scalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the TradeScalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the Trade
 
Manage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API GatewayManage your APIs and Microservices with an API Gateway
Manage your APIs and Microservices with an API Gateway
 

Similar to GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven

Implementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using KubelessImplementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using KubelessAhmed Misbah
 
Serverless Architecture
Serverless ArchitectureServerless Architecture
Serverless Architecturejuly mon
 
DevOps and BigData Analytics
DevOps and BigData Analytics DevOps and BigData Analytics
DevOps and BigData Analytics sbbabu
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...HostedbyConfluent
 
Service fabric and azure service fabric mesh
Service fabric and azure service fabric meshService fabric and azure service fabric mesh
Service fabric and azure service fabric meshMikkel Mørk Hegnhøj
 
The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017Amazon Web Services
 
Cloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment WorkshopCloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment WorkshopManuel Garcia
 
Kafka connect 101
Kafka connect 101Kafka connect 101
Kafka connect 101Whiteklay
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)Amazon Web Services
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksAmazon Web Services
 
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐Pahud Hsieh
 
Setup kubernetes federation between clusters
Setup kubernetes federation between clustersSetup kubernetes federation between clusters
Setup kubernetes federation between clustersssuser75c76a2
 
What's New in AWS Serverless and Containers
What's New in AWS Serverless and ContainersWhat's New in AWS Serverless and Containers
What's New in AWS Serverless and ContainersAmazon Web Services
 
The Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid ConnectivityThe Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid ConnectivityDaniel Toomey
 
Continuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWSContinuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWSAmazon Web Services
 
Kubernetes 101 Workshop
Kubernetes 101 WorkshopKubernetes 101 Workshop
Kubernetes 101 WorkshopVishal Biyani
 
Introduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdfIntroduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdfAmazon Web Services
 

Similar to GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven (20)

Implementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using KubelessImplementing FaaS on Kubernetes using Kubeless
Implementing FaaS on Kubernetes using Kubeless
 
Serverless Architecture
Serverless ArchitectureServerless Architecture
Serverless Architecture
 
DevOps and BigData Analytics
DevOps and BigData Analytics DevOps and BigData Analytics
DevOps and BigData Analytics
 
Function as a Service
Function as a ServiceFunction as a Service
Function as a Service
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
 
Service fabric and azure service fabric mesh
Service fabric and azure service fabric meshService fabric and azure service fabric mesh
Service fabric and azure service fabric mesh
 
The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017The State of Serverless Computing | AWS Public Sector Summit 2017
The State of Serverless Computing | AWS Public Sector Summit 2017
 
Cloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment WorkshopCloud Foundry: Hands-on Deployment Workshop
Cloud Foundry: Hands-on Deployment Workshop
 
Kafka connect 101
Kafka connect 101Kafka connect 101
Kafka connect 101
 
Sas 2015 event_driven
Sas 2015 event_drivenSas 2015 event_driven
Sas 2015 event_driven
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
 
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐從劍宗到氣宗  - 談AWS ECS與Serverless最佳實踐
從劍宗到氣宗 - 談AWS ECS與Serverless最佳實踐
 
Setup kubernetes federation between clusters
Setup kubernetes federation between clustersSetup kubernetes federation between clusters
Setup kubernetes federation between clusters
 
Managing Your Cloud Assets
Managing Your Cloud AssetsManaging Your Cloud Assets
Managing Your Cloud Assets
 
What's New in AWS Serverless and Containers
What's New in AWS Serverless and ContainersWhat's New in AWS Serverless and Containers
What's New in AWS Serverless and Containers
 
The Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid ConnectivityThe Hitchhiker’s Guide to Hybrid Connectivity
The Hitchhiker’s Guide to Hybrid Connectivity
 
Continuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWSContinuous Integration and Deployment Best Practices on AWS
Continuous Integration and Deployment Best Practices on AWS
 
Kubernetes 101 Workshop
Kubernetes 101 WorkshopKubernetes 101 Workshop
Kubernetes 101 Workshop
 
Introduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdfIntroduction to Containers - AWS Startup Day Johannesburg.pdf
Introduction to Containers - AWS Startup Day Johannesburg.pdf
 

More from Rich Lee

COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google CloudCOSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google CloudRich Lee
 
2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發Rich Lee
 
Redis Cache design
Redis Cache designRedis Cache design
Redis Cache designRich Lee
 
Couchbase & FTS
Couchbase & FTSCouchbase & FTS
Couchbase & FTSRich Lee
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice ArchitectureRich Lee
 
Apache Spark Introduction
Apache Spark IntroductionApache Spark Introduction
Apache Spark IntroductionRich Lee
 
AWS IoT in action
AWS IoT in actionAWS IoT in action
AWS IoT in actionRich Lee
 
Realtime web development
Realtime web developmentRealtime web development
Realtime web developmentRich Lee
 
Event sourcing
Event sourcingEvent sourcing
Event sourcingRich Lee
 

More from Rich Lee (10)

COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google CloudCOSCUP 2023 Building Portable and Reliable Applications on Google Cloud
COSCUP 2023 Building Portable and Reliable Applications on Google Cloud
 
2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發2021 JCConf 使用Dapr簡化Java微服務應用開發
2021 JCConf 使用Dapr簡化Java微服務應用開發
 
Redis Cache design
Redis Cache designRedis Cache design
Redis Cache design
 
Couchbase & FTS
Couchbase & FTSCouchbase & FTS
Couchbase & FTS
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice Architecture
 
Apache Spark Introduction
Apache Spark IntroductionApache Spark Introduction
Apache Spark Introduction
 
AWS IoT in action
AWS IoT in actionAWS IoT in action
AWS IoT in action
 
Realtime web development
Realtime web developmentRealtime web development
Realtime web development
 
Event sourcing
Event sourcingEvent sourcing
Event sourcing
 

Recently uploaded

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownloadvrstrong314
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEJelle | Nordend
 
10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdfkalichargn70th171
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesNeo4j
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...rajkumar669520
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationWave PLM
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)Max Lee
 
GraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisGraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisNeo4j
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzisteffenkarlsson2
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareinfo611746
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
 
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980
 

Recently uploaded (20)

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
 
10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
GraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisGraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysis
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting software
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 

GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-driven

  • 2. About Me Rich Lee • Integration Architect at Cathay Financial Holdings • Microservices • Event-driven • Cloud Native App Development • Kafka & Event-driven consultant rich04230@gmail.com RICH0423 @rich04230
  • 3. Agenda • FaaS – Google Cloud Functions • Kafka Connect • Demo: Cloud and on-premises integration with Event-driven – Github repo
  • 5. Backend as a service approaches Application Runtime Containers OS Hardware Functions IaaS Data CaaS Application Runtime Containers OS Hardware Functions Data PaaS Application Runtime Containers OS Hardware Functions Data FaaS Application Runtime Containers OS Hardware Functions Data Developer Cloud Provider
  • 6. Function-as-a-Service (FaaS) • FaaS is focused on the event-driven computing paradigm wherein application code, or containers, only run in response to events or requests. Serverless Event-driven computing FaaS
  • 7. Function-as-a-Service (FaaS) container-based AWS Lambda GCP Cloud Functions Azure Functions Azure Fargate GCP Cloud Run
  • 8. Google Cloud Functions Cloud Functions is a serverless execution environment for building and connecting cloud services. • Function types: – Http functions – Background functions • Triggers: – HTTP – Cloud Sotrage – Cloud Pub/Sub
  • 9.
  • 11. Why Kafka Connect • Simplify and improve getting data in and out of Kafka – Fault Tolerance – Exactly Once – Distribution – Ordering
  • 12. Kafka Connect • Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other systems.
  • 13. Kafka Connect • The Kafka Connect API is an interface that simplifies and automates the integration of a new data source or sink to your Kafka cluster.
  • 14. Confluent HUB Discover and share Connectors
  • 15. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them!
  • 16. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them! • Connectors + Configurations --> Tasks – A task is linked to a connector configuration – A job configuration may spawn multiple tasks
  • 17. Kafka Connect Concepts • Kafka Connect Cluster has multiple loaded Connectors – Each connector is a re usable piece of code (java jars) – Many connectors exist in the open source world, leverage them! • Connectors + Configurations --> Tasks – A task is linked to a connector configuration – A job configuration may spawn multiple tasks • Tasks are executed by Kafka Connect Workers (servers) – A worker is a single java process – A worker can be standalone or in a cluster (distributed)
  • 21. Kafka Connect Concepts • Kafka Connect Workers – Standalone – Distributed Standalone Distributed Parallelism A single process runs your connectors and tasks Multiple workers run your connectors and tasks Configuration bundled with your process submitted using a REST API Environment for development and testing for production deployment of connectors Reliability / Scalability Not fault tolerant, no scalability Easy to scale, and fault tolerant
  • 22. Kafka Connectors • REST API – connector – connector-plugins Resource Type Endpoint Method Description connector /connectors GET Get a list of active connectors /connectors POST Create a new connector /connectors/{name} GET Get information about the connector /connectors/{name}/config GET Get the configuration for the connector /connectors/{name}/config PUT Create or update the configuration for an existing connector /connectors/{name}/status GET Get current status of the connector /connectors/{name}/restart POST Restart the connector and its tasks /connectors/{name}/ DELETE Delete a connector
  • 23.
  • 25. Kafka Connectors • GCP CloudPubSubConnector – sink connector – source connector
  • 26. GCP CloudPubSubConnector • Configure sink connector { "name": "cps-sink-connector1", "config": { "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector", "tasks.max": "1", "topics": "iot-topic", "cps.topic": "cps-iot-input", "cps.project": "app-project-294009", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "key.converter.schemas.enable": false, "value.converter.schemas.enable": false } } Cloud Pub/Sub configs
  • 27. GCP CloudPubSubConnector • Configure source connector { "name": "cps-source-connector1", "config": { "connector.class": "com.google.pubsub.kafka.source.CloudPubSubSourceConnector", "tasks.max": "1", "kafka.topic": "iot-processed-topic", "cps.subscription": "subscription2", "cps.project": "app-project-294009", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "key.converter.schemas.enable": false, "value.converter.schemas.enable": false } } Cloud Pub/Sub configs
  • 28. Reference • GCP Pubsub Connector • Kafka Connect • Creating Apache Kafka Connectors with the Kafka Connect API • Source and Sink for Kafka and PubSub