SlideShare a Scribd company logo
February 22, 2018
Building Data-Driven
Microservices
Karthik Ramasamy
Database and real-time architect at Greenplum,
Locomatix, Twitter
Co-creator of Heron real-time engine
Co-founder, Streamlio
@karthikz
Today’s presenter
• Understanding microservices
• Best practices for connecting microservices
• Approaches to microservices design
• Streamlio technology and approach
Agenda
Understanding microservices
Comparison: “Monolith” vs “Microservices”
Monolith
Puts all functionality
into a single process
Scales by replicating
monolith across nodes
Microservices
Each element of
functionality in a
different service
Scales by distributing &
replicating each service
across nodes
Challenges with monolithic software
Difficult to
scale
Long Build/
Test/Release
Cycles
Operations
nightmares
Architecture is
hard to
maintain and
evolve
New releases
take months
Long time to
add new
features
Lack of agility
Lack of
Innovation
Anatomy of a microservice
Data Store
(e.g) Postgres,
Cassandra
Application/
Logic
code, libraries,
etc,
POST
GET
Public API
Connecting microservices
Comparison
Monolithic
• Most connections internal to
application
• Limited external connections
• Internal connections loosely
defined
Microservices
• Most connections external
• Greater stability of connection
interfaces required
Breaking the “monolith”
Structure application as a
set of loosely coupled,
collaborating services
Services communicate
using synchronous or
asynchronous protocols
Services can be developed
and deployed independent
of one another
Each service has its own
database in order to be
decoupled from other
services
• Decompose by use case and define services that are
responsible for particular actions (e.g) a shipping service
responsible for shipping complete orders
• Decompose by resources by defining a service that is
responsible for all operations on entities/resources of a
given type (e.g. account service that is responsible for
managing user accounts)
Decomposing into services
Option 1
Option 2
Approaches to microservices
• Takes orders from
customers
• Verifies inventory
• Checks credit
• Ships the order
E-Commerce application (monolithic architecture)
Inventory
Shipping
Account
API
Gateway
Webfront
UI
Approaches to micro services
Heavyweight
services
Single queue
per service
Shared queue
across services
Streamlio platform
approach
1 2 3 4
Approach 1: Heavyweight services
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Approach 1: Pros and cons
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Pros
• End-to-end service
ownership with the team
Cons
• Gateway has to manage
connections to services
• Possible diverse connection types
- makes coordination harder
• Heavy weight processing still
happens in the gateway
Approach 2: Single queue per service
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Queue
Queue
Queue
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Queue
Queue
Queue
Approach 2: Pros and cons
Pros
• e2e service ownership
with the team
• Single connection type
to message queue
Cons
• Heavy weight processing still happens at the
gateway
• One message queue cluster per service
• Too many message queue clusters to manage
• Messaging expertise needed for every team
• More complex and expensive to manage
Approach 3: Shared queue across services
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Message
Queue
Approach 3: Pros and cons
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Message
Queue
Pros
• Single message cluster shared
across multiple teams
• No messaging expertise needed
per team
Cons
• Heavy weight processing still happens
at the gateway
• Messaging cluster has to scale for
several services
• Without isolation capabilities,
contention across teams
Streamlio approach
Key technologies
Orchestration Messaging
Storage Process
communicate and pass
information among services
transform, aggregate
and enrich data flow
save state for micro-
services, calculations and
message replay
schedule, run, scale up
and scale down
microservices
Microservices
Interfaces
APIs Libraries & Connectivity
Streamlio platform
Real-time processing
Messaging & queuing
Stream storage
ConnectorsClientData SourceStormKafka Functional
Management
Resource Management
Metadata
Security
Monitoring
Orchestration
Powered by
Powered by
Powered by
Built on proven open-source technology
• Deployed in production for
3+ years at Yahoo
• Handling >2M topics
• Processing >100B
messages / day
• Deployed across 8
datacenters globally
• Deployed in production at
Twitter for 2+ years
• Used by ~50 teams
• Processing >2T events,
>20 PB / day
• In production at Twitter,
Yahoo, Salesforce, Huawei
• Processing >1.5T records,
>15PB / day at Yahoo
Using Streamlio to tie together microservices
Inventory
micro-service
Shipping
micro-service
Account
micro-service
API
Gateway
Webfront
UI
Requests/
Responses
Requests/
Responses
Streamlio approach: Advantages
• Single platform shared across multiple teams
• Single platform provides messaging, compute and storage capabilities
• No heavy weight processing at the gateway - offloaded to platform
• Single platform can scale to 1000’s of microservices
• Isolation capabilities for each team to manage their service in the platform
• Enterprise class features builtin - resilient, self healing, security, data protection
• Ease of operations thereby reducing total cost of ownership
Using Streamlio for multiple applications
API
Gateway
Requests/
Responses
Application 1
Application 2
Application 3
Using Streamlio across data centers
API
Gateway
Requests/
Responses
Application 1
Application 2
API
Gateway
Requests/
Responses
Application 1
Application 2
Karthik Ramasamy
Cofounder
Streamlio
@karthikz
Q&A
• Microservices
• Video Series: “Microservices: perceptions
and myths” (streaml.io/blog)
• Streamlio
• Streamlio Solution Brief: streaml.io/
resources
• Intro to Apache Pulsar: streaml.io/
product/messaging
How to learn more
Building data-driven microservices

More Related Content

What's hot

Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
Dalibor Wijas
 
Domain Driven Design
Domain Driven Design Domain Driven Design
Domain Driven Design
Araf Karsh Hamid
 
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
DataWorks Summit
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
Ryan Blue
 
From Data Warehouse to Lakehouse
From Data Warehouse to LakehouseFrom Data Warehouse to Lakehouse
From Data Warehouse to Lakehouse
Modern Data Stack France
 
Monolithic architecture
Monolithic architectureMonolithic architecture
Monolithic architecture
SRM University Delhi-NCR sonepat
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
Alex Ivy
 
Google Vertex AI
Google Vertex AIGoogle Vertex AI
Google Vertex AI
VikasBisoi
 
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Flink Forward
 
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Databricks
 
cLoki: Like Loki but for ClickHouse
cLoki: Like Loki but for ClickHousecLoki: Like Loki but for ClickHouse
cLoki: Like Loki but for ClickHouse
Altinity Ltd
 
How to build a successful Data Lake
How to build a successful Data LakeHow to build a successful Data Lake
How to build a successful Data Lake
DataWorks Summit/Hadoop Summit
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
James Serra
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
Jeffrey T. Pollock
 
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
HostedbyConfluent
 
Building an Observability platform with ClickHouse
Building an Observability platform with ClickHouseBuilding an Observability platform with ClickHouse
Building an Observability platform with ClickHouse
Altinity Ltd
 
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
Databricks
 
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Introduction to DataFusion  An Embeddable Query Engine Written in RustIntroduction to DataFusion  An Embeddable Query Engine Written in Rust
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Andrew Lamb
 
TechEvent Databricks on Azure
TechEvent Databricks on AzureTechEvent Databricks on Azure
TechEvent Databricks on Azure
Trivadis
 

What's hot (20)

Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
Domain Driven Design
Domain Driven Design Domain Driven Design
Domain Driven Design
 
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
Streamline Data Governance with Egeria: The Industry's First Open Metadata St...
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
 
From Data Warehouse to Lakehouse
From Data Warehouse to LakehouseFrom Data Warehouse to Lakehouse
From Data Warehouse to Lakehouse
 
Monolithic architecture
Monolithic architectureMonolithic architecture
Monolithic architecture
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
Google Vertex AI
Google Vertex AIGoogle Vertex AI
Google Vertex AI
 
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
 
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
 
cLoki: Like Loki but for ClickHouse
cLoki: Like Loki but for ClickHousecLoki: Like Loki but for ClickHouse
cLoki: Like Loki but for ClickHouse
 
How to build a successful Data Lake
How to build a successful Data LakeHow to build a successful Data Lake
How to build a successful Data Lake
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
 
Building an Observability platform with ClickHouse
Building an Observability platform with ClickHouseBuilding an Observability platform with ClickHouse
Building an Observability platform with ClickHouse
 
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
 
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Introduction to DataFusion  An Embeddable Query Engine Written in RustIntroduction to DataFusion  An Embeddable Query Engine Written in Rust
Introduction to DataFusion An Embeddable Query Engine Written in Rust
 
TechEvent Databricks on Azure
TechEvent Databricks on AzureTechEvent Databricks on Azure
TechEvent Databricks on Azure
 

Similar to Building data-driven microservices

Microservices-101
Microservices-101Microservices-101
Microservices-101
Subhashish Bhattacharjee
 
Do I Need A Service Mesh.pptx
Do I Need A Service Mesh.pptxDo I Need A Service Mesh.pptx
Do I Need A Service Mesh.pptx
PINGXIONG3
 
Microservice intro
Microservice introMicroservice intro
Microservice intro
ramesh_sharma
 
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
AFAS Software
 
QCon 2015 - Microservices Track Notes
QCon 2015 - Microservices Track Notes QCon 2015 - Microservices Track Notes
QCon 2015 - Microservices Track Notes
Abdul Basit Munda
 
Do You Need A Service Mesh?
Do You Need A Service Mesh?Do You Need A Service Mesh?
Do You Need A Service Mesh?
NGINX, Inc.
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice Architecture
Engin Yoeyen
 
API’s and Micro Services 0.5
API’s and Micro Services 0.5API’s and Micro Services 0.5
API’s and Micro Services 0.5
Richard Hudson
 
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
India Scrum Enthusiasts Community
 
2019 hashiconf seattle_consul_ioc
2019 hashiconf seattle_consul_ioc2019 hashiconf seattle_consul_ioc
2019 hashiconf seattle_consul_ioc
Pierre Souchay
 
MicroServices architecture @ Ctrip v1.1
MicroServices architecture @ Ctrip v1.1MicroServices architecture @ Ctrip v1.1
MicroServices architecture @ Ctrip v1.1
William Yang
 
Introduction to Microservices
Introduction to MicroservicesIntroduction to Microservices
Introduction to Microservices
MahmoudZidan41
 
The Overview of Microservices Architecture
The Overview of Microservices ArchitectureThe Overview of Microservices Architecture
The Overview of Microservices Architecture
Paria Heidari
 
Microservices and Serverless for Mega Startups - DevOps IL Meetup
Microservices and Serverless for Mega Startups - DevOps IL MeetupMicroservices and Serverless for Mega Startups - DevOps IL Meetup
Microservices and Serverless for Mega Startups - DevOps IL Meetup
Boaz Ziniman
 
gkkCloudtechnologyassociate(cta)day 1
gkkCloudtechnologyassociate(cta)day 1gkkCloudtechnologyassociate(cta)day 1
gkkCloudtechnologyassociate(cta)day 1
Anne Starr
 
Securing elastic applications_on_mobile_devices
Securing elastic applications_on_mobile_devicesSecuring elastic applications_on_mobile_devices
Securing elastic applications_on_mobile_devices
firzhan naqash
 
Agility with Microservices and DevOps
Agility with Microservices and DevOpsAgility with Microservices and DevOps
Agility with Microservices and DevOps
Archana Joshi
 
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Evolution of the IBM Cloud Console: From Monolith to Microservices and BeyondEvolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Tony Erwin
 
Testing the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the CloudTesting the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the Cloud
Nagarro
 
Cloud Computing & Business Intelligence
Cloud Computing & Business IntelligenceCloud Computing & Business Intelligence
Cloud Computing & Business Intelligence
Sudip Chatterjee
 

Similar to Building data-driven microservices (20)

Microservices-101
Microservices-101Microservices-101
Microservices-101
 
Do I Need A Service Mesh.pptx
Do I Need A Service Mesh.pptxDo I Need A Service Mesh.pptx
Do I Need A Service Mesh.pptx
 
Microservice intro
Microservice introMicroservice intro
Microservice intro
 
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
Alex Thissen (Xpirit) - Een verschuiving in architectuur: op weg naar microse...
 
QCon 2015 - Microservices Track Notes
QCon 2015 - Microservices Track Notes QCon 2015 - Microservices Track Notes
QCon 2015 - Microservices Track Notes
 
Do You Need A Service Mesh?
Do You Need A Service Mesh?Do You Need A Service Mesh?
Do You Need A Service Mesh?
 
Microservice Architecture
Microservice ArchitectureMicroservice Architecture
Microservice Architecture
 
API’s and Micro Services 0.5
API’s and Micro Services 0.5API’s and Micro Services 0.5
API’s and Micro Services 0.5
 
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
Agile Tour Pune 2015: Agility with Microservices and Devops: Archana Joshi an...
 
2019 hashiconf seattle_consul_ioc
2019 hashiconf seattle_consul_ioc2019 hashiconf seattle_consul_ioc
2019 hashiconf seattle_consul_ioc
 
MicroServices architecture @ Ctrip v1.1
MicroServices architecture @ Ctrip v1.1MicroServices architecture @ Ctrip v1.1
MicroServices architecture @ Ctrip v1.1
 
Introduction to Microservices
Introduction to MicroservicesIntroduction to Microservices
Introduction to Microservices
 
The Overview of Microservices Architecture
The Overview of Microservices ArchitectureThe Overview of Microservices Architecture
The Overview of Microservices Architecture
 
Microservices and Serverless for Mega Startups - DevOps IL Meetup
Microservices and Serverless for Mega Startups - DevOps IL MeetupMicroservices and Serverless for Mega Startups - DevOps IL Meetup
Microservices and Serverless for Mega Startups - DevOps IL Meetup
 
gkkCloudtechnologyassociate(cta)day 1
gkkCloudtechnologyassociate(cta)day 1gkkCloudtechnologyassociate(cta)day 1
gkkCloudtechnologyassociate(cta)day 1
 
Securing elastic applications_on_mobile_devices
Securing elastic applications_on_mobile_devicesSecuring elastic applications_on_mobile_devices
Securing elastic applications_on_mobile_devices
 
Agility with Microservices and DevOps
Agility with Microservices and DevOpsAgility with Microservices and DevOps
Agility with Microservices and DevOps
 
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Evolution of the IBM Cloud Console: From Monolith to Microservices and BeyondEvolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
 
Testing the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the CloudTesting the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the Cloud
 
Cloud Computing & Business Intelligence
Cloud Computing & Business IntelligenceCloud Computing & Business Intelligence
Cloud Computing & Business Intelligence
 

More from Streamlio

Infinite Topic Backlogs with Apache Pulsar
Infinite Topic Backlogs with Apache PulsarInfinite Topic Backlogs with Apache Pulsar
Infinite Topic Backlogs with Apache Pulsar
Streamlio
 
Apache Pulsar Overview
Apache Pulsar OverviewApache Pulsar Overview
Apache Pulsar Overview
Streamlio
 
Streamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache PulsarStreamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache Pulsar
Streamlio
 
Strata London 2018: Multi-everything with Apache Pulsar
Strata London 2018:  Multi-everything with Apache PulsarStrata London 2018:  Multi-everything with Apache Pulsar
Strata London 2018: Multi-everything with Apache Pulsar
Streamlio
 
Self Regulating Streaming - Data Platforms Conference 2018
Self Regulating Streaming - Data Platforms Conference 2018Self Regulating Streaming - Data Platforms Conference 2018
Self Regulating Streaming - Data Platforms Conference 2018
Streamlio
 
Introduction to Apache BookKeeper Distributed Storage
Introduction to Apache BookKeeper Distributed StorageIntroduction to Apache BookKeeper Distributed Storage
Introduction to Apache BookKeeper Distributed Storage
Streamlio
 
Event Data Processing with Streamlio
Event Data Processing with StreamlioEvent Data Processing with Streamlio
Event Data Processing with Streamlio
Streamlio
 
Stream-Native Processing with Pulsar Functions
Stream-Native Processing with Pulsar FunctionsStream-Native Processing with Pulsar Functions
Stream-Native Processing with Pulsar Functions
Streamlio
 
Distributed Crypto-Currency Trading with Apache Pulsar
Distributed Crypto-Currency Trading with Apache PulsarDistributed Crypto-Currency Trading with Apache Pulsar
Distributed Crypto-Currency Trading with Apache Pulsar
Streamlio
 
Evaluating Streaming Data Solutions
Evaluating Streaming Data SolutionsEvaluating Streaming Data Solutions
Evaluating Streaming Data Solutions
Streamlio
 
Autopiloting Realtime Processing in Heron
Autopiloting Realtime Processing in HeronAutopiloting Realtime Processing in Heron
Autopiloting Realtime Processing in Heron
Streamlio
 
Introduction to Apache Heron
Introduction to Apache HeronIntroduction to Apache Heron
Introduction to Apache Heron
Streamlio
 
Messaging, storage, or both? The real time story of Pulsar and Apache Distri...
Messaging, storage, or both?  The real time story of Pulsar and Apache Distri...Messaging, storage, or both?  The real time story of Pulsar and Apache Distri...
Messaging, storage, or both? The real time story of Pulsar and Apache Distri...
Streamlio
 

More from Streamlio (13)

Infinite Topic Backlogs with Apache Pulsar
Infinite Topic Backlogs with Apache PulsarInfinite Topic Backlogs with Apache Pulsar
Infinite Topic Backlogs with Apache Pulsar
 
Apache Pulsar Overview
Apache Pulsar OverviewApache Pulsar Overview
Apache Pulsar Overview
 
Streamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache PulsarStreamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache Pulsar
 
Strata London 2018: Multi-everything with Apache Pulsar
Strata London 2018:  Multi-everything with Apache PulsarStrata London 2018:  Multi-everything with Apache Pulsar
Strata London 2018: Multi-everything with Apache Pulsar
 
Self Regulating Streaming - Data Platforms Conference 2018
Self Regulating Streaming - Data Platforms Conference 2018Self Regulating Streaming - Data Platforms Conference 2018
Self Regulating Streaming - Data Platforms Conference 2018
 
Introduction to Apache BookKeeper Distributed Storage
Introduction to Apache BookKeeper Distributed StorageIntroduction to Apache BookKeeper Distributed Storage
Introduction to Apache BookKeeper Distributed Storage
 
Event Data Processing with Streamlio
Event Data Processing with StreamlioEvent Data Processing with Streamlio
Event Data Processing with Streamlio
 
Stream-Native Processing with Pulsar Functions
Stream-Native Processing with Pulsar FunctionsStream-Native Processing with Pulsar Functions
Stream-Native Processing with Pulsar Functions
 
Distributed Crypto-Currency Trading with Apache Pulsar
Distributed Crypto-Currency Trading with Apache PulsarDistributed Crypto-Currency Trading with Apache Pulsar
Distributed Crypto-Currency Trading with Apache Pulsar
 
Evaluating Streaming Data Solutions
Evaluating Streaming Data SolutionsEvaluating Streaming Data Solutions
Evaluating Streaming Data Solutions
 
Autopiloting Realtime Processing in Heron
Autopiloting Realtime Processing in HeronAutopiloting Realtime Processing in Heron
Autopiloting Realtime Processing in Heron
 
Introduction to Apache Heron
Introduction to Apache HeronIntroduction to Apache Heron
Introduction to Apache Heron
 
Messaging, storage, or both? The real time story of Pulsar and Apache Distri...
Messaging, storage, or both?  The real time story of Pulsar and Apache Distri...Messaging, storage, or both?  The real time story of Pulsar and Apache Distri...
Messaging, storage, or both? The real time story of Pulsar and Apache Distri...
 

Recently uploaded

Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
Alina Yurenko
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
sandeepmenon62
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
dakas1
 
What is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdfWhat is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdf
kalichargn70th171
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
KrishnaveniMohan1
 
Ensuring Efficiency and Speed with Practical Solutions for Clinical Operations
Ensuring Efficiency and Speed with Practical Solutions for Clinical OperationsEnsuring Efficiency and Speed with Practical Solutions for Clinical Operations
Ensuring Efficiency and Speed with Practical Solutions for Clinical Operations
OnePlan Solutions
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Streamlining End-to-End Testing Automation
Streamlining End-to-End Testing AutomationStreamlining End-to-End Testing Automation
Streamlining End-to-End Testing Automation
Anand Bagmar
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
Zycus
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies
 
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
dhavalvaghelanectarb
 
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
confluent
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
michniczscribd
 

Recently uploaded (20)

Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
 
bgiolcb
bgiolcbbgiolcb
bgiolcb
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
 
What is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdfWhat is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdf
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
 
Ensuring Efficiency and Speed with Practical Solutions for Clinical Operations
Ensuring Efficiency and Speed with Practical Solutions for Clinical OperationsEnsuring Efficiency and Speed with Practical Solutions for Clinical Operations
Ensuring Efficiency and Speed with Practical Solutions for Clinical Operations
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
 
Streamlining End-to-End Testing Automation
Streamlining End-to-End Testing AutomationStreamlining End-to-End Testing Automation
Streamlining End-to-End Testing Automation
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
 
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
 
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
 

Building data-driven microservices

  • 1. February 22, 2018 Building Data-Driven Microservices
  • 2. Karthik Ramasamy Database and real-time architect at Greenplum, Locomatix, Twitter Co-creator of Heron real-time engine Co-founder, Streamlio @karthikz Today’s presenter
  • 3. • Understanding microservices • Best practices for connecting microservices • Approaches to microservices design • Streamlio technology and approach Agenda
  • 5. Comparison: “Monolith” vs “Microservices” Monolith Puts all functionality into a single process Scales by replicating monolith across nodes Microservices Each element of functionality in a different service Scales by distributing & replicating each service across nodes
  • 6. Challenges with monolithic software Difficult to scale Long Build/ Test/Release Cycles Operations nightmares Architecture is hard to maintain and evolve New releases take months Long time to add new features Lack of agility Lack of Innovation
  • 7. Anatomy of a microservice Data Store (e.g) Postgres, Cassandra Application/ Logic code, libraries, etc, POST GET Public API
  • 9. Comparison Monolithic • Most connections internal to application • Limited external connections • Internal connections loosely defined Microservices • Most connections external • Greater stability of connection interfaces required
  • 10. Breaking the “monolith” Structure application as a set of loosely coupled, collaborating services Services communicate using synchronous or asynchronous protocols Services can be developed and deployed independent of one another Each service has its own database in order to be decoupled from other services
  • 11. • Decompose by use case and define services that are responsible for particular actions (e.g) a shipping service responsible for shipping complete orders • Decompose by resources by defining a service that is responsible for all operations on entities/resources of a given type (e.g. account service that is responsible for managing user accounts) Decomposing into services Option 1 Option 2
  • 13. • Takes orders from customers • Verifies inventory • Checks credit • Ships the order E-Commerce application (monolithic architecture) Inventory Shipping Account API Gateway Webfront UI
  • 14. Approaches to micro services Heavyweight services Single queue per service Shared queue across services Streamlio platform approach 1 2 3 4
  • 15. Approach 1: Heavyweight services Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI
  • 16. Approach 1: Pros and cons Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Pros • End-to-end service ownership with the team Cons • Gateway has to manage connections to services • Possible diverse connection types - makes coordination harder • Heavy weight processing still happens in the gateway
  • 17. Approach 2: Single queue per service Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Queue Queue Queue
  • 18. Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Queue Queue Queue Approach 2: Pros and cons Pros • e2e service ownership with the team • Single connection type to message queue Cons • Heavy weight processing still happens at the gateway • One message queue cluster per service • Too many message queue clusters to manage • Messaging expertise needed for every team • More complex and expensive to manage
  • 19. Approach 3: Shared queue across services Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Message Queue
  • 20. Approach 3: Pros and cons Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Message Queue Pros • Single message cluster shared across multiple teams • No messaging expertise needed per team Cons • Heavy weight processing still happens at the gateway • Messaging cluster has to scale for several services • Without isolation capabilities, contention across teams
  • 22. Key technologies Orchestration Messaging Storage Process communicate and pass information among services transform, aggregate and enrich data flow save state for micro- services, calculations and message replay schedule, run, scale up and scale down microservices Microservices
  • 23. Interfaces APIs Libraries & Connectivity Streamlio platform Real-time processing Messaging & queuing Stream storage ConnectorsClientData SourceStormKafka Functional Management Resource Management Metadata Security Monitoring Orchestration Powered by Powered by Powered by
  • 24. Built on proven open-source technology • Deployed in production for 3+ years at Yahoo • Handling >2M topics • Processing >100B messages / day • Deployed across 8 datacenters globally • Deployed in production at Twitter for 2+ years • Used by ~50 teams • Processing >2T events, >20 PB / day • In production at Twitter, Yahoo, Salesforce, Huawei • Processing >1.5T records, >15PB / day at Yahoo
  • 25. Using Streamlio to tie together microservices Inventory micro-service Shipping micro-service Account micro-service API Gateway Webfront UI Requests/ Responses Requests/ Responses
  • 26. Streamlio approach: Advantages • Single platform shared across multiple teams • Single platform provides messaging, compute and storage capabilities • No heavy weight processing at the gateway - offloaded to platform • Single platform can scale to 1000’s of microservices • Isolation capabilities for each team to manage their service in the platform • Enterprise class features builtin - resilient, self healing, security, data protection • Ease of operations thereby reducing total cost of ownership
  • 27. Using Streamlio for multiple applications API Gateway Requests/ Responses Application 1 Application 2 Application 3
  • 28. Using Streamlio across data centers API Gateway Requests/ Responses Application 1 Application 2 API Gateway Requests/ Responses Application 1 Application 2
  • 30. • Microservices • Video Series: “Microservices: perceptions and myths” (streaml.io/blog) • Streamlio • Streamlio Solution Brief: streaml.io/ resources • Intro to Apache Pulsar: streaml.io/ product/messaging How to learn more