Submit Search
Upload
排队排队--kafka
•
Download as PPTX, PDF
•
1 like
•
324 views
C
chernbb
Follow
introduce to kafka, about achitecture, producer, broker, consumer.
Read less
Read more
Internet
Report
Share
Report
Share
1 of 7
Download now
Recommended
Java类加载器
Java类加载器
Fu Cheng
Avro
Avro
Eric Turcotte
맛만 보자 Finagle이란
맛만 보자 Finagle이란
jbugkorea
java thrift
java thrift
Mesh Korea
Microservices in the Enterprise
Microservices in the Enterprise
Jesus Rodriguez
RPC protocols
RPC protocols
오석 한
Avro - More Than Just a Serialization Framework - CHUG - 20120416
Avro - More Than Just a Serialization Framework - CHUG - 20120416
Chicago Hadoop Users Group
Protobuf & Code Generation + Go-Kit
Protobuf & Code Generation + Go-Kit
Manfred Touron
Recommended
Java类加载器
Java类加载器
Fu Cheng
Avro
Avro
Eric Turcotte
맛만 보자 Finagle이란
맛만 보자 Finagle이란
jbugkorea
java thrift
java thrift
Mesh Korea
Microservices in the Enterprise
Microservices in the Enterprise
Jesus Rodriguez
RPC protocols
RPC protocols
오석 한
Avro - More Than Just a Serialization Framework - CHUG - 20120416
Avro - More Than Just a Serialization Framework - CHUG - 20120416
Chicago Hadoop Users Group
Protobuf & Code Generation + Go-Kit
Protobuf & Code Generation + Go-Kit
Manfred Touron
OpenFest 2016 - Open Microservice Architecture
OpenFest 2016 - Open Microservice Architecture
Nikolay Stoitsev
3 avro hug-2010-07-21
3 avro hug-2010-07-21
Hadoop User Group
G rpc lection1_theory_bkp2
G rpc lection1_theory_bkp2
eleksdev
G rpc lection1
G rpc lection1
eleksdev
RPC: Remote procedure call
RPC: Remote procedure call
Sunita Sahu
HTTP2 and gRPC
HTTP2 and gRPC
Guo Jing
아파치 쓰리프트 (Apache Thrift)
아파치 쓰리프트 (Apache Thrift)
Jin wook
Building High Performance APIs In Go Using gRPC And Protocol Buffers
Building High Performance APIs In Go Using gRPC And Protocol Buffers
Shiju Varghese
3 apache-avro
3 apache-avro
zafargilani
Enabling Googley microservices with HTTP/2 and gRPC.
Enabling Googley microservices with HTTP/2 and gRPC.
Alex Borysov
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
Codemotion
gRPC: The Story of Microservices at Square
gRPC: The Story of Microservices at Square
Apigee | Google Cloud
Msa読書会#3前半
Msa読書会#3前半
健仁 天沼
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
AboutYouGmbH
Apache kafka
Apache kafka
Rahul Jain
Apache big data 2016 - Speaking the language of Big Data
Apache big data 2016 - Speaking the language of Big Data
techmaddy
Driving containerd operations with gRPC
Driving containerd operations with gRPC
Docker, Inc.
GRPC 101 - DevFest Belgium 2016
GRPC 101 - DevFest Belgium 2016
Alex Van Boxel
Flume
Flume
chernbb
预测 副本
预测 副本
chernbb
Hbase拾荒者
Hbase拾荒者
chernbb
hadoop中的懒人Hive
hadoop中的懒人Hive
chernbb
More Related Content
Viewers also liked
OpenFest 2016 - Open Microservice Architecture
OpenFest 2016 - Open Microservice Architecture
Nikolay Stoitsev
3 avro hug-2010-07-21
3 avro hug-2010-07-21
Hadoop User Group
G rpc lection1_theory_bkp2
G rpc lection1_theory_bkp2
eleksdev
G rpc lection1
G rpc lection1
eleksdev
RPC: Remote procedure call
RPC: Remote procedure call
Sunita Sahu
HTTP2 and gRPC
HTTP2 and gRPC
Guo Jing
아파치 쓰리프트 (Apache Thrift)
아파치 쓰리프트 (Apache Thrift)
Jin wook
Building High Performance APIs In Go Using gRPC And Protocol Buffers
Building High Performance APIs In Go Using gRPC And Protocol Buffers
Shiju Varghese
3 apache-avro
3 apache-avro
zafargilani
Enabling Googley microservices with HTTP/2 and gRPC.
Enabling Googley microservices with HTTP/2 and gRPC.
Alex Borysov
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
Codemotion
gRPC: The Story of Microservices at Square
gRPC: The Story of Microservices at Square
Apigee | Google Cloud
Msa読書会#3前半
Msa読書会#3前半
健仁 天沼
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
AboutYouGmbH
Apache kafka
Apache kafka
Rahul Jain
Apache big data 2016 - Speaking the language of Big Data
Apache big data 2016 - Speaking the language of Big Data
techmaddy
Driving containerd operations with gRPC
Driving containerd operations with gRPC
Docker, Inc.
GRPC 101 - DevFest Belgium 2016
GRPC 101 - DevFest Belgium 2016
Alex Van Boxel
Viewers also liked
(18)
OpenFest 2016 - Open Microservice Architecture
OpenFest 2016 - Open Microservice Architecture
3 avro hug-2010-07-21
3 avro hug-2010-07-21
G rpc lection1_theory_bkp2
G rpc lection1_theory_bkp2
G rpc lection1
G rpc lection1
RPC: Remote procedure call
RPC: Remote procedure call
HTTP2 and gRPC
HTTP2 and gRPC
아파치 쓰리프트 (Apache Thrift)
아파치 쓰리프트 (Apache Thrift)
Building High Performance APIs In Go Using gRPC And Protocol Buffers
Building High Performance APIs In Go Using gRPC And Protocol Buffers
3 apache-avro
3 apache-avro
Enabling Googley microservices with HTTP/2 and gRPC.
Enabling Googley microservices with HTTP/2 and gRPC.
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
Introduction to gRPC - Mete Atamel - Codemotion Rome 2017
gRPC: The Story of Microservices at Square
gRPC: The Story of Microservices at Square
Msa読書会#3前半
Msa読書会#3前半
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Apache kafka
Apache kafka
Apache big data 2016 - Speaking the language of Big Data
Apache big data 2016 - Speaking the language of Big Data
Driving containerd operations with gRPC
Driving containerd operations with gRPC
GRPC 101 - DevFest Belgium 2016
GRPC 101 - DevFest Belgium 2016
More from chernbb
Flume
Flume
chernbb
预测 副本
预测 副本
chernbb
Hbase拾荒者
Hbase拾荒者
chernbb
hadoop中的懒人Hive
hadoop中的懒人Hive
chernbb
大数据保险 副本
大数据保险 副本
chernbb
我们需要你是这样的
我们需要你是这样的
chernbb
抱着马云大腿谈谈管理
抱着马云大腿谈谈管理
chernbb
微服务
微服务
chernbb
谈谈小米的营销
谈谈小米的营销
chernbb
大数据 数据挖掘
大数据 数据挖掘
chernbb
More from chernbb
(10)
Flume
Flume
预测 副本
预测 副本
Hbase拾荒者
Hbase拾荒者
hadoop中的懒人Hive
hadoop中的懒人Hive
大数据保险 副本
大数据保险 副本
我们需要你是这样的
我们需要你是这样的
抱着马云大腿谈谈管理
抱着马云大腿谈谈管理
微服务
微服务
谈谈小米的营销
谈谈小米的营销
大数据 数据挖掘
大数据 数据挖掘
排队排队--kafka
1.
排队排队!——kafka
2.
什么是kafka 不是《变形虫》的作者。 Kafka起源于LinkedIn,使用scala语言实现。 是一个分布式的、分区的、多复本的日志提交服务。 作为消息队列使用,类似RabbitMQ等。 负载高,可横向扩展。 可靠性较低,无排序,无确认。 RPC协议使用Avro,thrift协议。 Jakfa,java版本。
3.
kafka的架构 Kafka体系中有三种身份,Kafka只作为Broker。 Zookeeper作为集群管理,收集Consumer,Broker的信息,提 供查询。 producer推给broker,consumer从broker拉,broker是被动的。 Kafka可以分多副本,支持容灾。 消息分topic,不同的topic分不同的队列。同一topic,可以分 为多个partition,分为不同的队列中。 consumer分组,一组可以有多个。一个消息只能被一组消费 一次。不同组可以重复消费。 最多一次,至少一次,恰好一次。
4.
生产者 生产者要连接Kafka,需要保持和每个partition的leader的连接。 没有路由。 当一个topic分为多个partition时,producer需要指定partition。 生产者主动推送过去。 批量插入broker,速度快。 消息不要求时效性,不要求事务。 多副本的存在,只要消息插入到broker,就认为不会丢失。 接受的消息量一般随生产者的增加线性增加。
5.
代理者 被动,基本只维护队列。 分topic,每topic分多个partition,每个partition可以多副本, 有一个leader。leader尽量分散。 消息顺序写硬盘,分几个固定大小或者按照时间删掉旧的。 按照offset来分发消息,不保存offset的信息(zookeeper)。 In-Sync Replica,保证延迟不会太大。 可迁移topic,增加分区副本,或者增加分区。 只解决”fail/recover”,不处理“Byzantine”问题。
6.
消费者 一个消费者,绑定一个partition。一个partition,只能有一个 消费者。 通过zookeeper拿offset。 可以重设offset,特别适合灾难恢复。 消费者要到zookeeper连接,指定组id。 发送fetch到broker,并告知offset,broker返回。 API连接方便,指定zookeeper就可以。 当consumer或者broker数目变化时候,会导致连接的重新分配。 consumer的commit,会改变offset,可在执行前也可在执行后。
7.
flume的架构 类似。 cloudera的产品。
Download now