SlideShare a Scribd company logo
JAFKA

A fast MQ
overview
●   https://github.com/adyliu/jafka
●   271KB single jar
●   3.5MB with all dependencies and configurations
●   So simple, so fast




                           jafka mq                  2
Maven Usage

<dependency>
  <groupId>com.sohu.jafka</groupId>
  <artifactId>jafka</artifactId>
  <version>1.0-rc7</version>
</dependency>




                      jafka mq        3
Dependencies
●   Zkclient
    –   Zookeeper
●   Log4j
●   Jackson


●   Optional dependencies:
    –   Mx4j-tools
    –   jopt-simple
                          jafka mq   4
Future
●   A full kafka clone (http://incubator.apache.org/kafka)
●   Persisten messaging with O(1)
●   Constant time performance even with TB messages
●   High-throughput (100k/s)
●   Distributed messaging system
    –   producers/brokers/consumers
●   Auto load balancing
●   Simple message format



                                    jafka mq                 5
Message format (1)




       jafka mq      6
Message format (2)




       jafka mq      7
Log Implementation




       jafka mq      8
Architech and workflow




         jafka mq        9
Zookeeper nodes




      jafka mq    10
Offset length limit
●   Long.MAX_VALUE
    –   9223372036854775807 → 9*1018
●   If one message size is 1k
    –   Messages: 9*1015
    –   If producing 1 billion messages per day
        ●   Continuing days: 9*106
        ●   Continuing years: 24657



                                     jafka mq     11
About Filesystem
●   6 7200rpm SATA RAID-5 *
    –   300MB/s linear writes (300k/s 1KB per message)
    –   50k/s random writes




                                            jafka   mq   12
http://incubator.apache.org/kafka/design.html
sendfile() (1)
●   Common data transfer
    1. OS reads data from disk to pagecache
    2. Application reads data from pagecache to buffer
    3. Application writes data back into socket buffer
    4. OS copies data from socket buffer to NIC buffer
●   4 copies + 2 system_call



                              jafka mq                   13
sendfile() (2)
●    FileChannel.transferTo(long, long, WritableByteCh
     annel)
      –   Zero copy *




    https://www.ibm.com/developerworks/linux/library/j-zerocopy/
                                           jafka mq                14
Batch compression
●   GZIP message compression default




                        jafka mq       15
Sync or Async
●   Both supported
●   Setting at connecting
●   Batch sending while using asynchronous sending




                            jafka mq                 16
Usage
●   Simple & Funny




                      jafka mq   17
Simple usage - dependencies

   <dependency>
     <groupId>com.sohu.suc*</groupId>
     <artifactId>suc-mq</artifactId>
   </dependency>




    Version >= 2.0-SNAPSHOT
    ●




* our inner project
                         jafka mq       18
Simple usage - producer

Producers producers = Producers.buildProducer();
StringMessage data = new StringMessage("demo");
data.add("first message");
data.add("second message");
data.add("third message");
producers.send(data);




                           jafka mq                19
Simple usage - consumer

public static void main(String[] args) {

    Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() {
     @Override
     public void onMessage(String message) {
       System.out.println(message + " ==> " + new Date());
      }
    });
}




                                       jafka mq                                  20
Consumer => Queue
           Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() {
            public void onMessage(String message) {
Client A      System.out.println(message + " ==> " + new Date());
            }
           });




           Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() {
            public void onMessage(String message) {
Client A      System.out.println(message + " ==> " + new Date());
            }
           });




                                             jafka mq                                   21
Consumer => Topic
           Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() {
            public void onMessage(String message) {
              System.out.println(message + " ==> " + new Date());
Client A    }
           });




           Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() {
            public void onMessage(String message) {
Client B      System.out.println("[震子] ==> " + message);
            }
           });




                                                jafka mq                                  22
Full Producer code
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("serializer.class", StringEncoder.class.getName());
//
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
//
StringProducerData data = new StringProducerData("demo");
for(int i=0;i<100;i++) {
    data.add("Hello world #"+i);
}
try {
    long start = System.currentTimeMillis();
    for (int i = 0; i < 100; i++) {
       producer.send(data);
    }
    long cost = System.currentTimeMillis() - start;
    System.out.println("send 10000 message cost: "+cost+" ms");
} finally {
    producer.close();
}

                               jafka mq                                     23
Full Consumer Code
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("groupid", "test_group");
//
ConsumerConfig consumerConfig = new ConsumerConfig(props);
ConsumerConnector connector = Consumer.create(consumerConfig);
//
Map<String, List<MessageStream<String>>> topicMessageStreams =
connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder());
List<MessageStream<String>> streams = topicMessageStreams.get("demo");
//
ExecutorService executor = Executors.newFixedThreadPool(2);
final AtomicInteger count = new AtomicInteger();
for (final MessageStream<String> stream : streams) {
   executor.submit(new Runnable() {

          public void run() {
            for (String message : stream) {
               System.out.println(count.incrementAndGet() + " => " + message);
            }
          }
    });
}



                                                      jafka mq                     24
More Documents
●   https://github.com/adyliu/jafka/wiki




                           jafka mq        25
Open source
First Java open source in maven repository at groupId com.sohu.




                                  jafka mq                        26
About me
●   Ady Liu
●   Email: imxylz@gmail.com
●   Architech@SOHU.COM




                       jafka mq   27

More Related Content

What's hot

C#을 이용한 task 병렬화와 비동기 패턴
C#을 이용한 task 병렬화와 비동기 패턴C#을 이용한 task 병렬화와 비동기 패턴
C#을 이용한 task 병렬화와 비동기 패턴
명신 김
 
Reactive server with netty
Reactive server with nettyReactive server with netty
Reactive server with netty
Dmitriy Dumanskiy
 
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
Susan Potter
 
Apache zookeeper 101
Apache zookeeper 101Apache zookeeper 101
Apache zookeeper 101
Quach Tung
 
Cassandra summit 2013 - DataStax Java Driver Unleashed!
Cassandra summit 2013 - DataStax Java Driver Unleashed!Cassandra summit 2013 - DataStax Java Driver Unleashed!
Cassandra summit 2013 - DataStax Java Driver Unleashed!Michaël Figuière
 
Forgive me for i have allocated
Forgive me for i have allocatedForgive me for i have allocated
Forgive me for i have allocated
Tomasz Kowalczewski
 
Non blocking io with netty
Non blocking io with nettyNon blocking io with netty
Non blocking io with netty
Zauber
 
NoSQL and JavaScript: a Love Story
NoSQL and JavaScript: a Love StoryNoSQL and JavaScript: a Love Story
NoSQL and JavaScript: a Love Story
Alexandre Morgaut
 
Kubernetes networking - basics
Kubernetes networking - basicsKubernetes networking - basics
Kubernetes networking - basics
Juraj Hantak
 
Paris Cassandra Meetup - Cassandra for Developers
Paris Cassandra Meetup - Cassandra for DevelopersParis Cassandra Meetup - Cassandra for Developers
Paris Cassandra Meetup - Cassandra for DevelopersMichaël Figuière
 
GCD and OperationQueue.
GCD and OperationQueue.GCD and OperationQueue.
GCD and OperationQueue.
HSIEH CHING-FAN
 
In-depth caching in Varnish - GOG Varnish Meetup, march 2019
In-depth caching in Varnish - GOG Varnish Meetup, march 2019In-depth caching in Varnish - GOG Varnish Meetup, march 2019
In-depth caching in Varnish - GOG Varnish Meetup, march 2019
GOG.com dev team
 
Custom deployments with sbt-native-packager
Custom deployments with sbt-native-packagerCustom deployments with sbt-native-packager
Custom deployments with sbt-native-packager
GaryCoady
 
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
Nexus FrontierTech
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012
Joe Arnold
 
Blocks & GCD
Blocks & GCDBlocks & GCD
Blocks & GCD
rsebbe
 
Docker and Fargate
Docker and FargateDocker and Fargate
Docker and Fargate
Shinji Miyazato
 
YaJug - Cassandra for Java Developers
YaJug - Cassandra for Java DevelopersYaJug - Cassandra for Java Developers
YaJug - Cassandra for Java DevelopersMichaël Figuière
 
Pune-Cocoa: Blocks and GCD
Pune-Cocoa: Blocks and GCDPune-Cocoa: Blocks and GCD
Pune-Cocoa: Blocks and GCD
Prashant Rane
 
Geneva JUG - Cassandra for Java Developers
Geneva JUG - Cassandra for Java DevelopersGeneva JUG - Cassandra for Java Developers
Geneva JUG - Cassandra for Java DevelopersMichaël Figuière
 

What's hot (20)

C#을 이용한 task 병렬화와 비동기 패턴
C#을 이용한 task 병렬화와 비동기 패턴C#을 이용한 task 병렬화와 비동기 패턴
C#을 이용한 task 병렬화와 비동기 패턴
 
Reactive server with netty
Reactive server with nettyReactive server with netty
Reactive server with netty
 
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016
 
Apache zookeeper 101
Apache zookeeper 101Apache zookeeper 101
Apache zookeeper 101
 
Cassandra summit 2013 - DataStax Java Driver Unleashed!
Cassandra summit 2013 - DataStax Java Driver Unleashed!Cassandra summit 2013 - DataStax Java Driver Unleashed!
Cassandra summit 2013 - DataStax Java Driver Unleashed!
 
Forgive me for i have allocated
Forgive me for i have allocatedForgive me for i have allocated
Forgive me for i have allocated
 
Non blocking io with netty
Non blocking io with nettyNon blocking io with netty
Non blocking io with netty
 
NoSQL and JavaScript: a Love Story
NoSQL and JavaScript: a Love StoryNoSQL and JavaScript: a Love Story
NoSQL and JavaScript: a Love Story
 
Kubernetes networking - basics
Kubernetes networking - basicsKubernetes networking - basics
Kubernetes networking - basics
 
Paris Cassandra Meetup - Cassandra for Developers
Paris Cassandra Meetup - Cassandra for DevelopersParis Cassandra Meetup - Cassandra for Developers
Paris Cassandra Meetup - Cassandra for Developers
 
GCD and OperationQueue.
GCD and OperationQueue.GCD and OperationQueue.
GCD and OperationQueue.
 
In-depth caching in Varnish - GOG Varnish Meetup, march 2019
In-depth caching in Varnish - GOG Varnish Meetup, march 2019In-depth caching in Varnish - GOG Varnish Meetup, march 2019
In-depth caching in Varnish - GOG Varnish Meetup, march 2019
 
Custom deployments with sbt-native-packager
Custom deployments with sbt-native-packagerCustom deployments with sbt-native-packager
Custom deployments with sbt-native-packager
 
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
Tech Talk #4 : Multi - threading and GCD ( grand central dispatch ) in iOS - ...
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012
 
Blocks & GCD
Blocks & GCDBlocks & GCD
Blocks & GCD
 
Docker and Fargate
Docker and FargateDocker and Fargate
Docker and Fargate
 
YaJug - Cassandra for Java Developers
YaJug - Cassandra for Java DevelopersYaJug - Cassandra for Java Developers
YaJug - Cassandra for Java Developers
 
Pune-Cocoa: Blocks and GCD
Pune-Cocoa: Blocks and GCDPune-Cocoa: Blocks and GCD
Pune-Cocoa: Blocks and GCD
 
Geneva JUG - Cassandra for Java Developers
Geneva JUG - Cassandra for Java DevelopersGeneva JUG - Cassandra for Java Developers
Geneva JUG - Cassandra for Java Developers
 

Viewers also liked

Jsr310 - Java 8 Date and Time API
Jsr310 - Java 8 Date and Time APIJsr310 - Java 8 Date and Time API
Jsr310 - Java 8 Date and Time API
Ady Liu
 
Java常见疑惑和陷阱
Java常见疑惑和陷阱Java常见疑惑和陷阱
Java常见疑惑和陷阱Ady Liu
 
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutor
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutorInside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutor
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutorAdy Liu
 
高性能远程调用解决方案
高性能远程调用解决方案高性能远程调用解决方案
高性能远程调用解决方案
Ady Liu
 
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
Cyrille Le Clerc
 
Erlang scheduler
Erlang schedulerErlang scheduler
Erlang scheduler
dennis zhuang
 

Viewers also liked (6)

Jsr310 - Java 8 Date and Time API
Jsr310 - Java 8 Date and Time APIJsr310 - Java 8 Date and Time API
Jsr310 - Java 8 Date and Time API
 
Java常见疑惑和陷阱
Java常见疑惑和陷阱Java常见疑惑和陷阱
Java常见疑惑和陷阱
 
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutor
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutorInside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutor
Inside.java.concurrency 35.thread pool.part8_future.scheduledthreadpoolexecutor
 
高性能远程调用解决方案
高性能远程调用解决方案高性能远程调用解决方案
高性能远程调用解决方案
 
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
 
Erlang scheduler
Erlang schedulerErlang scheduler
Erlang scheduler
 

Similar to Jafka guide

Developing Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaDeveloping Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache Kafka
Joe Stein
 
Hazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMSHazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMS
uzquiano
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
Joe Stein
 
LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1
Hajime Tazaki
 
Apache Kafka
Apache KafkaApache Kafka
Apache KafkaJoe Stein
 
Presentation Lfoppiano Pycon
Presentation Lfoppiano PyconPresentation Lfoppiano Pycon
Presentation Lfoppiano PyconLuca Foppiano
 
I can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and SpringI can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and Spring
Joe Kutner
 
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
HostedbyConfluent
 
Enabling Data Scientists to easily create and own Kafka Consumers
Enabling Data Scientists to easily create and own Kafka ConsumersEnabling Data Scientists to easily create and own Kafka Consumers
Enabling Data Scientists to easily create and own Kafka Consumers
Stefan Krawczyk
 
Stream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache KafkaStream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache Kafka
Abhinav Singh
 
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Zabbix
 
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
tdc-globalcode
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINE
kawamuray
 
Sandboxing WebKitGTK (GUADEC 2019)
Sandboxing WebKitGTK (GUADEC 2019)Sandboxing WebKitGTK (GUADEC 2019)
Sandboxing WebKitGTK (GUADEC 2019)
Igalia
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
DataWorks Summit/Hadoop Summit
 
KDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics TutorialKDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics Tutorial
Neera Agarwal
 
Monitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_TutorialMonitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_TutorialTim Vaillancourt
 
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
Universitat Politècnica de Catalunya
 
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
Universitat Politècnica de Catalunya
 

Similar to Jafka guide (20)

Developing Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaDeveloping Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache Kafka
 
Hazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMSHazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMS
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Presentation Lfoppiano Pycon
Presentation Lfoppiano PyconPresentation Lfoppiano Pycon
Presentation Lfoppiano Pycon
 
I can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and SpringI can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and Spring
 
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
 
Enabling Data Scientists to easily create and own Kafka Consumers
Enabling Data Scientists to easily create and own Kafka ConsumersEnabling Data Scientists to easily create and own Kafka Consumers
Enabling Data Scientists to easily create and own Kafka Consumers
 
Stream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache KafkaStream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache Kafka
 
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
 
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
TDC2016POA | Trilha Arquitetura - Apache Kafka: uma introdução a logs distrib...
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINE
 
Sandboxing WebKitGTK (GUADEC 2019)
Sandboxing WebKitGTK (GUADEC 2019)Sandboxing WebKitGTK (GUADEC 2019)
Sandboxing WebKitGTK (GUADEC 2019)
 
KAFKA Quickstart
KAFKA QuickstartKAFKA Quickstart
KAFKA Quickstart
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
 
KDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics TutorialKDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics Tutorial
 
Monitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_TutorialMonitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_Tutorial
 
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
Software Frameworks for Deep Learning (D1L7 2017 UPC Deep Learning for Comput...
 
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
Deep Learning for Computer Vision: Software Frameworks (UPC 2016)
 

Recently uploaded

Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 

Recently uploaded (20)

Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 

Jafka guide

  • 2. overview ● https://github.com/adyliu/jafka ● 271KB single jar ● 3.5MB with all dependencies and configurations ● So simple, so fast jafka mq 2
  • 3. Maven Usage <dependency> <groupId>com.sohu.jafka</groupId> <artifactId>jafka</artifactId> <version>1.0-rc7</version> </dependency> jafka mq 3
  • 4. Dependencies ● Zkclient – Zookeeper ● Log4j ● Jackson ● Optional dependencies: – Mx4j-tools – jopt-simple jafka mq 4
  • 5. Future ● A full kafka clone (http://incubator.apache.org/kafka) ● Persisten messaging with O(1) ● Constant time performance even with TB messages ● High-throughput (100k/s) ● Distributed messaging system – producers/brokers/consumers ● Auto load balancing ● Simple message format jafka mq 5
  • 6. Message format (1) jafka mq 6
  • 7. Message format (2) jafka mq 7
  • 8. Log Implementation jafka mq 8
  • 10. Zookeeper nodes jafka mq 10
  • 11. Offset length limit ● Long.MAX_VALUE – 9223372036854775807 → 9*1018 ● If one message size is 1k – Messages: 9*1015 – If producing 1 billion messages per day ● Continuing days: 9*106 ● Continuing years: 24657 jafka mq 11
  • 12. About Filesystem ● 6 7200rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes jafka mq 12 http://incubator.apache.org/kafka/design.html
  • 13. sendfile() (1) ● Common data transfer 1. OS reads data from disk to pagecache 2. Application reads data from pagecache to buffer 3. Application writes data back into socket buffer 4. OS copies data from socket buffer to NIC buffer ● 4 copies + 2 system_call jafka mq 13
  • 14. sendfile() (2) ● FileChannel.transferTo(long, long, WritableByteCh annel) – Zero copy * https://www.ibm.com/developerworks/linux/library/j-zerocopy/ jafka mq 14
  • 15. Batch compression ● GZIP message compression default jafka mq 15
  • 16. Sync or Async ● Both supported ● Setting at connecting ● Batch sending while using asynchronous sending jafka mq 16
  • 17. Usage ● Simple & Funny jafka mq 17
  • 18. Simple usage - dependencies <dependency> <groupId>com.sohu.suc*</groupId> <artifactId>suc-mq</artifactId> </dependency> Version >= 2.0-SNAPSHOT ● * our inner project jafka mq 18
  • 19. Simple usage - producer Producers producers = Producers.buildProducer(); StringMessage data = new StringMessage("demo"); data.add("first message"); data.add("second message"); data.add("third message"); producers.send(data); jafka mq 19
  • 20. Simple usage - consumer public static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); } jafka mq 20
  • 21. Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { Client A System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { Client A System.out.println(message + " ==> " + new Date()); } }); jafka mq 21
  • 22. Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); Client A } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() { public void onMessage(String message) { Client B System.out.println("[震子] ==> " + message); } }); jafka mq 22
  • 23. Full Producer code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("serializer.class", StringEncoder.class.getName()); // ProducerConfig config = new ProducerConfig(props); Producer<String, String> producer = new Producer<String, String>(config); // StringProducerData data = new StringProducerData("demo"); for(int i=0;i<100;i++) { data.add("Hello world #"+i); } try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms"); } finally { producer.close(); } jafka mq 23
  • 24. Full Consumer Code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("groupid", "test_group"); // ConsumerConfig consumerConfig = new ConsumerConfig(props); ConsumerConnector connector = Consumer.create(consumerConfig); // Map<String, List<MessageStream<String>>> topicMessageStreams = connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder()); List<MessageStream<String>> streams = topicMessageStreams.get("demo"); // ExecutorService executor = Executors.newFixedThreadPool(2); final AtomicInteger count = new AtomicInteger(); for (final MessageStream<String> stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } } }); } jafka mq 24
  • 25. More Documents ● https://github.com/adyliu/jafka/wiki jafka mq 25
  • 26. Open source First Java open source in maven repository at groupId com.sohu. jafka mq 26
  • 27. About me ● Ady Liu ● Email: imxylz@gmail.com ● Architech@SOHU.COM jafka mq 27