Jafka guide

7,497 views

Published on

A fast Message-Queue base on zookeeper.

Jafka mq is a distributed publish-subscribe messaging system cloning from Apache Kafka.

So it has following features:

(1)Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.

(2)High-throughput: even with very modest hardware single broker can support hundreds of thousands of messages per second.

(3)Explicit support for partitioning messages over broker servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.

(4)Simple message format for many language clients.

Published in: Technology, Education
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,497
On SlideShare
0
From Embeds
0
Number of Embeds
4,289
Actions
Shares
0
Downloads
76
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Jafka guide

  1. 1. JAFKAA fast MQ
  2. 2. overview● https://github.com/adyliu/jafka● 271KB single jar● 3.5MB with all dependencies and configurations● So simple, so fast jafka mq 2
  3. 3. Maven Usage<dependency> <groupId>com.sohu.jafka</groupId> <artifactId>jafka</artifactId> <version>1.0-rc7</version></dependency> jafka mq 3
  4. 4. Dependencies● Zkclient – Zookeeper● Log4j● Jackson● Optional dependencies: – Mx4j-tools – jopt-simple jafka mq 4
  5. 5. Future● A full kafka clone (http://incubator.apache.org/kafka)● Persisten messaging with O(1)● Constant time performance even with TB messages● High-throughput (100k/s)● Distributed messaging system – producers/brokers/consumers● Auto load balancing● Simple message format jafka mq 5
  6. 6. Message format (1) jafka mq 6
  7. 7. Message format (2) jafka mq 7
  8. 8. Log Implementation jafka mq 8
  9. 9. Architech and workflow jafka mq 9
  10. 10. Zookeeper nodes jafka mq 10
  11. 11. Offset length limit● Long.MAX_VALUE – 9223372036854775807 → 9*1018● If one message size is 1k – Messages: 9*1015 – If producing 1 billion messages per day ● Continuing days: 9*106 ● Continuing years: 24657 jafka mq 11
  12. 12. About Filesystem● 6 7200rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes jafka mq 12http://incubator.apache.org/kafka/design.html
  13. 13. sendfile() (1)● Common data transfer 1. OS reads data from disk to pagecache 2. Application reads data from pagecache to buffer 3. Application writes data back into socket buffer 4. OS copies data from socket buffer to NIC buffer● 4 copies + 2 system_call jafka mq 13
  14. 14. sendfile() (2)● FileChannel.transferTo(long, long, WritableByteCh annel) – Zero copy * https://www.ibm.com/developerworks/linux/library/j-zerocopy/ jafka mq 14
  15. 15. Batch compression● GZIP message compression default jafka mq 15
  16. 16. Sync or Async● Both supported● Setting at connecting● Batch sending while using asynchronous sending jafka mq 16
  17. 17. Usage● Simple & Funny jafka mq 17
  18. 18. Simple usage - dependencies <dependency> <groupId>com.sohu.suc*</groupId> <artifactId>suc-mq</artifactId> </dependency> Version >= 2.0-SNAPSHOT ●* our inner project jafka mq 18
  19. 19. Simple usage - producerProducers producers = Producers.buildProducer();StringMessage data = new StringMessage("demo");data.add("first message");data.add("second message");data.add("third message");producers.send(data); jafka mq 19
  20. 20. Simple usage - consumerpublic static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } });} jafka mq 20
  21. 21. Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) {Client A System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) {Client A System.out.println(message + " ==> " + new Date()); } }); jafka mq 21
  22. 22. Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date());Client A } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() { public void onMessage(String message) {Client B System.out.println("[震子] ==> " + message); } }); jafka mq 22
  23. 23. Full Producer codeProperties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("serializer.class", StringEncoder.class.getName());//ProducerConfig config = new ProducerConfig(props);Producer<String, String> producer = new Producer<String, String>(config);//StringProducerData data = new StringProducerData("demo");for(int i=0;i<100;i++) { data.add("Hello world #"+i);}try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms");} finally { producer.close();} jafka mq 23
  24. 24. Full Consumer CodeProperties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("groupid", "test_group");//ConsumerConfig consumerConfig = new ConsumerConfig(props);ConsumerConnector connector = Consumer.create(consumerConfig);//Map<String, List<MessageStream<String>>> topicMessageStreams =connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder());List<MessageStream<String>> streams = topicMessageStreams.get("demo");//ExecutorService executor = Executors.newFixedThreadPool(2);final AtomicInteger count = new AtomicInteger();for (final MessageStream<String> stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } } });} jafka mq 24
  25. 25. More Documents● https://github.com/adyliu/jafka/wiki jafka mq 25
  26. 26. Open sourceFirst Java open source in maven repository at groupId com.sohu. jafka mq 26
  27. 27. About me● Ady Liu● Email: imxylz@gmail.com● Architech@SOHU.COM jafka mq 27

×