Jafka guide

  • 5,338 views
Uploaded on

A fast Message-Queue base on zookeeper. …

A fast Message-Queue base on zookeeper.

Jafka mq is a distributed publish-subscribe messaging system cloning from Apache Kafka.

So it has following features:

(1)Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.

(2)High-throughput: even with very modest hardware single broker can support hundreds of thousands of messages per second.

(3)Explicit support for partitioning messages over broker servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.

(4)Simple message format for many language clients.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,338
On Slideshare
0
From Embeds
0
Number of Embeds
15

Actions

Shares
Downloads
59
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. JAFKAA fast MQ
  • 2. overview● https://github.com/adyliu/jafka● 271KB single jar● 3.5MB with all dependencies and configurations● So simple, so fast jafka mq 2
  • 3. Maven Usage<dependency> <groupId>com.sohu.jafka</groupId> <artifactId>jafka</artifactId> <version>1.0-rc7</version></dependency> jafka mq 3
  • 4. Dependencies● Zkclient – Zookeeper● Log4j● Jackson● Optional dependencies: – Mx4j-tools – jopt-simple jafka mq 4
  • 5. Future● A full kafka clone (http://incubator.apache.org/kafka)● Persisten messaging with O(1)● Constant time performance even with TB messages● High-throughput (100k/s)● Distributed messaging system – producers/brokers/consumers● Auto load balancing● Simple message format jafka mq 5
  • 6. Message format (1) jafka mq 6
  • 7. Message format (2) jafka mq 7
  • 8. Log Implementation jafka mq 8
  • 9. Architech and workflow jafka mq 9
  • 10. Zookeeper nodes jafka mq 10
  • 11. Offset length limit● Long.MAX_VALUE – 9223372036854775807 → 9*1018● If one message size is 1k – Messages: 9*1015 – If producing 1 billion messages per day ● Continuing days: 9*106 ● Continuing years: 24657 jafka mq 11
  • 12. About Filesystem● 6 7200rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes jafka mq 12http://incubator.apache.org/kafka/design.html
  • 13. sendfile() (1)● Common data transfer 1. OS reads data from disk to pagecache 2. Application reads data from pagecache to buffer 3. Application writes data back into socket buffer 4. OS copies data from socket buffer to NIC buffer● 4 copies + 2 system_call jafka mq 13
  • 14. sendfile() (2)● FileChannel.transferTo(long, long, WritableByteCh annel) – Zero copy * https://www.ibm.com/developerworks/linux/library/j-zerocopy/ jafka mq 14
  • 15. Batch compression● GZIP message compression default jafka mq 15
  • 16. Sync or Async● Both supported● Setting at connecting● Batch sending while using asynchronous sending jafka mq 16
  • 17. Usage● Simple & Funny jafka mq 17
  • 18. Simple usage - dependencies <dependency> <groupId>com.sohu.suc*</groupId> <artifactId>suc-mq</artifactId> </dependency> Version >= 2.0-SNAPSHOT ●* our inner project jafka mq 18
  • 19. Simple usage - producerProducers producers = Producers.buildProducer();StringMessage data = new StringMessage("demo");data.add("first message");data.add("second message");data.add("third message");producers.send(data); jafka mq 19
  • 20. Simple usage - consumerpublic static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } });} jafka mq 20
  • 21. Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) {Client A System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) {Client A System.out.println(message + " ==> " + new Date()); } }); jafka mq 21
  • 22. Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date());Client A } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() { public void onMessage(String message) {Client B System.out.println("[震子] ==> " + message); } }); jafka mq 22
  • 23. Full Producer codeProperties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("serializer.class", StringEncoder.class.getName());//ProducerConfig config = new ProducerConfig(props);Producer<String, String> producer = new Producer<String, String>(config);//StringProducerData data = new StringProducerData("demo");for(int i=0;i<100;i++) { data.add("Hello world #"+i);}try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms");} finally { producer.close();} jafka mq 23
  • 24. Full Consumer CodeProperties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("groupid", "test_group");//ConsumerConfig consumerConfig = new ConsumerConfig(props);ConsumerConnector connector = Consumer.create(consumerConfig);//Map<String, List<MessageStream<String>>> topicMessageStreams =connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder());List<MessageStream<String>> streams = topicMessageStreams.get("demo");//ExecutorService executor = Executors.newFixedThreadPool(2);final AtomicInteger count = new AtomicInteger();for (final MessageStream<String> stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } } });} jafka mq 24
  • 25. More Documents● https://github.com/adyliu/jafka/wiki jafka mq 25
  • 26. Open sourceFirst Java open source in maven repository at groupId com.sohu. jafka mq 26
  • 27. About me● Ady Liu● Email: imxylz@gmail.com● Architech@SOHU.COM jafka mq 27