The document discusses ZeroMQ for publishing and subscribing to messages, ZooKeeper for leader election to ensure only one publisher, and FlatBuffers for fast serialization. It provides an overview of each technology, sample code, and compares them to similar messaging, coordination, and serialization systems. The presentation includes a weather application demo that uses these three components together, with ZeroMQ for messaging, ZooKeeper for leader election of the publisher, and FlatBuffers for serialization.

1. ZeroMQ, ZooKeeper &
FlatBuffers
Ravi Okade @ Code Camp NYC

3. Agenda:
ZeroMQ Pub-Sub
ZooKeeper Leader Election
FlatBuffers Serialization
Download sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing

4. Demo Weather Application
- Publishes weather info via ZeroMQ
- Publisher is a Console app. WPF and Console Subscribers.
- Starts with a simple string serialization
- Then switch to FlatBuffers for serialization
- Then add ZooKeeper:
- Publishers use ZK to ensure only one publisher at a time
- Subscribers listen to ZK for any change in the publisher and re-initialize ZeroMQ
- All code runs on a single machine for the demo

5. ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromq
https://www.infoq.com/author/Pieter-Hintjens
http://zguide.zeromq.org/page:all#Why-We-Needed-ZeroMQ

6. I am confused with all these products ending with MQ!
- ZeroMQ, ActiveMQ, RabbitMQ, QPID, Kafka..
Some good references:
- https://www.predic8.com/activemq-hornetq-rabbitmq-apollo-qpid-
comparison.htm
- http://bipinkunjumon.blogspot.com/2013/05/zeromq-and-activemq-
comparison.html
- https://tomasz.janczuk.org/2015/09/from-kafka-to-zeromq-for-log-
aggregation.html

7. ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromq

8. Publisher (publishes weather data by zipcode)

9. Subscriber (subscribes to weather data by zipcode)

10. Features
- Handles slow subscriber
- Resilient to publisher aborts
- Multiple language support (.NET, Java, Python..)
- Subscription filters
- Fast!

11. NetMQ https://github.com/zeromq/netmq
- .NET port of zeromq

12. FlatBuffers Intro
https://google.github.io/flatbuffers/
https://google.github.io/flatbuffers/md__internals.html

13. FlatBuffers
- Create Schema:
namespace codecamp;
table Weather
{
zipCode:string;
temperature:int;
humidity:int;
}
root_type Weather;
- Compile:
flatc -n weather.schema

14. Compare FlatBuffers with:
http://real-logic.github.io/simple-binary-encodinghttps://capnproto.org/

15. ZooKeeper
ZooKeeper is a distributed, open-
source coordination service for
distributed applications.
ZooKeeper is very popular for leader
election implementation and is widely
used in the Hadoop eco-system.

16. What else is ZooKeeper used for?
- Service discovery
- Leader election (this demo)
- Configuration repository, group membership
- Barriers
- Queue, Priority Queue
- Locks
- Two phase commit

17. ZooKeeper Leader Election
Publishers
Leader
ZooKeeper
Service
Subscribers
Co-ordinating
who is the leader
Notifying when
leader changes

18. ZooKeeper leader election strategy
- Like a DMV token system
- Zookeeper assigns an id to each publisher
- The process with lowest id is the leader
- If any process dies, Zookeeper sends a notification; the process with next lowest
id becomes the leader
- Note - ZooKeeper is an ordered system and guarantees consistency; hence each id
is assigned only once

19. ZooKeeper is also used for:
- Service Discovery
- Distributed Key-value store (database)
- Configuration store
- Many more: Barriers/Locks, Queues, 2phase Commit:
https://zookeeper.apache.org/doc/trunk/recipes.html

20. Compare ZooKeeper with:
https://www.consul.io/
https://coreos.com/etcd/
Apache Curator - a framework over Zookeeper
http://curator.apache.org/
Eureka (by Netflix)
https://github.com/Netflix/eurek
a/wiki/

21. References
https://github.com/ewhauser/zookeeper
https://github.com/zeromq/clrzmq (.NET Client for 0MQ)
https://github.com/zeromq/clrzmq4 (newer .NET Client for 0MQ)
Download sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing