After all the attention databases have been getting over the last years, it is high time to give some thought to queues. We will kick off with some considerations why you need queues in distributed systems and what their limitations are — in particular the at-least-once and at-most-once decision. Next we discuss our specific use case and why * we started off with ActiveMQ, * it's working ok for us, * we are looking for a better solution. While looking for a better solution, we considered Amazon SQS and RabbitMQ, but finally selected Hazelcast — which seemed to do everything for us. After the initial phase of enchantment, we came to realize that Hazelcast is actually not the right tool for us and why we do not want to fully rely on it. Luckily, Disque has just been released and looks really promising. And we have already started migrating to it, even though it's currently marked as alpha code.

1. Ataleofqueuesfrom ActiveMQ over Hazelcast to Disque
Philipp Krenn @xeraa

2. Vienna

3. Vienna

4. Vienna

5. ViennaDB
Papers We Love Vienna

6. Electronic Data Interchange EDI

8. Automotive
Retail

9. Transmission&messageflow
AS2, X.400, OFTP2, FTP, SMTP, IMAP,
web services,...
Documentformat
EDIFACT, X12, ODETTE, VDA,...

12. Architecture
+--------+
| | +----+
| +--------+ | |
| | | | | |
| | +--------+ v +
| | | | | | +------------> Processing
+-|-|----+ | | +
| | | | |
+-|------+ | |
| | <--------------+
+--------+
Connectors

14. Flow
Camel component / protocol implementation
+ ^
| |
| |
v +
Camel routing
+ ^
| |
| |
v +
Queue
+ ^
| |
| |
v +
Processing

15. Queues
Decoupling producer and consumer

16. Networkedmessagequeueslike
ActiveMQ,RabbitMQ,ZeroMQ,anda
hostofotherJavainspiredsoftware
tumorsarecrutchesofsystemsdesign.
— Ted Dziuba, http://widgetsandshit.com/teddziuba/2011/02/the-case-
against-queues.html

17. "TheBlockingConsumer"

18. "CollectingDatafor
OfflineProcessing"

19. "EverybodyLovesSystem
Complexity"

20. Thereareonlytwohardproblemsin
distributedsystems:
2.Exactly-oncedelivery
1.Guaranteedorderofmessages
2.Exactly-oncedelivery
— Mathias Verraes, https://twitter.com/mathiasverraes/status/
632260618599403520

21. Order
It works like it would on a single
machine
http://book.mixu.net/distsys/time.html

22. Exactlyonce

23. Removal/ACK
Before processing: at-most-once
A!er processing: at-least-once

24. At-most-once
Requires strict consistency with
communication overhead
Replicated messages and liveness
Network split — majority

25. At-least-once
Easier to distribute
Idempotent consumption
Timeout tuning

26. http://antirez.com/news/78
antirez 430 days ago

27. Cheating
Order

28. Cheating
1. Receive
2a. Persist in RDS (MySQL)
2b. Camel routing + queue
3. Acknowledge

29. Goals
Reliability
Liveness
At-most-once

30. Machines
Wait
Retry

31. AmazonSimpleQueue
Service(SQS)
Fast, reliable, scalable, and fully
managed

33. But...
At-least-once
64KB (now 256KB) payload

35. MasterSlave
Shared file system
JDBC
Replicated LevelDB
+ ZooKeeper

36. <bean id="mysql-ds" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.mysql.jdbc.Driver"/>
<property name="jdbcUrl" value="jdbc:mysql://{{rds_hostname}}:
3306/{{rds.activemq.database}}?relaxAutoCommit=true"/>
<property name="user" value="{{rds.activemq.user}}"/>
<property name="password" value="{{rds.activemq.pass}}"/>
<property name="acquireRetryAttempts" value="100"/>
<property name="testConnectionOnCheckout" value="true"/>
<property name="testConnectionOnCheckin" value="false"/>
<property name="idleConnectionTestPeriod" value="3600"/>
<property name="maxStatements" value="0"/>
<property name="minPoolSize" value="2"/>
<property name="maxPoolSize" value="20"/>
<property name="maxIdleTimeExcessConnections" value="600"/>
<property name="numHelperThreads" value="6"/>
<property name="acquireRetryDelay" value="5000"/>
<property name="maxConnectionAge" value="10800"/>
<property name="preferredTestQuery" value="SELECT 1;"/>
</bean>

38. RDSdown!

41. In-memorydatagridSet, list, map, queue, topic, lock,
atomic long,...
Query, aggregate, MapReduce
Hibernate 2nd level cache, session
replication

42. Eierlegende
Wollmilchsau

43. "egg-layingwool-
milk-sow"

44. Great
Does everything for Java apps
Multi-master HA
Fast

45. Badfeeling
Very few queue users
Bugs with IP / DNS changes
Painful upgrades

46. Currentlyouractionistocreate
bugforthesescenariosandtryto
ensureexactly-once.
— Enes Akar, https://groups.google.com/forum/#!msg/hazelcast/
u_KLHVnvT_U/Qx5Km8COk_oJ

47. Kafka
Real-time focus
At-least-once
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent
+Producer

48. Kestrel
Dead !

49. RabbitMQ
Too much middleware

50. Intheeventofnetworkfailure
(oranodecrashing),messages
canbeduplicated,andconsumers
mustbepreparedtohandlethem.
— https://www.rabbitmq.com/reliability.html

52. [...]noonetrytouseNRedis
independentnodesandthe
offeredprimitivesasabuilding
blockforadistributedsystem[...]
— Salvatore Sanfilippo, http://antirez.com/news/78

53. Disque,anin-memory,
distributedjobqueue
— https://github.com/antirez/disque

54. Great
Documentation
Does one thing
Multi-master HA
Configurable ACK
Similar to Redis

55. Driver:Spinach
"A scalable Java Disque client" based
on lettuce, "A scalable Java Redis
client"

56. !!
Some teething problems

57. CurrentconfigurationPer message
Replication factor: 1
Retry time: 0
No persistence

58. Currentuse
Billing since 07/2015
Full ActiveMQ replacement soon?

59. WARNING:ThisisalphacodeNOT
suitableforproduction.
— https://github.com/antirez/disque

60. Nobodylikesqueues

61. Thankyou!
Questions?!
@xeraa

62. ImageCredit
· Schnitzel https://flic.kr/p/9m27wm
· Architecture https://flic.kr/p/6dwCAe
· Conchita https://flic.kr/p/nBqSHT
· Paper https://flic.kr/p/83thLf
· Airli! https://commons.wikimedia.org/wiki/File:C-54landingattemplehof.jpg
· Beer https://flic.kr/p/aBSmtY
· Gum https://flic.kr/p/ALQ3b
· Eierlegende Wollmilchsau https://flic.kr/p/GzQTT
· British https://flic.kr/p/hzxLvw