1.
/ @laclefyoshi / ysaeki@r.recruit.co.jp

2.
•
•
•
•
•
•
2

3.
• 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (DEIM 2014)
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• Kinesis Family (R-JAWS Bigdata; 2016)
3

4.
5

5.
• Queue Queueing system
•
• Message
• 1
• Broker
•
• Producer
•
• Consumer
•
• Protocol
•
7

6.
•
•
•
8

7.
9
Apache ActiveMQ
Apache Kafka
RabbitMQ
NATS
NSQ
Redis
ZeroMQ
Nanomsg

8.
Apache Kafka
A distributed streaming platform
10

9.
Apache ActiveMQ
The most popular and powerful open source
messaging and Integration Patterns server
11

10.
RabbitMQ
Robust messaging for applications
Easy to use
12

11.
NATS
For cloud native applications, IoT messaging,
and microservices architectures
13

12.
NSQ
A realtime distributed messaging platform
14

13.
Redis
In-memory data structure store,
used as database, cache and message broker
15

14.
ZeroMQ
An embeddable networking library
but acts like a concurrency framework
16

15.
Nanomsg
A socket library that provides
several common communication patterns
17

16.
19

17.
20

18.
:
•
•
• Lambda Architecture
:
•
•
21

19.
22

20.
:
•
•
:
•
23

21.
24

22.
:
•
•
:
•
25

23.
27

24.
: ……
https://cloud.google.com/blog/big-data/2016/09/
apache-kafka-for-gcp-users-connectors-for-pubsub-dataflow-and-bigquery
28
Pub/Sub
Dataflow/Apache Beam
BigQuery
Apache Kafka

25.
29

26.
30
:
•
:
•

27.
31

28.
32
:
•
:
•

29.
2016/10/25
official / 3rd party
33

30.
• JIRA Github
• Apache Flink Apache Spark
Apache Bahir (3rd party)
• Apache Kafka
34

31.
1000
• [ 1] → [ 1 → 1]
• AWS: EC2 /t2.medium x 2 ( VPC)
• github:laclefyoshi/queues-benchmark
• Docker
• Python
36

32.
1000
37
•
•
•
• 1000 Byte 100000 Byte
1000 Byte
• 1000

33.
38

34.
40

35.
41

36.
Queue makes everyone happy!!