This document summarizes Cassandra clusters at Yahoo Japan. It discusses four main issues: 1) the C10k problem solved using a Cassandra proxy, 2) bootstrap problems solved by improving driver connection behavior, 3) slow queries in multi-tenant clusters addressed using detection tools, and 4) streaming traffic addressed using new throughput parameters. It also discusses trials of special hypervisors and bare metal clusters to optimize Cassandra performance on OpenStack infrastructure.

1. Cassandra @

2. Satoshi Konno
http://www.cybergarage.org
• Engineering Manager of NoSQL Team @ Yahoo! Japan
• Open Source Software Developer for
Virtual Reality, IoT and Cloud Computing
• Doctor's Course Student @ JAIST
Défago Lab : The φ accrual failure detector
About me
2

3. Agenda
• Company Profile
• Summary of C* Clusters
• Issues and Solutions of C*
• Next Generation Infrastructures for C*

4. Company Profile
4

5. Founded : January 31, 1996
Businesses : Internet Advertising
e-Commerce
Members Services, etc.
Web Services : 100+
Smartphone Apps: 50+ (iOS), 50+ (Android)
Employees : 5,800+ (as of June 30, 2016)
Head Office : Chiyoda-ku, Tokyo, Japan
Company Profile
5

6. Shareholder Composition
6
An independent and public company in the Japanese Market
U.S. Japan
35.5
%
42.9
%
Market Cap
$22 billion
Market Cap
$29 billion
Market Cap
$60 billion

7. 18th Largest Internet Company in market cap
7
0
100
200
300
400
500
600
bilion U.S. dollars
http://www.statista.com/statistics/277483/market-value-of-the-largest-internet-companies-worldwide/

8. 19years
16
17
18
Revenue ¥652B, Operating Income ¥171B (FY2015)
Continued Growth Sustained

9. 60%
Consumer
32%
%
Others
8 %
Marketing Solutions
Revenue Portfolio
(FY2015)

10. Extensive Reach to a Wide Range of Users
10
80%
80% of all Japanese Internet users use Yahoo! JAPAN
Nielsen NetView June 2015 : Data by Brands. Access from home and work using PCs (excl. internet applications)

11. Many Strong Services
11
Media
US
Search Video Answer Mail
JP
US
JP
Membership C2C Payment C2C EC B2C EC Local
Search Knowledge search MailNews
YAHUOKU!Premium Wallet Loco

12. Summary of C* Clusters
12

13. Yahoo! JAPAN Database Platforms
13
300+
Systems
NoSQL
Team
100+
Services

14. OSS Database Platforms
14
300+
Systems
180
Systems
MySQL 630
DBs
100
Systems
Cassandra 130
DBs
30
70
60
40
Yahoo Japan
NoSQL
Team
RDB
Team

15. Cassandra @ Yahoo! JAPAN
15
2010 2012 2014 2016
Service
Departments
Our
Team
0.5 0.8 1.x
0.8 1.x 2.x 3.x
NoSQL
Team

16. Our Cassandra Clusters
16
30
Clusters
30TB
Usages
1000+
Nodes
300,000
Read/sec
100,000
Write/sec
2016
10
Nodes /
Cluster
160
Nodes /
Cluster
…
1
Shared
Cluster
30
Special
Clusters
30
Systems
50
Systems
3
DCs

17. Our Use Case Summary on Cassandra
17
100
Systems
20
Database Caching
10
Advertising Services
40
User Databases
50
Service Databases
Browsing History
Impression Data
・・・・
Meta Data
Aggregated Data
・・・・
Generated Data
Session Data
Meta Data
Aggregated Data
・・・・
Generated Data
Recommendation
Demographic Data
Life Log
・・・・
Preference Data
Behavior History

18. Our Issues and Solutions
18

19. ISSUE #1 : C10k Problem – C* Proxy
19
PC + Tablet
3.36B PV
Smart Device
3.45B PV
6.8 Billion PV /month

20. ISSUE #1 : C10k Problem – C* Proxy
20
Yahoo Japan Services
..........
10 〜 200 Front-end Servers / Service
PHOTO：AFLO

21. ISSUE #1 : C10k Problem – C* Proxy
• PROBLEM : 200 front-end servers * 128 processes
* 2 (C* request + C* heart beat)
=51,200 connections / node
21PHOTO：AFLO
200 Front-end Servers
128 processes
51,200 connections !

22. ISSUE #1 : C10k Problem – C* Proxy
• PROBLEM : 200 front-end servers * 128 processes
* 2 (C* request + C* heart beat)
=51,200 connections / node
22PHOTO：AFLO

23. ISSUE #1 : C10k Problem – C* Proxy
• PROBLEM : 200 front-end servers * 128 processes
* 2 (C* request + C* heart beat)
=51,200 connections / node
23
Process down
PHOTO：AFLO

24. ISSUE #1 : C10k Problem – C* Proxy
• SOLUTION : 200 front-end servers * 128 processes
* 1 proxy * 2 (C* request + C* heart beat)
=400 connections / node
24
200 front-end servers
1 proxy
400 connections !
128 processes
PHOTO：AFLO

25. ISSUE #2 : Boostrap Problem - Driver
• Heavy Services : ↑3000qps/node
= C* cluster with real servers (SSD is recommended)
• Light Services : ↓1000qps/node and ↓3GB/node
= C * cluster with virtual servers on OpenStack
25
Heavy Service Light Service
CPU = Good
vCPU = Cheap

26. ISSUE #2 : Boostrap Problem - Driver
• PROBLEM : All processes in each front-end server
tries to connect a new C* node which is added into
the cluster at the same time ...
26
..........
! ! !
! ! !
vCPU = Cheap
PHOTO：AFLO

27. ISSUE #2 : Boostrap Problem - Driver
• PROBLEM : The authentication of C* based on
BCrypt is heavy processing for the vCPU nodes.
27
..........
!
vCPU : Authentication (BCrypt) is heavy !
! !
! ! !
PHOTO：AFLO

28. ISSUE #2 : Boostrap Problem - Driver
• PROBLEM : Most processes can not connect to C*
clusters on OpenStack due to the authentication
processing, and the processes will timeout and
repeat to connect without waiting endlessly …
28
All vCPU Usages = 100% !
PHOTO：AFLO
vCPU : Authentication (BCrypt) is heavy !
Timeout ! Retry !

29. ISSUE #2 : Boostrap Problem - Driver
• SOLUTION : Improving the C* drivers not to connect
simultaneously when the connection is failed.
29
..........
!! !
! ! !
PHOTO：AFLO

30. ISSUE #3 : Multi-tenancy – Slow Query
• Small Services : (↓500qps and ↓10GB) / keyspace
= Shared C* cluster with real servers
30
Shared
Cluster
50
Services

31. ISSUE #3 : Multi-tenancy – Slow Query
• PROBLEM : Couldn’t find the causal service of the
high loading queries in the multi-tenancy cluster.
31
Shared
Cluster Which
services ?
QUERY
QUERY
PHOTO：AFLO

32. ISSUE #3 : Multi-tenancy – Slow Query
• SOLUTION : CASSANDRA-12403 - Slow query
detecting
32
Shared
Cluster
Service Remove
Special
Cluster
QUERY
PHOTO：AFLO
Slow Query !

33. ISSUE #4 : Multi-racking – Inbound Params
• PROBLEM : Our C* clusters are build with other
services in a same rack or under a same core switch.
33PHOTO：AFLO

34. ISSUE #4 : Multi-racking – Inbound Params
• PROBLEM : C* Streaming occurs when the node is
added or remove by the our operation or the failure
detection.
34
Streaming
PHOTO：AFLO

35. ISSUE #4 : Multi-racking – Inbound Params
• PROBLEM : The streaming of C* rises a heavy traffic,
and it troubles the other services.
35
Streaming
Streaming
Streaming
Stop C*
streaming !
PHOTO：AFLO
stream_throughput_outbound
stream_throughput_outbound
stream_throughput_outbound

36. ISSUE #4 : Multi-racking – Inbound Params
• SOLUTION : CASSANDRA-11303 - New inbound
throughput parameters for streaming
36
Streaming
Streaming
Streaming
PHOTO：AFLO
stream_throughput_outbound
stream_throughput_outbound
stream_throughput_outbound
stream_throughput_inbound
stream_throughput_inbound
stream_throughput_inbound

37. Next Generation
Infrastructures for C*
37

38. • PURPOSE : To abstract our data center resources
using OpenStack.
Apps
Platforms
Infrastructures
APIAPI
API API API API
OpenStack @ Yahoo! JAPAN
38
50,000+
instances

39. Trial #1 : Special Hypervisor for C*
• PROBLEM : Our hypervisors of OpenStack has C*
and other service VMs.
39
Noisy
Neighbours

40. Trial #1 : Special Hypervisor for C*
• SOLUTION : Trying to offer the special hypervisors
which runs only C* VMs.
40
vCPU : 8+, Mem : 16GiB+
SSD : 100GiB+
Optimal
Flavors for C*
10Gbps x 2

41. TRIAL#2 : Bare Metal Clusters for C*
• PROBLEM : vCPU of OpenStack is cheap to run a C*
node in our special service environment such as the
many connections.
41
vCPU : Authentication (BCrypt) is heavy !

42. TRIAL #2 : Bare Metal Clusters for C*
• SOLUTION : Trying to offer the special bare metal
clusters which runs only C* using OpenStack Ironic.
42
Ironic
Xeon D-1541 2.1GHz (1CPU)
32GBMEM / SATA SSD 400GB
10Gbps x 2