MongoTokyo

Rakuten,inc 　 DU 　 Architect Group Hiroaki Kubota 　｜ 2011/1/18 KVS の性能 RDBMS のインデックス更に MapReduce を併せ持つ All-in-one NoSQL

Introduction

Who am I ?

Name ：窪田　博昭 Hiroaki Kubota
Company ： Rakuten Inc.
Unit ： ACT = Development Unit Architect Group
Mail: [email_address]
Hobby ： Futsal , Golf
Recent ： My physical power has gradual declined...
twitter : crumbjp
github: crumbjp
Profile Introduction

Introduction
Mongo’s characteristic
How to take advantage of the mongo for our service
Our new system “cockatoo”
MapReduce
Structure & Performance
Performance example ( on EC2 large )
Major problems...
Indexing
STALE
Diskspace
PHP client
Closing
Agenda Introduction

Mongo’s characteristics

READ performance is extremely good !
WRITE performance is so-so,but cannot be scalable.
To READ data immediately after it is WRITTEN is bad.
Very high availability !
Under development.
Maintenance tools are poor.
Some useless operations.
Mongo’s ... / Mongo has ... / Mongo is ... Mongo’s characteristic

How to take advantages of the Mongo
for the infoseek news

Our new system “ Cockatoo”
(used to be call “Albatross”)

For instance of our page

Page structure

Layout / Components Layout Components

Generic WEB structure Internet WEB API DB Request Call APIs Retrieve data

Cockatoo structure Internet WEB API ContentsDB MongoDB ReplSet Request LayoutDB MongoDB ReplSet Call APIs Retrieve data Get components SessionDB MongoDB ReplSet Memcache Gat page layout

Cockatoo structure Internet WEB API ContentsDB MongoDB ReplSet Request LayoutDB MongoDB ReplSet Call APIs Retrieve data Get components SessionDB MongoDB ReplSet Memcache Gat page layout Mongo’s READ performance is enough to cope with WEB PV. But WRITE performance is not enough.

Cockatoo structure Internet WEB API ContentsDB MongoDB ReplSet Request LayoutDB MongoDB ReplSet Call APIs Retrieve data Get components SessionDB MongoDB ReplSet Memcache Gat page layout

Cockatoo structure Internet WEB API ContentsDB MongoDB ReplSet Request LayoutDB MongoDB ReplSet Call APIs Retrieve data Get components SessionDB MongoDB ReplSet Memcache Gat page layout Zookeeper

Cockatoo structure Internet WEB API ContentsDB MongoDB ReplSet Request LayoutDB MongoDB ReplSet Call APIs Retrieve data Get components SessionDB MongoDB ReplSet Memcache Gat page layout Zookeeper Solr

Cockatoo structure CMS Developer API servers ContentsDB MongoDB ReplSet Set page layout HTML markup & API settings LayoutDB MongoDB ReplSet Deploy API Set components Batch servers Insert Data Set static contents

CMS Layout editor

MapReduce

We have never used MapReduce as regular operation.
However, We have used it for some irreglar case.
To search the invalid articles that should be removed because of someone’s mistakes...
To analyze the number of new articles posted a day.
To analyze the updated number an article.
We get start considering to use it regularly for the social data analyzing before long ...
Our usage MapReduce

Structure & Performance

Structure
Intel(R) Xeon(R) CPU X5650 2.67GHz 1core!!
4GB memory
50 GB disk space ( iScsi )
CentOS5.5 64bit
mongodb 1.8.0
ReplicaSet 5 nodes ( + 1 Arbiter)
Oplog size 1.2GB
Average object size 1KB
We are using very poor machine (Virtual machine) !!

Structure
We’ve also researched following environments...
Virtual machine 1 core
1kb data , 6,000,000 documents
8kb data , 200,000 documents
Virtual machine 3 core
1kb data , 6,000,000 documents
8kb data , 200,000 documents
EC2 large instance
2kb data , 60,000,000 documents. ( 100GB )
Researched environment

Performance
1~8 kb documents + 1 unique index
C = Number of CPU cores (Xeon 2.67 GHz)
DD = Score of ‘dd’ command (byte/sec)
S = Document size (byte)
GET qps = 4500 × C
SET(fsync) bytes/s = 0.05×DD ÷ S
SET(nsync) qps = 4500 BUT...
have chance of STALE
I found the formula for making a rough estimation of QPS

Performance example (on EC2 large)

Performance example (on EC2 large) Data-type { shop: 'someone', item: 'something', description: 'item explanation sentences...‘ }
EC2 large instance
2kb data , 60,000,000 documents. ( 100GB )
1 unique index
Environment and amount of data

Performance example (on EC2 large) Batch insert (1000 documents) fsync=true 17906 sec (=289 min) (=3358 docs/sec) Ensure index (background=false)
4049 sec (=67min)
primary 2101 sec (=35min)
secondary 1948 sec (=32min)

Performance example (on EC2 large) Add one node
5833sec (=97min)
Get files 2GB×48 2120 sec (=35min)
_id indexing 1406 sec (=23min)
uniq indexing 2251 sec (=38min)
other processes 56 sec (=1 min)

Performance example (on EC2 large) Group by
Reduce by unique index & map & reduce
368 msec
db.data.group({ key: { shop: 1}, cond: { shop: 'someone' }, reduce: function ( o , p ) { p.sum++; }, initial: { sum: 0 } });

Performance example (on EC2 large) MapReduce
Scan all data 3116sec (=52min)
number of key = 39092
db.data.mapReduce( function (){ emit ( this. shop,1); }, function (k,v){ var ret=0; v.forEach( function (value){ ret+=value; }); return ret; }, { query: {}, inline: 1, out: 'Tmp' } );

Major problems...

Indexing

Index probrem
Indexing is lock operation in default.
Indexing operation can run as background
on the primary. But...
It CANNOT run as background on the secondary
Moreover the all secondary’s indexing run
at the same time !!
Result in above...
All slave freezes ! orz...
Online indexisng is completely useless even if last version (2.0.2)

Present indexing ( default )

Index probrem Present indexing ( default ) Secondary Secondary Secondary Client Client Client Client Client Batch Primary save

Index probrem Present indexing ( default ) ensureIndex Primary Lock Indexing Secondary Secondary Secondary Client Client Client Client Client Batch Cannot write

Index probrem Present indexing ( default ) Cannot read !! SYNC SYNC SYNC Secondary Lock Indexing Secondary Lock Indexing Secondary Lock Indexing Client Client Client Client Client Primary Complete Batch finished

Index probrem Present indexing ( default ) Secondary Complete Secondary Complete Secondary Complete Client Client Client Client Client Primary Complete Batch

Present indexing ( background )

Index probrem Present indexing ( background ) Secondary Secondary Secondary Client Client Client Client Client Batch Primary save

Index probrem Present indexing ( background ) Primary Slowdown Indexing Secondary Secondary Secondary Client Client Client Client Client Batch ensureIndex( background ) Slow down...

Index probrem Present indexing ( background ) Cannot read !! SYNC SYNC SYNC Secondary Lock Indexing Secondary Lock Indexing Secondary Lock Indexing Client Client Client Client Client Primary Complete Batch finished

Index probrem Present indexing ( background ) Cannot read !! SYNC SYNC SYNC Secondary Lock Indexing Secondary Lock Indexing Secondary Lock Indexing Client Client Client Client Client Primary Complete Batch finished Background indexing don’t work on the secondaries

Index probrem Present indexing ( background ) Cannot read !! SYNC SYNC SYNC Secondary Lock Indexing Secondary Lock Indexing Secondary Lock Indexing Client Client Client Client Client Primary Complete Batch finished

Index probrem Present indexing ( background ) Secondary Complete Secondary Complete Secondary Complete Client Client Client Client Client Primary Complete Batch

Probable 2.1.X indexing

Index probrem
But not released formally.
So I checked out the source code up to date.
Certainlly it’ll be fixed !
Moreover it sounds like it’ll run as foreground
when slave status isn’t SECONDARY
(Does it means RECOVERING ?)
Accoding to mongodb.org this probrem will fix in 2.1.0

Index probrem Secondary Secondary Secondary Client Client Client Client Client Batch Primary save Probable 2.1.X indexing

Index probrem Probable 2.1.X indexing Secondary Secondary Secondary ensureIndex( background ) Client Client Client Client Client Batch Primary Slowdown Indexing Slow down...

Index probrem SYNC Slow down... SYNC SYNC Secondary Slowdown Indexing Secondary Slowdown Indexing Secondary Slowdown Indexing Probable 2.1.X indexing Client Client Client Client Client Primary Complete Batch finished

Index probrem Secondary Complete Secondary Complete Secondary Complete Client Client Client Client Client Primary Complete Probable 2.1.X indexing Batch

Index probrem
But I think it’s not enough.
I think it can bring failure to our system when
the all secondaries slowdown at the same time !!
So...
Background indexing 2.1.X

Ideal indexing

Index probrem Secondary Secondary Secondary Client Client Client Client Client Batch Primary save Ideal indexing

Index probrem Ideal indexing Secondary Secondary Secondary ensureIndex( background ) Client Client Client Client Client Batch Primary Slowdown Indexing Slow down...

Index probrem Ideal indexing ensureIndex Recovering Indexing Secondary Secondary Client Client Client Client Client finished Primary Complete Batch

Index probrem Ideal indexing ensureIndex Secondary Complete Recovering Indexing Secondary Client Client Client Client Client Batch Primary Complete

Index probrem Ideal indexing ensureIndex Secondary Complete Secondary Complete Recovering Indexing Client Client Client Client Client Batch Primary Complete

Index probrem Ideal indexing Secondary Complete Secondary Complete Secondary Complete Client Client Client Client Client Primary Complete Batch

Index probrem
It would be great if I can operate indexing manually
at each secondaries
But ... I easilly guess it’s difficult to apply for current Oplog

I suggest Manual indexing

Index probrem Secondary Secondary Secondary Client Client Client Client Client Batch Primary save Manual indexing

Index probrem Manual indexing Secondary Secondary Secondary Client Client Client Client Client Batch Primary Slowdown Indexing Slow down... ensureIndex( manual ,background )

Index probrem Manual indexing finished Primary Complete Batch Secondary Secondary Secondary Client Client Client Client Client

Index probrem Manual indexing finished Primary Complete Batch Secondary Secondary Secondary Client Client Client Client Client The secondaries don’t sync automatically

Index probrem Manual indexing finished Primary Complete Batch Secondary Secondary Secondary Client Client Client Client Client

Index probrem Manual indexing Recovering Indexing Secondary Secondary Client Client Client Client Client ensureIndex( manual ) Primary Complete Batch

Index probrem Manual indexing Secondary Complete Recovering Indexing Secondary Client Client Client Client Client ensureIndex( manual ) Primary Complete Batch

Index probrem Manual indexing Secondary Complete Secondary Complete Secondary Slowdown Indexing Client Client Client Client Client ensureIndex( manual,background ) Primary Complete Batch

Index probrem Manual indexing Secondary Complete Secondary Complete Secondary Slowdown Indexing Client Client Client Client Client ensureIndex( manual,background ) Primary Complete Batch It needs to support background operation Just in case,if the ReplSet has only one Secondary

Index probrem Manual indexing Secondary Complete Secondary Complete Secondary Slowdown Indexing Client Client Client Client Client ensureIndex( manual,background ) Primary Complete Batch

Index probrem Manual indexing Secondary Complete Secondary Complete Secondary Complete Client Client Client Client Client Primary Complete Batch

That’s all about Indexing problem

Struggle to control the sync

Unknown log & Out of control the ReplSet
Secondaries change status repeatedly in a moment
between Secondary and Recovering (1.8.0)
Then we found the strange line in the log...
[rsSync] replSet error RS102 too stale to catch up
We often suffered from going out of control the Secondaries...

What’s Stale ?
〈食品・飲料などが〉新鮮でない（⇔ fresh ）；
気の抜けた , 〈コーヒーが〉香りの抜けた ,
〈パンが〉ひからびた , 堅くなった ,
〈空気・臭 ( にお ) いなどが〉むっとする ,
いやな臭いのする
stale [stéil] ( レベル：社会人必須 ) powered by goo.ne.jp

What’s Stale ?
〈食品・飲料などが〉新鮮でない（⇔ fresh ）；
気の抜けた , 〈コーヒーが〉香りの抜けた ,
〈パンが〉ひからびた , 堅くなった ,
〈空気・臭 ( にお ) いなどが〉むっとする ,
いやな臭いのする
どうも非常によろしくないらしい・・・
stale [stéil] ( レベル：社会人必須 ) powered by goo.ne.jp

Mechanizm of being stale

ReplicaSet Primary Database Oplog Client Secondary Database mongod mongod Oplog

Replication (simple case)

ReplicaSet Primary Database Oplog Client Secondary Database mongod mongod Oplog

Insert & Replication 1 Primary Database Oplog Client Secondary Database Insert A A A Insert mongod mongod Oplog

Insert & Replication 1 Primary Database Oplog Client Secondary Database Insert A A A Oplog Insert A Sync

Replication (busy case)

Stale Primary Database Oplog Client Secondary Database Insert A A A Oplog Insert A mongod mongod

Insert & Replication 2 Primary Database Oplog Client Secondary Database Insert A B A Insert B Insert B A Oplog Insert A

Insert & Replication 2 Primary Database Oplog Client Secondary Database Insert A C A Insert B Insert B A C Insert C Oplog Insert A

Insert & Replication 2 Primary Database Oplog Client Secondary Database Insert A A A Update B Insert B A C Insert C Update A Oplog Insert A

Insert & Replication 2 Primary Database Oplog Client Secondary Database Insert A A B Insert B A C Insert C Update A Oplog Insert A Check Oplog

Insert & Replication 2 Primary Database Oplog Client Secondary Database Insert A A B Insert B C Insert C Update A Oplog Insert A Sync Insert B Insert C Update A A B C

Replication (more busy)

Stale Primary Database Oplog Client Secondary Database Insert A A A Oplog Insert A mongod mongod

Stale Primary Database Oplog Client Secondary Database Insert A B A Insert B Insert B A Oplog Insert A

Stale Primary Database Oplog Client Secondary Database Insert A C A Insert B Insert B A C Insert C Oplog Insert A

Stale Primary Database Oplog Client Secondary Database Insert A A A Update B Insert B A C Insert C Update A Oplog Insert A

Stale Primary Database Client Secondary Database C A Update B A C Oplog Insert A Oplog Insert A Insert B Insert C Update A Update C

Stale Primary Database Client Secondary Database D A Insert B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Insert A

Stale Primary Database Client Secondary Database A B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Check Oplog [Inset A] not found !! Insert A

Stale Primary Database Client Recovering Database A B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Check Oplog [Inset A] not found !! It cannot get infomation about [Insert B]. So cannot sync !! It’s called STALE Insert A

Stale
We can specify the oplog size as one of the command line option
Only at the first time per the dbpath
that is also specified as a command line.
Also we cannot change the oplog size
without clearing the dbpath.
Be careful !
We have to understand the importance of adjusting oplog size

Replication (Join as a new node)

InitialSync Primary Database Client A B C D Oplog Insert C Update A Update C Insert D mongod

InitialSync Primary Database Client Startup Database A B C Oplog D Oplog Insert C Update A Update C Insert D mongod mongod

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Insert C Update A Update C Insert D Get last Oplog Insert D

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Insert C Update A Update C Insert D Cloning DB Insert D A B C D

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Insert C Update A Update C Insert D Cloning DB Insert D A A B C D

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Insert C Update A Update C Insert D Cloning DB Insert D A A B C D E Insert E Insert E B

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Update A Update C Insert D Cloning DB complete Insert D A B C D B Update E Insert E Update B

InitialSync Primary Database Client Recovering Database A B C Oplog D Oplog Update C Insert D Check Oplog Insert D A B C D E Insert E Update B

InitialSync Primary Database Client Secondary Database A B C Oplog D Oplog Update C Insert D Sync Insert D E Insert E Update B Insert E Update B A B C D E

Additional infomation
Secondary will try to sync from other Secondaries
when it cannot reach the Primary or
might be stale against the Primary.
There is a bit of chance that sync problem not occured if the
secondary has old Oplog or larger Oplog space than Primary
From source code. ( I’ve never examed these... )

Sync from another secondary Primary Database Client Secondary Database A B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Insert A Secondary Database A B C D Insert B Insert C Update A Update C Insert D Insert A

Sync from another secondary Primary Database Client Secondary Database A B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Insert A Secondary Database A B C D Insert B Insert C Update A Update C Insert D Insert A Check Oplog [Inset A] not found !!

Sync from another secondary Primary Database Client Secondary Database A B A C Oplog Insert A D Insert B Insert C Update A Update C Insert D Insert A Secondary Database A B C D Insert B Insert C Update A Update C Insert D Insert A Check Oplog But found at the other secondary So it’s able to sync

Sync from the other secondary Primary Database Client Secondary A B C D Insert C Update A Update C Insert D Secondary Database A B C D Insert B Insert C Update A Update C Insert D Insert A Sync But found at the other secondary So it’s able to sync Database A B C D Insert C Update A Update C Insert D Insert B Insert A Insert B Insert A

That’s all about sync

Others...

Disk space

Disk space Data fragment into any DB files sparsely... We met the unfavorable circumstance in our DBs This circumstance appears at some of our collections around 3 months after we launched the services db.ourcol.storageSize() = 16200727264 (15GB) db.ourcol.totalSize() = 16200809184 db.ourcol.totalIndexSize() = 81920 db.outcol.dataSize() = 2032300 (2MB) What’s happen to them !!

Disk space Data fragment into any DB files sparsely... It’s seems like to be caused by the specific operation that insert , update and delete over and over. Anyway we have to shrink the using disk space regularly just like PostgreSQL’s vacume . But how to do it ?

Disk space Shrink the using disk spaces MongoDB offers some functions for this case. But couldn’t use in our case ! repairdatabase : Only runable on the Primary. It needs long time and BLOCK all operations !! compact : Only runable on the Secondary. Zero-fill the blank space instead of shrink disk spaces. So cannot shrink...

Disk space Our measurements
For temporary collection:
To issue drop-command regularly.
For other collections:
Get rid of one secondary from the ReplSet.
Shut down this.
Remove all DB files.
Join to the ReplSet.
Do these operations one after another.
Step down the Primary. (Change Primary node)
At last, do 1 – 4 operations on prior Primary.

Disk space Shrink operation Primary Bloated Secondary Bloated Secondary Bloated

Disk space Shrink operation Primary Bloated Dead Bloated Secondary Bloated shutdown mongod (kill -15)

Disk space Shrink operation Primary Bloated Dead Nothing Secondary Bloated delete DBPATH

Disk space Shrink operation Primary Bloated Startup Nothing Secondary Bloated start mongod

Disk space Shrink operation Primary Bloated Secondary Shrinked Secondary Bloated

Disk space Shrink operation Primary Bloated Secondary Shrinked Secondary Shrinked shutdown mongod delete DBPATH startup mongod

Disk space Shrink operation Secondary Bloated Primary Shrinked Secondary Shrinked step down

Disk space Shrink operation Primary Shrinked Secondary Shrinked Secondary Shrinked shutdown mongod delete DBPATH startup mongod

PHP client

PHP client We tried 1.1.4 and 1.2.2
1.1.4:
There is some critical bugs around connection pool.
We struggled to invalidate the broken connection.
I think, you should use 1.2.X instead of 1.1.X
1.2.2:
It seems like to be fixed around connection pool.
But there are 2 critical bugs !
Socket handle leak
Useless sleep
However, This version is relatively stable
as long as to fix these bugs

PHP client Patches https://github.com/crumbjp/Personal - mongo1.2.2.non-wait.patch - mongo1.2.2.sock-leak.patch

PHP client

Closing

Closing What’s MongoDB ? It has very good READ performance. We can use mongo instead of memcached. if we can allow the limited write performance. Die hard ! MongoDB have high availability even if under a severe stress.. Can use easilly without deep consideration We can manage to do anything after getting start to use. Let’s forget any awkward trivial things that have bothered us. How to treat the huge data ? How to put in the cache system ? How to keep the availablity ? And so on ....

Closing Keep in mind Sharding is challenging... It’s last resort ! It’s hard to operate. In particular, to maintain config-servers. [Mongos] is also difficult to keep alive. I want the way to failover Mongos. Mongo is able to run on the poor environment but... You should ONLY put aside the large diskspace Huge write is sensitive Adjust the oplog size carefully Indexing function has been unfinished Cannot apply index online

All right, Have fun !!

All right, Have fun !!
...with us at Rakuten

Thank you for your listening

http://d.hatena.ne.jp	884
http://www.10gen.com	200
http://f-kei.blogspot.com	75
http://blog.fkei.me	10
http://a0.twimg.com	9
http://webcache.googleusercontent.com	2
http://www.twylah.com	2
http://download.mongodb.org	2
http://www.alerts.mongodb.org	1
http://twitter.com	1
http://sxr.mongodb.org	1

MongoTokyo

by Hiroaki Kubota on Jan 18, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

11 Embeds 1,187

Statistics

MongoTokyo — Presentation Transcript