Presentation at MongoDB Tokyo 2012

Real examples of some services(using MongoDB )@just_do_neet 1

Today’s Agenda•Position Talk (5 minutes.)•Introduction of examples of using MongoDB (10 minutes x 3 .) - Photo-sharing app. - Messenger app.(Veriﬁcation only) - Analytics of web service•Conclusion (1 minutes.)•Q&A (remaining times.) 2

Position Talk 3

I think ‘MongoDB’ is.... MongoDB•Very useful storage. - The advantages of ‘Document oriented storage’ ‣ Storing complex data. ‣ Storing log file. - Have very powerful search features.$ db.musicians.find({person.first_name : ‘masashi’}).order({person.age : -1}).limit(10).forEach(..... 4

I think ‘MongoDB’ is.... MongoDB•But, not ﬁt for ‘Big-Data’ operations. - Huge data size, behavior of sharding, etc.... - Applying MongoDB in your own service requires tremendous courage. - There are few experts on ‘Mongo’. 5

Oops.... (´ ω ) http://www.slideshare.net/matsukaz/mongo-db-8707809 6

Three types of DataBase DB MongoDB•Merriman said ... - http://www.databasejournal.com/news/mongodb- nosql.html 7

I think ‘MongoDB’ is.... MongoDB•MongoDB is not a ‘Silver Bullet’. - global rock, data compression, security, etc...•Similarly, Other DB is not a ‘Silver Bullet’.•The best way is ‘using in the right place’. 8

Practical examples of using MongoDB1st. Photo-sharing app. 9

NAVER Photo AlbumNAVER Photo Albumhttp://photoalbum.naver.jp/ 10

NAVER Photo AlbumNAVER Photo Album 11

NAVER Photo Album NAVER Photo Album This ‘timeline’ datastored to MongoDB. 12

Handling complex data•‘Timeline‘ data structure is too complex. - Many types of activity. - Many types of user relation. - Must be able to be searched for old data. 13

Activity Patterns 14

Relation Patterns - Share A→B - Shared A←C - Both Friends A D 15

Search of past activity.•If delete uploaded photo, requires deleting data on ‘timeline’(every related users). - Delete row-data is easy. - Delete column-data is painful. - Search column-data(on every rel. users timeline) →delete column-data. →update row-data. 16

Architecture on RDBRDB•Oops... 17

Architecture on simple Key-Value StoreKVS•Very simple. But cannot be search column-data. 18

MondoDB can do it. MondoDB 19

Delete photo(column-data).#lookup photodb.activity.ﬁnd({feedList.feedData.photoList, photoId})#delete photodb.activity.remove({feedList.feedId, feedId})#re-insert timeline datadb.activity.insert(newFeedData) 20

Delete friends activity(column-data).#lookup friends activitydb.activity.ﬁnd({feedList.feedData.userList.userId, userId})#delete friends activitydb.activity.remove({feedList.feedId, feedId})#re-insert timeline datadb.activity.insert(newFeedData) 21

Delete album(row-data).#lookup albumdb.activity.ﬁnd({feedList.feedData.albumList.albumId,alubmId})#delete albumdb.activity.remove({feedList.feedId, feedId}) 22

Architecture on MongoDBMongoDB•Data schema has been guitely simpliﬁed. 23

That’s perfect! 24

Lessons learned from this case•Easy to handle complex data structure. - You can store multi-tier data. - You can use ‘dot-notation’ during the search.•Data schema is also simpliﬁed. 25

System architecture• MongoDB Ver.1.8.3 , Server× mongod× (ReplicaSet)• QuadCore×4 / mem 16GB / 1TB SATA HDD 26

More details•Presentation at Tokyo’s MongoDB study group (September 2011) - http://www.slideshare.net/naverjapan/mongodb-9422893 - This slide written by Japanese. 27

Veriﬁcation examples of using MongoDB2nd. Messenger app. 28

NAVER LineNAVER Linehttp://line.naver.jp/ 29

This app is...NAVER Line•Messenger app in the iOS and Android.•You can call freely and send free message.•This app is used world wide. - East asia(Japan, Korea, Taiwan,HongKong, etc..) - Mideast(Western Asia, Northern Africa)•Downloaded for 10,000,000 times iPhone and Android. (during 6 months) 30

Our problems...•Users are increasing rapidly.•Storing data are increasing rapidly.•Server are increasing rapidly. 31

We need scalable datastore. DB 32

Things required in ‘Scalable’ datastore•High availability. - Auto sharding. - Auto replication.•Liner-scale throughputs.•Easy for extension. / Easy for operation. 33

MondoDB can do it. MondoDB 34

MondoDB can do it? ? MondoDB 35

Veriﬁcation of ‘Scalable’ datastore•We had a veriﬁcation of two types datastore. - MongoDB - HBase 36

HBaseHBase•It’s a member of Hadoop Family. http://hbase.apache.org/ 37

Strong points and weak points of MongoDBMongoDB•Strong points. - Auto sharding. - Easy to be conﬁgured.•Weak Points? - Not stable for ‘sharding’. - Hypertrophy of data.(compression not supported.) 38

Strong points and weak point of HBaseHBase•Strong points. - Reliability. - Tuning is possible for many things. (ex. compression type, memory size, etc...)•Weak Points? - Single Point of Failure (NameNode) - Setup is not easy. 39

Architecture of MongoDB MongoDB http://www.mongodb.org/display/DOCS/Introduction 40

Architecture of HBase HBase http://lunarium.info/arc/index.php/HBase%E3%81%AE%E6%A7%8B%E9%80%A0 41

Scenario of veriﬁcation•Requirements User Count 10,000,000 Active User Count 5,000,000 User Request TPS 28,000 Total Command TPS 150,000 Record size 100,000,000•Number of Servers MongoDB (3 servers is 1 replica set) x 5 = 15 Management clusters x 3 HBase Data clusters x 12 42

Scenario of veriﬁcation•System architecture of mongodb 43

Scenario of veriﬁcation•System architecture of HBase 44

Results of veriﬁcation(qps) (qps) MongoDB HBase(default) HBase(tuning)25000 qps20000 qps15000 qps10000 qps 5000 qps 0 qps insert select delete•insert/delete case. HBase is faster.•In select case. this is no difference in terms of speed. (MongoDB is a little faster.) 45

Results of veriﬁcation(network resources) ( ) HQuorumPeer mongos (Zookeeper) mongod HRegionServer 46

Results of veriﬁcation(switch apparatus) ( )•blue : HBase case red : MongoDB case•As an urgent thing, too much trafﬁc for a switch. (in the case of MongoDB) 47

Results of veriﬁcation(data size) ( ) 700000000 MongoDB 600000000 MongoDB(fragment) 500000000 HBase HBase(fragment) 400000000 HBase(snappy) 300000000 200000000 100000000 0 size(1,000,000 record)•MongoDB data size is just ‘Mongo’.(oops...)•HBase is less unsatisfactory. (compression available.) 48

Finally , we are using HBase.(on this case.) HBase•There was an uneasy point in MongoDB. - Data size - Network trafﬁc.•We hope a compression function is mounted in MongoDB. (´ ω ) 49

Lessons learned from this case•MongoDB need a lot of system resources. - Disk size / Network trafﬁc ...•If there is a millionaire of Arab, it may be able to be used. But it is impossible for us to do it. 50

Practical examples of using MongoDB3rd. Analytics of web service. Web 51

We need ‘realtime analysis’ system. 52

MondoDB,ﬂuentd,and node.js can do it. MondoDB ﬂuentd node.js 53

Introduction of ‘ﬂuentd’ﬂuentd 54

fluentd - The Event Collector Servicefluentd•‘Fluent’ product made by a Japanese famous hacker ‘Sadayuki FURUHASHI‘ (Treasure Data Inc. ) http://www.scribd.com/doc/70897187/ Fluent-event-collector-update (English) http://blog.treasure-data.com/post/ 13047440992/fluentd-the-missing-log- collector (English) http://d.hatena.ne.jp/viver/20110929/p1 (Japanese) 55

fluentd - The Event Collector Servicefluentd•Features - Notify and Collect Event Logs on real-time (like ‘syslogd’). - Sending and Receiving Structured data(like ‘JSON’) - Very useful. - Very configurable. - Very extensible. 56

fluentd - Notify and Collect Event Logs tail(file) input buffering output file http forward Fluentd forward(tcp) exec exec copy•input : receiving event logs.•buffering : buffering event logs to memory or file.•output : sending to other systems. 57

Fluent - Notify and Collect Event Logs Apps fluentd Apps fluentd Logs Storage fluentd Apps fluentd•Apps : input = apps log(tail) / output = send to Logs(http)•Logs : input = receive from Apps(http) / output = local log(file) 58

ﬂuentd - Structured Data http://www.scribd.com/doc/70897187/ Fluent-event-collector-update•format : <time>t<tag><record(JSON hash)> 59

ﬂuentd - Structured Data http://www.scribd.com/doc/70897187/ Fluent-event-collector-update•format : <time>t<tag><record(JSON hash)> 60

fluentd - Install$ gem install fluentd$ yum install fluentd$ apt-get install fluentd$ git clone http://github.com/fluent/fluentd.git$ cd fluentd$ ./autogen.sh$ ./configure$ make$ sudo make install 61

fluentd - Configuration 1 <source> 2 type tail 3 format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<re ferer>[^"]*)" "(?<agent>[^"]*)")?$/ 4 time_format %d/%b/%Y:%H:%M:%S %z 5 path /usr/local/apache2/logs/access_log 6 tag apps.access 7 </source> 8 9 <match debug.**>10 type stdout11 </match>1213 <match apps.access>14 type roundrobin15 <store>16 type tcp17 host logs118 </store>19 </match>•<source> : definition receiving data.•<match> : definition sending data. 62

fluentd - Configuration 1 <source> 2 type tail 3 format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<re ferer>[^"]*)" "(?<agent>[^"]*)")?$/ 4 time_format %d/%b/%Y:%H:%M:%S %z 5 path /usr/local/apache2/logs/access_log 6 tag apps.access 7 </source> 8 9 <match debug.**>10 type stdout11 </match>1213 <match apps.access>14 type roundrobin15 <store>16 type tcp17 host logs118 </store>19 </match>•type : definition input / output processes. 63

fluentd - Configuration 1 <source> 2 type tail 3 format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<re ferer>[^"]*)" "(?<agent>[^"]*)")?$/ 4 time_format %d/%b/%Y:%H:%M:%S %z 5 path /usr/local/apache2/logs/access_log 6 tag apps.access 7 </source> 8 9 <match debug.**>10 type stdout11 </match>1213 <match apps.access>14 type roundrobin15 <store>16 type tcp17 host logs118 </store>19 </match>•tag : definition of name of event logs. 64

fluentd - Configuration13 <match apps.access> 13 <match apps.access>14 type roundrobin 14 type roundrobin15 <store> 15 <store>16 type tcp 16 type tcp17 host logs1 17 host logs118 </store> 18 host logs219 </match> 19 </store> 20 </match> Apps Apps fluentd fluentd fluentd Logs1 Apps fluentd Logs1 Apps fluentd fluentd fluentd Logs2 Apps Apps fluentd fluentd•roundrobin : multiple outputs using round-robin algorithm. 65

fluentd - Adding plugins$ fluent-gem search -rd fluent-plugin$ fluent-gem install <plugin name>•List of available fluent plugins. http://fluentd.org/plugin/•Document for plugin developers. http://fluentd.org/doc/devel.html 66

fluentd - Adding plugins$ fluent-gem search -rd fluent-plugin$ fluent-gem install <plugin name>•List of available fluent plugins. http://fluentd.org/plugin/•Document for plugin developers. http://fluentd.org/doc/devel.html https://github.com/doryokujin/fluent-plugin-aggregation 67

fluentd - Competitorfluentd•syslogd :•flume : https://cwiki.apache.org/FLUME/•scribe : https://github.com/facebook/scribe•kafka : http://incubator.apache.org/kafka/index.html 68

fluentd - More details•Please look at this sites. http://fluentd.org/doc/ http://www.scribd.com/doc/66633067/Fluent-event- collector http://www.ustream.tv/recorded/15866352 http://twitter.com/#!/search/%23fluentd 69

Architecture of ‘real-time analysis’ systemfluentd+MongoDB+node.js classifier fluentd Hadoop jubatus web web web web apache fluentd analysis fluentd Mongo DB storing log data and result of analysis. hoge.js visualization hoge.js hoge.js node.js 70

Demos 71

Demos•Demos : - Real-time visualizing using world map for ‘NAVER MATOME’ service. <ﬂuentd + mongodb + mongotail + maptail.js(node.js) > - Real-time monitoring stats of ‘NAVER MATOME’ service. <ﬂuentd + mongodb + node.js> 72

About ‘NAVER MATOME’NAVER•‘NAVER MATOME’ is famous ‘curation-service’ at Japan. http://matome.naver.jp/ 73

Demo case 1 - real-time visualizingﬂuentd•It’s a pretty demonstration. (not launched as public service.)•Real-time visualizing access_log using world map.•ﬂuentd + mongodb + mongo-tail + node.js(maptail.js) 74

mongo-tailmongo-tail•This tool is included in ‘fluent-plugin-mongo’ https://github.com/fluent/fluent-plugin-mongo/tree/ master/bin•“You can tail mongo capped collection.” $ mongo-tail -h <host> -p <port> -d <db name> -c <collection name> -f 75

node.js node•http://nodejs.org/ 76

maptail.jsmaptail.js•https://github.com/stagas/maptail•“Creates a server, monitors a tail -f output for IP addresses, GeoIPs them and sends them to a map.” 77

Architecture of ‘real-time visualizing’ demo Apps Hadoop Clusters Apps mongod node.js Apps ﬂuentd mongo-tail socket.io real-time real-time every 1 minute 78

fluent-plugin-mongomongodb$ fluent-gem install fluent-plugin-mongo <match apps.access> type mongo database hoge collection fuga host mongo1 port 27017 </match> Apps Apps fluentd fluentd fluentd Logs1 Apps fluentd Logs1 Apps fluentd fluentd fluentd mongo1 Apps Apps fluentd fluentd 79

More details•I am writing a blog article about this demo system. (Sorry Japanese only.) http://dayafterneet.blogspot.com/2011/11/ﬂuent- mongodb-nodejs-web.html 80

Demo case 2 - real-time monitoringﬂuentd•It’s a pretty demonstration. (not launched as public service.)•Real-time monitoring (stats of service). - user counts / frequent access pages / etc... - data refresh every few seconds.•ﬂuentd + mongodb + node.js 81

Lessons learned from this case•‘ﬂuentd + mongodb + node.js’ is an ideal harmony (on real-time web)•Reasons - ﬂuentd : Handling JSON-like data. (MessagePack) - mongodb : Storing JSON-like data. (BSON) - node.js : JavaScript environment. 82

MongoDB + node.js (+ ﬂuentd) = LoveMongoDB + node.js ( + ﬂuentd) = Love http://joyeur.com/2011/10/26/node-js-mongodb-love-guest-post-from-mongolab/ 83

Conclusion 84

Conclusion•‘MongoDB’ is very valuable database (in my opinion..). - Have many strong points. but have few weak points. - The best way is ‘using in the right place’.•Try to apply it on your system ! 85

Thank you for your attention. 86

http://tech.naver.jp	1039
http://d.hatena.ne.jp	351
http://www.10gen.com	130
http://f-kei.blogspot.com	76
http://blog.fkei.me	10
http://webcache.googleusercontent.com	7
http://b.hatena.ne.jp	1
http://207.46.192.232	1
http://sxr.mongodb.org	1

Presentation at MongoDB Tokyo 2012

by NAVER Japan on Jan 18, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

9 Embeds 1,616

Statistics

Presentation at MongoDB Tokyo 2012 — Presentation Transcript