MongoDB as Search Engine Repository @ MongoTokyo2011

asSearch Engine Document Repository Preferred Infrastructure, Inc. CTO Kazuki Ohta <kzk@preferred.jp> http://kzk9.net/ 1

Self Introduction• Kazuki Ohta, CTO at Preferred Infrastructure, Inc. (http://preferred.jp) • Interested in Data Intensive Computing • Graduated U-Tokyo in 2010 (System Software) • Parallel I/O Middleware for Massively Parallel HPC Environment • Summer Intern @ Argonne National Laboratory • ACM ICPC • Hadoop User Group ( http://hugjp.org/ )• Personal Site • http://kzk9.net/, @kzk_mover 2

Agenda 3

Agenda• Introduction of Sedue 3

Agenda• Introduction of Sedue• Problems We Had 3

Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved 3

Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved• Problems in the Integration Phase 3

Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved• Problems in the Integration Phase• Future Insight 3

Sedue Search Engine• Enterprise Distributed Search Engine • Developed at Preferred Infrastructure, Inc. • Multi-threaded C++ Server (0.3 million lines) • Often Handles Midscale Contents • 50 million documents/items• Around 30 customers • Media, Ad, E-Commerce, Digital Library, etc. 5

Sedue Data Model • Fixed Schema over De-Normalized Data • Field Deﬁnition + Index Deﬁnition • How the data is stored (name? type?) • How the data is indexedArticleID Title Content Search Recommend ID123 iPad2 iPad2 is coming! Filter ID124 MongoDB Durable in Single Server! ID125 MongoTokyo Today! Query 6

Sedue Schema (Sample)<schema> <fields> <field name=”article_id” type=”string” /> <field name=”title” type=”string” /> <field name=”contents” type=”string” /> <field name=”date” type=”datetime” /> </fields> <indexes> <index name=”search” type=”invertedindex” target=”content” /> <index name=”recommend” type=”doc2doc” target=”title, content” /> </indexes></schema> 7

Sedue Query (Sample)(search:iPad2)?date<today()?sort=date:desc QueryText Filter SortArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 8

Sedue Query (Sample)((search:iPad2)&(search:coming))?date<today()?sort=date:desc QueryText Filter Sort ArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 9

Sedue Query (Sample)(recommend:ID124)?date<today()?sort=date:desc QueryText Filter Sort ArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 10

This Data Model is Mapped to The Distributed System 11

Sedue Architecture Crawler 12

Sedue Architecture Crawler Distributed Repository 12

Sedue Architecture Crawler Distributed Repository Document Repository Proxy 12

Sedue Architecture Crawler Distributed Repository Document Indexer Repository Proxy 12

Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) Document Indexer Repository Proxy 12

Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) Document Searchar Indexer Repository Proxy 12

Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) DocumentQuery Searchar Indexer RepositoryServer Proxy 12

Sedue Architecture Crawler Distributed Distributed Repository File System (DFS)User Document Query Searchar Indexer Repository Server Proxy 12

Sedue Architecture Crawler Distributed Distributed Repository File System (DFS)User Document Query Searchar Indexer Repository Server Proxy Archive 12 Manager

Sedue Architecture• “Distributed Index-Query Mechanism” • Create indices, distribute them, query with them • Most types of search/recommendation algorithm ﬁts into this architecture • Otherwords: “Distributed Column-Oriented Database”• Once put the documents into Sedue, you can use search/ recommendation in One System • Register/Query is done via REST API 13

OK,now we developed the DistributedIndex-Query Engine! 14

However... 15

However...• THE PROBLEM: THE REAL WORLD 15

However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. 15

However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns 15

However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns • Especially in building vertical search over many sites (each has its own schema) 15

However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns • Especially in building vertical search over many sites (each has its own schema) • High Availability is required in some cases 15

Especially, Cross-Site Search BPITProITProNikkeiBusiness OnlinePC OnlineTechOnKenplatzECO JapanBPNet BP 16

ArticleI Title Content date FlagA FlagB FlagC FlagXX DID123 iPad2 iPad2 is coming! today 1ID124 MongoDB Durable in Single today Server!ID125 MongoTokyo Today! yesterday 0ID126 HBase 0.90 is out! 1ID127 Cassandra 1ID128 CouchDBID129 Ruby todayID130 Python N/A 0ID131 Haskell N/A 1ID132 D-Lang N/A 17

ArticleI Title Content date FlagA FlagB FlagC FlagXX DID123 iPad2 iPad2 is coming! today 1ID124 MongoDB Durable in Single today Server!ID125 MongoTokyo Today! yesterday 0ID126 HBase 0.90 is out! 1ID127 Cassandra 1 Sparse!!!ID128 CouchDBID129 Ruby todayID130 Python N/A 0ID131 Haskell N/A 1ID132 D-Lang N/A 18

One Lucky Thing:“Pluggable Storage Strategy” 19

Pluggable Storage Strategy• Important: We want to focus on developing application servers • we’re the search engine company, not the database company• DocumentRepository, DistributedFileSystem is pluggable! • Many, many NoSQL storages are emerging • Prepare the simple interface on top of them • You can select the underlying storage technology by the requirements of the system itself • by document volume, availability, consistency, etc. 20

At ﬁrst... (Repository) Online API Replication Column Sharding AdditionTokyo Cabinet (Table DB) ○ × ○ × MySQL × ○ Unfortunately, TokyoTyrant doen’t support Table Database at that time. 21

At ﬁrst... (DFS) API Setup Availability PerformanceNFS POSIX ○ costly costly libhdfsHDFS ○ sucks 22

http://www.mongodb.org/• OSS Document-Oriented Database • No Schema, BSON, Rich Query + B-TreeIndex • written in C++ • C, C++, Java, PHP, Python, Ruby COOL drivers • Embedded JavaScript Engine • db.insert({“category”:” ”}, MongoDB Sharding {“ ”: “ ”}) • db.articles.ﬁnd({“category”: “ ”}) • High Availability by ReplicaSet • High Scalability by Auto-Sharding 24

As Repository Online API Replication Column Sharding AdditionTokyo Cabinet (Table DB) ○ × ○ × MySQL × ○ MongoDB ○ ○ ○ ongoing (master-master) 25

GridFS• MongoDB as Blob-Storage • The contents is splitted into 256kb chunks, with some metadata. • Performance is not as high as HDFS, but still useful in mid-scale deployment. Chunk0 Large Blob Metadata Chunk1 26

As DFS API Setup Availability Performance NFS POSIX ○ costly costly libhdfsHDFS ○ sucksGridFS C++ ○ ○ 27

Now Sedue MongoDB • Use as Multiple Ways Repository • Repository + DFS • Easy setup!!! • 30million documentsUser • No Schema change is required DFS • Master-Master Replication • Backup once a week Sedue MongoDB 1.6 • 4 Production Deployments (Master-Master Replication) • 1 year 28

We had issues, but MongoDB is OSS!• SERVER-1408 (Fixed) • C++ Driver GridFS cannot store over 4G object.• SERVER-1372 (Fixed) • NULL check for auto_ptr<DBClientConnection> is missing• SERVER-1328 (Fixed) • scons install doesnt end with --preﬁx parameter?• SERVER-1232 (Fixed) • C++ GridFS Client should support larger Chunk Size• SERVER-2050 • Enables ScopedDbConnection to set the timeout. 29

Got the Mug! 30

How Long?• Prototype Version is in One Week • using C++ client API • about 500 lines• Production release in about 2 month • including bugﬁxes • mongo-user ML is really responsible • Eliot Horowitz merged my patch as quick as possible • The product itself is really stable than I expected (sorry) 31

How we store documents?• Most Straight Forward Way as Document DB • 30m documents, 4M limit each...{ Internal DocumentID (Indexed) # Internal Fields Internal ShardingID (Indexed) “__docid”: 32132, “__arcid”: 3, # Data Fields “title”: “MongoDB 1.8 is released!”, “content”: “Single Server Durability is supported”} 32

DocID Numbering• Counter by Atomic Increment Operation • docid++ 33

Query• Query by DocumentID • db.datadb.find({“__docid”: 12345}) = 1 doc• Query by ShardingID • db.datadb.find({“__arcid”: 3}) = <3m doc• These two fields have index! • Usage is more like K-V lookup, not the complex query • ShardingID query accesses whole disk structure now • Split by collection is ideal, but more hard to maintain 34

Problems... 35

Problem: Disk Consumption• MongoDB consumes the disk space a lot• Allocate some GBs (conﬁgurable), for the replication logs• Mostly append architecture • In-place modiﬁcation is supported, if smaller than the original size• No compression scheme • want LZO/gzip support! 36

Problem: Consistency• Fire-and-Forget Write Behavior • Normally, mongodb insert doesn’t ensure the success at the server-side • Need to call getLastError() to ensure it, but slower • In replicated environment, you can specify minimum number of servers which succeeded the write operation• ReplicaSet mechanism is somewhat in the blackbox? • What consistency it provides? Fail-over mechanism? • Finally chose master-master replication. But will be obsoleted? 37

1 billion Docs in MongoDB 38

Sharding• Scaling without no application modiﬁcation 39

Sharding• Test with 2 nodes (8G mem, 1 SATA disk) • 150 Doc Register / sec • Upto 50 million documents • Gradually slowing down... • More latency than non-sharding setup • More parallelism, More node?• This results is early 1.7 release • Now enhanced a lot? 40

Conclusion• Sedue is “Distributed Index-Query Engine” • Headache about Frequently Changing Schema• Sedue MongoDB • As DocumentRepository + Blob Storage • MongoDB handles real data well in some cases • Future: Sharding for More Large Deployment 41

We’re Hiring!• Engineers • Core Search Engine Developer • C++ Expert • Distributed Systems Expert • Professional Support and Service • UNIX/Linux Expert • Summer Intern Student• Contact Me • kzk@preferred.jp , @kzk_mover • PFI: @preferred_jp • SedueTeam: @nobu_k, @eiichiroi, @repeatedly 42

http://research.preferred.jp	866
http://d.hatena.ne.jp	449
http://webcache.googleusercontent.com	14
http://translate.googleusercontent.com	2
http://www.mefeedia.com	1
http://paper.li	1
http://a0.twimg.com	1

MongoDB as Search Engine Repository @ MongoTokyo2011

by PFI Marketing on Mar 01, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 1,334

Statistics

MongoDB as Search Engine Repository @ MongoTokyo2011 — Presentation Transcript