MongoDB as Search Engine Repository @ MongoTokyo2011

11,100 views
10,990 views

Published on

MongoDB as Search Engine Repository @ MongoTokyo2011

http://www.10gen.com/conferences/mongotokyo2011

Published in: Technology
1 Comment
21 Likes
Statistics
Notes
No Downloads
Views
Total views
11,100
On SlideShare
0
From Embeds
0
Number of Embeds
2,324
Actions
Shares
0
Downloads
172
Comments
1
Likes
21
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • MongoDB as Search Engine Repository @ MongoTokyo2011

    1. 1. asSearch Engine Document Repository Preferred Infrastructure, Inc. CTO Kazuki Ohta <kzk@preferred.jp> http://kzk9.net/ 1
    2. 2. Self Introduction• Kazuki Ohta, CTO at Preferred Infrastructure, Inc. (http://preferred.jp) • Interested in Data Intensive Computing • Graduated U-Tokyo in 2010 (System Software) • Parallel I/O Middleware for Massively Parallel HPC Environment • Summer Intern @ Argonne National Laboratory • ACM ICPC • Hadoop User Group ( http://hugjp.org/ )• Personal Site • http://kzk9.net/, @kzk_mover 2
    3. 3. Agenda 3
    4. 4. Agenda• Introduction of Sedue 3
    5. 5. Agenda• Introduction of Sedue• Problems We Had 3
    6. 6. Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved 3
    7. 7. Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved• Problems in the Integration Phase 3
    8. 8. Agenda• Introduction of Sedue• Problems We Had• How MongoDB Solved• Problems in the Integration Phase• Future Insight 3
    9. 9. 4
    10. 10. Sedue Search Engine• Enterprise Distributed Search Engine • Developed at Preferred Infrastructure, Inc. • Multi-threaded C++ Server (0.3 million lines) • Often Handles Midscale Contents • 50 million documents/items• Around 30 customers • Media, Ad, E-Commerce, Digital Library, etc. 5
    11. 11. Sedue Data Model • Fixed Schema over De-Normalized Data • Field Definition + Index Definition • How the data is stored (name? type?) • How the data is indexedArticleID Title Content Search Recommend ID123 iPad2 iPad2 is coming! Filter ID124 MongoDB Durable in Single Server! ID125 MongoTokyo Today! Query 6
    12. 12. Sedue Schema (Sample)<schema> <fields> <field name=”article_id” type=”string” /> <field name=”title” type=”string” /> <field name=”contents” type=”string” /> <field name=”date” type=”datetime” /> </fields> <indexes> <index name=”search” type=”invertedindex” target=”content” /> <index name=”recommend” type=”doc2doc” target=”title, content” /> </indexes></schema> 7
    13. 13. Sedue Query (Sample)(search:iPad2)?date<today()?sort=date:desc QueryText Filter SortArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 8
    14. 14. Sedue Query (Sample)((search:iPad2)&(search:coming))?date<today()?sort=date:desc QueryText Filter Sort ArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 9
    15. 15. Sedue Query (Sample)(recommend:ID124)?date<today()?sort=date:desc QueryText Filter Sort ArticleID Title Content date ID123 iPad2 iPad2 is coming! today ID124 MongoDB Durable in Single Server! today ID125 MongoTokyo Today! yesterday 10
    16. 16. This Data Model is Mapped to The Distributed System 11
    17. 17. Sedue Architecture Crawler 12
    18. 18. Sedue Architecture Crawler Distributed Repository 12
    19. 19. Sedue Architecture Crawler Distributed Repository Document Repository Proxy 12
    20. 20. Sedue Architecture Crawler Distributed Repository Document Indexer Repository Proxy 12
    21. 21. Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) Document Indexer Repository Proxy 12
    22. 22. Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) Document Searchar Indexer Repository Proxy 12
    23. 23. Sedue Architecture Crawler Distributed Distributed Repository File System (DFS) DocumentQuery Searchar Indexer RepositoryServer Proxy 12
    24. 24. Sedue Architecture Crawler Distributed Distributed Repository File System (DFS)User Document Query Searchar Indexer Repository Server Proxy 12
    25. 25. Sedue Architecture Crawler Distributed Distributed Repository File System (DFS)User Document Query Searchar Indexer Repository Server Proxy Archive 12 Manager
    26. 26. Sedue Architecture• “Distributed Index-Query Mechanism” • Create indices, distribute them, query with them • Most types of search/recommendation algorithm fits into this architecture • Otherwords: “Distributed Column-Oriented Database”• Once put the documents into Sedue, you can use search/ recommendation in One System • Register/Query is done via REST API 13
    27. 27. OK,now we developed the DistributedIndex-Query Engine! 14
    28. 28. However... 15
    29. 29. However...• THE PROBLEM: THE REAL WORLD 15
    30. 30. However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. 15
    31. 31. However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns 15
    32. 32. However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns • Especially in building vertical search over many sites (each has its own schema) 15
    33. 33. However...• THE PROBLEM: THE REAL WORLD • Schema is changed once a week. • Real data lacks most columns • Especially in building vertical search over many sites (each has its own schema) • High Availability is required in some cases 15
    34. 34. Especially, Cross-Site Search BPITProITProNikkeiBusiness OnlinePC OnlineTechOnKenplatzECO JapanBPNet BP 16
    35. 35. ArticleI Title Content date FlagA FlagB FlagC FlagXX DID123 iPad2 iPad2 is coming! today 1ID124 MongoDB Durable in Single today Server!ID125 MongoTokyo Today! yesterday 0ID126 HBase 0.90 is out! 1ID127 Cassandra 1ID128 CouchDBID129 Ruby todayID130 Python N/A 0ID131 Haskell N/A 1ID132 D-Lang N/A 17
    36. 36. ArticleI Title Content date FlagA FlagB FlagC FlagXX DID123 iPad2 iPad2 is coming! today 1ID124 MongoDB Durable in Single today Server!ID125 MongoTokyo Today! yesterday 0ID126 HBase 0.90 is out! 1ID127 Cassandra 1 Sparse!!!ID128 CouchDBID129 Ruby todayID130 Python N/A 0ID131 Haskell N/A 1ID132 D-Lang N/A 18
    37. 37. One Lucky Thing:“Pluggable Storage Strategy” 19
    38. 38. Pluggable Storage Strategy• Important: We want to focus on developing application servers • we’re the search engine company, not the database company• DocumentRepository, DistributedFileSystem is pluggable! • Many, many NoSQL storages are emerging • Prepare the simple interface on top of them • You can select the underlying storage technology by the requirements of the system itself • by document volume, availability, consistency, etc. 20
    39. 39. At first... (Repository) Online API Replication Column Sharding AdditionTokyo Cabinet (Table DB) ○ × ○ × MySQL × ○ Unfortunately, TokyoTyrant doen’t support Table Database at that time. 21
    40. 40. At first... (DFS) API Setup Availability PerformanceNFS POSIX ○ costly costly libhdfsHDFS ○ sucks 22
    41. 41. 23
    42. 42. http://www.mongodb.org/• OSS Document-Oriented Database • No Schema, BSON, Rich Query + B-TreeIndex • written in C++ • C, C++, Java, PHP, Python, Ruby COOL drivers • Embedded JavaScript Engine • db.insert({“category”:” ”}, MongoDB Sharding {“ ”: “ ”}) • db.articles.find({“category”: “ ”}) • High Availability by ReplicaSet • High Scalability by Auto-Sharding 24
    43. 43. As Repository Online API Replication Column Sharding AdditionTokyo Cabinet (Table DB) ○ × ○ × MySQL × ○ MongoDB ○ ○ ○ ongoing (master-master) 25
    44. 44. GridFS• MongoDB as Blob-Storage • The contents is splitted into 256kb chunks, with some metadata. • Performance is not as high as HDFS, but still useful in mid-scale deployment. Chunk0 Large Blob Metadata Chunk1 26
    45. 45. As DFS API Setup Availability Performance NFS POSIX ○ costly costly libhdfsHDFS ○ sucksGridFS C++ ○ ○ 27
    46. 46. Now Sedue MongoDB • Use as Multiple Ways Repository • Repository + DFS • Easy setup!!! • 30million documentsUser • No Schema change is required DFS • Master-Master Replication • Backup once a week Sedue MongoDB 1.6 • 4 Production Deployments (Master-Master Replication) • 1 year 28
    47. 47. We had issues, but MongoDB is OSS!• SERVER-1408 (Fixed) • C++ Driver GridFS cannot store over 4G object.• SERVER-1372 (Fixed) • NULL check for auto_ptr<DBClientConnection> is missing• SERVER-1328 (Fixed) • scons install doesnt end with --prefix parameter?• SERVER-1232 (Fixed) • C++ GridFS Client should support larger Chunk Size• SERVER-2050 • Enables ScopedDbConnection to set the timeout. 29
    48. 48. Got the Mug! 30
    49. 49. How Long?• Prototype Version is in One Week • using C++ client API • about 500 lines• Production release in about 2 month • including bugfixes • mongo-user ML is really responsible • Eliot Horowitz merged my patch as quick as possible • The product itself is really stable than I expected (sorry) 31
    50. 50. How we store documents?• Most Straight Forward Way as Document DB • 30m documents, 4M limit each...{ Internal DocumentID (Indexed) # Internal Fields Internal ShardingID (Indexed) “__docid”: 32132, “__arcid”: 3, # Data Fields “title”: “MongoDB 1.8 is released!”, “content”: “Single Server Durability is supported”} 32
    51. 51. DocID Numbering• Counter by Atomic Increment Operation • docid++ 33
    52. 52. Query• Query by DocumentID • db.datadb.find({“__docid”: 12345}) = 1 doc• Query by ShardingID • db.datadb.find({“__arcid”: 3}) = <3m doc• These two fields have index! • Usage is more like K-V lookup, not the complex query • ShardingID query accesses whole disk structure now • Split by collection is ideal, but more hard to maintain 34
    53. 53. Problems... 35
    54. 54. Problem: Disk Consumption• MongoDB consumes the disk space a lot• Allocate some GBs (configurable), for the replication logs• Mostly append architecture • In-place modification is supported, if smaller than the original size• No compression scheme • want LZO/gzip support! 36
    55. 55. Problem: Consistency• Fire-and-Forget Write Behavior • Normally, mongodb insert doesn’t ensure the success at the server-side • Need to call getLastError() to ensure it, but slower • In replicated environment, you can specify minimum number of servers which succeeded the write operation• ReplicaSet mechanism is somewhat in the blackbox? • What consistency it provides? Fail-over mechanism? • Finally chose master-master replication. But will be obsoleted? 37
    56. 56. 1 billion Docs in MongoDB 38
    57. 57. Sharding• Scaling without no application modification 39
    58. 58. Sharding• Test with 2 nodes (8G mem, 1 SATA disk) • 150 Doc Register / sec • Upto 50 million documents • Gradually slowing down... • More latency than non-sharding setup • More parallelism, More node?• This results is early 1.7 release • Now enhanced a lot? 40
    59. 59. Conclusion• Sedue is “Distributed Index-Query Engine” • Headache about Frequently Changing Schema• Sedue MongoDB • As DocumentRepository + Blob Storage • MongoDB handles real data well in some cases • Future: Sharding for More Large Deployment 41
    60. 60. We’re Hiring!• Engineers • Core Search Engine Developer • C++ Expert • Distributed Systems Expert • Professional Support and Service • UNIX/Linux Expert • Summer Intern Student• Contact Me • kzk@preferred.jp , @kzk_mover • PFI: @preferred_jp • SedueTeam: @nobu_k, @eiichiroi, @repeatedly 42

    ×