Your SlideShare is downloading. ×
0
EDU 2.0Migrating to CloudSearch          Graham Glass, Founder
Background●   Cloud-hosted E-learning Platform (LMS)●   www.edu20.org for academia●   www.edu20.com for businesses●   1,00...
Search●   Originally using Sphinx/ThinkingSphinx●   All data came from RDS/MySQL●   One Sphinx per app server●   Full inde...
CloudSearch●   Decided to move to CloudSearch●   Simple, scaleable●   No Sphinx servers to manage●   Reliable, fast, delta...
Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial...
Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 by...
Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add  search update records to database● Sep...
Example Hooksdef after_create  super  update_searchenddef update_search if ((type = material_class).searchable? && (scope ...
Example DB update def self.add(type, ids)   begin     search_update = SearchUpdate.new(:class_name => type.name, :ids =>(i...
End Result● Migrated all of search in about two weeks,  spending 1-2 hours a day.● edu20.org: 7,000,000 documents, m2.xlar...
EDU2.0 and Amazon CloudSearch
Upcoming SlideShare
Loading in...5
×

EDU2.0 and Amazon CloudSearch

175

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
175
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "EDU2.0 and Amazon CloudSearch"

  1. 1. EDU 2.0Migrating to CloudSearch Graham Glass, Founder
  2. 2. Background● Cloud-hosted E-learning Platform (LMS)● www.edu20.org for academia● www.edu20.com for businesses● 1,000,000+ users● 15,000 new users a week● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.
  3. 3. Search● Originally using Sphinx/ThinkingSphinx● All data came from RDS/MySQL● One Sphinx per app server● Full indexing once a day● Noticeable slowdown during indexing● Sphinx daemons would sometimes fail
  4. 4. CloudSearch● Decided to move to CloudSearch● Simple, scaleable● No Sphinx servers to manage● Reliable, fast, delta-indexing● Easy to index anything, including DynamoDB
  5. 5. Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial contents● Then index everything except for high volume items (messages, postings), which were migrated last of all.● Finally, index messages and postings
  6. 6. Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 bytes
  7. 7. Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add search update records to database● Separate workers update search every 15 minutes with records from database● Had issues with XML characters
  8. 8. Example Hooksdef after_create super update_searchenddef update_search if ((type = material_class).searchable? && (scope != None)) SearchUpdate.add(type, material_id) endend
  9. 9. Example DB update def self.add(type, ids) begin search_update = SearchUpdate.new(:class_name => type.name, :ids =>(ids.kind_of?(Array) ? ids.join(,) : ids), :operation => Update) search_update.save! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end
  10. 10. End Result● Migrated all of search in about two weeks, spending 1-2 hours a day.● edu20.org: 7,000,000 documents, m2.xlarge● edu20.com: 500,000 documents, m1.small● Simplified architecture, positioned for scalability and DynamoDB● Only downside: $500/month for search
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×