• Like

Thanks for flagging this SlideShare!

Oops! An error has occurred.

EDU 2.0 - Migrating to CloudSearch

  • 449 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
449
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. EDU 2.0Migrating to CloudSearch Graham Glass, Founder
  • 2. Background● Cloud-hosted E-learning Platform (LMS)● www.edu20.org for academia● www.edu20.com for businesses● 1,000,000+ users● 15,000 new users a week● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.
  • 3. Search● Originally using Sphinx/ThinkingSphinx● All data came from RDS/MySQL● One Sphinx per app server● Full indexing once a day● Noticeable slowdown during indexing● Sphinx daemons would sometimes fail
  • 4. CloudSearch● Decided to move to CloudSearch● Simple, scaleable● No Sphinx servers to manage● Reliable, fast, delta-indexing● Easy to index anything, including DynamoDB
  • 5. Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial contents● Then index everything except for high volume items (messages, postings), which were migrated last of all.● Finally, index messages and postings
  • 6. Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 bytes
  • 7. Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add search update records to database● Separate workers update search every 15 minutes with records from database● Had issues with XML characters
  • 8. Example Hooksdef after_create super update_searchenddef update_search if ((type = material_class).searchable? && (scope != None)) SearchUpdate.add(type, material_id) endend
  • 9. Example DB update def self.add(type, ids) begin search_update = SearchUpdate.new(:class_name => type.name, :ids =>(ids.kind_of?(Array) ? ids.join(,) : ids), :operation => Update) search_update.save! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end
  • 10. End Result● Migrated all of search in about two weeks, spending 1-2 hours a day.● edu20.org: 7,000,000 documents, m2.xlarge● edu20.com: 500,000 documents, m1.small● Simplified architecture, positioned for scalability and DynamoDB● Only downside: $500/month for search