Your SlideShare is downloading. ×
0

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

EDU 2.0 - Migrating to CloudSearch

473

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
473
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. EDU 2.0Migrating to CloudSearch Graham Glass, Founder
  • 2. Background● Cloud-hosted E-learning Platform (LMS)● www.edu20.org for academia● www.edu20.com for businesses● 1,000,000+ users● 15,000 new users a week● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.
  • 3. Search● Originally using Sphinx/ThinkingSphinx● All data came from RDS/MySQL● One Sphinx per app server● Full indexing once a day● Noticeable slowdown during indexing● Sphinx daemons would sometimes fail
  • 4. CloudSearch● Decided to move to CloudSearch● Simple, scaleable● No Sphinx servers to manage● Reliable, fast, delta-indexing● Easy to index anything, including DynamoDB
  • 5. Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial contents● Then index everything except for high volume items (messages, postings), which were migrated last of all.● Finally, index messages and postings
  • 6. Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 bytes
  • 7. Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add search update records to database● Separate workers update search every 15 minutes with records from database● Had issues with XML characters
  • 8. Example Hooksdef after_create super update_searchenddef update_search if ((type = material_class).searchable? && (scope != None)) SearchUpdate.add(type, material_id) endend
  • 9. Example DB update def self.add(type, ids) begin search_update = SearchUpdate.new(:class_name => type.name, :ids =>(ids.kind_of?(Array) ? ids.join(,) : ids), :operation => Update) search_update.save! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end
  • 10. End Result● Migrated all of search in about two weeks, spending 1-2 hours a day.● edu20.org: 7,000,000 documents, m2.xlarge● edu20.com: 500,000 documents, m1.small● Simplified architecture, positioned for scalability and DynamoDB● Only downside: $500/month for search

×