Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EDU 2.0 - Migrating to CloudSearch


Published on

  • Be the first to comment

EDU 2.0 - Migrating to CloudSearch

  1. 1. EDU 2.0Migrating to CloudSearch Graham Glass, Founder
  2. 2. Background● Cloud-hosted E-learning Platform (LMS)● for academia● for businesses● 1,000,000+ users● 15,000 new users a week● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.
  3. 3. Search● Originally using Sphinx/ThinkingSphinx● All data came from RDS/MySQL● One Sphinx per app server● Full indexing once a day● Noticeable slowdown during indexing● Sphinx daemons would sometimes fail
  4. 4. CloudSearch● Decided to move to CloudSearch● Simple, scaleable● No Sphinx servers to manage● Reliable, fast, delta-indexing● Easy to index anything, including DynamoDB
  5. 5. Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial contents● Then index everything except for high volume items (messages, postings), which were migrated last of all.● Finally, index messages and postings
  6. 6. Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 bytes
  7. 7. Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add search update records to database● Separate workers update search every 15 minutes with records from database● Had issues with XML characters
  8. 8. Example Hooksdef after_create super update_searchenddef update_search if ((type = material_class).searchable? && (scope != None)) SearchUpdate.add(type, material_id) endend
  9. 9. Example DB update def self.add(type, ids) begin search_update = =>, :ids =>(ids.kind_of?(Array) ? ids.join(,) : ids), :operation => Update)! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end
  10. 10. End Result● Migrated all of search in about two weeks, spending 1-2 hours a day.● 7,000,000 documents, m2.xlarge● 500,000 documents, m1.small● Simplified architecture, positioned for scalability and DynamoDB● Only downside: $500/month for search