This document discusses migrating an e-learning platform's search from Sphinx to Amazon CloudSearch. The platform has over 1 million users and indexes data from MySQL and DynamoDB. The migration involved testing with one item type, using a script to upload initial contents, then indexing all other types except high-volume messages and postings which were migrated last. Ruby on Rails was integrated using a gem and background workers update the search every 15 minutes. The migration took two weeks and simplified the architecture for better scalability.

1. EDU 2.0
Migrating to CloudSearch
Graham Glass, Founder

2. Background
● Cloud-hosted E-learning Platform (LMS)
● www.edu20.org for academia
● www.edu20.com for businesses
● 1,000,000+ users
● 15,000 new users a week
● Customers include Disney, Large
Universities, California School Districts,
small Kindergartens, etc.

4. Search
● Originally using Sphinx/ThinkingSphinx
● All data came from RDS/MySQL
● One Sphinx per app server
● Full indexing once a day
● Noticeable slowdown during indexing
● Sphinx daemons would sometimes fail

5. CloudSearch
● Decided to move to CloudSearch
● Simple, scaleable
● No Sphinx servers to manage
● Reliable, fast, delta-indexing
● Easy to index anything, including DynamoDB

6. Migration
● 14 different types were being indexed
● Decided to index just one item for testing
● Use script to upload initial contents
● Then index everything except for high
volume items (messages, postings), which
were migrated last of all.
● Finally, index messages and postings

7. Configuration
● Two search domains, one for each site
● 20 index fields (only 2 text)
● Truncate messages/postings to 1000 bytes

8. Rails Integration
● Used aws_cloud_search gem
● Added hooks into object model to add
search update records to database
● Separate workers update search every 15
minutes with records from database
● Had issues with XML characters

9. Example Hooks
def after_create
super
update_search
end
def update_search
if ((type = material_class).searchable? && (scope != 'None'))
SearchUpdate.add(type, material_id)
end
end

10. Example DB update
def self.add(type, ids)
begin
search_update = SearchUpdate.new(:class_name => type.name, :ids =>
(ids.kind_of?(Array) ? ids.join(',') : ids), :operation => 'Update')
search_update.save!
rescue Exception => exception
puts "SearchUpdate.add exception: #{exception.message}"
end
end

11. End Result
● Migrated all of search in about two weeks,
spending 1-2 hours a day.
● edu20.org: 7,000,000 documents, m2.xlarge
● edu20.com: 500,000 documents, m1.small
● Simplified architecture, positioned for
scalability and DynamoDB
● Only downside: $500/month for search