The document provides an overview of Snapguide, which attracts 1.5 million unique users monthly and processes around 2000 requests per minute using technologies like Python, MySQL, and AWS. It details the implementation of a CloudSearch for guide data with features like autocomplete and enhanced search queries, ranking popular guides featured based on various relevance criteria. Additionally, it discusses future work, including adding new domains and improving the search-based suggestion engine.

1. Share what you know
Sam Kimbrel sam@snapguide.com
Software Engineer
Monday, April 1, 13

2. What is Snapguide?
• 1.5 million uniques/month
• ~2000 reqs/min across app
and web
• Python (Pyramid/uWSGI/
nginx)
• MySQL/Redis
• Built primarily on AWS: EC2,
RDS, S3, SQS, SNS,
CloudSearch, CloudFront
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

3. daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

4. daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

5. daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

6. Snapguide on CloudSearch
• Beta trial users after mentioning Solr on the phone
(seriously!)
• Primary data set: guides
• Facets: guide topic, “featured” boolean, visibility/ACL
flags
• “autocomplete” search (more later)
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

7. {
"lang": "en",
"fields": {
"step_count": "14",
"author_external_id": "qS878yliQ4mxg_9uHt2AZg",
"author": "Claire Hesseltine",
"items": [
"Preheat oven to 325 degrees Fahrenheit.",
...
],
"title": "Make Brown Butter Sea Salt Cookies",
"featured": 1,
"summary": "The brown butter adds a nutty, caramel-like taste
to these delicious cookies.",
"topic": [
"desserts"
],
"main_image_uuid": "43d201c8fd4b4833b83d3f95d112f1c1",
"like_count": 761,
"public": "true"
},
"version": 1364333310,
"type": "add",
"id": "9eabff97e32c4244a8205da3fba442e9"
} daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

8. Queries
• Guide text search:
q=cookies
• Guide search with topic:
q=cookies&facet=topic&bq=topic:‘desserts’
• “Typeahead”/suggestion search:
bq=(or ‘paper flower’ ‘paper flower*’)
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

9. Result Ranking
• Use “Compare Rank Expressions”
• text_relevance is your friend
• Goals:
• Boost popular/featured guides
• Make title/summary matches worth more than item
(supplies, step text) matches
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

10. min(
cs.text_relevance(
{"weights":
{"title":2.5, "author": 1.5, "items":
0.1, "summary": 1.5},
"default_weight":1}),
1000)
+ min(200, like_count / 10)
+ 100*featured
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

11. Offline index updates
• Extracting guide data to update document is slow
• Remove update from online web request process
• Internal-only API endpoints
• SQS
• queue_consumer daemon
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

12. Offline index updates
Web server SQS
Queue consumer
Snapguide
DB/Redis
Web server
(dedicated to queues) CloudSearch
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

13. Performance
SSL is painful
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

14. Performance
but physical proximity (us-west-1) is
awesome
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

15. Future work
• Add more domains (users, new features)
• Search-based suggestion engine
• Improved ranking/scoring — crawl our social graph
daniel@snapguide.com • confidential do not distribute
Monday, April 1, 13

16. Questions?
www.snapguide.com
Monday, April 1, 13