The document outlines an overview of Amazon Web Services (AWS) and its capabilities for building custom search engines using its infrastructure and tools. It discusses historical developments in web technology and compares past and present processes for setting up websites and search engines. The document emphasizes the ease of leveraging AWS’s resources and tools like the Alexa Web Search Platform for web crawling and data processing.

1. Build Your Own Search Engine Jeff Barr Web Services Evangelist Amazon Web Services NGW044

2. Agenda Amazon Web Services Overview Looking Back Build Your Own Search Engine Q&A

3. Introduction And Background Software development background Veteran of several startups Visual Studio team at Microsoft (DHTML, XML, Web Services) 3.5 Years with Amazon Amazon Web Services Evangelist

4. What Is Amazon? Online Retailer Over 55 million active customer accounts Seven countries: US, UK, Germany, Japan, France, Canada, China Technology Consumer Multi-National Web Sites Vast Data Warehouse – 25 TB World-Class Logistics – 21 fulfillment centers; 9 million ft2 Technology Provider Hundreds of thousands of Amazon Associates Over 1,050,000 active seller accounts Over 150,000 software developers registered to use Amazon Web Services

5. What Is Alexa? Amazon subsidiary since 1999 Alexa Toolbar Web metrics Traffic rankings Web crawling

6. What Is Amazon Web Services? APIs that give developers programmatic access to Amazon’s data and technology Building-block web services Web-scale infrastructure E-commerce capability Content, data, and information New business models Customer-created content

7. AWS Product Family Amazon E-Commerce Service Complete access to Amazon’s product catalog Free + Associates commissions paid Amazon Historical Pricing Data warehouse access for product pricing Monthly Fee Amazon Mechanical Turk Artificial Artificial Intelligence 10% Commission Paid workforce Amazon Simple Queue Service IT building block In beta Amazon S3 Storage for the internet Charge by storage/bandwidth usage Alexa Web Information Service Data warehouse access for web crawl data 10K calls per month free, then 15 cents per 1000 calls Alexa Top Sites Top sites by Alexa traffic rank Charges by URL Alexa Web Search Platform Roll your own search engine Pay for time, storage, bandwidth

8. Amazon S3 Simple Storage Service Storage for the internet - web service to read and write data 15 cents per Gigabyte-Month to store data 20 cents per Gigabyte to access data Private and public storage Scalable, reliable, cost-effective, and simple!

9. Looking Back

10. Getting Online History Lesson 1996 vs. 2006 Lot has changed Let’s take a look

11. Going Online Then and Now What does is take to bring a simple web site online? Domain registration DNS support Network connection Server Hardware Development Tools Publicity Vehicle Monetization System

12. Then And Now Domain Registration Then Expensive ($70/year) Single vendor Multi-step, multi-day process Now Cheap ($10 or less / year) Dozens of vendors Single step, 10 minute process

13. Then And Now DNS Support Then Leech off of friend or university Long propagation times Complicated Days to understand & set up Now Free services (e.g. ZoneEdit) Very short propagation time Minutes to understand & set up

14. Then Versus Now Network Connection Then 9600 baud modem ISDN T1 Expensive Now DSL Dedicated hosting Cheap

15. Then Versus Now Server Hardware Then Start with dedicated PC Upgrade to expensive Sun hardware Now Build your own PC Hosting providers (EV1, BocaCom, Server Beach) Expensive Sun hardware

17. Then And Now Development Tools Then Text Editor Shell Window Now Visual Web Developer HTML Kit Front Page

18. Then Versus Now Publicity Vehicle Then Yahoo What’s New Usenet Press Release Wired Magazine Now Blogs / RSS / Pings Link sites Word of Mouth

19. Then Versus Now Monetization System Then Money? We are purists and we are doing this for fun! Banner ads Ad sales people Large sites only Now Pay per click Self serve Monetize page views

20. Then Building a Search Engine Lots of Servers Lots of Bandwidth Lots of Software Lots of Money Lots of Intellectual Capital Lots of Time

21. Now Building a Search Engine Use our infrastructure Leverage Alexa’s Crawl Alexa Web Search Platform 300 TB Archive 10 Billion web pages Pay as you go

22. AWSP Alexa Web Search Platform Build your own search engine! Process Specify pages to access within the 300TB archive Write parallelizable application to process pages Publish results as XML feed or as web service Pricing – everything costs $1 50 GB of data processing 1 CPU Hour 1 GB of data downloaded 4000 web service requests

23. AWSP Concepts Interactive Node - Development User Store – 12 TB of storage Compute Node – Processing Data Store 4 billion documents per crawl 3 crawls @ 100 TB In Process Current Previous All document types (HTML, Media, XML) Document header data

24. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

25. Great Ideas Vertical search engine Search engine optimization (SEO) Search engine marketing (SEM) Research < your idea here >

26. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

27. Write Code Run on Interactive Node Linux command line Interactive application development Use Collection API for data retrieval Use any language Libraries for C, Java, Perl Execution framework Application processes one document

28. Write Code Code can Examine document Examine headers Write to a collection Write to <stdout> Store data to Amazon S3

29. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

30. Test Code Run small test on Interactive Node Use predefined document collection Ensure proper functioning Measure document processing time

31. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

32. Identify Pages Choose a crawl Choose pages within the crawl by URL Linkage Alexa Traffic Rank (Top N) Redirection status Content Define a Collection

40. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

41. Schedule Job Allocate compute cluster resources Time Processors (1-10) Each processor 3.6 GHz CPU 4 GB of RAM 500 GB of local disk storage Charged at $1 per CPU hour

44. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

45. Run Job Job runs at specified time Code instances created on each node Job output combined automatically Collection Compute Node #1 Compute Node N ... Combine Results

46. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

47. Check Results Monitor progress using portal Final status email Log files Output

48. AWSP Design Process Great Idea Write Code Test Code Identify Pages Schedule Job Run Job Check Results Publish Results

49. Publishing Results Store data to S3 Create a new index for AWIS use Publish data for access via web search

50. Q & A

52. For More Information: AWSP: websearch.alexa.com Alexa Blog: awis.blogspot.com AWS Blog: aws.typepad.com Amazon Web Services: aws.amazon.com © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

53. © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.