Download to read offline
Uploaded byAnthony Brown
Writing a Search Engine. How hard could it be?
The document details the experience of writing a search engine using Azure and F# over a weekend, highlighting the challenges and constraints faced, such as budget and time limitations. It covers the necessary components for building a search engine, including data indexing, web scraping, and utilizing Azure services like Azure Search and Azure Service Bus. Key takeaways emphasize the effectiveness of Azure in simplifying infrastructure issues and the role of F# in addressing software challenges, resulting in a functional, if basic, search engine prototype.
More Related Content
Similar to Writing a Search Engine. How hard could it be?
Writing a Search Engine. How hard could it be?
- 1. WRITING A SEARCHENGINE. HOW HARD COULD IT BE? ANTHONY BROWN @BRUINBROWN93 ANTHONY@COMPOSITIONAL-IT.COM
- 2. ABOUT ABOUT ME ▸ Consultantat Compositional IT ▸ F# dev for ~3 years now ▸ Interested in Big Data, IoT, Cloud and Distributed Systems
- 3. COMPOSITIONAL IT FUNCTIONAL FIRST.CLOUD READY. @COMPOSITIONALIT
- 4. HOW HARD COULDIT BE? Every software developer ever INTRODUCTION
- 5. IT’S ONLY ANOPERATING SYSTEM, ALL IT DOES IS RUNS PROGRAMS! Everybody when Windows blue screens INTRODUCTION
- 6. IT’S ONLY AMULTIPLAYER ONLINE VIDEO GAME! Anybody playing a game when lag spikes hit TEXT
- 7. IT’S ONLY 2LINES OF JAVASCRIPT Backend developer needing to make a small API change INTRODUCTION
- 8. DUDE. HOLD MYBEER. Drunk people 10 seconds before making a terrible mistake
- 9.
- 10. WHAT NEXT? HIT UPGOOGLE.
- 11. WHAT TO DOIN LONDON THIS WEEKEND?
- 14. WRITING A SEARCHENGINE. HOW HARD COULD IT BE?
- 15. WRITING A SEARCHENGINE WITH AZURE AND F# IN A WEEKEND.
- 16.
- 17. THIS WAS AWEEKEND PROJECT.
- 18. YOU SHOULD EXPECT: -HACKY CODE.
- 19. YOU SHOULD EXPECT: -DEMOS TO FAIL.
- 20. YOU SHOULD NOTEXPECT: - A DEEP DIVE INTO SEARCH ENGINE TECH.
- 21. SEARCH ENGINE BACKGROUND CONSTRAINTS ▸Not a priority ▸ Can’t cost more than £85 per month ▸ No operations investment ▸ Limit to the weekend
- 22. BACKGROUND EVERYTHING I KNOWABOUT HOW SEARCH ENGINES WORK ▸ ▸ ▸ ▸ ▸ ▸
- 23. THE ANATOMY OFA LARGE- SCALE HYPER TEXTUAL WEB SEARCH ENGINE SERGEY BRIN LARRY PAGE
- 25. IT’S 2016. THE WEB’SCHANGED. A LOT.
- 26.
- 27.
- 28.
- 29.
- 30. WHAT’S NEW? + PLATFORMAS A SERVICE
- 31.
- 32. WHAT’S NEW? - PERSONALHOSTING
- 33. SEARCH ENGINE BACKGROUND WHAT’SIMPORTANT? ▸ Search ▸ Scraping ▸ Page rank
- 34. SEARCH IMPLEMENTATION HOW TOFIND A NEEDLE IN A HAYSTACK ▸ Take all of your documents ▸ Record all of the words which occur within a file ▸ Invert that index ▸ List of all words and the documents they appear in ▸ For all words in the search query, find the files which appear in every inverted index
- 35. SOUNDS EASY RIGHT? IDON’T CARE ABOUT IT.
- 36. AZURE SEARCH MANAGED SEARCHAS A SERVICE
- 37. AZURE SEARCH WHAT DOESAZURE SEARCH GIVE US? ▸ Hosted Search as a Service ▸ HTTP API for indexing and retrieving documents ▸ Ability to scale out (more replicas, more indexes) ▸ Free basic tier
- 38. AZURE SEARCH INTHE AZURE PORTAL.
- 39.
- 40. WE HAVE SEARCH. WHATNEXT?
- 41. INDEXING DATA WHAT ISA CRAWLER ▸ Autonomously find every web page on the internet ▸ Pull the content from that web page and index it ▸ Read the links on that page and index those links ▸ Recursively process until every page on the internet has been reached
- 42. THE PROBLEM? THE INTERNET’SPRETTY BIG.
- 43. AZURE SERVICE BUS DISTRIBUTEDMESSAGE QUEUES
- 44. INDEXING DATA WHAT DOESAZURE SERVICE BUS GIVE US? ▸ Scalable durable queues and topics with guaranteed availability ▸ .Net APIs to communicate with the service bus ▸ Free basic tier
- 45. WORKING WITH A SERVICEBUS QUEUE.
- 46. WE NEED TOBE GOOD CITIZENS. WE DON’T WANT TO DDOS A SINGLE WEBSITE DURING CRAWLING.
- 47. SERVICE BUS PROVIDESSUPPORT FOR MESSAGE DE-DUPLICATION BASED ON CONTENT.
- 48. WE DON’T WANTTO SCRAPE THROUGH EVERY WEB PAGE IN THE WORLD.
- 49. WE DON’T WANTTO INDEX: - GOOGLE SEARCH QUERIES
- 50. WE DON’T WANTTO INDEX: - PROTECTED CONTENT
- 51. WE DON’T WANTTO INDEX: - IRRELEVANT CONTENT
- 52. DEALING WITH THE ROBOTS.TXTFILE WRITING BASIC PARSERS IN F#
- 53. BEING A WELLBEHAVED SCRAPER WHAT IS ROBOTS.TXT? ▸ Text file standard for telling web scrapers what they should scrape ▸ Opt-in - crawlers can ignore the robots.txt file ▸ Simple file stored at the root of the web server
- 54.
- 55.
- 56. HTML AND INFORMATION RETRIEVAL QUERYINGHTML DOCUMENTS WITH HTML AGILITY PACK
- 57. WE HAVE AHTML FILE. WE NEED THE CONTENT OUT OF IT.
- 58. INFORMATION RETRIEVAL FROMHTML DOCUMENTS WORKING WITH THE HTML AGILITY PACK ▸ Provides a simple query layer over HTML documents ▸ Works with well formatted and poorly formatted HTML ▸ Provides XPath support over the document ▸ Allows for querying for individual properties and elements
- 59. EXTRACTING LINKS FROM AHTML DOCUMENT
- 60. EXTRACTING ALL OFTHE CONTENT FROM AN HTML DOCUMENT
- 61. WE NOW HAVEA WEB SCRAPER. WE NEED TO RUN THE WEB SCRAPER.
- 62. AZURE WEBJOBS SIMPLE HOSTINGOF LONG RUNNING PROCESSES
- 63. AZURE WEB JOBS WHATARE WEB JOBS? ▸ A means of hosting basic executables in the cloud ▸ Provides simplified deployment and monitoring ▸ Pricing per minute of usage
- 64. WE NOW HAVEA SEARCH ENGINE. KIND OF.
- 65. SEARCH IS A RECOMMENDATIONPROBLEM.
- 66. HOW DO WERECOMMEND CONTENT TO USERS?
- 67. PAGE RANK FINDING THEMOST INFLUENTIAL SITES ON THE INTERNET
- 68. PAGE RANK WHAT ISPAGE RANK? ▸ Stanford’s patented algorithm ▸ Helps you find the most influential websites on the internet ▸ Websites with lots of links to them are more influential
- 69. THE PROBLEM? THERE’S LOTSOF WEBSITES ON THE INTERNET.
- 70. THERE’S EVEN MORELINKS BETWEEN WEBSITES.
- 71. WE HAVE AHUGE LINK GRAPH. WE NEED TO PROCESS THAT GRAPH.
- 72. BIG DATA PROCESSINGWITH MBRACE AND CLOUDFLOWS.
- 73. WE HAVE AQUERY WHICH NEEDS TO RUN DAILY. WE NEED TO ORCHESTRATE IT.
- 74. AZURE FUNCTIONS +AZURE RESOURCE MANAGER USING AZURE FUNCTIONS FOR DEVOPS
- 75. DEVOPS WHAT IS AZURERESOURCE MANAGER? ▸ Declarative way of describing Azure infrastructure ▸ REST APIs to deploy infrastructure template files ▸ APIs to see current deployment status
- 76. DEVOPS WHAT IS AZUREFUNCTIONS? ▸ Lightweight scripting of Azure web jobs ▸ Allows for running scripts in response to certain events ▸ Billing based on number of function invocations
- 77. DEVOPS USING AZURE FUNCTIONSFOR DEVOPS ▸ Set up a timer triggered Azure Function ▸ Deploy an Mbrace cluster through Azure Resource Manager ▸ Send an event when the job completes ▸ Second Azure Function for deleting the MBrace cluster
- 78. AZURE FUNCTIONS AND AZURERESOURCE MANAGER.
- 79. WE NOW HAVEEVERYTHING IN PLACE FOR A SEARCH ENGINE. NOBODY CAN ACCESS IT THOUGH.
- 80. AZURE FUNCTIONS SERVERLESS WEBAPIS WITH AZURE FUNCTIONS
- 81. AZURE FUNCTIONS CAN OPERATEON HTTP REQUESTS.
- 82. NO LONG TERM HOSTINGCOSTS.
- 83.
- 84.
- 85. HTTP API AZURE SEARCH LINKDATABASE PAGERANK CLUSTER ORCHESTRATOR AZURE SERVICEBUS INDEXER PAGERANK IMPORTER PAGERANK SCORE STORE
- 86. PLENTY OF ROOMFOR IMPROVEMENTS.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93. WHAT’S NEXT FORIT? NOTHING.
- 94. PRODUCTISING A GOOGLE COMPETITORIS BASICALLY IMPOSSIBLE.
- 95. IN SUMMARY WRAPPING UP& KEY TAKEAWAYS
- 96. AZURE + F#= <3
- 97. AZURE MAKES HARD INFRASTRUCTUREPROBLEMS SIMPLE.
- 98. F# MAKES HARDSOFTWARE PROBLEMS SIMPLE.
- 99. TOGETHER THEY MAKE HARDPROBLEMS SIMPLE.
- 100. IT’S NOT GOOGLE. BUTIT TOOK 1 DEV 2 DAYS.
- 101. CLOUD IS THEEPITOME OF BUSINESS AGILITY
- 103.
- 104.