The document outlines steps for web scraping restaurants from TripAdvisor using GenStage in Elixir. It describes parsing pagination entries from the city page to extract links to individual restaurant pages. These restaurant pages are then parsed to extract various data fields like address, rating, images and save to a database. The scraping process is designed as a parallel pipeline using GenStage to process pages concurrently and populate an application hierarchy to organize the workflow.

1. Web Scraping with
GenStage
by: Simon Escobar Benitez

2. Goal

3. Steps
• Get City Page
• Parse Pagination Entries
• Parse Pagination Pages
• Parse Restaurant Pages
• Save to DB (left as an exercise)

4. Get City Page
• Manual Step (https://www.tripadvisor.com/
Restaurants-g297478-
Medellin_Antioquia_Department.html)
• Can be automated via WebCrawling using a
headless web browser and automating searches.

5. Parse Pagination Entries
Pagination Links:
• /RestaurantSearch-g297478-
Medellin_Antioquia_Department.html
• /RestaurantSearch-g297478-oa30-
Medellin_Antioquia_Department.html
• /RestaurantSearch-g297478-oa60-
Medellin_Antioquia_Department.html
• /RestaurantSearch-g297478-oa1380-
Medellin_Antioquia_Department.html

6. Format
• /RestaurantSearch-g297478-oa#{NUMBER}-
Medellin_Antioquia_Department.html
• Where #{NUMBER} is the Pagination number going
from 0 up to the total number of entries by 30

7. CODE

8. Parse Pagination Pages
• Get pagination page
• Get all the place entries in each page (only the link)

9. CODE

10. Parse Restaurant Pages
• Parse Address
• Parse City
• Parse Country
• Parse Phone
• Parse Tags
• Parse Rating
• Parse Price Range
• Parse Images
• Parse Category Tags
• Parse Hours Open
• Parse Location

11. CODE

12. Designing the pipeline

13. Parallel pipeline

14. Application Hierarchy

16. CODE