Tutorial on scraping weather data using Python and BeautifulSoup. Show various techniques and thought processes behind scraping the information

1. Make It Rain With Web
Scraping
Weather Data
CTPUG - 24 February 2018Gavin Wiener

2. Who Am I

3. What is Web-Scraping
“Web scraping, web harvesting, or web data extraction is data
scraping used for extracting data from websites”
https://en.wikipedia.org/wiki/Web_scraping

4. Throw Your Hands in The Air Like You Just Don't
Care
Who has web scraped?
Why did you web scrape?

5. Reasons
● Poor (or No) API (Application Programming Interface)
○ “Where there is a will, there is way”
● Poor design
○ Non-responsive
○ Cluttered
○ E.g. https://whereisthemyciti.com/
● Curiosity
○ Python
○ Web-scraping
○ New skill

6. Tools
● Python
○ https://www.python.org/
● Requests
○ http://docs.python-requests.org/en/master/user/install/
● BeautifulSoup
○ https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Minimum previous knowledge required

7. Disclaimer
● Read the website’s policy
○ robots.txt
○ Automatic scraping
● Do not circumvent paid APIs
○ Might be illegal
○ Time is better used

8. The Website

9. https://www.worldweatheronline.com/cape-town-weather/western-cape/za.aspx?day=20&tp=1#hourly

10. Get Data - Investigate URLs
https://www.worldweatheronline.com/cape-town-weather/we
stern-cape/za.aspx?day=20&tp=1#hourly

11. Goal
● Like a puzzle
○ Patterns
○ Repetition
● Good HTML structure
○ Web-framework templates

12. Identify Structure - Inspecting Elements

14. Single Instance

15. Isolate Elements

18. The Goods

19. The Goods

20. The Goods

21. I Can Haz Your Data - Code
Getting the Rows

22. The Goods - Banner

23. The Goods - Banner

24. The Goods - Banner

25. The Goods - Banner

26. I Can Haz Your Data - Code

27. I Can Haz Your Data - Raw

28. Summary
1. Identify the structure, and interesting components e.g. <table>, ids, classes
2. Identify how to reach the data e.g. urls
3. ‘Scrape’ the data with code e.g. code
4. Profit??

29. Summary - Web Scraping Sometimes #2
● Inconsistent
○ Structure can change
● Code can be messy
● Lots of data manipulation
○ Paying for a well-maintained API is better than headaches

30. gavinwiener@gmail.com
http://github.com/divisionMax/