Slides from my talk on web scraping to BrisJS the Brisbane JavaScript meetup. You can find the code on GitHub: https://github.com/ashleydavis/brisjs-web-scraping-talk

1. Web scraping
Generally a bad idea

2. Web scraping
If it sounds painful
That’s because it is

3. Web scraping
Should I do it?
No

4. Thanks for coming
Any questions?

5. What is web scraping
● Programmatically extracting data from web pages

6. Example

7. Web scraping is a horrible idea
● The scripts are tightly linked to the HTML
● The scripts fragile and prone to breaking
● Identifying HTML elements to extract is messy work
● Legal gray area
● You could be blocked from the web site

8. Sometimes web scraping is all we have
● The data isn’t accessible any other way
● We still need the data

9. Benefits of web scraping
● Automation
● Scalability

10. Techniques to demonstrate
1. Simple technique
○ For simple/static web pages
2. Advanced technique
○ JavaScript must execute
○ Interaction
○ Authentication

11. Tools
1. Simple technique
○ request-promise
○ cheerio
2. Advanced technique
○ nightmare (headless browser)
○ cheerio

12. Live coding
The code:
https://github.com/ashleydavis/brisjs-web-scraping-talk
The pages to scrape:
Simple: https://quotes.wsj.com/AU/XASX/CBA
Advanced: https://www.asx.com.au/asx/share-price-research/company/CBA

13. Production issues...
Performance
● Cache the Nightmare object / batch requests
● Disable image download
Debugging
● Show the Electron window
● Enable devtools
● Handle errors from Nightmare
● Display logging from the headless browser

14. Resources
● Code
○ github.com/ashleydavis/brisjs-web-scraping-talk
● Contact
○ Email: ashley@codecapers.com.au
○ Twitter: @ashleydavis75
○ GitHub:
■ ashleydavis
■ data-forge
● Data Wrangling with JavaScript
○ datawranglingwithjavascript.com
● The Data Wrangler
○ the-data-wrangler.com
My book