The document discusses the challenges and pitfalls of web scraping, highlighting it as a generally poor practice due to its fragility, potential legal issues, and risk of being blocked from websites. However, it acknowledges that web scraping may be necessary when data is otherwise inaccessible and outlines both simple and advanced techniques for implementing scraping. The content also provides resources and contact information for further reference.

1. Web scraping
Generally a bad idea

2. Web scraping
If it sounds painful
That’s because it is

3. Web scraping
Should I do it?
No

4. Thanks for coming
Any questions?

5. What is web scraping
● Programmatically extracting data from web pages

6. Example

7. Web scraping is a horrible idea
● The scripts are tightly linked to the HTML
● The scripts fragile and prone to breaking
● Identifying HTML elements to extract is messy work
● Legal gray area
● You could be blocked from the web site

8. Sometimes web scraping is all we have
● The data isn’t accessible any other way
● We still need the data

9. Benefits of web scraping
● Automation
● Scalability

10. Techniques to demonstrate
1. Simple technique
○ For simple/static web pages
2. Advanced technique
○ JavaScript must execute
○ Interaction
○ Authentication

11. Tools
1. Simple technique
○ request-promise
○ cheerio
2. Advanced technique
○ nightmare (headless browser)
○ cheerio

12. Live coding
The code:
https://github.com/ashleydavis/brisjs-web-scraping-talk
The pages to scrape:
Simple: https://quotes.wsj.com/AU/XASX/CBA
Advanced: https://www.asx.com.au/asx/share-price-research/company/CBA

13. Production issues...
Performance
● Cache the Nightmare object / batch requests
● Disable image download
Debugging
● Show the Electron window
● Enable devtools
● Handle errors from Nightmare
● Display logging from the headless browser

14. Resources
● Code
○ github.com/ashleydavis/brisjs-web-scraping-talk
● Contact
○ Email: ashley@codecapers.com.au
○ Twitter: @ashleydavis75
○ GitHub:
■ ashleydavis
■ data-forge
● Data Wrangling with JavaScript
○ datawranglingwithjavascript.com
● The Data Wrangler
○ the-data-wrangler.com
My book