Recommended
More Related Content
What's hot
What's hot (19)
More from Elizabeth (Lizzie) Siegle
More from Elizabeth (Lizzie) Siegle (20)
Recently uploaded
Recently uploaded (20)
SF Python Holiday Party 2018
- 3. What is Beautiful Soup? @LIZZIEPIKA - Web-scraping library - Extract desired data from page - Clean content
- 4. @LIZZIEPIK A How will We Use it?- Find every link on a webpage
- 5. @LIZZIEPIK A Some of our Tools -Requests to access HTML - bs4 to parse HTML
- 12. @LIZZIEPIK A Text +1(813)906-5107 “harry potter” “crazy rich asians” “jane austen” etc. @LIZZIEPIK A
Editor's Notes
- Hi everyone, I’m Lizzie, and today we’ll be talking about how to parse HTML to extract book quotes (or any data you wish) with Beautiful Soup
- So first off, what is this mysterious Python module? Has anyone used beautiful soup? hmm..Slytherin gif instead?
- Simple yet powerful 3rd-party Python library to parse specific content from a webpage in HTML, XML, and other markup languages. It does the backend job of removing the HTML markup for us. Yay!
- For example, you can use it to find all the links of a website, every “a” element with a “href” attribute, use to extract data like stock indices, sports stats, specific words, etc. We’ll use it to find book quotes from goodreads.com/quotes! How many people use Goodreads? Read?
- Requests to access HTML pages to scrape bs4 (Beautiful Soup 4) to parse HTML pages Install Requests, bs4 on the command line with pip
- With requests.get, we open the webpage. Then we make a BeautifulSoup object representing the document as a nested data structure. An optional second parameter would be a parser. “html.parser" is the default parser used with BeautifulSoup, but other parsers you could use include lxml or html5lib but both require an external dependency.
- Next, we want to get quotes from the page. If you visit https://goodreads.com/quotes and right click -> view page source you would see HTML classes are nested like this.
- With this nested structure in mind we will find every div with the class "quoteText." We can do this with soup(“div"), the shorthand for Beautiful Soup's find_all method:
- Let's loop through the quotes and find each quote’s author. As shown in the HTML code we inspected, the only a tag in div_quotes is the author. If an a tag exists then we know an author exists. We can access all that text inside a tag as a single Unicode string with the get_text() method. If we can't find an author, we skip it because it may not be a good quote.
- default: Python 3 uses Unicode, so every string is a sequence of Unicode characters. We loop through a given tag's children by calling .contents on the Beautifulsoup object, + then encode each child as ASCII: ignore any foreign Unicode characters. If a line starts with a tag symbol then characters that are not part of the quote are ignored. Otherwise, that line is added to the quote to return
- To clean and format each quote, strip() removes leading + trailing characters which could mess up our data. We also format the quote along with the author, and add on a "#" character to know where to parse the list. Finally, we filter through all the quotes to find and return the quotes, including characters considered printable like digits, letters, punctuation, or whitespace.
- Time for audience participation!! Get out your phones Text this number a book title, series, or author
- hype repo twitter, email, around conf