The document discusses various techniques for web scraping and analyzing scraped data, including using libraries like Requests, Selenium, BeautifulSoup, and tools like PostgreSQL, HDFS, and Spark. It provides code examples and recommendations for scraping etiquette, parsing different data formats, storing and analyzing large datasets.

1. Caso Ideal: WGET $URL > /tmp/mypage.html
Nunca es el caso ideal
self.conn = pycurl.Curl()
# Restart connection if less than 1 byte/s is received during "timeout"
seconds
if isinstance(self.timeout, int):
self.conn.setopt(pycurl.LOW_SPEED_LIMIT, 1)
self.conn.setopt(pycurl.LOW_SPEED_TIME, self.timeout)
self.conn.setopt(pycurl.URL, API_ENDPOINT_URL)
self.conn.setopt(pycurl.USERAGENT, USER_AGENT)
# Using gzip is optional but saves us bandwidth.
self.conn.setopt(pycurl.ENCODING, 'deflate, gzip')
self.conn.setopt(pycurl.POST, 1)
self.conn.setopt(pycurl.POSTFIELDS, urllib.urlencode(POST_PARAMS))
self.conn.setopt(pycurl.HTTPHEADER, ['Host: stream.twitter.com',
● Necesitamos PUTs, no Solo GETS
'Authorization: %s' %
self.get_oauth_header()])
# self.handle_tweet is the method that are called when new tweets arrive
self.conn.setopt(pycurl.WRITEFUNCTION, self.handle_tweet)
● A veces queremos scrappear un Stream, con
reconexiones
● Hay que enviar cabeceras, cookies de sesion...
● ¡En la DeepWeb hace falta user y password!

2. Requests: “HTTP for Humans”
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content­type']
'application/json; charset=utf8'
>>> r.encoding
'utf­8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}
http://docs.python-requests.org/en/latest/

3. ¡ajax, sesiones, navegación!
● Si con curl o requests no basta, hay que emular
un navegador.
● Webdriver, en Selenium
http://www.seleniumhq.org/projects/webdriver/
WebDriver driver = new FirefoxDriver();
// And now use this to visit Google
driver.get("http://www.google.com");
// Alternatively the same thing can be done like this
// driver.navigate().to("http://www.google.com");
// Find the text input element by its name
WebElement element = driver.findElement(By.name("q"));
// Enter something to search for
element.sendKeys("Cheese!");
// Now submit the form. WebDriver will find the form for us from the element
element.submit();

4. Nettiquette
(Para que no digas que nunca te lo han dicho).
● Mira el /robots.txt de los sitios que vayas a
scrappear.
● Honestamente, habria que mirar tambien las cabeceras
x-robots en HTTP y las tag robots en el HTML
● Controla la velocidad. Si el sitio va lento, baja la
presion.
● Y al reves, para más velocidad: usar multiples IP, usar
mutiples scrappers, lanzar proxies en la nube...
¿Httplib2 + squid?
● Indica en el UserAgent una forma de
contactarte. Email, web.

5. Parsing
● Html/xml: Sax, Xpath, …
● Json: .loads(), etc …
● JS en el server: nodejs
● BeautifulSoup
import xml.etree.ElementTree as ET
xsltproc :-(
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0">
<xsl:output method="text"/>
<xsl:template match="/">
<xsl:for­each
select="response/results/<xsl:value­of
select="field[@id='content']" />
<xsl:text>&#xA;</xsl:text>
</xsl:for­each>
</xsl:template>
</xsl:stylesheet>
http://www.crummy.com/software/BeautifulSoup/
for link in soup.find_all('a'):
print(link.get('href'))
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc)
print(soup.prettify())

6. Almacenando y Analizando
● Postgresql: tiene extensiones json y GIS
● Mysql: …
select position,json­>'
user'­>>'
screen_name', json­>>'
text' from georaw
where cod_prov='28' and st_Distance_Sphere(position::geometry,
st_makepoint(­3.73679,40.44439))
< 50;
● Hdfs/hive/etc: si tienes mas de una máquina.
– (o una con muchos cores)
– (o podrias tenerlas y quieres usar spark, mapreduce, etc)
./bin/spark­shell
­­total­executor­cores
7
sc.textFile("hdfs://localhost:9000/user/hadoopsingle/geoRaw").filter(line =>
line.contains("trafico")).count
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
import hiveContext._
import org.apache.spark.sql.catalyst.expressions._
val TableHQL = hiveContext.hql ("FROM raw.csv SELECT id,
type,length").groupBy(..........).persist()
TableHQL.map{case Row( id, t,l) => (l.asInstanceOf[Double] * 0.30) }.reduce(_+_)

7. ● API (Continuará...)