A simple introduction to scraperwiki with attention for the NOT developers

1. Summer School
"Data journalism e visualizzazione
grafica dei dati"
29 July 2011 – Flavon (TN)
A introduction to
for not developers
Maurizio Napolitano <napo@fbk.eu>

2. Description in the
name
SCRAPER
WIKI
source
http://www.modot.org/central/major_projects/July2006photos.htm
source
http://www.commoncraft.com/video/wikis

3. Wiki like Wikipedia
Scraper like ???
a scraper extract data
from a content

4. Legal aspect
Scraper sites may violate
copyright law.
Even taking content from an open content site can be a
copyright violation, if done in a way which does not respect
the license.
For instance, the GNU Free Documentation License (GFDL)
and Creative Commons ShareAlike (CC-BY-SA) licenses
require that a republisher inform readers of the license
conditions, and give credit to the original author.
http://en.wikipedia.org/wiki/Scraper_site

5. .. then scraperwiki is ...
https://scraperwiki.com/
A place where share scrapers … and data :)

6. ScraperWiki legal
aspect
Use
6. You agree that, in using the ScraperWiki site and services, you will
not interfere with the legal rights
[...]
Intellectual Property
9. Subject to the following paragraphs, the source code of the
ScraperWiki site, and all other copyrightable materials that form a part
of it is released under the GNU Affero General Public License.
10. All scraping code hosted on the site is licensed under the GNU
General Public License. You hereby license all scraping code you
create using ScraperWiki under the same licence.
11. You agree to assert no additional intellectual property rights,
including copyright and database right, in any scraped data other than
those which subsisted in the relevant web sites before the running of
the relevant scraper and which were held by you at that time.
12. You grant us a non-exclusive, worldwide, licence to use any data
that you store on our site, for the purposes of administering the site.
https://scraperwiki.com/terms_and_conditions/

7. ScraperWiki legal
aspect
USE
6.You agree [..] you will not interfere with
the legal rights
[...]
INTELLECTUAL PROPERTY
9. […] the source code of the ScraperWiki [..] is released
under the GNU Affero General Public License.
10. All
scraping code […] is licensed under the GNU
General Public License.
11.You agree to assert no additional
intellectual property rights [...]
12. You grant us a non-exclusive, worldwide, licence to use any data
that you store on our site, for the purposes of administering the site.

8. HOW CREATE A
SCRAPER?

9. The NOT developers

10. The technical
approach
http://unstats.un.org/unsd/demographic/products/socind/education.htm

11. Behind the page
HTML
code

12. Where are the data?
There is a structure
behind!!!

13. The algorithm!!!
Download th web page Read the information
Find the right position
Extract the data
Create a CSV file
data1;data2;data3
[...]
dataN1;dataN2;dataN3

14. Example: python code
https://scraperwiki.com/docs/python/python_intro_tutorial/

15. … and everything run
in the cloud!!!

16. The code in the cloud
https://scraperwiki.com/scrapers/mlb_rosters/

17. Sharing & ReUse

18. Enjoy!!!
httpS://scraperwiki.com/

19. Thanks!
A introduction to ScraperWiki for NOT developers by
Maurizio Napolitano <napo@fbk.eu>
is licensed under a
Creative Commons Attribuzione 3.0 Unported License.
Created for
Summer School
"Data journalism e visualizzazione
grafica dei dati"
29 July 2011 – Flavon (TN)