The Web Observatory and the process of data gathering.

1. THE WEB OBSERVATORY
DATA GATHERING WITH
EUGENE SIOW & XIN WANG
29 JANUARY 2016

2. WHAT IS THE WEB OBSERVATORY?
SEARCH + ACCESS

3. DATA | WO | APP
OPEN
PRIVATE
NoSQL
SQL
STREAMS
LINKED DATA
JS
PYTHON
NODE

4. GATHERING DATA WITH SCRAPING
DATA ON WEBPAGES
DATA I CAN USE
transform
(
)

5. THE PROCESS OF SCRAPING
INVESTIGATE THE STRUCTURE OF THE PAGE
CHECK IF THERE IS AN API
{APPLICATION PROGRAMMING INTERFACE}
USE CHROME’S INSPECTOR OR FIREBUG
EXTRACT, TRANSFORM, LOAD
WHAT IS YOUR DESIRED END FORMAT?

6. HANDS-ON RESOURCES
codepen.io/xgfd/pen/wMyQWb
github.com/eugenesiow/datathon2016/wiki
DATA-DRIVEN APPS USING THE WO
DATA GATHERING
webobservatory.soton.ac.uk
THE SOTON WEB OBSERVATORY
BACKGROUNDS FROM THE HUBBLE SPACE TELESCOPE