The document discusses the use of weather and climate data beyond meteorology, highlighting sources and tools for accessing and analyzing this data, such as APIs and Jupyter notebooks. It also explores the connection between weather and traffic collisions, suggesting that historical weather data can be used to build predictive models for accidents. Techniques like linear regression and random forest are introduced for analyzing the relationship between weather conditions and traffic incidents.

1. Weather and Climate
Data: Not Just for
Meteorologists
Margriet Groenendijk
ODSC Europe
London – 13 October 2017

2. Weather
Forecast
@MargrietGr

3. Weather
Forecast
in a
notebook
@MargrietGr

4. Jupyter notebooks
@MargrietGr

5. Data
The Weather
Company API
Get free access here:
https://ibm.com/cloud
@MargrietGr

6. Demo
@MargrietGr

7. But ...
Where does weather
data come from?
@MargrietGr

8. Observations and Models
@MargrietGr

9. Weather Data
http://www.metoffice.gov.uk/datapoint/
https://www.wunderground.com/weather/
api/
https://business.weather.com/products/th
e-weather-company-data-packages
https://climexp.knmi.nl
http://www.ecmwf.int/en/forecasts/datase
ts
@MargrietGr

10. But …
I want a map
@MargrietGr

11. netcdf binary files (*.nc)
@MargrietGr

12. netcdf binary files (*.nc)
@MargrietGr

13. from netCDF4 import Dataset, num2date
import numpy as np
cfile =
'assets/HadCRUT.4.5.0.0.median.nc'
dataset = Dataset(cfile)
print dataset.data_model
print dataset.variables
Data is here:
https://crudata.uea.ac.uk/cru/data/temperature/@MargrietGr

14. NETCDF4
OrderedDict([(u'latitude', <type
'netCDF4._netCDF4.Variable'>
float32 latitude(latitude)
standard_name: latitude
long_name: latitude
point_spacing: even
units: degrees_north
axis: Y
unlimited dimensions:
ts: days since 1850-1-1 00:00:00
calendar: gregorian
start_year: 1850
...
@MargrietGr

15. import scipy
import matplotlib
from pylab import *
from mpl_toolkits.basemap import Basemap,
addcyclic, shiftgrid, maskoceans
# define the area to plot and projection to use
m =
Basemap(llcrnrlon=-180,llcrnrlat=-
60,urcrnrlon=180,urcrnrlat=80,projection='mill')
@MargrietGr
# covert the latitude, longitude and temperatures to
raster coordinates to be plotted
t1=temperature[0,:,:]
t1,lon=addcyclic(t1,lons)
january,longitude=shiftgrid(180.,t1,lon,start=False)
x,y=np.meshgrid(longitude,lats)
px,py=m(x,y)

16. rcParams['font.size']=12
rcParams['figure.figsize']=[8.0, 6.0]
figure()
palette=cm.RdYlBu_r
rmin=-30.; rmax=30.
ncont=20
dc=(rmax-rmin)/ncont
vc=arange(rmin,rmax+dc,dc)
pal_norm=matplotlib.colors.Normalize(vmin = rmin, vmax
= rmax, clip = False)
m.drawcoastlines(linewidth=0.5)
m.drawmapboundary(fill_color=(1.0,1.0,1.0))
cf=m.pcolormesh(px, py, january, cmap = palette)
cbar=colorbar(cf,orientation='horizontal',
shrink=0.95)
cbar.set_label('Mean Temperature in January')
tight_layout()

17. @MargrietGr

18. This was all about measurements
What about forecasts
and predictions?
@MargrietGr

19. Climate Models
@MargrietGr

20. All parameters Atmospheric
parameters
Carbon cycle
parameters
Sulphur cycle
and oceanic parameters
Global
Temperature
anomalies
1850-2100
@MargrietGr
Climate Model Experiments

21. What can I use
weather data
for?
@MargrietGr

22. @MargrietGr
Historic
Weather Data
Build model
Predict with
Weather
Forecast Data

23. Example
Traffic
Collisions
and Weather
www.pexels.com/photo/blur-cars-dew-drops-125510/@MargrietGr

24. NYPD Traffic
Collisions
1,119,577
Collisions since July
2012
https://data.cityofnewyork.us/Public-
Safety/NYPD-Motor-Vehicle-Collisions/h9gi-
nx95
@MargrietGr

25. Weather
Historic
Weather
https://business.weather.com/products/
weather-data-packages
@MargrietGr

26. Explore Data with PixieDust
Load into Pandas DataFrame
@MargrietGr

27. Explore Data with PixieDust
@MargrietGr

28. @MargrietGr
Explore Weather Data with PixieDust

29. Hypothesis
The number of traffic
collisions is influenced by
the weather
@MargrietGr

30. Combine collisions and weather data
newdf =
pd.merge_asof(collisions.sort_values(by='Date'),
weather.sort_values(by='date'),
left_on='Date',right_on='date',tolerance=pd.
Timedelta('1h'))
@MargrietGr

31. Combine collisions and weather data
times = pd.DatetimeIndex(collisions.Date)
collisions1 =
collisions.groupby(
[times.year,times.month,times.day,times.hour]).
sum()
collisions1['Hour']=collisions1.index
@MargrietGr

32. Combine collisions and weather data
@MargrietGr

33. Analysis with scikit-learn
@MargrietGr
from sklearn import preprocessing
min_max_scaler = preprocessing.MinMaxScaler()
np_scaled =
min_max_scaler.fit_transform(X_collisions)
X_normalized =
pd.DataFrame(np_scaled)X_normalized.head()

34. Simple linear regression
@MargrietGr
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test =
train_test_split(X_normalized, y_collisions,
random_state=42)
lr = LinearRegression(fit_intercept=True,
normalize=False)
lr.fit(X_train, y_train)
Training set score: 0.14
Test set score: 0.13

35. Random Forest
@MargrietGr
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=5,
random_state=2)
forest.fit(X_train, y_train)
Accuracy on training set: 0.948
Accuracy on test set: 0.041

36. @MargrietGr
Tree feature importances

37. Additional Data Needed
Speed limit
Number of lanes
Traffic volume
Population density
Road quality
Usage by trucks and buses
Pavement rating
Width
https://data.cityofnewyork.us/Transportation/Street-
Pavement-Rating/2cav-chmn
For each collision find
nearest road based on
Latitude and Longitude
@MargrietGr

38. Hope you will now
consider weather and
climate data in your data
science projects!
@MargrietGr

39. @MargrietGr
Historic
Weather Data
Build model
Predict with
Weather
Forecast Data

40. Thank you!
Margriet Groenendijk, PhD
Data Scientist
Developer Advocate
mgroenen@uk.ibm.com
@MargrietGr
Slides
https://www.slideshare.net/MargrietGr
oenendijk/presentations
Blog
https://medium.com/ibm-watson-
data-lab
IBM Data Science Experience
https://datascience.ibm.com
PixieDust
https://ibm-watson-data-
lab.github.io/pixiedust/
Notebooks
https://github.com/ibm-watson-data-
lab/python-notebooks
IBM Cloud
https://ibm.com/cloud
@MargrietGr