Weather is part of our everyday lives. Who doesn’t check the rain radar before heading out, or the weather forecast when planning a weekend away? But where does this data come from, and what is it made of? The answer is a mix of measurements, models and statistics, meaning that the use of weather and climate data can get complex very quickly. These slides provide a brief overview of the science behind weather and climate forecasts and provides you with the tools to get started with weather data - even if you aren't a meteorologist. Learn how to connect weather data to other data sources, how to visualize weather and climate data in an interactive weather dashboard embedded in a Python notebook, and other ways you can use weather data for yourself, from examples using weather APIs, maps, PixieDust and Machine Learning.

1. Weather and Climate
Data: Not Just for
Meteorologists
Margriet Groenendijk
ODSC Europe
London – 13 October 2017

2. Weather
Forecast
@MargrietGr

3. Weather
Forecast
in a
notebook
@MargrietGr

4. Jupyter notebooks
@MargrietGr

5. Data
The Weather
Company API
Get free access here:
https://ibm.com/cloud
@MargrietGr

6. Demo
@MargrietGr

7. But ...
Where does weather
data come from?
@MargrietGr

8. Observations and Models
@MargrietGr

9. Weather Data
http://www.metoffice.gov.uk/datapoint/
https://www.wunderground.com/weather/
api/
https://business.weather.com/products/th
e-weather-company-data-packages
https://climexp.knmi.nl
http://www.ecmwf.int/en/forecasts/datase
ts
@MargrietGr

10. But …
I want a map
@MargrietGr

11. netcdf binary files (*.nc)
@MargrietGr

12. netcdf binary files (*.nc)
@MargrietGr

13. from netCDF4 import Dataset, num2date
import numpy as np
cfile =
'assets/HadCRUT.4.5.0.0.median.nc'
dataset = Dataset(cfile)
print dataset.data_model
print dataset.variables
Data is here:
https://crudata.uea.ac.uk/cru/data/temperature/@MargrietGr

14. NETCDF4
OrderedDict([(u'latitude', <type
'netCDF4._netCDF4.Variable'>
float32 latitude(latitude)
standard_name: latitude
long_name: latitude
point_spacing: even
units: degrees_north
axis: Y
unlimited dimensions:
ts: days since 1850-1-1 00:00:00
calendar: gregorian
start_year: 1850
...
@MargrietGr

15. import scipy
import matplotlib
from pylab import *
from mpl_toolkits.basemap import Basemap,
addcyclic, shiftgrid, maskoceans
# define the area to plot and projection to use
m =
Basemap(llcrnrlon=-180,llcrnrlat=-
60,urcrnrlon=180,urcrnrlat=80,projection='mill')
@MargrietGr
# covert the latitude, longitude and temperatures to
raster coordinates to be plotted
t1=temperature[0,:,:]
t1,lon=addcyclic(t1,lons)
january,longitude=shiftgrid(180.,t1,lon,start=False)
x,y=np.meshgrid(longitude,lats)
px,py=m(x,y)

16. rcParams['font.size']=12
rcParams['figure.figsize']=[8.0, 6.0]
figure()
palette=cm.RdYlBu_r
rmin=-30.; rmax=30.
ncont=20
dc=(rmax-rmin)/ncont
vc=arange(rmin,rmax+dc,dc)
pal_norm=matplotlib.colors.Normalize(vmin = rmin, vmax
= rmax, clip = False)
m.drawcoastlines(linewidth=0.5)
m.drawmapboundary(fill_color=(1.0,1.0,1.0))
cf=m.pcolormesh(px, py, january, cmap = palette)
cbar=colorbar(cf,orientation='horizontal',
shrink=0.95)
cbar.set_label('Mean Temperature in January')
tight_layout()

17. @MargrietGr

18. This was all about measurements
What about forecasts
and predictions?
@MargrietGr

19. Climate Models
@MargrietGr

20. All parameters Atmospheric
parameters
Carbon cycle
parameters
Sulphur cycle
and oceanic parameters
Global
Temperature
anomalies
1850-2100
@MargrietGr
Climate Model Experiments

21. What can I use
weather data
for?
@MargrietGr

22. @MargrietGr
Historic
Weather Data
Build model
Predict with
Weather
Forecast Data

23. Example
Traffic
Collisions
and Weather
www.pexels.com/photo/blur-cars-dew-drops-125510/@MargrietGr

24. NYPD Traffic
Collisions
1,119,577
Collisions since July
2012
https://data.cityofnewyork.us/Public-
Safety/NYPD-Motor-Vehicle-Collisions/h9gi-
nx95
@MargrietGr

25. Weather
Historic
Weather
https://business.weather.com/products/
weather-data-packages
@MargrietGr

26. Explore Data with PixieDust
Load into Pandas DataFrame
@MargrietGr

27. Explore Data with PixieDust
@MargrietGr

28. @MargrietGr
Explore Weather Data with PixieDust

29. Hypothesis
The number of traffic
collisions is influenced by
the weather
@MargrietGr

30. Combine collisions and weather data
newdf =
pd.merge_asof(collisions.sort_values(by='Date'),
weather.sort_values(by='date'),
left_on='Date',right_on='date',tolerance=pd.
Timedelta('1h'))
@MargrietGr

31. Combine collisions and weather data
times = pd.DatetimeIndex(collisions.Date)
collisions1 =
collisions.groupby(
[times.year,times.month,times.day,times.hour]).
sum()
collisions1['Hour']=collisions1.index
@MargrietGr

32. Combine collisions and weather data
@MargrietGr

33. Analysis with scikit-learn
@MargrietGr
from sklearn import preprocessing
min_max_scaler = preprocessing.MinMaxScaler()
np_scaled =
min_max_scaler.fit_transform(X_collisions)
X_normalized =
pd.DataFrame(np_scaled)X_normalized.head()

34. Simple linear regression
@MargrietGr
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test =
train_test_split(X_normalized, y_collisions,
random_state=42)
lr = LinearRegression(fit_intercept=True,
normalize=False)
lr.fit(X_train, y_train)
Training set score: 0.14
Test set score: 0.13

35. Random Forest
@MargrietGr
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=5,
random_state=2)
forest.fit(X_train, y_train)
Accuracy on training set: 0.948
Accuracy on test set: 0.041

36. @MargrietGr
Tree feature importances

37. Additional Data Needed
Speed limit
Number of lanes
Traffic volume
Population density
Road quality
Usage by trucks and buses
Pavement rating
Width
https://data.cityofnewyork.us/Transportation/Street-
Pavement-Rating/2cav-chmn
For each collision find
nearest road based on
Latitude and Longitude
@MargrietGr

38. Hope you will now
consider weather and
climate data in your data
science projects!
@MargrietGr

39. @MargrietGr
Historic
Weather Data
Build model
Predict with
Weather
Forecast Data

40. Thank you!
Margriet Groenendijk, PhD
Data Scientist
Developer Advocate
mgroenen@uk.ibm.com
@MargrietGr
Slides
https://www.slideshare.net/MargrietGr
oenendijk/presentations
Blog
https://medium.com/ibm-watson-
data-lab
IBM Data Science Experience
https://datascience.ibm.com
PixieDust
https://ibm-watson-data-
lab.github.io/pixiedust/
Notebooks
https://github.com/ibm-watson-data-
lab/python-notebooks
IBM Cloud
https://ibm.com/cloud
@MargrietGr