Ian Huston and Alexander Kagoshima are data scientists at Pivotal Labs who use Cloud Foundry to deploy data science projects. They discuss using Cloud Foundry to easily control incoming data and bind external data services. They also provide examples of building prediction APIs and deploying machine learning models on Cloud Foundry to power data-driven applications.

2. Data Science on Cloud Foundry
Ian Huston @ianhuston
Alexander Kagoshima @akagoshima

3. Who are we?
• Data Scientists at Pivotal Labs
• Using Cloud Foundry since 2013
• Working with enterprises to get value out
of their data

4. Image by Drew Conway: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram

5. Data Scientist (n.):
Person who is better at statistics than any
software engineer and better at software
engineering than any statistician.
- Josh Wills

6. Typical Projects
Risk
Analysis
Predictive
Maintenance
Understanding
Your Customer

8. Data Services
Easy control of incoming data

9. Data Services
Bind and scale system services
– Databases, NoSQL, message queues etc.
$
cf
create-­‐service
rediscloud
PLAN_NAME
INSTANCE_NAME
$
cf
bind-­‐service
APP_NAME
INSTANCE_NAME
Add User Provided Services
– Standalone Hadoop or Apache Spark cluster,
Big Data System
$
cf
cups
SERVICE_INSTANCE
-­‐p
"host,
port,
username,
password"
Data Service
App
App
App
App
App
App

10. Deploy a Model Prediction API
Control distributed computation

11. h"ps://github.com/ihuston/python-­‐conda-­‐buildpack
Install
PyData
packages
with
binary
builds
using
conda

12. h"ps://github.com/alexkago/cf-­‐buildpack-­‐r
R
interpreter
and
package
setup,
ready
for
RShiny

13. Siloed
Data
Siloed
Systems
Distributed
Big Data
Platform
HOW TO
DEPLOY
MODELS?
Data Extract
?
(Model
development
happens here!)
(Business
needs model
predictions
here!)

14. App
App
App
App
App
Big Data Platform
Big Data Storage

15. R
E
S
T
A
P
I
Send data as JSON
Data
Ingest
Model
Create Model
Redis
Kicking off
periodic
retraining
Save training
data
Save model
object
Send JSON data
without label
Receive prediction
from trained model
instance
Deployed at:
http://dsoncf.cfapps.io
Code:
https://github.com/pivotalsoftware/ds-cfpylearning
PREDICTION API
ARCHITECTURE
$
cf
create-­‐service
rediscloud
PLAN_NAME
INSTANCE_NAME

16. MODEL
INTERFACE

17. Data Driven Applications

18. SIMPLE HTML + JS
MODEL
PREDICTIONS
http://ds-demo-transport.cfapps.io

19. RSHINY APP
INTERACTIVE
EXPLORATION
https://ak-insurance-demo.cfapps.io:4443/

20. Show off your data
science related Cloud
Foundry apps:
Twitter: @dsoncf
http://dsoncf.com

21. @ianhuston
@akagoshima

22. R
E
S
T
A
P
I
Send data as JSON
Data
Ingest
Model
Create Model
Redis
Kicking off
periodic
retraining
Save training
data
Save model
object
Send JSON data
without label
Receive prediction
from trained model
instance
Deployed at:
http://dsoncf.cfapps.io
Code:
https://github.com/pivotalsoftware/ds-cfpylearning
Visualization
PREDICTION API
ARCHITECTURE

23. Data Services
Bind and scale system services
– Databases, NoSQL, message queues etc.
$
cf
create-­‐service
rediscloud
PLAN_NAME
INSTANCE_NAME
$
cf
bind-­‐service
APP_NAME
INSTANCE_NAME
Add User Provided Services
– Standalone Hadoop or Apache Spark cluster, Big Data System
$
cf
cups
SERVICE_INSTANCE
-­‐p
"host,
port,
username,
password"