Download to read offline
Uploaded byIan Huston
CFSummit: Data Science on Cloud Foundry
Ian Huston and Alexander Kagoshima are data scientists at Pivotal Labs who use Cloud Foundry to deploy data science projects. They discuss using Cloud Foundry to easily control incoming data and bind external data services. They also provide examples of building prediction APIs and deploying machine learning models on Cloud Foundry to power data-driven applications.
More Related Content
Similar to CFSummit: Data Science on Cloud Foundry
CFSummit: Data Science on Cloud Foundry
- 2. Data Science onCloud Foundry Ian Huston @ianhuston Alexander Kagoshima @akagoshima
- 3. Who are we? • Data Scientists at Pivotal Labs • Using Cloud Foundry since 2013 • Working with enterprises to get value out of their data
- 4. Image by DrewConway: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
- 5. Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. - Josh Wills
- 6.
- 8. Data Services Easy controlof incoming data
- 9. Data Services Bind andscale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password" Data Service App App App App App App
- 10. Deploy a ModelPrediction API Control distributed computation
- 11. h"ps://github.com/ihuston/python-‐conda-‐buildpack Install PyData packages with binary builds using conda
- 12. h"ps://github.com/alexkago/cf-‐buildpack-‐r R interpreter and package setup, ready for RShiny
- 13. Siloed Data Siloed Systems Distributed Big Data Platform HOW TO DEPLOY MODELS? Data Extract ? (Model development happens here!) (Business needs model predictions here!)
- 14. App App App App App Big Data Platform BigData Storage
- 15. R E S T A P I Send data asJSON Data Ingest Model Create Model Redis Kicking off periodic retraining Save training data Save model object Send JSON data without label Receive prediction from trained model instance Deployed at: http://dsoncf.cfapps.io Code: https://github.com/pivotalsoftware/ds-cfpylearning PREDICTION API ARCHITECTURE $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME
- 16.
- 17.
- 18. SIMPLE HTML +JS MODEL PREDICTIONS http://ds-demo-transport.cfapps.io
- 19.
- 20. Show off yourdata science related Cloud Foundry apps: Twitter: @dsoncf http://dsoncf.com
- 21.
- 22. R E S T A P I Send data asJSON Data Ingest Model Create Model Redis Kicking off periodic retraining Save training data Save model object Send JSON data without label Receive prediction from trained model instance Deployed at: http://dsoncf.cfapps.io Code: https://github.com/pivotalsoftware/ds-cfpylearning Visualization PREDICTION API ARCHITECTURE
- 23. Data Services Bind andscale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"