SlideShare a Scribd company logo
1© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
1© Copyright 2014 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2014 Pivotal. All rights reserved.
Cloud Foundry
for Data Science
Ian Huston, Data Scientist
2© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Who am I?
Ÿ  Ian Huston
Ÿ  @ianhuston
Ÿ  github.com/ihuston
Ÿ  www.ianhuston.net
Ÿ  Talk resources:
http://tinyurl.com/cf4ds-talk
Ÿ  Data Scientist
Ÿ  Previously a theoretical
physicist using Python for
numerical simulations & HPC
Ÿ  Use Python/R/SQL/… for
predictive analytics and
machine learning
3© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
What is Cloud Foundry?
http://cloudfoundry.org
Open Source Cloud Platform
Simple App Deployment,
Scaling & Availability
No Cloud Provider Lock In
4© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
How can CF help data scientists?
Ÿ  Jamie is a data scientist who has just finished some
analysis. They want to put up a simple internal web app with
Javascript visualisations connected to internal data stores.
Ÿ  Sam is a data engineer who wants to set up a REST API to
expose a production machine learning model as a service.
Ÿ  Alex is a data scientist who has an existing RShiny or
Python app that they want to make available with multiple
instances.
5© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Cloud Foundry is a Platform
You bring the apps, the rest
is taken care of!
Source: Albert Barron (IBM),
https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service
6© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Cloud Foundry Foundation: Industry Standard
Gold
Silver
7© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
CF for data scientists & developers
Easily deploy your web app
cf	
  push	
  myapp	
  
Scale up and out quickly
cf	
  scale	
  myapp	
  –i	
  5	
  –m	
  1G	
  
Create and bind services
cf	
  bind-­‐service	
  myapp	
  redis	
  
	
  
8© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Simple Python Flask App Demo
Ÿ  Simple one page “Hello World” web app
Ÿ  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8
Ÿ  Demonstrates:
–  Installation of requirements
–  Scaling properties
Ÿ  Need to Provide:
–  App files
–  Dependencies listed in requirements.txt file
–  Optional manifest.yml file with configuration for deployment
9© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Official Language Support on Cloud Foundry
Go
Ruby
10© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Buildpacks – How to Customise CF
Use Heroku-like buildpacks to add additional runtimes:
Ÿ  Community Buildpacks for Clojure, Haskell, .NET, Erlang,…
Ÿ  R Buildpack
–  https://github.com/alexkago/cf-buildpack-r
–  Works with RShiny apps
Ÿ  PyData Buildpack
–  https://github.com/ihuston/python-conda-buildpack
–  Uses conda for hard to install dependencies
11© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Comparison with Docker
runtime layer
OS image
application layer
Docker
system brings fixed
host OS Kernel
* Devs may bring a custom buildpack
runtime layer*
OS image
application layer
CF Buildpack
system brings fixed
host OS Kernel
App container
System Provides
Dev Provides
12© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Services now available
These are available on Pivotal CF (Pivotal’s packaged Cloud Foundry offering).
See http://run.pivotal.io for the services available on Pivotal Web Services.
13© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Data Services
Ÿ  Easy to bind and scale services
–  Databases, NoSQL, message queues etc.
$	
  cf	
  create-­‐service	
  rediscloud	
  PLAN_NAME	
  INSTANCE_NAME	
  
$	
  cf	
  bind-­‐service	
  APP_NAME	
  INSTANCE_NAME	
  
Ÿ  Details provided in VCAP_APP_SERVICES env variable:
{	
  "rediscloud":	
  [	
  
	
  	
  	
  	
  {	
  "name":	
  "rediscloud-­‐42",	
  "label":	
  "rediscloud",	
  "plan":	
  "20mb",	
  
	
  	
  	
  	
  	
  	
  "credentials":	
  {	
  "port":	
  "6379",	
  	
  
	
  "hostname":	
  "pub-­‐redis-­‐6379.us-­‐east-­‐1-­‐2.3.ec2.redislabs.com",	
  
	
  	
  	
  	
  	
  	
  	
  	
  "password":	
  "your_redis_password”	
  	
  }	
  	
  }	
  ]	
  
}	
  
14© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
What can you build with CF?
Some ideas:
Ÿ  Interactive data exploration apps
Ÿ  APIs for a machine learning model
Ÿ  Internet of Things data collection and modelling tools
Ÿ  Microservices architecture for data analysis
Ÿ  Interface to a natural language system like IBM’s Watson
15© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Examples
Ÿ  Simple HTML + JS
app
Ÿ  Uses data in S3
Ÿ  Scale out when
heavy load
expected with auto
load balancing
http://ds-demo-transport.cfapps.io
16© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Examples
Ÿ  RShiny app
Ÿ  Interactive controls
with very little
additional coding.
Ÿ  Runs RandomForest
model in background
https://ak-insurance-demo.cfapps.io:4443/
17© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Send me your examples!
Show off your data science related Cloud Foundry
apps:
Twitter: @dsoncf
http://dsoncf.com
http://DataScienceOnCloudFoundry.com
18© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
Try Cloud Foundry today!
Pivotal Web Services: http://run.pivotal.io
IBM Bluemix: http://bluemix.net
Anynines: http://anynines.com
HP Helion: http://horizon.hpcloud.com
19© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
More info and resources
Ÿ  Material for this talk: http://tinyurl.com/cf4ds-talk
Ÿ  Cloud Foundry: http://cloudfoundry.org
Ÿ  Python Flask Mega Meta Tutorial for DS:
http://tinyurl.com/flaskDS
Ÿ  Twelve Factor Apps: http://12factor.net/
Ÿ  Meetups: http://cloud-foundry.meetup.com/

More Related Content

What's hot

Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks
 
Breathing New Life into Apache Oozie with Apache Ambari Workflow Manager
Breathing New Life into Apache Oozie with Apache Ambari Workflow ManagerBreathing New Life into Apache Oozie with Apache Ambari Workflow Manager
Breathing New Life into Apache Oozie with Apache Ambari Workflow Manager
DataWorks Summit
 
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
DataWorks Summit
 
CF Korea Meetup - Spring Cloud Services
CF Korea Meetup - Spring Cloud ServicesCF Korea Meetup - Spring Cloud Services
CF Korea Meetup - Spring Cloud Services
Jay Lee
 
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
Timothy Spann
 
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on KubernetesApache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
DataWorks Summit
 
Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10
Timothy Spann
 
Big Data on OpenStack
Big Data on OpenStackBig Data on OpenStack
Big Data on OpenStack
Nati Shalom
 
Hadoop Everywhere & Cloudbreak
Hadoop Everywhere & CloudbreakHadoop Everywhere & Cloudbreak
Hadoop Everywhere & Cloudbreak
Sean Roberts
 
Hadoop on Docker
Hadoop on DockerHadoop on Docker
Hadoop on Docker
Rakesh Saha
 
Manage democratization of the data - Data Replication in Hadoop
Manage democratization of the data - Data Replication in HadoopManage democratization of the data - Data Replication in Hadoop
Manage democratization of the data - Data Replication in Hadoop
DataWorks Summit
 
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Animesh Singh
 
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
Cloud Operations with Streaming Analytics using Apache NiFi and Apache FlinkCloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
DataWorks Summit
 
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
Puppet
 
Lessons learned running a container cloud on YARN
Lessons learned running a container cloud on YARNLessons learned running a container cloud on YARN
Lessons learned running a container cloud on YARN
DataWorks Summit
 
Connect Data and Devices with Apache NiFi
Connect Data and Devices with Apache NiFiConnect Data and Devices with Apache NiFi
Connect Data and Devices with Apache NiFi
Data Works MD
 
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifiTracking crime as it occurs with apache phoenix, apache hbase and apache nifi
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi
Timothy Spann
 
Resilient microservices with Kubernetes - Mete Atamel
Resilient microservices with Kubernetes - Mete AtamelResilient microservices with Kubernetes - Mete Atamel
Resilient microservices with Kubernetes - Mete Atamel
ITCamp
 
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with KubernetesExtending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with Kubernetes
Nicola Ferraro
 
Creating the Internet of Your Things
Creating the Internet of Your ThingsCreating the Internet of Your Things
Creating the Internet of Your Things
DataWorks Summit/Hadoop Summit
 

What's hot (20)

Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
 
Breathing New Life into Apache Oozie with Apache Ambari Workflow Manager
Breathing New Life into Apache Oozie with Apache Ambari Workflow ManagerBreathing New Life into Apache Oozie with Apache Ambari Workflow Manager
Breathing New Life into Apache Oozie with Apache Ambari Workflow Manager
 
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
 
CF Korea Meetup - Spring Cloud Services
CF Korea Meetup - Spring Cloud ServicesCF Korea Meetup - Spring Cloud Services
CF Korea Meetup - Spring Cloud Services
 
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
 
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on KubernetesApache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
 
Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10
 
Big Data on OpenStack
Big Data on OpenStackBig Data on OpenStack
Big Data on OpenStack
 
Hadoop Everywhere & Cloudbreak
Hadoop Everywhere & CloudbreakHadoop Everywhere & Cloudbreak
Hadoop Everywhere & Cloudbreak
 
Hadoop on Docker
Hadoop on DockerHadoop on Docker
Hadoop on Docker
 
Manage democratization of the data - Data Replication in Hadoop
Manage democratization of the data - Data Replication in HadoopManage democratization of the data - Data Replication in Hadoop
Manage democratization of the data - Data Replication in Hadoop
 
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
 
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
Cloud Operations with Streaming Analytics using Apache NiFi and Apache FlinkCloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
 
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
Infrastructure-as-Code with Puppet Enterprise in the Cloud - PuppetConf 2014
 
Lessons learned running a container cloud on YARN
Lessons learned running a container cloud on YARNLessons learned running a container cloud on YARN
Lessons learned running a container cloud on YARN
 
Connect Data and Devices with Apache NiFi
Connect Data and Devices with Apache NiFiConnect Data and Devices with Apache NiFi
Connect Data and Devices with Apache NiFi
 
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifiTracking crime as it occurs with apache phoenix, apache hbase and apache nifi
Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi
 
Resilient microservices with Kubernetes - Mete Atamel
Resilient microservices with Kubernetes - Mete AtamelResilient microservices with Kubernetes - Mete Atamel
Resilient microservices with Kubernetes - Mete Atamel
 
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with KubernetesExtending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with Kubernetes
 
Creating the Internet of Your Things
Creating the Internet of Your ThingsCreating the Internet of Your Things
Creating the Internet of Your Things
 

Similar to Cloud Foundry for Data Science

Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud Foundry
Ian Huston
 
HP Helion Webinar #2
HP Helion Webinar #2 HP Helion Webinar #2
HP Helion Webinar #2
BeMyApp
 
Pivotal Cloud Platform Roadshow Keynote
Pivotal Cloud Platform Roadshow KeynotePivotal Cloud Platform Roadshow Keynote
Pivotal Cloud Platform Roadshow Keynote
cornelia davis
 
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
OpenStack Korea Community
 
Manchester geek night pcf 101
Manchester geek night   pcf 101Manchester geek night   pcf 101
Manchester geek night pcf 101
Sufyaan Kazi
 
PCF Cloud-Native Workshop Slides
PCF Cloud-Native Workshop SlidesPCF Cloud-Native Workshop Slides
PCF Cloud-Native Workshop Slides
VMware Tanzu
 
OpenStack + Cloud Foundry for the OpenStack Boston Meetup
OpenStack + Cloud Foundry for the OpenStack Boston MeetupOpenStack + Cloud Foundry for the OpenStack Boston Meetup
OpenStack + Cloud Foundry for the OpenStack Boston Meetup
ragss
 
Cloud Roundtable | Pivoltal: Agile platform
Cloud Roundtable | Pivoltal: Agile platformCloud Roundtable | Pivoltal: Agile platform
Cloud Roundtable | Pivoltal: Agile platform
Codemotion
 
Serverless survival kit
Serverless survival kitServerless survival kit
Serverless survival kit
Steve Houël
 
SD Times - Docker v2
SD Times - Docker v2SD Times - Docker v2
SD Times - Docker v2
Alvin Richards
 
Sst hackathon express
Sst hackathon expressSst hackathon express
Sst hackathon express
Aeshan Wijetunge
 
Using Databases and Containers From Development to Deployment
Using Databases and Containers  From Development to DeploymentUsing Databases and Containers  From Development to Deployment
Using Databases and Containers From Development to Deployment
Aerospike, Inc.
 
Ian huston getting started with cloud foundry
Ian huston   getting started with cloud foundryIan huston   getting started with cloud foundry
Ian huston getting started with cloud foundry
Jessica Willis
 
Ian Huston - "Deploying your data driven web app on Cloud Foundry"
Ian Huston - "Deploying your data driven web app on Cloud Foundry" Ian Huston - "Deploying your data driven web app on Cloud Foundry"
Ian Huston - "Deploying your data driven web app on Cloud Foundry"
Sheamus McGovern
 
Industrial IoT bootcamp
Industrial IoT bootcampIndustrial IoT bootcamp
Industrial IoT bootcamp
Lothar Schubert
 
HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3 HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3
BeMyApp
 
Github Projects Overview and IBM Streams V4.1
Github Projects Overview and IBM Streams V4.1Github Projects Overview and IBM Streams V4.1
Github Projects Overview and IBM Streams V4.1
lisanl
 
How to deploy machine learning models into production
How to deploy machine learning models into productionHow to deploy machine learning models into production
How to deploy machine learning models into production
DataWorks Summit
 
The Platform for Building Great Software
The Platform for Building Great SoftwareThe Platform for Building Great Software
The Platform for Building Great Software
Platform CF
 
Pivotal One: The Platform For Building Great Software
Pivotal One: The Platform For Building Great Software Pivotal One: The Platform For Building Great Software
Pivotal One: The Platform For Building Great Software
VMware Tanzu
 

Similar to Cloud Foundry for Data Science (20)

Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud Foundry
 
HP Helion Webinar #2
HP Helion Webinar #2 HP Helion Webinar #2
HP Helion Webinar #2
 
Pivotal Cloud Platform Roadshow Keynote
Pivotal Cloud Platform Roadshow KeynotePivotal Cloud Platform Roadshow Keynote
Pivotal Cloud Platform Roadshow Keynote
 
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
[2015-11월 정기 세미나] Cloud Native Platform - Pivotal
 
Manchester geek night pcf 101
Manchester geek night   pcf 101Manchester geek night   pcf 101
Manchester geek night pcf 101
 
PCF Cloud-Native Workshop Slides
PCF Cloud-Native Workshop SlidesPCF Cloud-Native Workshop Slides
PCF Cloud-Native Workshop Slides
 
OpenStack + Cloud Foundry for the OpenStack Boston Meetup
OpenStack + Cloud Foundry for the OpenStack Boston MeetupOpenStack + Cloud Foundry for the OpenStack Boston Meetup
OpenStack + Cloud Foundry for the OpenStack Boston Meetup
 
Cloud Roundtable | Pivoltal: Agile platform
Cloud Roundtable | Pivoltal: Agile platformCloud Roundtable | Pivoltal: Agile platform
Cloud Roundtable | Pivoltal: Agile platform
 
Serverless survival kit
Serverless survival kitServerless survival kit
Serverless survival kit
 
SD Times - Docker v2
SD Times - Docker v2SD Times - Docker v2
SD Times - Docker v2
 
Sst hackathon express
Sst hackathon expressSst hackathon express
Sst hackathon express
 
Using Databases and Containers From Development to Deployment
Using Databases and Containers  From Development to DeploymentUsing Databases and Containers  From Development to Deployment
Using Databases and Containers From Development to Deployment
 
Ian huston getting started with cloud foundry
Ian huston   getting started with cloud foundryIan huston   getting started with cloud foundry
Ian huston getting started with cloud foundry
 
Ian Huston - "Deploying your data driven web app on Cloud Foundry"
Ian Huston - "Deploying your data driven web app on Cloud Foundry" Ian Huston - "Deploying your data driven web app on Cloud Foundry"
Ian Huston - "Deploying your data driven web app on Cloud Foundry"
 
Industrial IoT bootcamp
Industrial IoT bootcampIndustrial IoT bootcamp
Industrial IoT bootcamp
 
HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3 HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3
 
Github Projects Overview and IBM Streams V4.1
Github Projects Overview and IBM Streams V4.1Github Projects Overview and IBM Streams V4.1
Github Projects Overview and IBM Streams V4.1
 
How to deploy machine learning models into production
How to deploy machine learning models into productionHow to deploy machine learning models into production
How to deploy machine learning models into production
 
The Platform for Building Great Software
The Platform for Building Great SoftwareThe Platform for Building Great Software
The Platform for Building Great Software
 
Pivotal One: The Platform For Building Great Software
Pivotal One: The Platform For Building Great Software Pivotal One: The Platform For Building Great Software
Pivotal One: The Platform For Building Great Software
 

More from Ian Huston

Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural LanguagesData Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Ian Huston
 
Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)
Ian Huston
 
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
Ian Huston
 
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field InflationCalculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Ian Huston
 
Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011
Ian Huston
 
Second Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-rollSecond Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-roll
Ian Huston
 
Inflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big BangInflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big Bang
Ian Huston
 
Cosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical SimulationsCosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical Simulations
Ian Huston
 
Cosmo09 presentation
Cosmo09 presentationCosmo09 presentation
Cosmo09 presentation
Ian Huston
 

More from Ian Huston (9)

Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural LanguagesData Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
 
Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)
 
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at...
 
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field InflationCalculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
 
Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011
 
Second Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-rollSecond Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-roll
 
Inflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big BangInflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big Bang
 
Cosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical SimulationsCosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical Simulations
 
Cosmo09 presentation
Cosmo09 presentationCosmo09 presentation
Cosmo09 presentation
 

Recently uploaded

Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
moinahousna
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 

Recently uploaded (20)

Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 

Cloud Foundry for Data Science

  • 1. 1© Copyright 2015 Pivotal. All rights reserved. @ianhuston 1© Copyright 2014 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2014 Pivotal. All rights reserved. Cloud Foundry for Data Science Ian Huston, Data Scientist
  • 2. 2© Copyright 2015 Pivotal. All rights reserved. @ianhuston Who am I? Ÿ  Ian Huston Ÿ  @ianhuston Ÿ  github.com/ihuston Ÿ  www.ianhuston.net Ÿ  Talk resources: http://tinyurl.com/cf4ds-talk Ÿ  Data Scientist Ÿ  Previously a theoretical physicist using Python for numerical simulations & HPC Ÿ  Use Python/R/SQL/… for predictive analytics and machine learning
  • 3. 3© Copyright 2015 Pivotal. All rights reserved. @ianhuston What is Cloud Foundry? http://cloudfoundry.org Open Source Cloud Platform Simple App Deployment, Scaling & Availability No Cloud Provider Lock In
  • 4. 4© Copyright 2015 Pivotal. All rights reserved. @ianhuston How can CF help data scientists? Ÿ  Jamie is a data scientist who has just finished some analysis. They want to put up a simple internal web app with Javascript visualisations connected to internal data stores. Ÿ  Sam is a data engineer who wants to set up a REST API to expose a production machine learning model as a service. Ÿ  Alex is a data scientist who has an existing RShiny or Python app that they want to make available with multiple instances.
  • 5. 5© Copyright 2015 Pivotal. All rights reserved. @ianhuston Cloud Foundry is a Platform You bring the apps, the rest is taken care of! Source: Albert Barron (IBM), https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service
  • 6. 6© Copyright 2015 Pivotal. All rights reserved. @ianhuston Cloud Foundry Foundation: Industry Standard Gold Silver
  • 7. 7© Copyright 2015 Pivotal. All rights reserved. @ianhuston CF for data scientists & developers Easily deploy your web app cf  push  myapp   Scale up and out quickly cf  scale  myapp  –i  5  –m  1G   Create and bind services cf  bind-­‐service  myapp  redis    
  • 8. 8© Copyright 2015 Pivotal. All rights reserved. @ianhuston Simple Python Flask App Demo Ÿ  Simple one page “Hello World” web app Ÿ  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8 Ÿ  Demonstrates: –  Installation of requirements –  Scaling properties Ÿ  Need to Provide: –  App files –  Dependencies listed in requirements.txt file –  Optional manifest.yml file with configuration for deployment
  • 9. 9© Copyright 2015 Pivotal. All rights reserved. @ianhuston Official Language Support on Cloud Foundry Go Ruby
  • 10. 10© Copyright 2015 Pivotal. All rights reserved. @ianhuston Buildpacks – How to Customise CF Use Heroku-like buildpacks to add additional runtimes: Ÿ  Community Buildpacks for Clojure, Haskell, .NET, Erlang,… Ÿ  R Buildpack –  https://github.com/alexkago/cf-buildpack-r –  Works with RShiny apps Ÿ  PyData Buildpack –  https://github.com/ihuston/python-conda-buildpack –  Uses conda for hard to install dependencies
  • 11. 11© Copyright 2015 Pivotal. All rights reserved. @ianhuston Comparison with Docker runtime layer OS image application layer Docker system brings fixed host OS Kernel * Devs may bring a custom buildpack runtime layer* OS image application layer CF Buildpack system brings fixed host OS Kernel App container System Provides Dev Provides
  • 12. 12© Copyright 2015 Pivotal. All rights reserved. @ianhuston Services now available These are available on Pivotal CF (Pivotal’s packaged Cloud Foundry offering). See http://run.pivotal.io for the services available on Pivotal Web Services.
  • 13. 13© Copyright 2015 Pivotal. All rights reserved. @ianhuston Data Services Ÿ  Easy to bind and scale services –  Databases, NoSQL, message queues etc. $  cf  create-­‐service  rediscloud  PLAN_NAME  INSTANCE_NAME   $  cf  bind-­‐service  APP_NAME  INSTANCE_NAME   Ÿ  Details provided in VCAP_APP_SERVICES env variable: {  "rediscloud":  [          {  "name":  "rediscloud-­‐42",  "label":  "rediscloud",  "plan":  "20mb",              "credentials":  {  "port":  "6379",      "hostname":  "pub-­‐redis-­‐6379.us-­‐east-­‐1-­‐2.3.ec2.redislabs.com",                  "password":  "your_redis_password”    }    }  ]   }  
  • 14. 14© Copyright 2015 Pivotal. All rights reserved. @ianhuston What can you build with CF? Some ideas: Ÿ  Interactive data exploration apps Ÿ  APIs for a machine learning model Ÿ  Internet of Things data collection and modelling tools Ÿ  Microservices architecture for data analysis Ÿ  Interface to a natural language system like IBM’s Watson
  • 15. 15© Copyright 2015 Pivotal. All rights reserved. @ianhuston Examples Ÿ  Simple HTML + JS app Ÿ  Uses data in S3 Ÿ  Scale out when heavy load expected with auto load balancing http://ds-demo-transport.cfapps.io
  • 16. 16© Copyright 2015 Pivotal. All rights reserved. @ianhuston Examples Ÿ  RShiny app Ÿ  Interactive controls with very little additional coding. Ÿ  Runs RandomForest model in background https://ak-insurance-demo.cfapps.io:4443/
  • 17. 17© Copyright 2015 Pivotal. All rights reserved. @ianhuston Send me your examples! Show off your data science related Cloud Foundry apps: Twitter: @dsoncf http://dsoncf.com http://DataScienceOnCloudFoundry.com
  • 18. 18© Copyright 2015 Pivotal. All rights reserved. @ianhuston Try Cloud Foundry today! Pivotal Web Services: http://run.pivotal.io IBM Bluemix: http://bluemix.net Anynines: http://anynines.com HP Helion: http://horizon.hpcloud.com
  • 19. 19© Copyright 2015 Pivotal. All rights reserved. @ianhuston More info and resources Ÿ  Material for this talk: http://tinyurl.com/cf4ds-talk Ÿ  Cloud Foundry: http://cloudfoundry.org Ÿ  Python Flask Mega Meta Tutorial for DS: http://tinyurl.com/flaskDS Ÿ  Twelve Factor Apps: http://12factor.net/ Ÿ  Meetups: http://cloud-foundry.meetup.com/