February EPD Webinar: How do I...use PiCloud for cloud computing?

How do I...use PiCloud for cloud computing?
Ken Elkabany • February 5, 2010

January 5, 2010 release

• Python 2.6
• 75+ libraries SciPy, NumPy, matplotlib, IPython, Mayavi...
• NumPy 1.4 faster (20-30%) import times, better handling of
special floating point types

• 64-bit available on all platforms in addition to 32-bit
• PiCloud’s cloud library for cloud computing

Ken Elkabany
Co-Founder & CEO, PiCloud

• UC Berkeley

• Founded PiCloud in 2008

Presented by
Ken Elkabany, CEO
ken@picloud.com

Feb. 5, 2010 EPD Presentation: How do I use PiCloud?

Who are we?
2

 PiCloud, Inc.
 Founded in 2009, and based in Sunnyvale, CA
 Our mission: Simplify cloud computing
 The PiCloud Cloud-Computing Platform
 Integrated into the Python programming language
 Partnership with Enthought, Inc. to include our cloud
library in EPD (Enthought Python Distribution)
 Beta-release

© Copyright 2010 by PiCloud, Inc.

What is cloud computing?
3

 Cloud computing is a style of computing in which
dynamically scalable and often virtualized
resources are provided as a service over the
Internet.


What does a cloud look like?
4

 Low upfront costs
 Serversare not purchased, only rented (utility
computing)
 Multi-tenant
 Centralization of infrastructure geographically
 High-levels of utilization and efficiency

 Scalable and Agile
 On-demand provisioning of resources
 Secure, Reliable, Sustainable


Cloud Computing Ecosystem
5

 Cloud infrastructures
 Allow for the provisioning of
virtualized resources
 Compute: Boot up and log into
a Linux or Windows server
 Storage: Specialized services
like S3
 The PiCloud platform sits on
top of infrastructure,
abstracting its details away


The PiCloud Abstraction
6

Input Output
Input Output


Our motivation
7

 AutoTagger
 Facebook photo-tagging assistant
 PiCloud’s First Customer

 Basic Functionality
 Detect and extract faces from
a single photo
 Repeatedly apply face detection
to all photos in an album


Our motivation
8

Functional view: I need to detect faces in Server view: I need 20 high performance virtual servers to
these photos for the user detect faces in these photos in parallel

Design Objectives:
•Load balancing
•Auto-Scaling
•Performance Monitoring, introspection
PiCloud •Error detection, reporting, and handling.

Roadmap
9

 Brief tour of the website
 Example 1: First steps with PiCloud
 Using the web and console interface
 Example 2: Monte Carlo PI
 Example 3: Basic twitter analyzer
 Conclusion


Example 1: First steps
10

 Run a simple function on the cloud
>>> def func(): # define the function
>>> return 3*3
>>>
>>> import cloud # import our library
>>> cloud.setkey(API_KEY, API_SECRET_KEY) # set your key (can do in cfgs)
>>> jid = cloud.call(func) # returns job id
>>> cloud.status(jid) # returns job status
‘processing’
‘done’
>>> jid = cloud.result(jid) # returns the result
9


11

 3 Simple Steps
 Import our library
 Set your api key (this is how we identify you)

 Pass your function into cloud.call

 That’s all you need to run any function on the cloud!
 Your function can do anything that Python allows
you to do. (except open listening sockets)


If you’re trying to following along…
12

 But you don’t have an API Key yet, try this:
import cloud
cloud.start_simulator()
# do not call cloud.setkey


Web interface
13

 http://www.picloud.com/accounts/
 Manage API Keys
 View functions/jobs
 Status(queued, processing, done)
 Resource usage (profiling)

 Track compute usage with analytics
 Python C-extensions support
 Documentation


14

 What if there’s an exception?
>>> def func(): # define the function
>>> return ‘a’ + 3
>>>
>>> import cloud # import our library
>>> cloud.setkey(API_KEY, API_SECRET_KEY) # set your key (can do in cfgs)
>>> jid = cloud.call(func) # returns job id
‘error’
>>> cloud.result(jid) # returns job status
cloud.cloud.CloudException: Job 7199: Traceback (most recent call last):
File "<stdin>", line 2, in func
TypeError: cannot concatenate 'str' and 'int' objects


Under the hood
15

Application Load Workers
Using PiCloud balancers, (compute
schedulers servers)

Cloud Library PiCloud
Identifies and Transmits data Interface
and source dependencies

Distributed Distributed
Database File System,
Monitoring DHT
Analyze workload

You

Under the hood
16

 How does our advanced system translate over to
tangible benefits for you?
 Robust

 Scalable

 Secure

 Multi-tenant
 Higher utilization -> cheaper
 Optimized

 Easy maintenance


Using the simulator
17

 Motivation
 Copying the state of your Python interpreter to PiCloud
can be costly, and should be optimized to a minimum
 We’ll warn you automatically if you are sending over 1MB
of information
 Inmany cases, faster to test things locally (and there
are no charges!)
 How do I use it?
cloud.start_simulator()


Example 2: Monte Carlo π
18

 Estimate PI using the Monte Carlo method
 Randomly throw darts at the board below

4 * (num _ darts _ landing _ in _ circle )
PI 
total _ darts _ thrown

 Derived from
r 2 
P (dart _ in _ circle )  
(2 r ) 2
4


Example 2: Monte Carlo PI
19

 Map a function across a series of data points
def monteCarlo(num_test):
"""
Throw num_test darts at a 1x1 rectangle
Return how many appear within the quarter circle
"""
numInCircle = 0
y = random.random()

for _ in xrange(num_test):
x = random.random()
y = random.random()
if x*x + y*y < 1.0:
numInCircle += 1

return numInCircle


20

 1 line modification to run this on the cloud
import random
import cloud

numMaps = 6
numTests = 10000000

def calcPi():

"""Send off numMaps monteCarlo simulations in parallel"""
jids = cloud.map(monteCarlo, [numTests for _ in range(numMaps)], _high_cpu=True)
numInCircleResults = cloud.result(jids)
numInCircle = sum(numInCircleResults)

pi = (4 * numInCircle) / float(numTests* numMaps)
return pi

if __name__ == '__main__':
pi = calcPi()

print 'Pi determined to be %s' % pi

21

 How much did this cost?
 Use cloud.info to get cpu time used (runtime)
 Cost = (cpu time) * rate

 Cost = 60 seconds * 0.0013 per minute = $0.0013


22

 What was the _high_cpu attribute for?
jids = cloud.map(monteCarlo, [numTests for _ in range(numMaps)], _high_cpu=True)

 Automatically run your code with more CPU power
 Default is 1 CPU unit (1-1.2ghz Xeon processor)
 _high_cpu is 2.5 CPU units


Example 3: Twitter Analyzer
23

 Determine relationships between followers, #
following, and statuses using linear regression
 We use cloud.map to scrape twitter for user data
 Run linear regressions in parallel using
cloud.call
 Graph statuses/followers with pylab
 NOTE: To use this example, you will need to modify
the posted file to use your own Twitter account
information.


Conclusion
24

 You can now leverage the compute power of a
cluster of servers with only a couple lines of code!
 We’ve given you a taste of the features PiCloud has
to offer, but there are many more!
 If you have not already, register at www.picloud.com
 Questions? ken@picloud.com


February EPD Webinar: How do I...use PiCloud for cloud computing?

More Related Content

Similar to February EPD Webinar: How do I...use PiCloud for cloud computing?

More from Enthought, Inc.

Recently uploaded

February EPD Webinar: How do I...use PiCloud for cloud computing?