Day of Cloud: Amazon EC2
Upcoming SlideShare
Loading in...5
×
 

Day of Cloud: Amazon EC2

on

  • 4,705 views

One of five presentations at Chicago's Day of Cloud mini-conference. Chris McAvoy (http://www.psclistens.com) demonstrates cloud computing with Amazon services.

One of five presentations at Chicago's Day of Cloud mini-conference. Chris McAvoy (http://www.psclistens.com) demonstrates cloud computing with Amazon services.

Statistics

Views

Total Views
4,705
Views on SlideShare
4,419
Embed Views
286

Actions

Likes
0
Downloads
77
Comments
0

11 Embeds 286

http://www.psclistens.com 140
http://weblog.lonelylion.com 90
http://psclistens.com 23
http://www.slideshare.net 14
http://pscgroup.com 6
http://test.psclistens.com 4
http://www.psctou.com 3
http://www.pscgroup.com 3
http://www.lmodules.com 1
http://www.psclisten.com 1
http://pkreis 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • lots of examples at http://code.google.com/p/boto/wiki/BotoS3Examples My own S3 backup script: #!/usr/bin/env pythonimport osimport botofrom boto.s3.key import Keyc = boto.connect_s3()b = c.get_bucket('mcavoy-photos')import dbmdef backup(): db = dbm.open('picture_backup_log', 'c') os.chdir('/Users/cmcavoy/Pictures/iPhoto Library') for dirpath, dirnames, filenames in os.walk('Data'): for f in filenames: try: if db[f] == 'True': print "%s logged in database, skipping" % f continue except: pass k = Key(b) f_full = os.path.join(dirpath, f) k.key = f_full print "%s %s" % (f_full, k.exists()) if not k.exists(): print "uploading %s" % f_full k.set_contents_from_filename(f_full) print "done uploading %s" % f_full db[f] = 'True' db.close()if __name__ == '__main__': backup()
  • this will be discussion only, cloudfront is a CDN service that’s attached to S3. It allows you to distribute files from S3 across multiple datacenters, similar to any reasonable CDN service.
  • will go pretty quickly on this one as well. I wrote a script that moves a mysql database to simpledb ( http://code.activestate.com/recipes/576548/ ): import botoimport MySQLdbfrom MySQLdb import cursorsimport threadingdb_user = 'root'db_name = 'eve'db = MySQLdb.connect(user=db_user,db=db_name,cursorclass=cursors.DictCursor)c = db.cursor()sdb = boto.connect_sdb(access_key, secret_key)def get_or_create_domain(domain): try: d = sdb.get_domain(domain) except boto.exception.SDBResponseError: d = sdb.create_domain(domain) return ddef get_primary_key(table_name, cursor): """ Returns a dictionary of fieldname -> infodict for the given table, where each infodict is in the format: {'primary_key': boolean representing whether it's the primary key, 'unique': boolean representing whether it's a unique index} """ cursor.execute("SHOW INDEX FROM %s" % table_name) indexes = {} for row in cursor.fetchall(): if row['Key_name'] == 'PRIMARY': return row['Column_name'] raise("Table %s does not have a primary key" % table_name)class BotoWorker(threading.Thread): def __init__(self, name, record, domain): self.domain = domain self.name = name self.record = record threading.Thread.__init__(self) def run(self): print "inserting %s into %s" % (self.name, self.domain) item = self.domain.new_item(self.name) for key, value in self.record.items(): try: item[key] = value except UnicodeDecodeError: item[key] = 'unicode error'def main(): c.execute("show tables like 'invtypes';") for table in c.fetchall(): print table table = table["Tables_in_%s (invtypes)" % db_name] print "loading data from %s" % table total = c.execute("select * from %s" % table) print "fetched %s items from mysql" % total complete = 0 for record in c.fetchall(): name = record.pop(get_primary_key(table, c)) thread_started = False while not thread_started: if threading.activeCount() < 30: print "got a thread %s" % threading.activeCount() BotoWorker(name=name, record=record, domain=get_or_create_domain(table)).start() thread_started = True complete += 1 print "%s complete of %s" % (complete, total)if __name__ == '__main__': main()
  • Another quick one. I’ll use examples from the boto documentation.
  • The bulk of the presentation is here.
  • boto is a python library wrapper for amazon web services, including cloud web services. All the examples we’ll look at today are written in boto.

Day of Cloud: Amazon EC2 Day of Cloud: Amazon EC2 Presentation Transcript

  • Amazon Cloud Services
    • Chris McAvoy
    • Vice-President, Open Source
    • PSC Group LLC
  • Agenda
    • Services Overview
    • Case Studies
    • Techniques
    • Questions
  • We’ll talk about
    • API & Python boto library
    • S3 & CloudFront
    • SimpleDB (briefly)
    • Simple Queue Service (briefly)
    • Elastic Compute Cloud
    View slide
  • We won’t talk about
    • e-commerce / fulfillment API
    • Elastic MapReduce
    • Mechanical Turk
    View slide
  • Services
  • S3
    • Storage
    • Wildly popular
    • Both secure and public storage
    • Lots of support (django pluggable storage)
  • S3 Pricing
  • CloudFront
    • Content Distribution Network (CDN)
    • Allows distribution of S3 hosted files
  • SimpleDB
    • No-SQL (but, a SQL like interface)
    • Erlang
    • Schema-less
  • Simple Queue Service (SQS)
    • Hard to find examples of people using it
  • EC2
    • By far the most exciting Amazon cloud service
    • Windows & Linux on-demand servers
    • Support from IBM, Microsoft, Sun, Redhat, Oracle
    • Virtual Private Cloud
  • Amazon Machine Image
    • Not persistent!
    • Windows / Linux
    • Most flavors of Linux
    • Use prebuilt images, or build your own
    • Many images supported by large vendors
  • EC2 Pricing Small 1.7 gb / 1cpu $0.10 $72 Large 7.5 gb / 4 cpu $0.40 $288 X Large 15gb / 8 cpu $0.80 $576 High CPU 1.7 gb / 5 cpu $0.20 $144 High CPU X Large 7 gb / 20 cpu $0.80 $576
  • Elastic IP
    • Static IP
    • Elastic sounds cloudier than static
  • Elastic Block Store
    • Persistent disk
    • Again, cloudy sounding
  • Elastic Load Balancer
    • Load balancer
    • You can’t associate elastic ip’s with the load balancer
  • boto
    • code.google.com/p/boto
    • Python based library for accessing AWS services
    • Well maintained, easy to use
  • Case Studies
  • Crains
  • Crains
  • Social Web Manager
  • Techniques
  • Amazon Web Console
  • RightScale
  • Roll Your Own
  • Questions?
    • [email_address]
    • http://www.psclistens.com
    • http://weblog.lonelylion.com
    • http://github.com/cmcavoy/Day-of-Cloud-AWS-Examples ( http://bit.ly/rn4GP )
    • @cmcavoy