Day of Cloud: Amazon EC2

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

more share options
Embed For WordPress.com

Without related presentations

lots of examples at http://code.google.com/p/boto/wiki/BotoS3Examples My own S3 backup script: #!/usr/bin/env pythonimport osimport botofrom boto.s3.key import Keyc = boto.connect_s3()b = c.get_bucket('mcavoy-photos')import dbmdef backup(): db = dbm.open('picture_backup_log', 'c') os.chdir('/Users/cmcavoy/Pictures/iPhoto Library') for dirpath, dirnames, filenames in os.walk('Data'): for f in filenames: try: if db[f] == 'True': print "%s logged in database, skipping" % f continue except: pass k = Key(b) f_full = os.path.join(dirpath, f) k.key = f_full print "%s %s" % (f_full, k.exists()) if not k.exists(): print "uploading %s" % f_full k.set_contents_from_filename(f_full) print "done uploading %s" % f_full db[f] = 'True' db.close()if __name__ == '__main__': backup()

this will be discussion only, cloudfront is a CDN service that’s attached to S3. It allows you to distribute files from S3 across multiple datacenters, similar to any reasonable CDN service.

will go pretty quickly on this one as well. I wrote a script that moves a mysql database to simpledb ( http://code.activestate.com/recipes/576548/ ): import botoimport MySQLdbfrom MySQLdb import cursorsimport threadingdb_user = 'root'db_name = 'eve'db = MySQLdb.connect(user=db_user,db=db_name,cursorclass=cursors.DictCursor)c = db.cursor()sdb = boto.connect_sdb(access_key, secret_key)def get_or_create_domain(domain): try: d = sdb.get_domain(domain) except boto.exception.SDBResponseError: d = sdb.create_domain(domain) return ddef get_primary_key(table_name, cursor): """ Returns a dictionary of fieldname -> infodict for the given table, where each infodict is in the format: {'primary_key': boolean representing whether it's the primary key, 'unique': boolean representing whether it's a unique index} """ cursor.execute("SHOW INDEX FROM %s" % table_name) indexes = {} for row in cursor.fetchall(): if row['Key_name'] == 'PRIMARY': return row['Column_name'] raise("Table %s does not have a primary key" % table_name)class BotoWorker(threading.Thread): def __init__(self, name, record, domain): self.domain = domain self.name = name self.record = record threading.Thread.__init__(self) def run(self): print "inserting %s into %s" % (self.name, self.domain) item = self.domain.new_item(self.name) for key, value in self.record.items(): try: item[key] = value except UnicodeDecodeError: item[key] = 'unicode error'def main(): c.execute("show tables like 'invtypes';") for table in c.fetchall(): print table table = table["Tables_in_%s (invtypes)" % db_name] print "loading data from %s" % table total = c.execute("select * from %s" % table) print "fetched %s items from mysql" % total complete = 0 for record in c.fetchall(): name = record.pop(get_primary_key(table, c)) thread_started = False while not thread_started: if threading.activeCount() < 30: print "got a thread %s" % threading.activeCount() BotoWorker(name=name, record=record, domain=get_or_create_domain(table)).start() thread_started = True complete += 1 print "%s complete of %s" % (complete, total)if __name__ == '__main__': main()

Another quick one. I’ll use examples from the boto documentation.

The bulk of the presentation is here.

boto is a python library wrapper for amazon web services, including cloud web services. All the examples we’ll look at today are written in boto.

Favorites, Groups & Events

Day of Cloud: Amazon EC2 - Presentation Transcript

Amazon Cloud Services
- Chris McAvoy
- Vice-President, Open Source
- PSC Group LLC
Agenda
- Services Overview
- Case Studies
- Techniques
- Questions
We’ll talk about
- API & Python boto library
- S3 & CloudFront
- SimpleDB (briefly)
- Simple Queue Service (briefly)
- Elastic Compute Cloud
We won’t talk about
- e-commerce / fulfillment API
- Elastic MapReduce
- Mechanical Turk
Services
S3
- Storage
- Wildly popular
- Both secure and public storage
- Lots of support (django pluggable storage)
S3 Pricing
CloudFront
- Content Distribution Network (CDN)
- Allows distribution of S3 hosted files
SimpleDB
- No-SQL (but, a SQL like interface)
- Erlang
- Schema-less
Simple Queue Service (SQS)
- Hard to find examples of people using it
EC2
- By far the most exciting Amazon cloud service
- Windows & Linux on-demand servers
- Support from IBM, Microsoft, Sun, Redhat, Oracle
- Virtual Private Cloud
Amazon Machine Image
- Not persistent!
- Windows / Linux
- Most flavors of Linux
- Use prebuilt images, or build your own
- Many images supported by large vendors
EC2 Pricing Small 1.7 gb / 1cpu $0.10 $72 Large 7.5 gb / 4 cpu $0.40 $288 X Large 15gb / 8 cpu $0.80 $576 High CPU 1.7 gb / 5 cpu $0.20 $144 High CPU X Large 7 gb / 20 cpu $0.80 $576
Elastic IP
- Static IP
- Elastic sounds cloudier than static
Elastic Block Store
- Persistent disk
- Again, cloudy sounding
Elastic Load Balancer
- Load balancer
- You can’t associate elastic ip’s with the load balancer
boto
- code.google.com/p/boto
- Python based library for accessing AWS services
- Well maintained, easy to use
Case Studies
Crains
Crains
Social Web Manager
Techniques
Amazon Web Console
RightScale
Roll Your Own
Questions?
- [email_address]
- http://www.psclistens.com
- http://weblog.lonelylion.com
- http://github.com/cmcavoy/Day-of-Cloud-AWS-Examples ( http://bit.ly/rn4GP )
- @cmcavoy