Day of Cloud: Amazon EC2


Published on

One of five presentations at Chicago's Day of Cloud mini-conference. Chris McAvoy ( demonstrates cloud computing with Amazon services.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • lots of examples at My own S3 backup script: #!/usr/bin/env pythonimport osimport botofrom boto.s3.key import Keyc = boto.connect_s3()b = c.get_bucket('mcavoy-photos')import dbmdef backup(): db ='picture_backup_log', 'c') os.chdir('/Users/cmcavoy/Pictures/iPhoto Library') for dirpath, dirnames, filenames in os.walk('Data'): for f in filenames: try: if db[f] == 'True': print "%s logged in database, skipping" % f continue except: pass k = Key(b) f_full = os.path.join(dirpath, f) k.key = f_full print "%s %s" % (f_full, k.exists()) if not k.exists(): print "uploading %s" % f_full k.set_contents_from_filename(f_full) print "done uploading %s" % f_full db[f] = 'True' db.close()if __name__ == '__main__': backup()
  • this will be discussion only, cloudfront is a CDN service that’s attached to S3. It allows you to distribute files from S3 across multiple datacenters, similar to any reasonable CDN service.
  • will go pretty quickly on this one as well. I wrote a script that moves a mysql database to simpledb ( ): import botoimport MySQLdbfrom MySQLdb import cursorsimport threadingdb_user = 'root'db_name = 'eve'db = MySQLdb.connect(user=db_user,db=db_name,cursorclass=cursors.DictCursor)c = db.cursor()sdb = boto.connect_sdb(access_key, secret_key)def get_or_create_domain(domain): try: d = sdb.get_domain(domain) except boto.exception.SDBResponseError: d = sdb.create_domain(domain) return ddef get_primary_key(table_name, cursor): """ Returns a dictionary of fieldname -> infodict for the given table, where each infodict is in the format: {'primary_key': boolean representing whether it's the primary key, 'unique': boolean representing whether it's a unique index} """ cursor.execute("SHOW INDEX FROM %s" % table_name) indexes = {} for row in cursor.fetchall(): if row['Key_name'] == 'PRIMARY': return row['Column_name'] raise("Table %s does not have a primary key" % table_name)class BotoWorker(threading.Thread): def __init__(self, name, record, domain): self.domain = domain = name self.record = record threading.Thread.__init__(self) def run(self): print "inserting %s into %s" % (, self.domain) item = self.domain.new_item( for key, value in self.record.items(): try: item[key] = value except UnicodeDecodeError: item[key] = 'unicode error'def main(): c.execute("show tables like 'invtypes';") for table in c.fetchall(): print table table = table["Tables_in_%s (invtypes)" % db_name] print "loading data from %s" % table total = c.execute("select * from %s" % table) print "fetched %s items from mysql" % total complete = 0 for record in c.fetchall(): name = record.pop(get_primary_key(table, c)) thread_started = False while not thread_started: if threading.activeCount() < 30: print "got a thread %s" % threading.activeCount() BotoWorker(name=name, record=record, domain=get_or_create_domain(table)).start() thread_started = True complete += 1 print "%s complete of %s" % (complete, total)if __name__ == '__main__': main()
  • Another quick one. I’ll use examples from the boto documentation.
  • The bulk of the presentation is here.
  • boto is a python library wrapper for amazon web services, including cloud web services. All the examples we’ll look at today are written in boto.
  • Day of Cloud: Amazon EC2

    1. 1. Amazon Cloud Services <ul><li>Chris McAvoy </li></ul><ul><li>Vice-President, Open Source </li></ul><ul><li>PSC Group LLC </li></ul>
    2. 2. Agenda <ul><li>Services Overview </li></ul><ul><li>Case Studies </li></ul><ul><li>Techniques </li></ul><ul><li>Questions </li></ul>
    3. 3. We’ll talk about <ul><li>API & Python boto library </li></ul><ul><li>S3 & CloudFront </li></ul><ul><li>SimpleDB (briefly) </li></ul><ul><li>Simple Queue Service (briefly) </li></ul><ul><li>Elastic Compute Cloud </li></ul>
    4. 4. We won’t talk about <ul><li>e-commerce / fulfillment API </li></ul><ul><li>Elastic MapReduce </li></ul><ul><li>Mechanical Turk </li></ul>
    5. 5. Services
    6. 6. S3 <ul><li>Storage </li></ul><ul><li>Wildly popular </li></ul><ul><li>Both secure and public storage </li></ul><ul><li>Lots of support (django pluggable storage) </li></ul>
    7. 7. S3 Pricing
    8. 8. CloudFront <ul><li>Content Distribution Network (CDN) </li></ul><ul><li>Allows distribution of S3 hosted files </li></ul>
    9. 9. SimpleDB <ul><li>No-SQL (but, a SQL like interface) </li></ul><ul><li>Erlang </li></ul><ul><li>Schema-less </li></ul>
    10. 10. Simple Queue Service (SQS) <ul><li>Hard to find examples of people using it </li></ul>
    11. 11. EC2 <ul><li>By far the most exciting Amazon cloud service </li></ul><ul><li>Windows & Linux on-demand servers </li></ul><ul><li>Support from IBM, Microsoft, Sun, Redhat, Oracle </li></ul><ul><li>Virtual Private Cloud </li></ul>
    12. 12. Amazon Machine Image <ul><li>Not persistent! </li></ul><ul><li>Windows / Linux </li></ul><ul><li>Most flavors of Linux </li></ul><ul><li>Use prebuilt images, or build your own </li></ul><ul><li>Many images supported by large vendors </li></ul>
    13. 13. EC2 Pricing Small 1.7 gb / 1cpu $0.10 $72 Large 7.5 gb / 4 cpu $0.40 $288 X Large 15gb / 8 cpu $0.80 $576 High CPU 1.7 gb / 5 cpu $0.20 $144 High CPU X Large 7 gb / 20 cpu $0.80 $576
    14. 14. Elastic IP <ul><li>Static IP </li></ul><ul><li>Elastic sounds cloudier than static </li></ul>
    15. 15. Elastic Block Store <ul><li>Persistent disk </li></ul><ul><li>Again, cloudy sounding </li></ul>
    16. 16. Elastic Load Balancer <ul><li>Load balancer </li></ul><ul><li>You can’t associate elastic ip’s with the load balancer </li></ul>
    17. 17. boto <ul><li> </li></ul><ul><li>Python based library for accessing AWS services </li></ul><ul><li>Well maintained, easy to use </li></ul>
    18. 18. Case Studies
    19. 19. Crains
    20. 20. Crains
    21. 21. Social Web Manager
    22. 22. Techniques
    23. 23. Amazon Web Console
    24. 24. RightScale
    25. 25. Roll Your Own
    26. 26. Questions? <ul><li>[email_address] </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> ( ) </li></ul><ul><li>@cmcavoy </li></ul>