Your SlideShare is downloading. ×
Rackspace & Akamai vs. Amazon & CloudFront for a Django site
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Rackspace & Akamai vs. Amazon & CloudFront for a Django site

3,707
views

Published on

Comparison of the 2 major cloud providers for storing and serving static files and the challenges for a Django site. …

Comparison of the 2 major cloud providers for storing and serving static files and the challenges for a Django site.
LA Django Nov 2013


0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,707
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
34
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Moving  to  the  Clouds:   Amazon  S3  &  Cloud  Front   vs.  Rackspace  Cloud  Files  &  Akamai     LA  Django Nov  19  2013 Sep  Dehpour www.linkedin.com/in/sepehr www.github.com/erasmose  
  • 2. About  the  presenter Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Sep  Dehpour  is  the  primary  engineer  on  a  Django  site  that  has  over   1.5  million  page  views  per  day.   His  responsibilities  range  from  systems  administration,  Chef   DevOps,  performance  optimizations  to  new  feature  development.   Sep  is  an  avid  skier  and  likes  to  think  he  is  a  good  surfer  too,  but   that  shit  ain't  easy.
  • 3. Rackspace  &  Akamai  vs.  Amazon  &  CloudFront Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 In  this  talk:  Comparison  of  the  2  major  cloud  providers   for  storing  and  serving  sta1c  files  and  the  challenges   for  a  Django  site.
  • 4. Tradi1onal Serving  Sta1c  files Cloud Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End CDN Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Nginx/Apache Origin
  • 5. Storing  &  Serving  Sta1c  files   Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End User CDN Origin Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13
  • 6. Comparison  in  glance Intro Tradi1onal  vs.  Cloud Pay as you Rackspace  vs.  Amazon Akamai Akamai CDN s3  objects Tools Store files End grow from 10¢/GB/Mo Pay as you grow from 9.5¢/GB/Mo at 12¢/GB at 213 PoP Cloudfront CDN at 12¢/GB and media of any size Store files and media of any size Highly scalable, redundant Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Highly scalable, redundant
  • 7. CDN  Comparison  in  glance Intro Tradi1onal  vs.  Cloud Akamai is delivering 20-30% of web Rackspace  vs.  Amazon Akamai Apple, Netflix, Amazon.com s3  objects Tools 1,200 points of presence (PoP) End Nasa, PBS 40 super PoP But Rackspace plan uses 213 PoP Akamai NetSession Interface Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 peer-to-peer no peer-to-peer Tip:  Use  hQp://www.cdnplanet.com/tools/cdnfinder    to  find  the  CDN  used  to  serve  sta1c  content. Put  the  full  path  to  an  image  or  CSS  file  to  see  what  CDN  is  serving  it.  
  • 8. CDN  Speed  Comparison  1Mb  object Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13
  • 9. CDN  Speed  Comparison  12Kb  object Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13
  • 10. Akamai  Visualizing hQp://www.akamai.com/html/technology/visualizing_akamai.html Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13
  • 11. So  what’s  exactly  on  S3/Cloud  Files? Container{ Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Containers at the root level holding objects Object { “key”:”path to file to simulate a nested folder structure”, “etag”:MD5 checksum, “content_type”:Mime Type, “ttl”: time to expire on CDN Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 }, ... }
  • 12. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 13. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 14. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 15. CloudFuse Virtual File System Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Unlike  tradi1onal  file  systems  that  essen1ally  save  data  to  and  retrieve   data  from  disk,  virtual  filesystems  do  not  actually  store  data  themselves.   They  act  as  a  view  or  transla1on  of  an  exis1ng  file  system  or  storage   device. S3fs
  • 16. Buggy: CloudFuse Virtual File System Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Unlike  tradi1onal  file  systems  that  essen1ally  save  data  to  and  retrieve   data  from  disk,  virtual  filesystems  do  not  actually  store  data  themselves.   They  act  as  a  view  or  transla1on  of  an  exis1ng  file  system  or  storage   device. S3fs (?)
  • 17. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 18. S3cmd Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Command line http://s3tools.org/s3cmd Run s3cmd --configure Run s3cmd ls to list all your buckets. Make a bucket with s3cmd mb s3://my-new-bucket-name List the contents of the bucket s3cmd ls s3://bucket_name Upload a file into the bucket s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml
  • 19. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 20. Pyrax Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Command line import pyrax pyrax.set_credential_file("/home/user/.pyrax.cfg") cf = pyrax.cloudfiles print "list of containers:", cf.list_containers() cont = cf.get_container("my_container") #uploading a folder local = "path to a local folder" cf.sync_folder_to_container(folder_path=local, container=cont, include_hidden=False)
  • 21. Pyrax Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Command line #uploading file pth = "path to main.css on local" file_name_on_cloud = "static/assets/stylesheets/ main.css" chksum = pyrax.utils.get_checksum(pth) obj = cont.upload_file(pth,obj_name= file_name_on_cloud, etag=chksum) print "Calculated checksum:", chksum print "Stored object etag:", obj.etag
  • 22. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 23. Boto Command line from boto.s3.connection import S3Connection Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End from boto.s3.key import Key import sys conn = S3Connection('AWS credentials', 'AWS credentials') bucket = conn.get_bucket('bucketname') rs = bucket.list("Keys starting with this string") Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 for key in rs: print key.name
  • 24. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 25. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Django (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 26. Django Storages Django #s3utils.py from storages.backends.s3boto import S3BotoStorage class StaticRootS3BotoStorage(S3BotoStorage): location = 'static' Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon class MediaRootS3BotoStorage(S3BotoStorage): Akamai location = 'media' s3  objects ----------------------------------------------------------------------Tools #settings.py End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 DEFAULT_FILE_STORAGE = "my_project.s3utils.MediaRootS3BotoStorage" STATICFILES_STORAGE = "my_project.s3utils. StaticRootS3BotoStorage" AWS_STORAGE_BUCKET_NAME = storage_bucket_name AWS_BUCKET_NAME = storage_bucket_name AWS_S3_CUSTOM_DOMAIN = custom_domain
  • 27. Django Storages Django Always use Django default_storage so you can easily switch the back-end storage Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End from django.core.files.storage import default_storage from django.core.files.base import ContentFile path = default_storage.save('/path/to/file', ContentFile('new content')) path u'/path/to/file' default_storage.size(path) 11 default_storage.open(path).read() 'new content' default_storage.delete(path) Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 default_storage.exists(path) False
  • 28. Django Storages Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Django But the 3rd party app that we were using did custom file writing instead of using Django default_storage
  • 29. Django Storages Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Django And we had a few days till deploying
  • 30. Toolsets/libraries Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Rackspace control panel CloudFuse Swift Client (written in Python) Pyrax Django Cumulus Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 Web Toolkit Virtual File System Command line Python Django Python (other) AWS control panel S3fs S3cmd (written in Python) Boto Django Storages Go s3
  • 31. Go S3 Python Go S3: A light wrapper around Python Boto https://github.com/erasmose/go_s3 Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End When to use Go_s3? You have custom file read/write in your apps or 3rd party apps that are not compatible with “Django Storages” and you don’t have time to go through making it use “Django default storage” so it gets compatible with “Django Storages” or You want to have full control over what goes on S3 and its access control (ACL) Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13
  • 32. Go S3 Python Go S3: A light wrapper around Python Boto https://github.com/erasmose/go_s3 #in  your  django  settings: Intro AWS_ACCESS_KEY_ID  =  "your  AWS_ACCESS_KEY_ID" Tradi1onal  vs.  Cloud AWS_SECRET_ACCESS_KEY  =  "your  AWS_SECRET_ACCESS_KEY" Rackspace  vs.  Amazon AWS_STORAGE_BUCKET_NAME  =  "your  AWS_STORAGE_BUCKET_NAME" -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ Akamai #inside  a  custom  save()  for  a  Django  model: s3  objects Tools go_s3.connect_to_s3() try: End        go_s3.upload_file(file_full_path,  full_path_s3,  acl='private',   del_after_upload=True)        print  go_s3.get_grants(full_path_on_s3) except:        logger.error("Unexpected  error  dealing  with  s3  upload:  ",   exc_info=True)                         Sep Dehpour linkedin.com/in/sepehr finally: github.com/erasmose CAPABILITIES        go_s3.close_connection() Wednesday, November 20, 13
  • 33. Go S3 Python ACL canned access control policy Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13 a.  private:  Owner  gets  FULL_CONTROL.  No  one  else  has  any   access  rights. b.  public-­‐read:  Owners  gets  FULL_CONTROL  and  the   anonymous  principal  is  granted  READ  access. c.  public-­‐read-­‐write:  Owner  gets  FULL_CONTROL  and  the   anonymous  principal  is  granted  READ  and  WRITE  access. d.  authenticated-­‐read:  Owner  gets  FULL_CONTROL  and  any   principal  authenticated  as  a  registered  Amazon  S3  user   is  granted  READ  access
  • 34. Intro Tradi1onal  vs.  Cloud Rackspace  vs.  Amazon Akamai s3  objects Tools End Thanks  to  TrueCar  for  hos1ng LA  Django Nov  19  2013 Sep  Dehpour www.linkedin.com/in/sepehr www.github.com/erasmose   Sep Dehpour linkedin.com/in/sepehr github.com/erasmose CAPABILITIES Wednesday, November 20, 13