MetaCDN: Creating a ‘Cloud
 Storage’ Mashup for High
  Performance, Low Cost
     Content Delivery
Dr. James Broberg (brob...
Content Delivery
   Networks (CDNs)
• What is a CDN?
 • Content Delivery Networks (CDNs) such as
    Akamai, Mirror Image ...
Existing CDN
          providers
• Akamai is the clear leader in coverage and
  market share (approx. 80%)
  • Price is pr...
Storage Clouds
           ‘Storage as a Service’

 Now!



Late ‘08


 ??????
SaaS Comparison
Introducing MetaCDN
• What if we could create a low-cost, high
  performance overlay CDN using these
  storage clouds?
  •...
How MetaCDN works
        AmazonS3Connector                                                          NirvanixConnector
cre...
How MetaCDN works
    Amazon S3 &                                            Mosso Cloud                                  ...
MetaCDN Manager
• The MetaCDN Manager ensures that:
 • All current deployments are meeting
    QoS targets (where applicab...
MetaCDN QoS
         Monitor
• The MetaCDN QoS Monitor:
 • Tracks the performance of participating
    providers at all ti...
MetaCDN Allocator
• The MetaCDN Allocator allows users to
  deploy files either directly or from a public
  origin URL, wit...
MetaCDN Database

MetaCDN   1              0:M      CDN        1            1    CDN
                has                  ...
MetaCDN Web Portal
• Developed using Java Enterprise and Java
  Server Faces (JSF) technologies
• MySQL back-end to store ...
MetaCDN Web Service
• Makes all functionality available via Web
  Services (SOAP & REST/HTTP)
  • Web interface is useful ...
MetaCDN Load
          Redirector
• We have created a unified namespace to
  simply deployment, routing, management
• Curre...
MetaCDN Load
             Redirector (cont.)
MetaCDN end-user                                        DNS Server           ...
Redirector Overhead
          1.8
               from USA
                 S3 USA
                MetaCDN
          1.6


...
Redirector Overhead
             from Australia
           0.9
                                          Nirvanix SDN #3
 ...
Evaluating MetaCDN
     performance
• Ran tests over 24 hour period (mid-week)
• Downloading test replicas (1KB, 10MB) 30
...
Summary of Results -
           Throughput (KB/s)
                S3    S3   SDN   SDN   SDN   SDN   Coral
               ...
Summary of results -
          Response Time (Sec)
                S3    S3   SDN   SDN    SDN   SDN   Coral
             ...
Summary of results
       (cont)
• Clients benefited greatly from local replicas
• Results are consistent in terms of respo...
MetaCDN features in
planning / development
• Support as many providers as possible
 • Ensures redundancy, reliability, cov...
MetaCDN features in
planning / development
• Create Apache module (mod_metacdn) to
  utilise at content origin sites
• Mod...
Collaborations
• Always looking for people to collaborate
  on the project
• Work to be done on:
 • Load balancing / redir...
Acknowledgements

• Thanks to reviewers for useful feedback.
• Australian Research Council (ARC) for
  funding the project...
Thanks
Latest information will be available at:
www.metacdn.org


International Workshop on Cloud
Computing (Cloud 2009):
...
MetaCDN
MetaCDN
MetaCDN
MetaCDN
Upcoming SlideShare
Loading in …5
×

MetaCDN

1,856 views

Published on

by James Broberg - Presentation given at the 2nd International Workshop on Web APIs and Mashups (at ICSOC2008) on December 1st, 2008 in Sydney, Australia. http://www.icsoc-mashups.org/

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,856
On SlideShare
0
From Embeds
0
Number of Embeds
107
Actions
Shares
0
Downloads
65
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

MetaCDN

  1. 1. MetaCDN: Creating a ‘Cloud Storage’ Mashup for High Performance, Low Cost Content Delivery Dr. James Broberg (brobergj@csse.unimelb.edu.au) http://www.csse.unimelb.edu.au/~brobergj http://www.metacdn.org
  2. 2. Content Delivery Networks (CDNs) • What is a CDN? • Content Delivery Networks (CDNs) such as Akamai, Mirror Image and Limelight place web server clusters in numerous geographical locations to improve the responsiveness and locality of the content it hosts for end-users.
  3. 3. Existing CDN providers • Akamai is the clear leader in coverage and market share (approx. 80%) • Price is prohibitive for SME, NGO, Gov... • Anecdotally 2−15 times more expensive than Cloud Storage, and require 1−2 year commitments and min. data use (10TB+) • Academic CDNs include Coral, Codeen, Globule.... • No SLA / QoS provided, only ‘best effort’
  4. 4. Storage Clouds ‘Storage as a Service’ Now! Late ‘08 ??????
  5. 5. SaaS Comparison
  6. 6. Introducing MetaCDN • What if we could create a low-cost, high performance overlay CDN using these storage clouds? • Exploit the strengths and coverage footprints of multiple providers • Cherry pick providers based on customers QoS needs / cost budget • MetaCDN provides this while hiding the complexity from users
  7. 7. How MetaCDN works AmazonS3Connector NirvanixConnector createFolder(foldername, location) DefaultConnector createFolder(foldername, location) deleteFolder(foldername) DEPLOY_USA deleteFolder(foldername) createFile(file, foldername, DEPLOY_EU createFile(file, foldername, location, date) DEPLOY_ASIA location, date) createFile(fileURL, foldername, DEPLOY_AUS createFile(fileURL, foldername, location, date) createFolder(foldername, location) location, date) renameFile(filename, newname, deleteFolder(foldername) renameFile(filename, newname, location) throws createFile(file, foldername, location) FeatureNotSupportedException location, date) createTorrent(file) throws createTorrent(file) createFile(fileURL, foldername, FeatureNotSupportedException createTorrent(fileURL) location, date) createTorrent(fileURL) throws deleteFile(file, location) renameFile(filename, newname, FeatureNotSupportedException listFilesAndFolders() location) deleteFile(file, location) deleteFilesAndFolders() createTorrent(file) listFilesAndFolders() deleteFilesAndFolders() createTorrent(fileURL) <<exception>> deleteFile(file, location) FeatureNotSupportedException listFilesAndFolders() FeatureNotSupportedException(msg) deleteFilesAndFolders()
  8. 8. How MetaCDN works Amazon S3 & Mosso Cloud Shared/Private Nirvanix SDN Coral CDN CloudFront Files CDN Host WebDAVConnector JetS3t toolkit Nirvanix SDK Cloud Files SDK Java stub Java SDK Java SDK Java SDK MetaCDN.org Open Source Nirvanix, Inc Mosso, Inc SCPConnector Java stub MetaCDN.org AmazonS3Connector NirvanixConnector CloudFilesConnector CoralConnector FTPConnector Java stub Java stub Java stub Java stub Java stub MetaCDN.org MetaCDN.org MetaCDN.org MetaCDN.org MetaCDN.org MetaCDN MetaCDN MetaCDN QoS MetaCDN MetaCDN Manager Monitor Allocator Database Web Portal Load Redirector Web Service Java (JSF/EJB) based portal Random redirection SOAP Web Service Support HTTP POST Geographical redirection RESTful Web Service New/view/modify deployment Least cost redirection Programmatic access
  9. 9. MetaCDN Manager • The MetaCDN Manager ensures that: • All current deployments are meeting QoS targets (where applicable) • Replicas are removed when no longer required (minimising cost) • A users’ budget has not been exceeded, by tracking usage (i.e. storage/download)
  10. 10. MetaCDN QoS Monitor • The MetaCDN QoS Monitor: • Tracks the performance of participating providers at all times • Monitors and records throughput and response time from a variety of locations • Ensures that upstream providers are meeting their Service Level Agreements
  11. 11. MetaCDN Allocator • The MetaCDN Allocator allows users to deploy files either directly or from a public origin URL, with the following options: • Maximise coverage and performance • Deploy content in specific locations • Cost optimised deployment • Quality of Service (QoS) optimised
  12. 12. MetaCDN Database MetaCDN 1 0:M CDN 1 1 CDN has for User Credentials Provider 1 1 1 hosted has has by M 1 M 1 deployed 0:M MetaCDN Coverage Content as Replica locations
  13. 13. MetaCDN Web Portal • Developed using Java Enterprise and Java Server Faces (JSF) technologies • MySQL back-end to store persistent data • Web portal acts as the entry point to the system and application-level load balancer • Most suited for small or ad-hoc deployments, and especially useful for less technically inclined content creators.
  14. 14. MetaCDN Web Service • Makes all functionality available via Web Services (SOAP & REST/HTTP) • Web interface is useful for novices and for ad-hoc deployments, but doesn’t scale • Larger customers have 1,000’s - 10,000 - 100,000s of files that need deployment • Let them automate their deployment and management via Web Services! • Perfect for Mashup developers!!!
  15. 15. MetaCDN Load Redirector • We have created a unified namespace to simply deployment, routing, management • Currently, file deployment results in multiple URLs, each mapping to a replica • http://metacdn-eu-user.s3.amazonaws.com/myfile.mp4 • http://metacdn-us-user.s3.amazonaws.com/myfile.mp4 • http://node3.nirvanix.com/MetaCDN/user/myfile.mp4 • Single namespace is created for fine control • http://www.metacdn.org/FileMapper?itemid=2
  16. 16. MetaCDN Load Redirector (cont.) MetaCDN end-user DNS Server MetaCDN gateway Resolve www.metacdn.org Return IP of closest MetaCDN gateway, www-na.metacdn.org GET http://metacdn.org/MetaCDN/FileMapper?itemid=1 processRequest () geoRedirect () HTTP 302 Redirect to http://metacdn-us-username.s3.amazonaws.com/filename.pdf Resolve metacdn-us-username.s3.amazonaws.com Amazon S3 USA Return IP of metacdn-us-username.s3.amazonaws.com GET http://metacdn-us-username.s3.amazonaws.com/filename.pdf Return replica
  17. 17. Redirector Overhead 1.8 from USA S3 USA MetaCDN 1.6 1.4 1.2 1 Seconds 0.8 0.6 0.4 0.2 0 0 5 10 15 20 25 Hour
  18. 18. Redirector Overhead from Australia 0.9 Nirvanix SDN #3 MetaCDN 0.85 0.8 0.75 Seconds 0.7 0.65 0.6 0.55 0 5 10 15 20 25 Hour
  19. 19. Evaluating MetaCDN performance • Ran tests over 24 hour period (mid-week) • Downloading test replicas (1KB, 10MB) 30 times per hour, take average and conf. inter. • 10MB - Throughput, 1KB - Response Time • Ran test from 6 locations: Melbourne (AUS), Paris (FRA),Vienna (AUT), San Diego & New Jersey (USA), Seoul (KOR) • Replicas located across US, EU, ASIA, AUS
  20. 20. Summary of Results - Throughput (KB/s) S3 S3 SDN SDN SDN SDN Coral US EU #1 #2 #3 #4 Melbourne Australia 264.3 389.1 30 366.8 408.5 405.6 173.7 Paris France 703.1 2116 483.8 2948 416.8 1043 530.2 Vienna Austria 490.7 1347 288.4 2271 211 538.7 453.4 Seoul South Korea 312.8 376.1 466.5 411.9 2456 588.2 152 San Diego USA 1234 323.5 5946 380.1 506.1 820.4 338.5 New Jersey USA 2381 1949 860.8 967.1 572.8 4230 636.4
  21. 21. Summary of results - Response Time (Sec) S3 S3 SDN SDN SDN SDN Coral US EU #1 #2 #3 #4 Melbourne Australia 1.378 1.458 0.663 0.703 1.195 0.816 5.452 Paris France 0.533 0.2 0.538 0.099 1.078 0.316 3.11 Vienna Austria 0.723 0.442 0.585 0.099 1.088 0.406 3.171 Seoul South Korea 1.135 1.21 0.856 0.896 1 0.848 3.318 San Diego USA 0.232 0.455 0.23 0.361 0.775 0.319 4.655 New Jersey USA 0.532 0.491 0.621 0.475 1.263 0.516 1.916
  22. 22. Summary of results (cont) • Clients benefited greatly from local replicas • Results are consistent in terms of response time and throughput with previous studies of dedicated CDNs • Back-end providers have sufficient performance and reliability to be used to host replicas in the MetaCDN system
  23. 23. MetaCDN features in planning / development • Support as many providers as possible • Ensures redundancy, reliability, coverage and choice (reduces single vendor lock- in) • Windows Azure Storage Service support will be added ASAP. • Other storage & delivery services can be rapidly added as they are released....
  24. 24. MetaCDN features in planning / development • Create Apache module (mod_metacdn) to utilise at content origin sites • Module would utilise MetaCDN web services to maintain optimal deployment • Maximise performance during peak load via intelligent replica deployment • Minimise storage cost during periods of low demand
  25. 25. Collaborations • Always looking for people to collaborate on the project • Work to be done on: • Load balancing / redirection algorithms • Intelligent Caching / replication algorithms • Security / Access Control of content • Improving MetaCDN Web Services • Please contact me if you are interested...
  26. 26. Acknowledgements • Thanks to reviewers for useful feedback. • Australian Research Council (ARC) for funding the project. • Cory & Barry (Nirvanix) and Eric (Rackspace/Mosso) for development support.
  27. 27. Thanks Latest information will be available at: www.metacdn.org International Workshop on Cloud Computing (Cloud 2009): http://www.gridbus.org/cloud2009

×