A Technical Overview of DuraCloud

3,163 views

Published on

Interested in learning more about the technical architecture of DuraCloud? Curious about how it runs in the cloud? This presentation details the technical underpinnings of DuraCloud.
Presented by:
Bill Branan, DuraCloud Lead Developer, DuraSpace

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,163
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
36
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • The Media Streaming service works the same, but the action occurs in Amazon CloudFront rather than in Elastic MapReduce
  • DurAdmin could be replaced by any application that makes calls to DuraCloud
  • A Technical Overview of DuraCloud

    1. 1. A Technical Overview Bill Branan DuraCloud Technical Lead
    2. 2. Agenda• Introduction• Storage• Services• Reporting• UI• Security• Tools• Roadmap
    3. 3. Introduction• DuraCloud is: – Hosted service • Runs on cloud-based compute systems • Connects to cloud-based storage systems • Provides a service platform – Open source software suite• Webinar Focus: – Components of software platform – Interacting with hosted DuraCloud
    4. 4. DuraCloud in ContextUser Data Center Data File System Repository DuraCloud Management Console • Account Management • Instance Management
    5. 5. DuraCloud Storage Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service ContainerDuraCloud Instance
    6. 6. DuraCloud Storage User Data Center Amazon Storage Amazon S3 Adapter REST API Rackspace Rackspace Storage Storage Adapter Cloud Files Provider Storage Interface Mediation Azure Storage Microsoft Azure Adapter Storage SDSC StorageStorage Management Adapter SDSC Cloud (DuraStore) Storage
    7. 7. Storage REST Interface Space Actions Content Actions• Add Space • Add Content• Get/Set Space Properties • Get/Set Content Properties• Get Spaces List • Get Content• Get Space Content List • Copy Content• Get/Set Space Access • Delete Content• Delete Space Storage Provider Interface Other Actions • Get Stores • Get Tasks List • Perform Task
    8. 8. Storage AdaptersTransform calls from DuraCloud interface intocalls understood by a specific storage provider• Determine how DuraCloud “spaces” translate into top level provider containers (buckets)• Define how space and content properties are stored and retrieved• Manage eventual consistency concerns
    9. 9. DuraCloud Services Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service ContainerDuraCloud Instance
    10. 10. DuraCloud Services Media Streamer Duplicate Duplicate on on Demand Change Bit Bit BitIntegrity Integrity IntegrityChecker Checker Checker Tools Bulk Image Image Transformer Transformer Bulk Image Server … so how are they deployed?
    11. 11. DuraCloud Service Deployment Services Service Management Available? (DuraService) 1 REST API 2 Service List Service Manager Services In Plan? 4 Service Registry Service List 3 Service BundlePart 1: Service Config Ask which services are available Service Plan
    12. 12. DuraCloud Service Deployment Deploy Service Management Service (DuraService) Service 1 Config REST API 2 Service ManagerPart 2: Retrieve Deploy Service Service Bundle Deploy Service 4 Service Service Bundle Config Service Service Registry Bundle Service Container Service Bundle 3 Service Bundle Service Configurator Deployed Deployed Service Config Service Service Config Service Plan
    13. 13. Service REST Interface• Get Services• Deploy Service• Get (Deployed) Service• Get Deployed Service Properties• Update Service Configuration• UnDeploy Service
    14. 14. DuraCloud DuraCloud Instance Distributed Services Services Media Streamer Duplicate Duplicate on on Demand Change Bit Bit Bit Integrity Integrity Integrity Checker Checker Checker Tools Bulk Image Image Transformer Transformer Bulk Image Server • Runs primarily outside of• Runs on DuraCloud Instance DuraCloud Instance• Connects to DuraStore • Connects to DuraStore• Can be direct Java service • Makes use of cloud network• Can be deployed web app or computation features
    15. 15. DuraCloud Bulk Service Deployment Service Management 1 (DuraService) 2 REST APITask Management Service Manager (DuraStore) REST API Run Service Registry Service 4 Run Task 3 6 Hadoop 5 Bundle Job Service Container Config Hadoop Deployed Service Plan Service Config
    16. 16. DuraCloud Reporting Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service ContainerDuraCloud Instance
    17. 17. DuraCloud Report Generation Report Management (DuraReport) Storage Service Report Report Builder BuilderDepositReports Message Broker Storage Data REST API REST API Service Storage Management Event Service Management Messages (DuraStore) (DuraService)
    18. 18. Report REST Interface Storage Report Service Report• Start Storage Report • Get Deployed Services• Get Latest Storage Report Report• Get Storage Report List • Get Latest Completed Services Report• Get Storage Report • Get Completed Services• Get Storage Report Info Report List• Cancel Storage Report • Get Services Report• Schedule Storage Report• Cancel Storage Report Schedule
    19. 19. DuraCloud UI Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service ContainerDuraCloud Instance
    20. 20. DuraCloud UI Interaction Administrative User Interface (DurAdmin) Store Client Service Client Report Client REST API REST API REST APIStorage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport)
    21. 21. DuraCloud Security DurAdmin Storage Provider DuraStore DuraService DuraReport Application SecurityStorage Provider Access Control Instance Firewall Transport Security User Data Center
    22. 22. DuraCloud Tools Transfer to DuraCloud Transfer from DuraCloud• Upload Tool • Retrieval Tool – GUI to select files and – Retrieve files from folders one, selected, or all – Visual transfer progress spaces• Sync Tool • Stitch Tool – Monitors for changes – Retrieve large files – Run one time or – Acts on a single file continually• Chunk Tool – Transfer large files – Acts on a single file
    23. 23. DuraCloud ToolsUser Data Center Store Client Store Client Chunk Tool Stitch Tool Sync Tool Retrieval Tool Upload Tool REST API Storage Management (DuraStore)
    24. 24. Near Term Road MapNew features• Access control at the space level• Improved format identification• Automated bit integrity checking• Email notification on service completion• Multi-Account DuraCloud instances• Content indexing and Search – Improved discovery – Improved content browsing• Folder-based navigation• Improved handling of very large files• Multi-Item updates/deletesIntegrations• SDSC storage provider (production version)• Fedora-in-the-cloud prototype• DSpace-in-the-cloud prototype
    25. 25. Long Term Road MapNew Features• Video transcoding• Document format transformation• Improved image support services• Media streaming access controls• Metadata extraction servicesIntegrations• Azure storage provider (production version)• Eucalyptus Walrus storage provider• Rackspace compute instances• Managed Fedora-in-the-cloud• Managed DSpace-in-the-cloud
    26. 26. Questions?
    27. 27. Upcoming DuraCloud Webinars DSpace and DuraCloud November 30, 2011 at 1pm ET Fedora and DuraCloud January 11, 2012 at 1pm ET
    28. 28. Thank You! Further Questions? info@duracloud.org

    ×