Interested in learning more about the technical architecture of DuraCloud? Curious about how it runs in the cloud? This presentation details the technical underpinnings of DuraCloud. Presented by: Bill Branan, DuraCloud Lead Developer, DuraSpace

1. A Technical Overview
Bill Branan
DuraCloud Technical Lead

2. Agenda
• Introduction
• Storage
• Services
• Reporting
• UI
• Security
• Tools
• Roadmap

3. Introduction
• DuraCloud is:
– Hosted service
• Runs on cloud-based compute systems
• Connects to cloud-based storage systems
• Provides a service platform
– Open source software suite
• Webinar Focus:
– Components of software platform
– Interacting with hosted DuraCloud

4. DuraCloud in Context
User Data Center
Data
File System
Repository
DuraCloud
Management Console
• Account Management
• Instance Management

5. DuraCloud Storage
Administrative User Interface (DurAdmin)
Storage Management Service Management Report Management
(DuraStore) (DuraService) (DuraReport)
REST API
REST API REST API
Service Container
DuraCloud Instance

6. DuraCloud Storage
User Data Center
Amazon Storage Amazon S3
Adapter
REST API
Rackspace Rackspace
Storage Storage Adapter Cloud Files
Provider
Storage Interface
Mediation Azure Storage Microsoft Azure
Adapter
Storage
SDSC Storage
Storage Management Adapter SDSC Cloud
(DuraStore) Storage

7. Storage REST Interface
Space Actions Content Actions
• Add Space • Add Content
• Get/Set Space Properties • Get/Set Content Properties
• Get Spaces List • Get Content
• Get Space Content List • Copy Content
• Get/Set Space Access • Delete Content
• Delete Space
Storage Provider Interface
Other Actions
• Get Stores
• Get Tasks List
• Perform Task

8. Storage Adapters
Transform calls from DuraCloud interface into
calls understood by a specific storage provider
• Determine how DuraCloud “spaces” translate
into top level provider containers (buckets)
• Define how space and content properties are
stored and retrieved
• Manage eventual consistency concerns

9. DuraCloud Services
Administrative User Interface (DurAdmin)
Storage Management Service Management Report Management
(DuraStore) (DuraService) (DuraReport)
REST API
REST API REST API
Service Container
DuraCloud Instance

10. DuraCloud Services
Media
Streamer
Duplicate Duplicate
on on
Demand Change
Bit Bit
Bit
Integrity Integrity
Integrity
Checker Checker
Checker
Tools Bulk
Image
Image
Transformer
Transformer
Bulk
Image
Server
… so how are they deployed?

11. DuraCloud Service Deployment
Services Service Management
Available? (DuraService)
1
REST API
2
Service List Service Manager
Services In
Plan?
4
Service Registry
Service List
3 Service Bundle
Part 1: Service Config
Ask which services are available
Service Plan

12. DuraCloud Service Deployment
Deploy Service Management
Service (DuraService)
Service
1 Config REST API
2
Service Manager
Part 2: Retrieve
Deploy Service Service
Bundle
Deploy Service 4
Service Service
Bundle Config Service Service Registry
Bundle
Service Container Service
Bundle 3 Service Bundle
Service Configurator
Deployed
Deployed Service Config
Service Service Config
Service Plan

13. Service REST Interface
• Get Services
• Deploy Service
• Get (Deployed) Service
• Get Deployed Service Properties
• Update Service Configuration
• UnDeploy Service

14. DuraCloud DuraCloud
Instance Distributed
Services Services
Media
Streamer
Duplicate Duplicate
on on
Demand Change
Bit Bit
Bit
Integrity Integrity
Integrity
Checker Checker
Checker
Tools Bulk
Image
Image
Transformer
Transformer
Bulk
Image
Server
• Runs primarily outside of
• Runs on DuraCloud Instance
DuraCloud Instance
• Connects to DuraStore
• Connects to DuraStore
• Can be direct Java service
• Makes use of cloud network
• Can be deployed web app
or computation features

15. DuraCloud Bulk Service Deployment
Service Management
1 (DuraService)
2
REST API
Task Management Service Manager
(DuraStore)
REST API
Run Service Registry
Service 4
Run Task 3
6 Hadoop 5
Bundle
Job
Service Container Config
Hadoop Deployed Service Plan
Service Config

16. DuraCloud Reporting
Administrative User Interface (DurAdmin)
Storage Management Service Management Report Management
(DuraStore) (DuraService) (DuraReport)
REST API
REST API REST API
Service Container
DuraCloud Instance

17. DuraCloud Report Generation
Report Management
(DuraReport)
Storage Service
Report Report
Builder Builder
Deposit
Reports
Message
Broker
Storage Data
REST API REST API
Service
Storage Management Event Service Management
Messages
(DuraStore) (DuraService)

18. Report REST Interface
Storage Report Service Report
• Start Storage Report • Get Deployed Services
• Get Latest Storage Report Report
• Get Storage Report List • Get Latest Completed
Services Report
• Get Storage Report
• Get Completed Services
• Get Storage Report Info Report List
• Cancel Storage Report • Get Services Report
• Schedule Storage Report
• Cancel Storage Report
Schedule

19. DuraCloud UI
Administrative User Interface (DurAdmin)
Storage Management Service Management Report Management
(DuraStore) (DuraService) (DuraReport)
REST API
REST API REST API
Service Container
DuraCloud Instance

20. DuraCloud UI Interaction
Administrative User Interface (DurAdmin)
Store Client Service Client Report Client
REST API REST API REST API
Storage Management Service Management Report Management
(DuraStore) (DuraService) (DuraReport)

21. DuraCloud Security
DurAdmin
Storage
Provider
DuraStore DuraService DuraReport
Application Security
Storage Provider
Access Control Instance Firewall
Transport Security
User Data Center

22. DuraCloud Tools
Transfer to DuraCloud Transfer from DuraCloud
• Upload Tool • Retrieval Tool
– GUI to select files and – Retrieve files from
folders one, selected, or all
– Visual transfer progress spaces
• Sync Tool • Stitch Tool
– Monitors for changes – Retrieve large files
– Run one time or – Acts on a single file
continually
• Chunk Tool
– Transfer large files
– Acts on a single file

23. DuraCloud Tools
User Data Center
Store Client Store Client
Chunk Tool Stitch Tool
Sync Tool Retrieval Tool
Upload Tool
REST API
Storage Management (DuraStore)

24. Near Term Road Map
New features
• Access control at the space level
• Improved format identification
• Automated bit integrity checking
• Email notification on service completion
• Multi-Account DuraCloud instances
• Content indexing and Search
– Improved discovery
– Improved content browsing
• Folder-based navigation
• Improved handling of very large files
• Multi-Item updates/deletes
Integrations
• SDSC storage provider (production version)
• Fedora-in-the-cloud prototype
• DSpace-in-the-cloud prototype

25. Long Term Road Map
New Features
• Video transcoding
• Document format transformation
• Improved image support services
• Media streaming access controls
• Metadata extraction services
Integrations
• Azure storage provider (production version)
• Eucalyptus Walrus storage provider
• Rackspace compute instances
• Managed Fedora-in-the-cloud
• Managed DSpace-in-the-cloud

26. Questions?

27. Upcoming DuraCloud Webinars
DSpace and DuraCloud
November 30, 2011 at 1pm ET
Fedora and DuraCloud
January 11, 2012 at 1pm ET

28. Thank You!
Further Questions?
info@duracloud.org