My SRB
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
908
On Slideshare
908
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • 18

Transcript

  • 1. MySRB a Virtual Data Grid Solution (An interface to the Storage Resource Broker) Arcot Rajasekar SDSC
  • 2. Introduction
    • Motivation
    • SRB Background
    • mySRB
    SRB
  • 3. Data Handling Problems
    • Large Datasets; Large Number of Datasets; Scaling
    • Distributed, Heterogeneous Storage
    • Virtualization & Transparency
    • Collaboration, Access Control, Authentication, Security
    • Replication, Coherency, Synchronization
    • Fault Tolerance and Load Distribution
    • Scheduling, Caching & Data Placements
    • Data Migration over Time & Space
    • Data/Collection Curation
    • Uniform Name Space
    • Handling Legacy Data and Data/Resource Evolution
    • User-friendly Interfaces
  • 4. Metadata Problems
    • Types of Metadata – Relational to XML to unstructured
    • Standardized to User-defined Metadata
    • Large Number of Attributes;
    • Large Size; Scaling
    • Federation - integration over space
    • Evolution - integration over time
    • Evolution - integration over contexts
    • Discovery and Search
    • Presentation – user friendly
    • Extraction and Maintenance
  • 5. Virtual Data Grid Requirements
    • Seamless access to data and information stored at local and remote sites
    • Virtualization of data, collection and meta information
    • Handle Dataset Scaling – size & number
    • Integrate Data Collections & Associated Metadata
    • Handle Multiplicity of Platforms, Resource & Data Types
    • Handle Seamless Authentication
    • Handle Access Control
    • Provide Auditing Facilities
    • Handle Legacy Data & Methods
  • 6. Solution SRB SDSC Storage Resource Broker & Meta-data Catalog SRB Archives HPSS, ADSM, UniTree, DMF Databases DB2, Oracle, Sybase File Systems Unix, NT, Mac OSX Application C, C++, Linux I/O Unix Shell Dublin Core Resource, Mthd, User User Defined Application Meta-data Remote Proxies DataCutter Metadata Extraction Java, NT Browsers Web Prolog Predicate MCAT HRM
  • 7. Features SRB
    • Data Virtualization
      • Uniform Data Name Space & Virtual Collection-concept
    • Resource Transparency
      • Local or Remote, Resource Type & Access Method
    • Location/Format Transparency
      • Path Names, Schemas, Containers
    • User Transparency
      • Uniform User Name Space
      • Tickets & Groups
    • Method Transparency
    • Other Features
      • Rich Access Control
      • Cross-Domain Authentication
      • Segmented & Replicated Data Mgmt
      • Data Discovery
    Virtual Data Grid (SRB) MCAT Duke UCLA NCMIR CalTech SDSC
  • 8. Solution SRB
    • Federated Server Architecture
    • Uniform Access Interface – Thread-safe Client
      • Programmatic API (C, C++, Java, Perl-through-C)
      • GUI (Java for Unix, Windows Browser [ NT, Me, 98, 2000 ])
      • Web Support (CGI-Scripts, Portals )
      • Command Line Interface (Unix, DOS)
    • Metadata Catalog (Oracle, DB2, Sybase, SQLServer)
      • Handles transparencies, authentication, access control,
      • replication, container support, …
      • User-defined Metadata
    • Multi-Platform Support
      • Unix, Linux, Windows, MacOSX (from Cray to Desktop)
      • HPSS, ADSM, UniTree, …, UnixFS, NTFS,…, Oracle, DB2,…, FTP,HTTP
    • Integration of Data Handling & Digital Library Functionalities
  • 9. SRB Space DL DR - Data Repository DL - Dig Library MC - Meta Catalog SRB SRB SRB SRB SRB SRB SRB SRB SRB SRB DR DR DR DR DR DR DL DL DL MC Client Client Client Client Client Client
  • 10. SRB Projects
    • Digital Libraries
      • UCB, Umich, UCSB, Stanford,CDL
      • NSF NSDL - UCAR / DLESE
    • NASA Information Power Grid
    • DOE ASCI Data Visualization Corridor
    • Astronomy
      • National Virtual Observatory
      • 2MASS Project (2 Micron All Sky Survey)
    • Particle Physics
      • Particle Physics Data Grid (DOE)
      • GriPhyN
      • SLAC Synchrotron Data Repository
    • Medicine
      • Visible Embryo (NLM)
    • Earth Systems Sciences
      • ESIPS
      • LTER
    • Persistent Archives
      • NARA
      • LOC
    • Neuro Science & Molecular Science
      • TeleScience, Brain Images, BIRN
      • JCSG (SSRL/SLAC), AfCS, …
  • 11. MySRB
    • Web-bases Access to the SRB
    • Secure HTTP
    • Uses Cookies for Session Control
    • Self Registration of Users Supported
      • Currently limited to SDSC users
    • Self Registration of Resources (soon)
    • Access to Both Data and Metadata
  • 12. MySRB Features
    • Data & File Management
    • Collection Creation and Management
    • Collection of Varied Objects
      • Files, SQL Objects, Databases, URLs, directories, archives, …
    • Metadata Handling
    • Browsing & Querying Interface
    • Access Control
    • Version Control (soon)
    • Support proxy (remote) operations
  • 13. Data Management
    • Browse in Hierarchical Collections
    • Registration of
    • (remote) Legacy Files & Directories
    • Registration of SQL Objects
    • Registration of URLs
    • Data Movement Operations
      • Ingest & Re-Ingest, Delete, Unlink
      • Replicate, Copy, Move, S-Link
    • Access Control Operations
      • Read, Write, Own, Curate, Annotate, …
      • Ticket-based Access
    • Version Control Operations (soon)
      • Read Lock, Write Lock, Unlock
      • Check In Check Out
  • 14. Types of Meta data
    • System-level Metadata
      • Size, resource, owner, date, access control, …
    • User-defined Meta data
      • for data & collections
      • <name,value,unit> triples
      • No limits in number of metadata
      • Support for Collection-level schemas
        • Comments, default values, drop-down lists
      • Support for Standardized Schemas
        • (eg. Dublin Core)
    • Annotations
      • Supports textual annotations
      • Annotator, date, context also registered
  • 15. Meta Data Management
    • Insert, Update and Delete of Metadata
    • Access Control for Metadata (soon in mySRB)
    • Querying across system-level, user-defined metadata and annotations
      • Query under collections & across collections
    • Browsing on user-defined metadata
    • Metadata supported for legacy files & directories
    • Extract Metadata (using proxy operations)
  • 16. Conclusion
    • Demonstration: goto mySRB
    • For Additional Information:
      • http://www.npaci.edu/dice/srb