MySRB a Virtual Data Grid Solution (An interface to the Storage Resource Broker) Arcot Rajasekar SDSC
Introduction <ul><li>Motivation </li></ul><ul><li>SRB Background </li></ul><ul><li>mySRB </li></ul>SRB
Data Handling Problems  <ul><li>Large Datasets; Large Number of Datasets; Scaling </li></ul><ul><li>Distributed, Heterogen...
Metadata Problems <ul><li>Types of Metadata – Relational to XML to unstructured </li></ul><ul><li>Standardized to User-def...
Virtual Data Grid Requirements <ul><li>Seamless access to data and information stored  at local and remote sites </li></ul...
Solution SRB SDSC Storage Resource Broker & Meta-data Catalog SRB Archives HPSS, ADSM, UniTree, DMF Databases DB2, Oracle,...
Features  SRB <ul><li>Data Virtualization </li></ul><ul><ul><li>Uniform Data Name Space & Virtual Collection-concept </li>...
Solution SRB  <ul><li>Federated Server Architecture </li></ul><ul><li>Uniform Access Interface – Thread-safe Client </li><...
SRB Space DL  DR - Data Repository DL - Dig Library MC - Meta Catalog SRB SRB SRB SRB SRB SRB SRB SRB SRB SRB DR DR DR DR ...
SRB  Projects <ul><li>Digital Libraries </li></ul><ul><ul><li>UCB, Umich, UCSB, Stanford,CDL </li></ul></ul><ul><ul><li>NS...
MySRB <ul><li>Web-bases Access to the SRB </li></ul><ul><li>Secure HTTP </li></ul><ul><li>Uses Cookies for Session Control...
MySRB Features <ul><li>Data & File Management </li></ul><ul><li>Collection Creation and Management </li></ul><ul><li>Colle...
Data Management <ul><li>Browse in Hierarchical Collections </li></ul><ul><li>Registration of  </li></ul><ul><li>(remote) L...
Types of Meta data <ul><li>System-level Metadata </li></ul><ul><ul><li>Size, resource, owner, date, access control, … </li...
Meta Data Management <ul><li>Insert, Update and Delete of Metadata </li></ul><ul><li>Access Control for Metadata (soon in ...
Conclusion <ul><li>Demonstration:  goto mySRB </li></ul><ul><li>For Additional Information: </li></ul><ul><ul><li>http://w...
Upcoming SlideShare
Loading in...5
×

My SRB

771

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
771
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 18
  • My SRB

    1. 1. MySRB a Virtual Data Grid Solution (An interface to the Storage Resource Broker) Arcot Rajasekar SDSC
    2. 2. Introduction <ul><li>Motivation </li></ul><ul><li>SRB Background </li></ul><ul><li>mySRB </li></ul>SRB
    3. 3. Data Handling Problems <ul><li>Large Datasets; Large Number of Datasets; Scaling </li></ul><ul><li>Distributed, Heterogeneous Storage </li></ul><ul><li>Virtualization & Transparency </li></ul><ul><li>Collaboration, Access Control, Authentication, Security </li></ul><ul><li>Replication, Coherency, Synchronization </li></ul><ul><li>Fault Tolerance and Load Distribution </li></ul><ul><li>Scheduling, Caching & Data Placements </li></ul><ul><li>Data Migration over Time & Space </li></ul><ul><li>Data/Collection Curation </li></ul><ul><li>Uniform Name Space </li></ul><ul><li>Handling Legacy Data and Data/Resource Evolution </li></ul><ul><li>User-friendly Interfaces </li></ul>
    4. 4. Metadata Problems <ul><li>Types of Metadata – Relational to XML to unstructured </li></ul><ul><li>Standardized to User-defined Metadata </li></ul><ul><li>Large Number of Attributes; </li></ul><ul><li>Large Size; Scaling </li></ul><ul><li>Federation - integration over space </li></ul><ul><li>Evolution - integration over time </li></ul><ul><li>Evolution - integration over contexts </li></ul><ul><li>Discovery and Search </li></ul><ul><li>Presentation – user friendly </li></ul><ul><li>Extraction and Maintenance </li></ul>
    5. 5. Virtual Data Grid Requirements <ul><li>Seamless access to data and information stored at local and remote sites </li></ul><ul><li>Virtualization of data, collection and meta information </li></ul><ul><li>Handle Dataset Scaling – size & number </li></ul><ul><li>Integrate Data Collections & Associated Metadata </li></ul><ul><li>Handle Multiplicity of Platforms, Resource & Data Types </li></ul><ul><li>Handle Seamless Authentication </li></ul><ul><li>Handle Access Control </li></ul><ul><li>Provide Auditing Facilities </li></ul><ul><li>Handle Legacy Data & Methods </li></ul>
    6. 6. Solution SRB SDSC Storage Resource Broker & Meta-data Catalog SRB Archives HPSS, ADSM, UniTree, DMF Databases DB2, Oracle, Sybase File Systems Unix, NT, Mac OSX Application C, C++, Linux I/O Unix Shell Dublin Core Resource, Mthd, User User Defined Application Meta-data Remote Proxies DataCutter Metadata Extraction Java, NT Browsers Web Prolog Predicate MCAT HRM
    7. 7. Features SRB <ul><li>Data Virtualization </li></ul><ul><ul><li>Uniform Data Name Space & Virtual Collection-concept </li></ul></ul><ul><li>Resource Transparency </li></ul><ul><ul><li>Local or Remote, Resource Type & Access Method </li></ul></ul><ul><li>Location/Format Transparency </li></ul><ul><ul><li>Path Names, Schemas, Containers </li></ul></ul><ul><li>User Transparency </li></ul><ul><ul><li>Uniform User Name Space </li></ul></ul><ul><ul><li>Tickets & Groups </li></ul></ul><ul><li>Method Transparency </li></ul><ul><li>Other Features </li></ul><ul><ul><li>Rich Access Control </li></ul></ul><ul><ul><li>Cross-Domain Authentication </li></ul></ul><ul><ul><li>Segmented & Replicated Data Mgmt </li></ul></ul><ul><ul><li>Data Discovery </li></ul></ul><ul><ul><li>… </li></ul></ul>Virtual Data Grid (SRB) MCAT Duke UCLA NCMIR CalTech SDSC
    8. 8. Solution SRB <ul><li>Federated Server Architecture </li></ul><ul><li>Uniform Access Interface – Thread-safe Client </li></ul><ul><ul><li>Programmatic API (C, C++, Java, Perl-through-C) </li></ul></ul><ul><ul><li>GUI (Java for Unix, Windows Browser [ NT, Me, 98, 2000 ]) </li></ul></ul><ul><ul><li>Web Support (CGI-Scripts, Portals ) </li></ul></ul><ul><ul><li>Command Line Interface (Unix, DOS) </li></ul></ul><ul><li>Metadata Catalog (Oracle, DB2, Sybase, SQLServer) </li></ul><ul><ul><li>Handles transparencies, authentication, access control, </li></ul></ul><ul><ul><li>replication, container support, … </li></ul></ul><ul><ul><li>User-defined Metadata </li></ul></ul><ul><li>Multi-Platform Support </li></ul><ul><ul><li>Unix, Linux, Windows, MacOSX (from Cray to Desktop) </li></ul></ul><ul><ul><li>HPSS, ADSM, UniTree, …, UnixFS, NTFS,…, Oracle, DB2,…, FTP,HTTP </li></ul></ul><ul><li>Integration of Data Handling & Digital Library Functionalities </li></ul>
    9. 9. SRB Space DL DR - Data Repository DL - Dig Library MC - Meta Catalog SRB SRB SRB SRB SRB SRB SRB SRB SRB SRB DR DR DR DR DR DR DL DL DL MC Client Client Client Client Client Client
    10. 10. SRB Projects <ul><li>Digital Libraries </li></ul><ul><ul><li>UCB, Umich, UCSB, Stanford,CDL </li></ul></ul><ul><ul><li>NSF NSDL - UCAR / DLESE </li></ul></ul><ul><li>NASA Information Power Grid </li></ul><ul><li>DOE ASCI Data Visualization Corridor </li></ul><ul><li>Astronomy </li></ul><ul><ul><li>National Virtual Observatory </li></ul></ul><ul><ul><li>2MASS Project (2 Micron All Sky Survey) </li></ul></ul><ul><li>Particle Physics </li></ul><ul><ul><li>Particle Physics Data Grid (DOE) </li></ul></ul><ul><ul><li>GriPhyN </li></ul></ul><ul><ul><li>SLAC Synchrotron Data Repository </li></ul></ul><ul><li>Medicine </li></ul><ul><ul><li>Visible Embryo (NLM) </li></ul></ul><ul><li>Earth Systems Sciences </li></ul><ul><ul><li>ESIPS </li></ul></ul><ul><ul><li>LTER </li></ul></ul><ul><li>Persistent Archives </li></ul><ul><ul><li>NARA </li></ul></ul><ul><ul><li>LOC </li></ul></ul><ul><li>Neuro Science & Molecular Science </li></ul><ul><ul><li>TeleScience, Brain Images, BIRN </li></ul></ul><ul><ul><li>JCSG (SSRL/SLAC), AfCS, … </li></ul></ul>
    11. 11. MySRB <ul><li>Web-bases Access to the SRB </li></ul><ul><li>Secure HTTP </li></ul><ul><li>Uses Cookies for Session Control </li></ul><ul><li>Self Registration of Users Supported </li></ul><ul><ul><li>Currently limited to SDSC users </li></ul></ul><ul><li>Self Registration of Resources (soon) </li></ul><ul><li>Access to Both Data and Metadata </li></ul>
    12. 12. MySRB Features <ul><li>Data & File Management </li></ul><ul><li>Collection Creation and Management </li></ul><ul><li>Collection of Varied Objects </li></ul><ul><ul><li>Files, SQL Objects, Databases, URLs, directories, archives, … </li></ul></ul><ul><li>Metadata Handling </li></ul><ul><li>Browsing & Querying Interface </li></ul><ul><li>Access Control </li></ul><ul><li>Version Control (soon) </li></ul><ul><li>Support proxy (remote) operations </li></ul>
    13. 13. Data Management <ul><li>Browse in Hierarchical Collections </li></ul><ul><li>Registration of </li></ul><ul><li>(remote) Legacy Files & Directories </li></ul><ul><li>Registration of SQL Objects </li></ul><ul><li>Registration of URLs </li></ul><ul><li>Data Movement Operations </li></ul><ul><ul><li>Ingest & Re-Ingest, Delete, Unlink </li></ul></ul><ul><ul><li>Replicate, Copy, Move, S-Link </li></ul></ul><ul><li>Access Control Operations </li></ul><ul><ul><li>Read, Write, Own, Curate, Annotate, … </li></ul></ul><ul><ul><li>Ticket-based Access </li></ul></ul><ul><li>Version Control Operations (soon) </li></ul><ul><ul><li>Read Lock, Write Lock, Unlock </li></ul></ul><ul><ul><li>Check In Check Out </li></ul></ul>
    14. 14. Types of Meta data <ul><li>System-level Metadata </li></ul><ul><ul><li>Size, resource, owner, date, access control, … </li></ul></ul><ul><li>User-defined Meta data </li></ul><ul><ul><li>for data & collections </li></ul></ul><ul><ul><li><name,value,unit> triples </li></ul></ul><ul><ul><li>No limits in number of metadata </li></ul></ul><ul><ul><li>Support for Collection-level schemas </li></ul></ul><ul><ul><ul><li>Comments, default values, drop-down lists </li></ul></ul></ul><ul><ul><li>Support for Standardized Schemas </li></ul></ul><ul><ul><ul><li>(eg. Dublin Core) </li></ul></ul></ul><ul><li>Annotations </li></ul><ul><ul><li>Supports textual annotations </li></ul></ul><ul><ul><li>Annotator, date, context also registered </li></ul></ul>
    15. 15. Meta Data Management <ul><li>Insert, Update and Delete of Metadata </li></ul><ul><li>Access Control for Metadata (soon in mySRB) </li></ul><ul><li>Querying across system-level, user-defined metadata and annotations </li></ul><ul><ul><li>Query under collections & across collections </li></ul></ul><ul><li>Browsing on user-defined metadata </li></ul><ul><li>Metadata supported for legacy files & directories </li></ul><ul><li>Extract Metadata (using proxy operations) </li></ul>
    16. 16. Conclusion <ul><li>Demonstration: goto mySRB </li></ul><ul><li>For Additional Information: </li></ul><ul><ul><li>http://www.npaci.edu/dice/srb </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×