Policy-based Data Management


Published on

Data grids are an emerging technology that enables the formation of sharable collections from data distributed across multiple storage resources. The integrated Rule Oriented Data System (iRODS) is a data grid developed by the DICE Center at UNC-CH. The iRODS data grid enforces management policies that control properties of the collection. Examples of policies include retention, disposition, distribution, replication, metadata extraction, time-dependent access controls, data processing, data redaction, and integrity checking. Policies can be defined that automate administrative functions (file migration and replication) and that validate assessment criteria (authenticity, integrity, chain of custody). iRODS is used to build data sharing environments, digital libraries, and preservation environments. The iRODS data grid is used at UNC-CH to support the Carolina Digital Repository, the LifeTime Library for the School of Information and Library Science, data grids for the Renaissance Computing Institute (RENCI), collaborations within North Carolina, and both national and international data sharing. At RENCI, the TUCASI data grid supports shared collections between UNC-CH, Duke, and NCSU. The RENCI data grid is federated with ten other data grids including the National Climatic Data Center, the Texas Advanced Computing Center data grid, and the Ocean Observatories Initiative data grid. International applications include the CyberSKA Square Kilometer Array for radio astronomy and the French National Institute for Nuclear Physics and Particle Physics. The collections that are assembled may contain hundreds of millions of files, and petabytes of data. A specific goal is the integration of institutional repositories with the national data infrastructure that is being assembled under the NSF DataNet program. The software is available as an open source distribution from http://irods.diceresearch.org.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Policy-based Data Management

  1. 1. Policy-Based Data Management Arcot Rajasekar Mike Conway Reagan W. Moore University of North Carolina at Chapel Hill [email_address]
  2. 2. Topics <ul><li>Policy-based data management </li></ul><ul><li>Integrated Rule-Oriented Data System </li></ul><ul><ul><li>DataNet Federation Consortium </li></ul></ul><ul><ul><li>SILS’s Lifetime Library </li></ul></ul><ul><li>Simple Demonstration </li></ul>
  3. 3. Expectations <ul><li>Data collection sizes will increase </li></ul><ul><ul><li>Now petabytes, soon exabytes </li></ul></ul><ul><ul><ul><li>1 PB/year = 32 MB/sec </li></ul></ul></ul><ul><ul><ul><li>1 PB/day = 11.6 GB/sec </li></ul></ul></ul><ul><li>Moving Data is becoming a problem </li></ul><ul><ul><ul><li>Need to keep data in a centralized location </li></ul></ul></ul><ul><ul><ul><ul><li>Store once </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Use from anywhere </li></ul></ul></ul></ul><ul><ul><ul><li>Need to do data analyses at the storage system </li></ul></ul></ul><ul><ul><ul><li>Digital Legacy is becoming a frightful nightmare </li></ul></ul></ul><ul><ul><ul><ul><li>How to control our digital lifecycle </li></ul></ul></ul></ul>
  4. 4. Applications <ul><li>Digital libraries </li></ul><ul><ul><li>Personal Libraries </li></ul></ul><ul><ul><li>email, photos, music, video, ebooks, documents, maps </li></ul></ul><ul><ul><li>social networking content </li></ul></ul><ul><ul><li>Need for continuous indexing of contents </li></ul></ul><ul><li>Scientific/Office data collections </li></ul><ul><ul><li>Extraction of features from data sets </li></ul></ul><ul><ul><li>Discovery and Access </li></ul></ul><ul><ul><li>Creation of derived data products </li></ul></ul><ul><ul><li>Sharing with collaborators and other users </li></ul></ul><ul><ul><li>Keeping it for reference & repurpose </li></ul></ul><ul><li>…… . </li></ul>
  5. 5. Policy-based Data Sharing Client Provider iRODS controlled workflows Provider iRODS controlled workflows Storage Storage Shared Collection Consensus on Policies and Procedures controls the shared data
  6. 6. Access distributed data with Web-based Browser or iRODS GUI or Command Line clients. iRODS Data Server Disk, Tape, etc. iRODS Middleware Overview of iRODS Architecture User w/ Client Can Search, Access, Add and Manage Data & Metadata iRODS Metadata Catalog Track information iRODS Rule Engine Track Policies
  7. 7. Data Virtualization Storage System Storage Protocol Access Interface Policy Enforcement Points Standard Micro-services Map from the actions requested by the client to multiple policy enforcement points. Map from policy to standard micro-services. Map from micro-services to standard Posix I/O operations. Map standard I/O operations to the protocol supported by the storage system Standard I/O Operations Data Grid
  8. 8. iRODS Distributed Data Management
  9. 9. iRODS Extensible Infrastructure <ul><li>Clients – specific to discipline and life cycle state </li></ul><ul><li>Policies – specific to discipline </li></ul><ul><li>Procedures – specific to discipline </li></ul><ul><li>Remaining infrastructure is generic </li></ul><ul><ul><li>Network transport Parallel I/O </li></ul></ul><ul><ul><li>Authentication / Authorization Single Sign-on </li></ul></ul><ul><ul><li>Distributed storage access Protocol mediation </li></ul></ul><ul><ul><li>Remote execution Deferred/periodic </li></ul></ul><ul><ul><li>Metadata management Catalog </li></ul></ul><ul><ul><li>Message passing Debugging/progress </li></ul></ul><ul><ul><li>Rule engine Policy control </li></ul></ul>
  10. 10. Generic Capabilities <ul><li>Replication </li></ul><ul><li>Registration of files into the data grid </li></ul><ul><li>Synchronization of remote directory </li></ul><ul><li>Managed file transport (iDrop) </li></ul><ul><li>Automated metadata extraction </li></ul><ul><li>Queries on metadata, tags </li></ul><ul><li>Server-side workflows (loop over result sets) </li></ul><ul><li>Parallel I/O streams & RBUDP transport </li></ul>
  11. 11. Policies <ul><li>Retention, disposition, distribution, arrangement </li></ul><ul><li>Authenticity, provenance, description </li></ul><ul><li>Integrity, replication, synchronization </li></ul><ul><li>Deletion, trash cans, versioning </li></ul><ul><li>Archiving, staging, caching </li></ul><ul><li>Authentication, authorization, redaction </li></ul><ul><li>Access, approval, IRB, audit trails, report generation </li></ul><ul><li>Assessment criteria, validation </li></ul><ul><li>Derived data product generation, format parsing </li></ul><ul><li>Federation of independent data grids </li></ul>
  12. 12. Highly Controlled Environment <ul><li>All accesses are authenticated </li></ul><ul><ul><li>GSI / Kerberos / Challenge-response / Shibboleth </li></ul></ul><ul><li>All operations are authorized </li></ul><ul><ul><li>ACLs on files, storage </li></ul></ul><ul><ul><li>User groups, storage groups </li></ul></ul><ul><li>All policies evaluate a constraint </li></ul><ul><ul><li>Constraints based on persistent state information and session information </li></ul></ul>
  13. 13. Applications <ul><li>Data grids – PB-size distributed collections </li></ul><ul><ul><li>Astronomy – NOAO, CyberSKA, LSST </li></ul></ul><ul><ul><li>High Energy Physics – BaBar, KEK </li></ul></ul><ul><ul><li>Earth Systems – NASA (MODIS data set) </li></ul></ul><ul><ul><li>Australian Research Collaboration Service </li></ul></ul><ul><ul><li>Plant biology – iPlant Collaborative </li></ul></ul><ul><li>Institutional repositories </li></ul><ul><ul><li>Carolina Digital Repository </li></ul></ul><ul><li>Libraries </li></ul><ul><ul><li>Texas Digital Libraries </li></ul></ul><ul><ul><li>Seismology - Southern California Earthquake Center </li></ul></ul><ul><li>Archives </li></ul><ul><ul><li>Ocean Observatories Initiative </li></ul></ul>
  14. 14. LIfeTime Library <ul><li>Student personal digital library </li></ul><ul><ul><li>Manage course material </li></ul></ul><ul><ul><li>Photograph collections </li></ul></ul><ul><ul><li>Video collections </li></ul></ul><ul><ul><li>Reference collection (soft links to information) </li></ul></ul><ul><li>Choose favorite access mechanism </li></ul><ul><ul><li>iDrop (synchronize local directory, share data) </li></ul></ul><ul><ul><li>iDrop cloud browser (add tags, metadata) </li></ul></ul><ul><ul><li>Unix tools (execute personal rules) </li></ul></ul>
  15. 15. LifeTime Library Storage System Storage Protocol Access Interface Policy Enforcement Points Standard Micro-services Multiple clients Policies to automate replication turn on versioning set audit trails enforce strict ACLs replicate metadata Micro-services to create thumbnails extract metadata assign organization Standard I/O Operations Data Grid
  16. 16. DataNet Federation Consortium <ul><li>Implement national data grid </li></ul><ul><ul><li>Federate existing discipline-specific data management systems to enable national research collaborations </li></ul></ul><ul><li>Enable collaborative research on shared data collections </li></ul><ul><ul><li>Manage collection life cycle as the user community broadens </li></ul></ul><ul><li>Integrate “live” research data into education initiatives </li></ul><ul><ul><li>Enable student research participation through control policies </li></ul></ul>Cyber-infrastructure Partners: Univ. of North Carolina, Chapel Hill Univ. of California, San Diego Arizona State University Drexel University Duke University University of Arizona University of South Carolina Science and Engineering Initiatives: Ocean Observatories Initiative the iPlant Collaborative CUAHSI CIBER-U Odum Social Science Institute Temporal Dynamics of Learning Center National Science Foundation Cooperative Agreement: OCI-0940841 Policy-based data management Project Shared Collection Processing Pipeline Digital Library Reference Collection Federation Collection Life Cycle
  17. 17. iRODS - Open Source Software <ul><li>Community driven software development </li></ul><ul><ul><li>Focus on features required by user communities </li></ul></ul><ul><ul><li>Focus on bug-free software </li></ul></ul><ul><ul><li>Focus on highly reliable software </li></ul></ul><ul><ul><li>Focus on highly extensible software </li></ul></ul><ul><ul><li>Approximately 3-4 software releases per year </li></ul></ul><ul><li>Distributed under a BSD license </li></ul><ul><ul><li>International collaborations on software development </li></ul></ul><ul><ul><li>IN2P3 (France), SHAMAN (UK), ARCS (Australia), Academia Sinica (Taiwan) </li></ul></ul><ul><li>Highly Successful </li></ul>
  18. 18. <ul><li>iRODS - Open Source Software </li></ul><ul><li>Reagan W. Moore </li></ul><ul><li>[email_address] </li></ul><ul><li>http://irods.diceresearch.org </li></ul>NSF OCI-0940841 “ DataNet Federation Consortium ” NSF OCI-1032732 &quot;SDCI Data Improvement: Improvement and Sustainability of iRODS Data Grid Software for Multi-Disciplinary Community Driven Application&quot; NSF OCI-0848296 “NARA Transcontinental Persistent Archives Prototype” NSF SDCI-0721400 “Data Grids for Community Driven Applications”