Storage Tank in Data Grid
Upcoming SlideShare
Loading in...5
×
 

Storage Tank in Data Grid

on

  • 559 views

 

Statistics

Views

Total Views
559
Views on SlideShare
559
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Storage Tank in Data Grid Storage Tank in Data Grid Presentation Transcript

    • Storage Tank in Data Grid
    Shin, SangYong(syshin, #6468) IBM Grid Computing August 23, 2003
  • Storage Architecture Model - app. data is in files - files stored on block storage - all managed by storage mgmt s/w Application File System Block Virtualization Storage Devices Block subsystem Storage Management
  • SAN Block Virtualization Today SAN Block Virtualization
    • Common view of block storage
    • No server impact on storage change
    Emerging
    • No common view of block storage
    • Server impact on storage change
    - IBM block virtualization is Lodestone
  • High-end Host Host Host Application Midrange Disk array Shark, Brand X ... FastT, Brand Y RAID Brick LVE = Lodestone Virtualization Engine LVE Disk array
    • Functions
      • Providing Virtual Disks
      • Online- Dynamic Volume Sizing
      • Advanced Copy Functions
      • Economic Disaster Recovery Solutions.
      • Different Level of Performance
      • Data Backup with low price disk
      • No Service Downtime
      • etc
    LVE LVE LVE Extending Lodestone for Grid
    • File Systems - Current Capabilities Vs Grid Requirements
    • GPFS
      • HPC, Engineering, Digital Media
      • Access from servers in a cluster
      • Concurrent multiple I/Os
      • AIX and Linux OS only
      • No access to other FS data
    • Storage Tank
      • Commercial, file sharing, DB serving
      • Access from servers on SAN
      • All servers and OSes
      • No access to other FS data
    • Grid requirements
      • Access from any machine, any OS, anywhere
      • Access to all file system data
    • Planned Approach:
      • Allow remote access to our file systems
      • Provide multi-site support
      • Integrate data from other sources
    SAN
    • NFSv4 support for our file systems
    • We believe NFSV4 will be an important protocol for the grid
      • has the necessary extensions for robust security and WAN access
      • is the first NFS protocol to come through the standards process
      • proposed standard in Dec. 2002; expected to be draft standard by 4Q03
    • Our plan is to provide NFSv4 support for our file systems (J2, GPFS and Storage Tank)
      • Best case will be late 2004
    • Storage Tank (ST) - a SAN file system
    SN ST agent AIX ST agent Solaris Backup Data Data data Meta - Meta data ST agent Linux
    • Capabilities:
      • access to ST data through Globus GridFTP interface
      • register ST files in Globus Replica Location Service
      • enabled to support OGSA services (e.g. replication)
      • centralized, policy-based storage management
      • cross-platform file sharing
      • performance comparable to local file system with direct client-to-storage data path
    ST agent Win2K ST Server ST Server ST Server GridFTP, NFS Prototypes: 2H02-1H03 Customer: CERN file attributes, file location info, control info LAN ST Clients
    • CERN Requirements
    • Data analysis of Large Hadron Collider (LHC) experiments
      • Basic unit of data is an LHC event
      • data represents physical collision between 2 protons
      • 1 to few MBs
      • stored within 1 GB files
      • event metadata stored in an RDBMS
    • Tiered structure
      • CERN is Tier 0
      • event data and metadata distributed to Tier 1 centers
      • physicists at Tier 2 centers analyze data at Tier 1 centers
    • 2.4 PB of disk and 14 PB of tape by 2007
    • Grid access (AFS/DFS like), simple storage management
    • IP SANs, not FC
    • Our Proposal
    • Use Storage Tank for basic storage infrastructure
    • Use iSCSI disks
      • FAStT with iSCSI gateway or 200i
    • DB2 for event metadata
    • Research extensions
      • NAS head for Storage Tank
      • Grid access to Storage Tank
      • Object Store prototype for disks
  • Single namespace across multiple sites - Replication of files for good performance - Extended protocols for consistency across replicas - Joint research w/ Johns Hopkins underway Control Network (IP) data Integrated ST/NAS Appliance ST Extensions Prototype: 1H04 Customer: CERN, JHU Branch office Extend ST to Multiple Sites – Distributed Storage Tank Tank NYC Fargo SFO SAN Meta-data Server Cluster Win2K ST Agent AIX ST Agent Solaris ST Agent Linux ST Agent Meta-data Server Cluster Control Network (IP) SAN Meta-data Server Cluster Win2K ST Agent AIX ST Agent Solaris ST Agent Linux ST Agent data
    • Ultimate Vision for Federated Grid File Systems
    . . . . . . Organization 1 Organization 2 file sources Access Server Client Client exporter exporter Access Server Client Client exporter exporter Proxy Server
    • Control Network (IP)
    data Control Network (IP) data Grid data repository GridFTP NFS NAS data repository NFS NFS Extend ST to access data from other file systems/sources SAN Meta-data Server Cluster Win2K ST Agent AIX ST Agent Solaris ST Agent Linux ST Agent SAN Meta-data Server Cluster Win2K ST Agent AIX ST Agent Solaris ST Agent Linux ST Agent Client Client Client Client
    • Storage Management in Grid Computing Environment
    CIM/XML OGSA CIM/XML
    • IBM storage management products today (TSM, TSRM, ITSANM) and planned products (Merlot) cover a reasonable set of functions
    • We are converging, with the industry, on CIM/XML as the standard for storage device management
    • In support of grid, we expect:
      • to convert our management solutions to Web/OGSA services
      • to enhance functionality
    We are just starting to focus on grid implications for storage management Applications Lodestone Storage Tank Shark, Tape, etc. CIM Provider Interface CIM Provider Interface CIM Provider Interface Storage Management Services OGSA-CIM Wrapper
    • Block subsystem
    Summary of Data Grid Extend ST & GPFS Lodestone Support OGSA upper interface Support CIM lower interface OGSA CIM Application File System Block Virtualization Storage Devices Block subsystem Storage Management