Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Se training storage grid webscale technical overview

Se training storage grid webscale technical overview

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Se training storage grid webscale technical overview

  1. 1. StorageGRID Webscale Technical Overview September 2014
  2. 2. Student Guide & Workshop & Internal Training & Confidential Update Daily https://goo.gl/VVmVZ0 5673-StorageGRID for Standard Enterprise Solution http://ouo.io/SROnEV Building Private Clouds on E-Series The Swift Object Opportunity http://ouo.io/vWrya Deployment Guide - Acuo Universal Clinical Platform with NetApp StorageGRID Integration Architecture http://ouo.io/TowHxU Deployment Guide - StorageGRID E-Series http://ouo.io/jhf1M Getting Started with StorageGRID CDMI Integrations http://ouo.io/dAORQj NetApp and the Object-Based Storage and Archiving Landscape http://ouo.io/wvoEgQ NetApp Distributed Content Repositories What Are We Doing in Real Life http://ouo.io/3y65M NetApp Portfolio Overview http://ouo.io/O9aKQ POC - StorageGRID Webscale Proof of Concept Guide http://ouo.io/pzljIH QRG_StorageGRID Webscale http://ouo.io/3dqRr SE Training - StorageGRID Webscale Technical Overview http://ouo.io/BqtKUU Solution Brief - NetApp StorageGRID Manage Large Pools of Mission-Critical Patient Data Across Healthcare Facilities http://ouo.io/KTDFz StorageGRID Webscale 10.0 Installation Overview http://ouo.io/hIYhJ StorageGRID Webscale Nonstop Object Storage for Enterprise and Cloud http://ouo.io/dMRoOr StorageGRID Webscale Object-Enabled Data Management http://ouo.io/nhuj4f Technical Report - Integration Guide for NetApp StorageGRID with McKesson Horizon Medical Imaging PACS http://ouo.io/9ncafj © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only2
  3. 3. Agenda  What is Object Storage  StorageGRID Webscale Introduction  Technical Overview  Key Concepts  Data flow © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only3
  4. 4. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only4 Object storage growth driven by macro trends… Unstructured data continues to grow •New sources of unstructured data growth media, mobile and applications •Data continues to be retained for long periods (archival, compliance etc) Unstructured data profile is changing •Simultaneous access to the same data is rarely required •Most data accessed a few times initially and then rarely accessed Data access is changing •Geographically dispersed access •Applications accessing data – don’t care about POSIX semantics, file locking Highly cost sensitive petabyte scale repositories •Driving tradeoffs between $/GB, latency, throughput, and data protection Storage being managed in a cloud ecosystem •Unified management & orchestration •Growth in cloud hosted applications that leverage object storage
  5. 5. Introducing StorageGRID Webscale  A new variant of StorageGRID  Target object store (cloud, archive, media) use cases at massive scale  New Features  Protocols – native support for S3 API  Scalability – 100 billion objects, 70 PB  Simplicity – modular scalable resilient architecture, simplified deployment  Proven track record for reliability and innovation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only5
  6. 6. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only6 Block File Object What is Object Storage? Different ways to address data Specific location on disks / memory Tracks Sectors Specific folder in fixed logical order File path File name Date Flexible container size Data and Metadata Unique ID
  7. 7. Object Storage Example: File vs. Object © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only7 Object BasedFile Based ValetParking Garage Daily Garage 1 Floor 4 Row N Space 53 /users/jsmith/car/garage1/floor4/rown/space53.file C:UsersjsmithGarage1Floor4RowNSpace53.file Object UID 317
  8. 8. Why Object storage? © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only8  Massive Scale  Billions of objects  Petabytes of data  Global namespace  Explosive growth  Respond to compliance and retention requirements  Cost effectively meet SLAs with intelligent data placement  Access from anywhere  Control access, security, and data integrity
  9. 9. Object storage used across different workloads Emerging Object storage segments in the hybrid cloud © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only9 Service Providers (XaaS) Media Repository (Media Redistribution) • Large object sizes (+250MB) • Distributed repositories • High data rates for redistribution • Time to first byte latency < 50ms Secure multi-tenancy (billing, isolation, authentication, & self-service) Control plane APIs and workflow automation New apps requiring RESTful interfaces (S3, Swift) Web Data Repositories • Small object (~KB) performance • Extremely high transaction load • Searchable, scalable metadata • High object counts Data Archives • Long access latency tolerance • Integration to tape / Glacier • Long retention periods • Erasure coding Scale seamlessly Ease of install & management Global namespace Cost
  10. 10. StorageGRID Webscale Object-Enabled Data Management © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only10
  11. 11. NetApp StorageGRID Webscale Software Defined Object Storage  Billions of objects and multi-petabyte  Architected for massive scale Built for the Hybrid Cloud  Global, Always-On data availability and durability  Support for cloud applications (S3, CDMI)  10th Generation object store proven with product deployments Dynamic Policy Engine  Intelligent, policy driven data management for optimal availability, performance and cost over the life cycle of data © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only11
  12. 12. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only12 StorageGRID Platform for Distributed Content Repositories StorageGRID Object Storage Software + E-Series Storage Array MULTIPLE: APPLICATIONS + SITES + PROTOCOLS MULTIPLE: TARGETS + TIERS MULTIPLE: TENANTS + POLICIES + ADMINISTRATORS Site 1 Site 2 … Site NSite 3 APPLICATIONS APPLICATIONS APPLICATIONS APPLICATION NetApp E-Series Tape
  13. 13. StorageGRID Webscale features © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only13 Hardware Obsolescence Protection Non-Disruptive Operations Object Integrity and Security Multi-Tenancy Global Object Namespace Services Automation Manageability Reliability Scalability ILM, Metadata driven policies Seamless Scaling Audit & Reporting
  14. 14. Technical Overview StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only14
  15. 15. Solution Topology © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only15 • Clients can access global object namespace via Gateway Nodes or Storage Nodes E2760 DE6600 Administrators REST API clients 10GbE LAN Per Node Resource Requirement VMDK (GB) vCPU RAM (GB) 100 8 24 100 8 24 300 8 24 Storage Gateway Admin DE6600 E2760 DE6600 DE6600 • StorageGRID Webscale nodes running in VMware hosts S3, CDMIHTTPS 16Gb FC switch WAN Router • E2760 block-based storage • SSDs for read cache and VM datastores • Mixed disk types (SSD, SAS, NL-SAS) for tiered storage pools
  16. 16. StorageGRID Architecture © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only16 App1 App2  Admin Nodes  Management Services: Configuration, Monitoring, Audit and Logging  Storage Nodes  Manages object storage including replication  API Gateway Nodes  Load balancing interface through which applications connect to the system  Archive Nodes  Interface to archive media storage such as tape DATA CENTER 1 DC2 DC3 Design the grid to scale for performance, capacity & resiliency STORAGE STORAGE ADMIN ARCHIVE API LOAD BALANCER
  17. 17.  Use NetApp StorageGRID Webscale Designer to architect the grid to your requirements  Quickly deploy the grid via NetApp StorageGRID Webscale Installer  Adapt your grid to changing requirements  Add sites  Add nodes and capacity  Support rolling upgrades Centralized Deployment & Rolling Upgrades Design, deploy, and maintain configuration control © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only17 DC1 DC2 DC4 DC3
  18. 18. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only18 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  19. 19. Client Connections, Object Identifiers & Metadata © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only19 Client CDMI S3 Read/Write StorageGRID Webscale System HTTPs Client Connections Object Identifier Object type: JPG Date modified: 07/21/ 2014 GPS Coordinates: Lat, Long Location: DC @ Seattle Metadata
  20. 20.  Scale beyond traditional application metadata to enable new capabilities  Flexibility:  Metadata is application defined  Up to 4096 fields can be created as requirements change (CDMI)  No lock-in to predefined schema  ILM engine evaluates the metadata and applies policies  S3 metadata available to the policy engine Extensive metadata management Metadata – Why it matters ?  Metadata is distributed throughout the grid  Increased scalability and resiliency  Faster retrieval and efficient ILM policy evaluation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only20 Object Identifier: 00006FFD00192A1200555FFEE12039468EBF622 D9402C4F962 Locations: Location 1: Data Center 1/DC1-S1/LDR Location 2: Data Center 2/DC2-S3/LDR Metadata CDMI/CVTE: 0 CDMI/META: {"application":"finance","doctype":"contract", "project":"45667}
  21. 21. Extensive metadata management Metadata Management  Metadata Includes – Object size, User Metadata, Bucket Name, Account Id  CDMI supports a large number of user metadata fields (up to 4096)  Policy support  Metadata can be used in polices (CDMI & S3)  No pre-configuration required for metadata, immediately available for policy use © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only21
  22. 22. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only22 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  23. 23. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only23 Sites (Link Cost Groups) Storage Pools StorageGRID Data Management Storage Pools Storage Grade TAPE Storage Grade SAS Storage Grade FLASH Munich, Germany San Francisco, USA Storage Node T1 Storage Node T2 Tape Storage Pool EU Storage Pool Storage Node S1Storage Node F1 Storage Node F2 Fast Storage Pool Vancouver, CAStorage Node T3Storage Node F3  Simple configuration of SLO based storage across multiple sites and storage grades
  24. 24. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only24 Customer Defined Storage grades StorageGRID Data Management Associating Storage Nodes with Storage Grades Creating Storage Grades  Customers can configure their own Storage Grades, and associate them with specific nodes
  25. 25. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only25 Site configuration – Not just a site StorageGRID Data Management  Customers can model network costs, creating powerful configurations  Possible Examples  Configure two sites to be treated as one,  Specify which sites are connected by high b/w pipes & low b/w pipes,  Prefer traffic to go in a particular direction
  26. 26. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only26 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  27. 27. Dynamic Policy Engine – An Overview Data management key concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only27  Manage policies not objects  Evaluate objects based on metadata such as:  Custom user application metadata  Method of ingest (S3 or CDMI)  Size of object  Last access time  Apply ILM rules to set:  Geography — Placement of an object  Storage grade — Type of storage used to store an object  Replication — Number of copies stored  Retention — Set time during which an object cannot be purged App1 Site1 Site2 Site3
  28. 28. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only28 Policy Management Data Management Key Concepts ILM Policy When If.. then.. • At Ingest • Objects at rest (already ingested) • After a read (enabling caching) • Metadata matches specific criteria • Move/Copy to one or more Storage Pools
  29. 29. ILM Rule – Specify the Conditions Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only29 When If..  Customers designate when a rule will be applied and what conditions must be met to trigger the action for placement and retention.
  30. 30. ILM Rule – Defining data placement and retention © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only30 then..  Simple configuration of complex placement rules, that are graphically displayed for easy understanding Data Management Key Concepts
  31. 31. Combine ILM rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only31 Data Management Key Concepts  Highly complex business logic can be implemented by combining multiple rules into a policy
  32. 32. ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only32 Application writes to grid via S3 Metadata is evaluated DC1 DC2 DC3  Store objects with S3 metadata “Bucket Name = ClientX” on ingest at DC1 on SSD and DC2 on SATA for 90 days  After 90 days store on DC1 on SATA and DC3 on Tape 1 x copy DC1SSD 1 x copy DC2SATA 90 Days later…. 1 x copy DC1SATA 1 x copy DC3Tape Bucket Name = ClientX
  33. 33. ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only33 DC1 DC2 DC3  What if requirements change?  A new Data Center is brought on line and now we must store a copy at DC4  We can edit the policy – and apply not just to new data, but re-evaluate existing data and create new replicas as needed  Without impact to the performance of the grid DC4
  34. 34.  Create ILM Rules to set customized service levels  GOLD = 2 x copies on SSD, 1 x copy on SATA  SILVER = 1 x copy on SAS, 2 x SATA  BRONZE = 1 x copy on SATA, 1 x copy on tape Set custom service levels via metadata SLA Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only34 {"sla":"gold"} DC1 DC2 DC3
  35. 35. Built-in Object Integrity Verification and Self Healing Durability at the object level  Digital fingerprint is calculated per object upon ingest  Interlocking layers of object-wide and sub-object level integrity protection  Object hash value  Content hash value  CRC checksum  HMAC message authentication digest © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only35 ID: 41E85A1D Data Metadata Data Metadata Fingerprint
  36. 36. Built-in Object Integrity Verification and Self Healing Durability at the object level  Continuous verification: on ingest, retrieval, replication, migration and at rest  Object failing integrity test is automatically replaced with another copy © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only36
  37. 37.  Real time audit feed can be used to create custom reports  Audit logs created in open format allows the use of partner products such as Splunk  Comprehensive audit feed for:  Chargeback and billing  Search integration  Custom reporting  Security diagnostics  Compliance events  Validate performance for SLAs Continuous and active monitoring Audit and Reporting © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only37
  38. 38. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only38  Analyze system performance and activity at every level  Grid Wide
  39. 39. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only39  Analyze system performance and activity at every level  Grid Wide  Site Specific
  40. 40. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only40  Analyze system performance and activity at every level  Grid Wide  Site Specific  Service Level
  41. 41. Data Flow Understanding how data flows through a StorageGRID Webscale system for different operations © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only41
  42. 42. Data flow overview Basic topology and business rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 42  Three departments  Sales  Finance  Marketing  Two pools per department  Capacity pool  Performance pool  Three sites  Two datacenter sites  One remote office  Policy - Finance  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
  43. 43. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Receives object write request from client along with custom metadata  Returns object ID to client and forms the file payload into an object  Packetization  Digital fingerprint  Compression (optional)  Encryption (optional) 43 0x05DFF4338ADCE6F5
  44. 44. Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 44  By default, creates semi-sync local copy for immediate redundancy  Metadata is stored and replicated  Replicate object as per ILM policy  Finance Policy Example  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
  45. 45. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Replication Metadata-driven ILM with optimal resource utilization © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Optimal resources in target storage pools are selected for the replication destination  While honoring ILM the grid considers  Network costs  Server utilization  Storage utilization 45
  46. 46. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Retrieval Requesting objects by the client from StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Receives object read request from client Determines optimal object location relative to request location Streams a copy of the object to the client and verifies integrity of the object on-the- fly 46
  47. 47. Thank You © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only47
  48. 48. Management Services Admin Node  NMS – Network Management System: Provides the administrative interface for configuration and monitoring of the grid.  CMN – Configuration Management Node: Manages system-wide configurations such as connection profiles, grid tasks, and system configuration options.  AMS - Audit Management System: Keeps logs of system activity and events. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only48
  49. 49. Object, Metadata Storage and Replication Storage Nodes  LDR - Local Distribution Router: Stores, moves, verifies, and retrieves object data stored on disks  DDS - Distributed Data Store: Stores, replicates, and protects metadata in the key value store. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only49
  50. 50. Object, Metadata Storage and Replication Storage Nodes - Continued  CMS - Content Management System: Manages object placement and replication based on ILM rules.  ADC - Administrative Domain Controller: Maintains topology information and provides authentication services. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only50
  51. 51. Load Balancing and Client Connectivity API Gateway Nodes  CLB - Connection Load Balancer: Acts as switchboard for connecting clients to the most efficient LDR service for ingest and retrieval.  Ports  8081 CDMI  8082 S3 © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only51
  52. 52. Data storage on archive media Archive Nodes  ARC – Archive: Communicates with archiving middleware to store and retrieve data to and from archive media such as tape. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only52

    Be the first to comment

    Login to see the comments

Se training storage grid webscale technical overview

Views

Total views

597

On Slideshare

0

From embeds

0

Number of embeds

5

Actions

Downloads

16

Shares

0

Comments

0

Likes

0

×