Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1© Copyright 2011 EMC Corporation. All rights reserved.
ATMOS
ARCHITECTURE
OVERVIEW
Mark O’Connell
2© Copyright 2011 EMC Corporation. All rights reserved.
Agenda
•  Why Object Storage
•  Architecture
•  Data Management at...
3© Copyright 2011 EMC Corporation. All rights reserved.
Trends in IT
•  Companies, information, and services are global
• ...
4© Copyright 2011 EMC Corporation. All rights reserved.
EMC Atmos – Managing Big Data in the
Cloud
•  Purpose-built cloud ...
5© Copyright 2011 EMC Corporation. All rights reserved.
Typical Customer Usage Models
Build and Manage Enterprise Cloud
St...
6© Copyright 2011 EMC Corporation. All rights reserved.
EMC Atmos Design Principles
•  Massively scalable infrastructure
–...
7© Copyright 2011 EMC Corporation. All rights reserved.
EMC Atmos Design Principles
•  Data services and access
methods
– ...
8© Copyright 2011 EMC Corporation. All rights reserved.
LAN access protocols
Atmos Native Client Access API
Firewall
LAN a...
9© Copyright 2011 EMC Corporation. All rights reserved.
CIFS
Atmos Native Client Access API
Firewall
LAN access
WAN access...
10© Copyright 2011 EMC Corporation. All rights reserved.
Atmos – Storage Architecture
One per node
Locally stores data
No ...
11© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Storage Architecture
•  MDS (MetaData Service) design is th...
12© Copyright 2011 EMC Corporation. All rights reserved.
Creating an Atmos
System
13© Copyright 2011 EMC Corporation. All rights reserved.
Architecture Overview – Building Blocks
•  Industry standard comp...
14© Copyright 2011 EMC Corporation. All rights reserved.
REST
clients
IFS clients CAS
clients
Architecture Overview – Virt...
15© Copyright 2011 EMC Corporation. All rights reserved.
Forming a System from the Building
Blocks
RMG 2
(London)
RMG 1
(B...
16© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Data Storage
17© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Object Creation and RMG
Interactions
1)  REST request recv’...
18© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Data Path – Data Placement
How is it decided where the
meta...
19© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Data Path – Policy Specification
Policy specifications deta...
20© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Data Path – Policy Selector
Policy selectors use criteria f...
21© Copyright 2011 EMC Corporation. All rights reserved.
Accessing Data in
Atmos
22© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Read interactions with RMG and
Policy
1)  REST read request...
23© Copyright 2011 EMC Corporation. All rights reserved.
Decomposition of an Atmos object
•  Blob
–  The actual data bits ...
24© Copyright 2011 EMC Corporation. All rights reserved.
Accessing Your Data
•  How can I identify the piece of data that ...
25© Copyright 2011 EMC Corporation. All rights reserved.
Accessing Your Data Using Metadata
•  Associate descriptive metad...
26© Copyright 2011 EMC Corporation. All rights reserved.
Accessing Your Data
•  Atmos provides multiple object access sche...
27© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Multi-tenancy
28© Copyright 2011 EMC Corporation. All rights reserved.
Architecture Overview
Atmos Tenancy Model
•  Foundation for cloud...
29© Copyright 2011 EMC Corporation. All rights reserved.
Blob storage
Atmos Tenancy Model
Tenants
•  Dedicated network con...
30© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Tenancy Model - Data
•  Tenants
–  Tenancy is a logical com...
31© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Tenancy Model - Data
•  Users
–  Authenticates to store/ret...
32© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Tenancy Model - Management
•  SysAdmin
–  SysAdmin creates ...
33© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Tenancy Model - Management
•  Subtenant admin
–  Subtenant ...
34© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Tenancy Model – Policy
Application
Storage Policy
Subtenant...
35© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Management Strategy
•  Atmos storage was designed to be cus...
36© Copyright 2011 EMC Corporation. All rights reserved.
Default Atmos GUI
•  Common operations available in the default G...
37© Copyright 2011 EMC Corporation. All rights reserved.
Customizing the Atmos GUI – Atmos
Online
•  Designed for a fully ...
38© Copyright 2011 EMC Corporation. All rights reserved.
Atmos Architecture Summary
•  A multi-site architecture for stori...
39© Copyright 2011 EMC Corporation. All rights reserved.
See Atmos in Action – Visit Booth 201
Theatre presentations every...
40© Copyright 2011 EMC Corporation. All rights reserved.
Additional Resources
Atmos Web landing page
http://www.emc.com/pr...
41© Copyright 2011 EMC Corporation. All rights reserved.
42© Copyright 2011 EMC Corporation. All rights reserved.
THANK YOU
Upcoming SlideShare
Loading in …5
×

Presentation atmos architecture overview

Download & Share Technology
Presentations http://goo.gl/k80oY0
Student Guide & Best http://goo.gl/6OkI77

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Presentation atmos architecture overview

  1. 1. 1© Copyright 2011 EMC Corporation. All rights reserved. ATMOS ARCHITECTURE OVERVIEW Mark O’Connell
  2. 2. 2© Copyright 2011 EMC Corporation. All rights reserved. Agenda •  Why Object Storage •  Architecture •  Data Management at Scale •  Multi-tenancy for management at scale
  3. 3. 3© Copyright 2011 EMC Corporation. All rights reserved. Trends in IT •  Companies, information, and services are global •  Growth of unstructured, digital information •  24x7, worldwide access to data •  Growth in data access points – laptops, mobiles, etc. •  Scale out architecture, grow with your data needs •  Huge worldwide networking infrastructural building blocks
  4. 4. 4© Copyright 2011 EMC Corporation. All rights reserved. EMC Atmos – Managing Big Data in the Cloud •  Purpose-built cloud storage platform •  Globally distributed •  Intelligent data management •  Web-scale
  5. 5. 5© Copyright 2011 EMC Corporation. All rights reserved. Typical Customer Usage Models Build and Manage Enterprise Cloud Storage•  Manage global unstructured data •  Protect data •  Reduce IT storage costs Make Existing Storage Cloud-Ready •  Utilize existing storage resources and hardware •  Atmos Cloud storage in virtual environment Provide Cloud Storage-as-a-service •  Service multiple consumers •  Provide secure multitenancy •  Meter and bill for usage Archive to the Cloud •  Isolate distributed archives •  Manage chargeback •  Ensure security and protect data
  6. 6. 6© Copyright 2011 EMC Corporation. All rights reserved. EMC Atmos Design Principles •  Massively scalable infrastructure –  Multiple petabytes; billions of objects –  Hundreds of sites, fully distributed –  Unified namespace •  Policy-based management –  Policy tells content where to go, what actions to take –  Integrated object based metadata –  Event driven policy evaluation
  7. 7. 7© Copyright 2011 EMC Corporation. All rights reserved. EMC Atmos Design Principles •  Data services and access methods –  Replication, GeoProtection, compression, de-duplication, spin-down –  Web Services (REST/SOAP) and file system access –  Full metering and billing infrastructure •  Management at scale –  Auto-config and auto-healing architecture –  Multi-tenancy –  Manage more with fewer people –  Partitioned management services allows sharing a common infrastructure amongst disparate users
  8. 8. 8© Copyright 2011 EMC Corporation. All rights reserved. LAN access protocols Atmos Native Client Access API Firewall LAN access WAN access REST REST Server Blob storage Runs on each node Runs across nodes Atmos High-Level Architecture
  9. 9. 9© Copyright 2011 EMC Corporation. All rights reserved. CIFS Atmos Native Client Access API Firewall LAN access WAN access Installable Filesystem NFS Atmos CAS REST GeoDrive Browser NFS CIFS Custom Apps REST Server CDP log collection and processing Blob storage Atmos - Access Architecture
  10. 10. 10© Copyright 2011 EMC Corporation. All rights reserved. Atmos – Storage Architecture One per node Locally stores data No communication with other SSs Atmos Native Client Access API Write K46DJ…3X2 Replicated Replicated, identifies an MDS Queues asynchronous replication requests Track online storage resources Policy Management Policy Management Policy Management Storage Service Storage Service Job Service Resource Management Service A-M N-Z Metadata Service Metadata Service Node 2Node 1 MDLS MDLS MetaData Location Service
  11. 11. 11© Copyright 2011 EMC Corporation. All rights reserved. Atmos Storage Architecture •  MDS (MetaData Service) design is the key to scalable global storage •  Each MDS manages a slice of the object namespace –  3x replicated for HA and availability –  MDS unavailability only affects some objects •  MDS can use any available storage server –  Storage outages do not affect ability to store new data –  No impact for reads unless no stored copies are available •  Normal operation favors MDSs local to the client –  Reduces latency for most operations while maintaining global availability •  REST protocol for globally scalable access
  12. 12. 12© Copyright 2011 EMC Corporation. All rights reserved. Creating an Atmos System
  13. 13. 13© Copyright 2011 EMC Corporation. All rights reserved. Architecture Overview – Building Blocks •  Industry standard components –  Processor, memory, etc details •  Standard data center environments •  Factory configured, racked, and cabled •  Flexible, cost-effective service model •  User serviceable •  Up to 30 drives per 3U drawer •  Up to 720 TB per rack •  120, 240, or 360 drive configurations WS 360WS 120/240 Compute Configuration •  1:15 Server-to-drive ratio •  120/240/480 TB capacity •  GbE or 10GbE connectivity Capacity Configuration •  1:60 Server-to-drive ratio •  360/720 TB capacity •  GbE or 10GbE connectivity
  14. 14. 14© Copyright 2011 EMC Corporation. All rights reserved. REST clients IFS clients CAS clients Architecture Overview – Virtual Building Block Ÿ  Identical software components and capabilities in a virtual (VMware ESX) environment − Interoperates with HW based Atmos deployments Ÿ  Extends any storage with a web services interface and object storage − Heterogeneous SAN, NAS, and Object access on a single platform − Enables scale out unified object storage across data centers or geographies IS1 IS4IS3IS2 RMG ESX Servers Atmos RMG, 4 IS, 8 Nodes FC/ NFS RES T LAN Protocols Atmos CAS
  15. 15. 15© Copyright 2011 EMC Corporation. All rights reserved. Forming a System from the Building Blocks RMG 2 (London) RMG 1 (Boston) RMG 3 (Raleigh) RMG 4 (Rome) IS1 IS4IS3IS2 RMG ESX Server s •  Each physical or virtual building block is typically one RMG –  RMG = Resource Management Group, unit of geographic scaling –  LAN connected nodes, typically collocated –  One RMG can house multiple racks •  RMGs organize the nodes into distinct locations –  Used in policy placement of data –  Resource availability tracked locally and then communicated globally •  Client requests localized via RMG •  Responses optimized to use resources local to the RMG
  16. 16. 16© Copyright 2011 EMC Corporation. All rights reserved. Atmos Data Storage
  17. 17. 17© Copyright 2011 EMC Corporation. All rights reserved. Atmos Object Creation and RMG Interactions 1)  REST request recv’d 2)  MDLS finds a Metadata Service (normally in local RMG), creates record 3)  Local data copy(s) stored, normally synchronous 4)  Asynchronous remote data copy(s) queued via Job Service 5)  Multiple remote asynchronous copies leverage RMG local resources 6)  A record is made of bandwidth and storage consumed RMG 2 (London) RMG 1 (Boston) 3 6 2 4 5 1
  18. 18. 18© Copyright 2011 EMC Corporation. All rights reserved. Atmos Data Path – Data Placement How is it decided where the metadata and data are placed? •  User policies guide the placement decision •  Can be based on –  Initiating user –  Location of request –  User specified metadata –  Or more •  Policy specifies –  Metadata location –  Data protection level –  Data protection location RMG 2 (London) RMG 1 (Boston) RMG 3 (Raleigh) RMG 4 (Rome) IS1 IS4IS3IS2 RMG ESX Server s
  19. 19. 19© Copyright 2011 EMC Corporation. All rights reserved. Atmos Data Path – Policy Specification Policy specifications detail how an object is to be stored within the system Metadata is normally placed in the RMG which receives the create request Specify data storage options – N way mirroring, erasure encoding, location, sync vs async, etc.
  20. 20. 20© Copyright 2011 EMC Corporation. All rights reserved. Atmos Data Path – Policy Selector Policy selectors use criteria from the object to choose how the object will be stored. Specify the policy specification, which controls where the data is stored Specify how the decision is made – user metadata value (s), system metadata, etc Specify if the policy should be applied when objects are created or when they are updated
  21. 21. 21© Copyright 2011 EMC Corporation. All rights reserved. Accessing Data in Atmos
  22. 22. 22© Copyright 2011 EMC Corporation. All rights reserved. Atmos Read interactions with RMG and Policy 1)  REST read request recv’d 2)  MDLS uses Object ID to identify Metadata Service 3)  Metadata Service returns locations of all replicas 4)  Closest replica used to satisfy the read 5)  A record is made of the bandwidth consumed RMG 2 (London) RMG 1 (Boston) 3 5 2 4 51 43 3 1
  23. 23. 23© Copyright 2011 EMC Corporation. All rights reserved. Decomposition of an Atmos object •  Blob –  The actual data bits of the object, protected according to policy •  System Metadata –  Standard information added automatically to the object –  E.g. time last modified, time created, size, name, storage policy, etc. •  ACL –  Security information about which UIDs may access this object –  May be freely changed by the object creator and others with access •  User Metadata –  Arbitrary information which can be associated with an object –  Can be used to drive policy decisions for the object –  Applications independent of the creator can add metadata •  Subject to object ACL permission –  Can be declared listable for easy object access
  24. 24. 24© Copyright 2011 EMC Corporation. All rights reserved. Accessing Your Data •  How can I identify the piece of data that I want? –  Use the object id •  May require an external database to manage the object ids –  Named objects (filesystem names) /home/finance/records/2011
  25. 25. 25© Copyright 2011 EMC Corporation. All rights reserved. Accessing Your Data Using Metadata •  Associate descriptive metadata with objects of interest •  Use listable metadata tags to quickly index similar items –  Walk the listable tag, possibly examining the object(s) to find the one of interest, e.g. •  Records from 2011 •  Records from blackjack table with dealer = “Mark O’Connell”
  26. 26. 26© Copyright 2011 EMC Corporation. All rights reserved. Accessing Your Data •  Atmos provides multiple object access schemes –  Object id •  Automatically assigned by the Atmos system •  Fastest mechanism to retrieve your data –  Named objects •  Optionally associate a name with a particular object •  Names are like pathnames (e.g. /mark/files/presentation.ppt) •  Supports directory list operations •  Additional overhead on R/W to translate name –  Listable tags •  Based on metadata in the object •  Quickly find a set of objects which share a certain property •  Useful for application specific index and search operations
  27. 27. 27© Copyright 2011 EMC Corporation. All rights reserved. Atmos Multi-tenancy
  28. 28. 28© Copyright 2011 EMC Corporation. All rights reserved. Architecture Overview Atmos Tenancy Model •  Foundation for cloud storage architecture –  Conceptual grouping of resources –  Hierarchical model enables flexible application integration –  Platform to support millions of users –  Provides robust reporting and resource management •  Improves operational efficiency –  Manage access and resources per tenant/subtenant –  Allows optimal policy definition and management
  29. 29. 29© Copyright 2011 EMC Corporation. All rights reserved. Blob storage Atmos Tenancy Model Tenants •  Dedicated network connections − Control of ingest bandwidth •  Controls storage usage (policies) − N+3, N+6, 2x mirror, etc •  Creates subtenants − Storage system + storage admin Subtenants •  Preconfigured storage system •  A distinct object namespace •  Can be metered and billed •  Manages/creates application connection credentials UIDs •  Represent an entity which can connect to the blob store and create/access objects •  Can be metered/billed N+3 2x mirror Tenant 1 Tenant 2 Tenant 3
  30. 30. 30© Copyright 2011 EMC Corporation. All rights reserved. Atmos Tenancy Model - Data •  Tenants –  Tenancy is a logical compartmentalization of data and resources –  Tenants have their own access nodes, security control, storage policies, and access to the data •  Not aware of any other resources except for its own •  No access to other tenant’s resources –  All underlying resources shared among Tenants •  Subtenants –  Each Subtenant owns a distinct storage environment •  Users, objects and filesystem •  UserID needs to be unique only within a subtenant –  Each subtenant manages a disjoint set of data •  Stronger than an ACL, like a virtual storage machine –  Subtenant inherits policy specifications and selectors from the Tenant –  Subtenant may reorder policy specifications
  31. 31. 31© Copyright 2011 EMC Corporation. All rights reserved. Atmos Tenancy Model - Data •  Users –  Authenticates to store/retrieve/manipulate data in the cloud –  Per-user listable tags (metadata organization) –  May share files with other users in the same subtenant (via ACLs)
  32. 32. 32© Copyright 2011 EMC Corporation. All rights reserved. Atmos Tenancy Model - Management •  SysAdmin –  SysAdmin creates tenants –  SysAdmin assigns nodes to tenants –  SysAdmin manages the physical state of the system •  Upgrades, adding nodes to the system, etc –  SysAdmin has no data access, SysAdmin cannot create/alter policies •  Tenant admin –  Tenant admin creates policy specifications – how data will be stored in the system –  Tenant admin creates subtenants (virtual storage environment) and subtenant admins –  Tenant admin assigns policies to a subtenant
  33. 33. 33© Copyright 2011 EMC Corporation. All rights reserved. Atmos Tenancy Model - Management •  Subtenant admin –  Subtenant admin allows access to the storage environment (creates/deletes users) –  Subtenant may have a level of control over storage policies which are in effect •  If the tenant admin allows multiple policies to be used by a subtenant storage environment, then the subtenant admin has a level of control over how the policies are applied •  Users –  No management rights
  34. 34. 34© Copyright 2011 EMC Corporation. All rights reserved. Atmos Tenancy Model – Policy Application Storage Policy Subtenants and Assigned Policy Selectors 2 Sync copies 3 Sync EC copies 1 Sync copy 2 Sync copies 1 async copy India 2 Sync EC copies 1 async EC India Engineering Default policy Normal Finance Criticality Low, use normal Default policy Payroll IT Beta Default policy Test Research Size > 1MB, use Silver Criticality High, use Gold Normal Payroll Test Silver Gold
  35. 35. 35© Copyright 2011 EMC Corporation. All rights reserved. Atmos Management Strategy •  Atmos storage was designed to be customized to the needs of multiple individual environments –  Tenant and subtenant model –  Multiple individuals can have multiple custom environments on one storage system •  Atmos management was designed to be similarly customized •  Full REST/POX API available for management operations –  Allows customizable management flows •  Management interface could be different for each tenant admin or subtenant admin –  Authenticate as SysAdmin, TenantAdmin, or SubtenantAdmin –  Management API can also be used to fully automate Atmos system management
  36. 36. 36© Copyright 2011 EMC Corporation. All rights reserved. Default Atmos GUI •  Common operations available in the default GUI –  Logins available for SysAdmin, TenantAdmin, SubtenantAdmin –  Different screens and information shown based on login
  37. 37. 37© Copyright 2011 EMC Corporation. All rights reserved. Customizing the Atmos GUI – Atmos Online •  Designed for a fully automated compute and storage service •  Customized workflow based on the Atmos system management APIs –  Allows automated onboarding of new customers, querying of billing information, etc •  Available at www.atmosonline.com
  38. 38. 38© Copyright 2011 EMC Corporation. All rights reserved. Atmos Architecture Summary •  A multi-site architecture for storing massive amounts of content •  Policy management allows control over data placement and protection •  Multi-tenancy for both data and management –  Allows customization of the storage for multiple, independent environments –  Allows a fine degree of logical separation of data –  Allows scoped management control for administrators
  39. 39. 39© Copyright 2011 EMC Corporation. All rights reserved. See Atmos in Action – Visit Booth 201 Theatre presentations every half hour Find Atmos-powered Storage-as-a- Service options available worldwide See how easy it is to integrate and customize Atmos via the open API and SDK Learn how Centera and Atmos work together for compliance via the cloud Sneak peek at new features – like GeoDrive! Meet with experts, peers and customers Enter to Win an iPad 2.0
  40. 40. 40© Copyright 2011 EMC Corporation. All rights reserved. Additional Resources Atmos Web landing page http://www.emc.com/products/family/atmos.htm Video: Why Beth Israel Deaconess Chooses Atmos for Cloud Computing http://www.emc.com/collateral/demos/microsites/mediaplayer-video/beth-israel- deaconess-emc.htm IDC Analyst Paper, EMC Atmos: Making Rain in the Clouds http://info.emc.com/mk/get/SDL? reg_src=web&P.ctp_program_execution.Source_ID=AMA00018247 Analyst Report: ESG: EMC Atmos Cloud Storage Helps Vistaprint Cut Storage Infrastructure Costs and Accelerate Production http://www.emc.com/collateral/analyst-reports/esg-solution-impact-analysis-emc- vistaprint.pdf Sandbox: Atmos Online Developer Network https://community.emc.com/community/edn/atmos
  41. 41. 41© Copyright 2011 EMC Corporation. All rights reserved.
  42. 42. 42© Copyright 2011 EMC Corporation. All rights reserved. THANK YOU

    Be the first to comment

    Login to see the comments

Download & Share Technology Presentations http://goo.gl/k80oY0 Student Guide & Best http://goo.gl/6OkI77

Views

Total views

653

On Slideshare

0

From embeds

0

Number of embeds

17

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×