Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series


Published on

Customers using Amazon S3 at large scale benefit greatly from storage management features. Storage lifecycle policies help them reduce storage costs. Cross-region replication makes it easier to copy data between AWS regions for compliance or disaster recovery. Event notifications allow automatic initiation of processes on objects as they arrive, or capture information about objects and log it for security purposes. In this session, you'll learn about these features, and we'll also introduce several new storage management features like S3 Object Tagging, S3 Analytics Storage Class Analysis, S3 Inventory, S3 CloudWatch Metrics, and S3 CloudTrail integration that give users unmatched visibility into what data they are storing and how that data is being used. These new features make it simpler to analyze usage by users, apps, or organizations, to highlight anomalies, and to optimize business process workflows. They also help identify opportunities to reduce costs, improve performance, and archive infrequently used data. In addition, they can provide insight into who is accessing data stored in S3.

Learning Objectives:
• Understand the portfolio of storage management features available in S3
• Learn to manage cloud storage based upon what the data is (instead of where it's stored) using S3 Object Tagging
• Understand the benefits of using S3 Inventory over the LIST API
• Examine how S3 Analytics Storage Class Analysis helps to build lifecycle policies to optimize storage consumption
• Learn about how new S3 integration with AWS CloudWatch and AWS CloudTrail help enhance performance and security

Published in: Technology
  • Be the first to comment

Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sr. Product Manager, AWS Deep Dive on S3 Storage Management Omair Gillani
  2. 2. Batches and Streams AWS Direct Connect AWS Snowball, Snowball Edge, Snowmobile 3rd Party Connectors Transfer Acceleration AWS Storage Gateway Amazon Kinesis Firehose File BlockObject Amazon GlacierAmazon S3 Amazon EC2 Instance Store (ephemeral) Amazon EFS Amazon EBS (persistent)
  3. 3. AWS Direct Connect AWS Snowball ISV Connectors Amazon Kinesis Firehose S3 Transfer Acceleration AWS Storage Gateway Data transfer into Amazon S3 AWS Snowmobile AWS Snowball Edge
  4. 4. Storage Gateway hybrid storage solutions Enables using standard storage protocols to access AWS storage services Customer Premises Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Enterprise storage Devices Application servers
  5. 5. What’s new? File gateway File interface to objects in your Amazon S3 buckets Virtual on-premises file server Store and retrieve Amazon S3 objects through standard file storage protocols (NFS v3 and v4.1) Local cache for low-latency access Direct access to objects with native S3 capabilities, such as versioning, lifecycle management, and cross region replication
  6. 6. What’s new? We’re reducing prices Effective December 1, 2016 Storage prices reduced Volume and virtual tape storage by up to 28% Archived virtual tape storage by up to 43% Flat per GB price replaces monthly gateway fee $0.01 per GB written to AWS storage by your gateway* Simplified virtual tape retrieval pricing $0.01 per GB retrieved replaces peak-rate fee * Up to $125/month per gateway
  7. 7. Cross-Region ReplicationLifecycle Policy S3 Object TagsEvent Notifications Amazon S3 CloudWatch Metrics S3 Inventory Audit with AWS CloudTrail S3 Data Events S3 Analytics Standard Standard - Infrequent Access Amazon Glacier Storage Management for S3
  8. 8. Understand your storage usage S3 InventoryAnalyze Logs with Amazon EMR S3 Analytics
  9. 9. S3 Inventory Save time Daily or Weekly delivery Delivery to S3 bucketCSV File Output Trigger business workflows and applications such as secondary index, garbage collection, data auditing, and offline analytics Half the price of LIST API at $0.0025 per million objects listed
  10. 10. S3 Inventory More information about your objects than provided by LIST API such as replication status, multipart upload flag, and delete marker Name Value Type Description Bucket String Bucket name. UTF-8 encoded. Key String Object key name. UTF-8 encoded. Version Id String Version Id of the object Is Latest Boolean true if object is the latest version (current version) of a versioned object, otherwise false Delete Marker Boolean true if object is a delete marker of a versioned object, otherwise false Size Long Object size in bytes Last Modified String Last modified timestamp. Format in ISO: YYYY-MM-DDTHH:mm:ss.SSSZ ETag String eTag in HEX encoded format StorageClass String Valid values: STANDARD, REDUCED_REDUNDANCY, GLACIER, STANDARD_IA. UTF-8 encoded. Multipart Uploaded Boolean true if object is uploaded by using multipart, otherwise false Replication Status String Valid values: REPLICA, COMPLETED, PENDING, FAILED. UTF-8 encoded.
  11. 11. S3 Inventory Setup notification when S3 Inventory is complete /Data /<InventoryFile>.gz /<InventoryFile>.gz … /<DayofReport> /manifest.json /manifest.checksum … AWS Lambda Amazon SQS Amazon SNS
  12. 12. Eventually consistent rolling snapshot S3 Inventory New objects may not be listed Recently deleted objects may still be included O1 O2 O3 O1 O2 O3 O1 O2 O1 O2 O3 NEW Validate before you act! Use HEAD OBJECT or GET OBJECT
  13. 13. S3 Analytics – Storage Class Analysis Analyze buckets, prefixes or tags $0.10 per million objects analyzed per month Daily Storage Class Analysis & Lifecycle candidates Data-driven storage management for S3 Export Analysis data to your S3 bucket
  14. 14. S3 Analytics – Storage Class Analysis
  15. 15. S3 Analytics – Storage Class Analysis
  16. 16. S3 Analytics – Storage Class Analysis Export to use BI tool of your choice
  17. 17. S3 Analytics – Storage Class Analysis Daily Export Dataset
  18. 18. Simple to configure S3 Analytics S3 Management Console PUT Bucket Analytics Multiple Policy Documents <AnalyticsConfiguration> <Id>...</Id> <Filter> ... </Filter> <StorageClassAnalysis> <DataExport> ... </DataExport> </StorageClassAnalysis> </AnalyticsConfiguration>
  19. 19. Monitor your storage Monitor and Alert with CloudWatch Audit your storage with CloudTrail Data Events Server Access Logs
  20. 20. CloudWatch metrics for S3 Operational & performance monitoring  Generate metrics for data of your choice  Entire bucket, Prefixes, and Tags  Up to 1,000 object groups  1-minute CloudWatch metrics  Alert and alarm on metrics
  21. 21. CloudWatch metrics for S3 Metric Name Metric value AllRequests Count PutRequests Count PostRequests Count GetRequests Count ListRequests Count DeleteRequests Count HeadRequests Count Metric Name Metric value BytesDownloaded MB BytesUploaded MB 4xxErrors Count 5xxErrors Count FirstByteLatency ms TotalRequestLatency ms $0.50 per metric per month
  22. 22. S3 Data Events in CloudTrail Perform security analysis, meet your IT auditing and compliance needs, and take immediate action on object-level activity to immediately improve security posture Pricing: $1 per million data events recorded and storage charges apply Log object level operations Changes to bucket configurations SNS notification for log delivery
  23. 23. Manage your data Cross-Region Replication Lifecycle Policies Event Notifications S3 Object Tags
  24. 24. Manage your data S3 Object Tags Easily manage and control access for Amazon S3 objects  Classify your data  Tag your objects with key-value pairs  Write policies once based on the type of data AnalyzeLifecycle PolicyAccess Control
  25. 25. Deep dive on tags  Tags are key-value pairs  Maximum 10 tags per object  Maximum key length—127 Unicode characters  Maximum value length—255 Unicode characters  Tag keys and values are case-sensitive. 2 ways to put tags via API  Put objects with tag parameter, or  add tag API after object is created Simple pricing  $0.01 per 10,000 tags per month
  26. 26. Lifecycle policies based on tags <LifecycleConfiguration> <Rule> <ID>sample-rule</ID> <Filter> <And> <Prefix>documents/</Prefix> <Tag> <Key>Project</Key> <Value>Delta</Value> </Tag> <Tag> <Key>Data type</Key> <Value>HPI</Value> </Tag> </And> </Filter> <Status>Enabled</Status> <Transition> <Days>365</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>3650</Days> </Expiration> </Rule> </LifecycleConfiguration>  Transition or expire storage using tags  Simplify S3 lifecycle policies  Filter with prefix, tag, or both
  27. 27. Thank you!