The document provides an overview of AzCopy, a command-line utility for managing Azure storage, allowing data transfer between storage accounts and local file systems. It highlights features such as journaling for resuming incomplete operations, handling JSON and CSV data exports/imports, and working with Azure storage tables through a data movement library. Additionally, it mentions best practices for HTTP communications and references upcoming training events.

1. Consulting/Training
Easy Copy with AzCopy
Using AzCopy to work with Azure Storage

2. Consulting/Training
 Cloud-native storage solution for Microsoft Azure
 Designed for durability, availability
 Pay-as-you-go: amount stored & egress (ingress is free)
 Grouped into “accounts”
 Up to 500 TB of data stored per account
 Up to 100 named accounts per subscription (can be increased via support)
 Storage Services for working with Blob, Table, Queue, File
Azure Storage

3. Consulting/Training
 REST API
 PowerShell, Azure CLI
 Azure Storage SDK
 1st & 3rd Party GUI Tools
 MSFT Storage Explorer List (updated 7/2015): http://j.mp/AzureStorageExplorers
 Direct-Ship Drives
 Azure Services
Working With Data In Azure Storage
And what if I want to move data between Storage Accounts?

4. Consulting/Training
 Command-line utility
 Copy data to/from Blob, File, Table storage
 Operates with local file system or between storage accounts
 Wrapper around .NET Storage API, with parallel execution, journaling,
progress
 http://aka.ms/downloadazcopy
Introducing AzCopy

5. Consulting/Training
AzCopy Fundamentals

6. Consulting/Training
 AzCopy uses a “Journal File”
 Default Location:
%SystemDrive%Users%username%AppDataLocalMicrosoftAzureAzCopy
 Can be repointed with /z
 If the file exists
 If the command matches, the journal contents are used to resume
 If the command does not match, you are prompted to overwrite the journal
 The journal file is deleted upon successful completion of an operation
Resuming Incomplete Operations

7. Consulting/Training
Resuming

8. Consulting/Training
 JSON or CSV specified with /PayloadFormat value
 CSV download includes a .schema.csv file for each data file
 <account name>_<table name>_<timestamp>.<volume index>_<crc>.json
 Volume Index = m_n
 Can Split based on partition key range with PKRS parameter – allows parallel
export
 Can Split based on size with SplitSize parameter (Size in MB, min =32)
 m = partition key range index, n = split index (zeros if omitted)
Working with Azure Storage Tables - Export

9. Consulting/Training
 JSON only
 Uses a manifest file to locate data files and perform validation
 Manifest file is created as part of Export
 <account name>_<table name>_<timestamp>.manifest
 Specify how to handle PK/RK collisions with EntityOperation –
InsertOrSkip, InserOrMerge, InsertOrReplace
Working with Azure Storage Tables - Import

10. Consulting/Training
Storage Tables

11. Consulting/Training
 Based on the core data movement framework that powers AzCopy
 Works with Blobs & Files
 NuGet & Source (with Samples)
 NuGet:
https://www.nuget.org/packages/Microsoft.Azure.Storage.DataMovement
 Git:
https://github.com/Azure/azure-storage-net-data-movement
Introducing the Data Movement Library
One VERY interesting sample – “S3ToAzureSample”

12. Consulting/Training
 TransferManager
 Copy, CopyDirectory, Download, DownlodDirectory, Upload, UploadDirectory
 Set Azure Storage coordinates using “standard” Azure Storage API calls
 Options (Recursive, SearchPattern, etc.)
 TransferContext (ProgressHandler, OverwriteCallback, etc.)
 There are some “best practices” you should follow to set up HTTP
communications the way DML needs it
Working With the Data Movement Library

13. Consulting/Training
The Data Movement API

14. Consulting/Training
April 16, Microsoft Office Alpharetta
https://GAB-ATL.eventbrite.com

15. Consulting/Training
Microsoft Ignite
September 26–30, 2016
Atlanta, GA
ignite.microsoft.com

16. Consulting/Training
Thank You