azure track -04- azure storage deep dive

1. Storage Services Yves Goeleven

2. Still alive?

3. Yves Goeleven Founder of MessageHandler.net • Windows Azure MVP

4. Exhibition theater @ kinepolis

5. www.messagehandler.net

6. What you might have expected Azure Storage Services • VM (Disk persistence) • Azure Files • CDN • StoreSimple • Backup & replica’s • Application storage • Much much more...

7. I assume you know what storage services does!

8. The real agenda Deep Dive Architecture • Global Namespace • Front End Layer • Partition Layer • Stream Layer

9. WARNING!

10. Level 666

11. Which discusses how Storage defies the rules of the CAP theorem • Distributed & scalable system • Highly available • Strongly Consistent • Partition tolerant • All at the same time!

12. That can’t be done, right?

13. Let’s dive in

14. Global namespace

15. Global namespace Every request gets sent to • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName

16. Global namespace DNS based service location • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • DNS resolution • Datacenter (f.e. Western Europe) • Virtual IP of a Stamp

17. Stamp Scale unit • A windows azure deployment • 3 layers • Typically 20 racks each • Roughly 360 XL instances • With disks attached • 2 Petabyte (before june 2012) • 30 Petabyte Intra-stamp replication

18. Incoming read request Arrives at Front End • Routed to partition layer • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • Reads usually from memory Partition Layer Front End Layer FE FE FE FE FE FE FE FE FE PS PS PS PS PS PS PS PS PS

19. Incoming read request 2 nodes per request • Blazingly fast • Flat “Quantum 10” network topology • 50 Tbps of bandwidth • 10 Gbps / node • Faster than typical ssd drive

20. Incoming Write request Written to stream layer • Synchronous replication • 1 Primary Extend Node • 2 Secondary Extend Nodes Front End Layer Stream Layer Partition Layer Primary Secondary Secondary PS PS PS PS PS PS PS PS PS EN EN EN EN EN EN EN

21. Geo replication Intra-stamp replicationIntra-stamp replication Inter-stamp replication

22. Location Service Stamp Allocation • Manages DNS records • Determines active/passive stamp configuration • Assigns accounts to stamps • Geo Replication • Disaster Recovery • Load Balancing • Account Migration

23. Front end layer

24. Front End Nodes Stateless role instances • Authentication & Authorization • Shared Key (storage key) • Shared Access Signatures • Routing to Partition Servers • Based on PartitionMap (in memory cache) • Versioning • x-ms-version: 2011-08-18 • Throttling Front End Layer FE FE FE FE FE FE FE FE FE

25. Partition Map Determine partition server • Used By Front End Nodes • Managed by Partition Manager (Partition Layer) • Ordered Dictionary • Key: http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • Value: Partition Server • Split across partition servers • Range Partition

26. Practical examples Blob • Each blob is a partition • Blob uri determines range • Range only matters for listing • Same container: More likely same range • https://myaccount.blobs.core.windows.net/mycontainer/subdir/file1.jpg • https://myaccount.blobs.core.windows.net/mycontainer/subdir/file2.jpg • Other container: Less likely • https://myaccount.blobs.core.windows.net/othercontainer/subdir/file1.jpg PS PS PS

27. Practical examples Queue • Each queue is a partition • Queue name determines range • All messages same partition • Range only matters for listing queues PS PS PS

28. Practical examples Table • Each group of rows with same PartitionKey is a partition • Ordering within set by RowKey • Range matters a lot! • Expect continuation tokens! PS PS PS

29. Continuation tokens Requests across partitions • All on same partition server • 1 result • Across multiple ranges (or to many results) • Sequential queries required • 1 result + continuation token • Repeat query with token until done • Will almost never occur in development! • As dataset is typically small PS PS PS

30. Partition layer

31. Partition layer Dynamic Scalable Object Index • Object Tables • System defined data structures • Account, Entity, Blob, Message, Schema, PartitionMap • Dynamic Load Balancing • Monitor load to each part of the index to determine hot spots • Index is dynamically split into thousands of Index Range Partitions based on load • Index Range Partitions are automatically load balanced across servers to quickly adapt to changes in load • Updates every 15 seconds

32. Consists of • Partition Master • Distributes Object Tables • Partitions across Partition Servers • Dynamic Load Balancing • Partition Server • Serves data for exactly 1 Range Partition • Manages Streams • Lock Server • Master Election • Partition Lock Partition layer Partition Layer PS PS PS PM PM LS LS

33. How it works Dynamic Scalable Object Index Partition Layer PS PS PS PM PM LS LS Front End Layer FE FE FE FE Account Container Blob Aaaa Aaaa Aaaa Harry Pictures Sunrise Harry Pictures Sunset Richard Images Soccer Richard Images Tennis Zzzz Zzzz zzzz A-H H’-R R’-Z Map

34. Not in the critical path! Dynamic Load Balancing Stream Layer Partition Layer Primary Secondary Secondary PS PS PS PS PS PS PS PS PS EN EN EN EN EN EN EN PM PM LS LS

35. Architecture • Log Structured Merge-Tree • Easy replication • Append only persisted streams • Commit log • Metadata log • Row Data (Snapshots) • Blob Data (Actual blob data) • Caches in memory • Memory table (= Commit log) • Row Data • Index (snapshot location) • Bloom filter • Load Metrics Partition Server Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics

36. HOT Data • Completely in memory Incoming Read request Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics

37. COLD Data • Partially Memory Incoming Read request Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data StreamRow Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics

38. Persistent First • Followed by memory Incoming Write Request Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics

39. Not in critical path! Checkpoint process Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Row Data StreamCommit Log Stream

40. Not in critical path! Geo replication process Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Row Data Stream

41. Stream layer

42. Append-Only Distributed File System • Streams are very large files • File system like directory namespace • Stream Operations • Open, Close, Delete Streams • Rename Streams • Concatenate Streams together • Append for writing • Random reads Stream layer

43. Concepts • Block • Min unit of write/read • Checksum • Up to N bytes (e.g. 4MB) • Extent • Unit of replication • Sequence of blocks • Size limit (e.g. 1GB) • Sealed/unsealed • Stream • Hierarchical namespace • Ordered list of pointers to extents • Append/Concatenate Stream layer Extent Stream //foo/myfile.data Extent Extent Extent Pointer 1 Pointer 2 Pointer 3 Block Block Block Block Block Block Block Block Block Block Block

44. Consists of • Paxos System • Distributed consensus protocol • Stream Master • Extend Nodes • Attached Disks • Manages Extents • Optimization Routines Stream layer Stream Layer EN EN EN EN SM SM SM Paxos

45. Allocation process • Partition Server • Request extent / stream creation • Stream Master • Chooses 3 random extend nodes based on available capacity • Chooses Primary & Secondary's • Allocates replica set • Random allocation • To reduce Mean Time To Recovery Extent creation Stream Layer EN EN EN SM Partition Layer PS Primary Secondary Secondary

46. Bitwise equality on all nodes • Synchronous • Append request to primary • Written to disk, checksum • Offset and data forwarded to secondaries • Ack Replication Process Journaling • Dedicated disk (spindle/ssd) • For writes • Simultaneously copied to data disk (from memory or journal) Stream Layer EN EN EN SM Partition Layer PS Primary Secondary Secondary

47. Ack from primary lost going back to partition layer • Retry from partition layer • can cause multiple blocks to be appended (duplicate records) Dealing with failure Unresponsive/Unreachable Extent Node (EN) • Seal the failed extent & allocate a new extent and append immediately ExtentExtent Stream //foo/myfile.data Extent Extent Pointer 1 Pointer 2 Pointer 3 Block Block Block Block Block Block Block Block Block Block Block Block Extent Block Block Block Extent Block

48. Replication Failures (Scenario 1) Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Block Seal 120 120 Primary Block Block Block Secondary Block Block Block Block Block Secondary Block Block Block Block

49. Replication Failures (Scenario 2) Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Seal 120 100 Primary Block Block Block Secondary Block Block Block Secondary Block Block Block ?

50. Data Stream • Partition Layer only reads from offsets returned from successful appends • Committed on all replicas • Row and Blob Data Streams • Offset valid on any replica • Safe to read Network partitioning during read Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Block

51. Log Stream • Logs are used on partition load • Check commit length first • Only read from • Unsealed replica if all replicas have the same commit length • A sealed replica Network partitioning during read Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block 1, 2 Seal Secondary Block Block Block Block Block Secondary Block Block Block Block

52. Downside to this architecture • Lot’s of wasted disk space • ‘Forgotten’ streams • ‘Forgotten’ extends • Unused blocks in extends after write errors • Garbage collection required • Partition servers mark used streams • Stream manager cleans up unreferenced streams • Stream manager cleans up unreferenced extends • Stream manager performs erasure coding • Reed Solomon algorithm • Allows recreation of cold streams after deleting extends Garbage Collection

53. Wrapup

54. Layering & co-design • Stream Layer • Availability with Partition/failure tolerance • For Consistency, replicas are bit-wise identical up to the commit length • Partition Layer • Consistency with Partition/failure tolerance • For Availability, RangePartitions can be served by any partition server • Are moved to available servers if a partition server fails • Designed for specific classes of partitioning/failures seen in practice • Process to Disk to Node to Rack failures/unresponsiveness • Node to Rack level network partitioning Beating the CAP theorem

55. Want to know more? • White paper: http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf • Video: http://www.youtube.com/watch?v=QnYdbQO0yj4 • More slides: http://sigops.org/sosp/sosp11/current/2011-Cascais/11-calder.pptx • Erasure coding: http://www.snia.org/sites/default/files2/SDC2012/presentations/Cloud/ ChengHuang_Ensuring_Code_in_Windows_Azure.pv3.pdf Resources

56. Still alive?

57. And take home the Lumia 1320 Present your feedback form when you exit the last session & go for the drink Give Me Feedback

58. Follow Technet Belgium @technetbelux Subscribe to the TechNet newsletter aka.ms/benews Be the first to know

59. Belgiums’ biggest IT PRO Conference

azure track -04- azure storage deep dive

ITProceed

azure track -04- azure storage deep dive