Blob sync. Optimized updating of blobs on Azure
•Download as PPTX, PDF•
0 likes•579 views
Optimize the way you update blobs on Azure Blob Storage. Only upload/download the deltas instead of wasting your bandwidth.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Blob sync. Optimized updating of blobs on Azure
- 1. SAVE BANDWIDTH (AND LEARN TO LOVE BLOBS)
- 2. CLOUD STORAGE 101 • DURABLE • HIGHLY AVAILABLE • ACCESS ANYWHERE (WITH CREDENTIALS) • SCALABLE
- 4. CLOUD STORAGE 101: BLOB BASICS • USE BLOCKS OF DATA TO CONSTRUCT BLOB • REPLACE BLOCKS IN EXISTING BLOBS
- 8. CLOUD STORAGE 101 UPLOAD ENTIRE BLOB AGAIN
- 9. CLOUD STORAGE 101 UPLOAD ENTIRE BLOB AGAIN
- 10. WHY?
- 11. CLOUD STORAGE 101 TRY AGAIN
- 15. CLOUD STORAGE 101 UPLOAD SINGLE BLOCK
- 16. BLOBSYNC AWESOMESAUCE • DETECTS CHANGES • DOES NOT NEED ORIGINAL FILE TO DETECT CHANGES • UPLOADS/DOWNLOADS CHANGES ONLY • A TRANSPARENT BLACKBOX… OPEN SOURCE BUT CAN TREAT AS A BLACK BOX
- 18. THEORY VS REALITY • THEORY Azure Blob Storage Local machine
- 19. THEORY VS REALITY • THEORY Azure Blob Storage Local machine 0 100 200 300 400
- 20. THEORY VS REALITY • THEORY Azure Blob Storage Local machine 0 100 200 300 400
- 21. THEORY VS REALITY • THEORY Azure Blob Storage Local machine 0 100 200 300 400
- 22. THEORY…. • IS ALL GOOD IN THEORY
- 23. THEORY VS REALITY • REALITY Azure Blob Storage Local machine 0 100 200 300 400
- 24. THEORY VS REALITY • REALITY Azure Blob Storage Local machine 0 100 200 300 400 A DB C A B’ C D
- 25. FINDING COMMON GROUND • HOW DO WE FIND MOVED BLOCKS?
- 26. FINDING COMMON GROUND • HOW DO WE FIND MOVED BLOCKS? • USE HASH/SIGNATURES FOR EACH BLOCK • SEARCH FOR SIGNATURE ALL THROUGHOUT FILE
- 27. THEORY VS REALITY • SEARCH LOCAL Azure Blob Storage Local machine 0 100 200 300 400 A DB C A B’ C D
- 28. THEORY VS REALITY • SEARCH LOCAL • EG. SEARCH FOR ‘C’ Local machine 0 100 200 300 400 A B’ C D
- 29. SUCCESS! • CAN NOW FIND BLOCKS EVEN WHEN MOVED
- 30. SUCCESS! • CAN NOW FIND BLOCKS EVEN WHEN MOVED • IF WE CAN FIND A BLOCK WE CAN DETERMINE IF WE CAN REUSE IT
- 31. SUCCESS! • CAN NOW FIND BLOCKS EVEN WHEN MOVED • IF WE CAN FIND A BLOCK WE CAN DETERMINE IF WE CAN REUSE IT • BUT…….
- 32. SUCCESS! • CAN NOW FIND BLOCKS EVEN WHEN MOVED • IF WE CAN FIND A BLOCK WE CAN DETERMINE IF WE CAN REUSE IT • BUT……. • MD5/SHA ETC ARE TOO SLOW TO DO THIS
- 33. • TOO SLOW? NO WAY! • EG • 100MB FILE/BLOB • BLOCK OF 100K • > 104M HASH CALCULATIONS. JUST TO FIND THAT ONE BLOCK
- 34. YOU HAVE TO ROLL WITH IT. • ROLLING SIGNATURE • EXTREMELY QUICK.
- 35. YOU HAVE TO ROLL WITH IT. • ROLLING SIGNATURE • EXTREMELY QUICK. • DUE TO FALSE POSITIVES USE MD5/SHA AS CONFIRMATION STEP
- 36. YOU HAVE TO ROLL WITH IT. • SIG = FUNC( 0 .. 4 )
- 37. YOU HAVE TO ROLL WITH IT. • SIG = FUNC( 0 .. 4 ) • CALCULATE SIG OF 1..5 BASED OFF OLD SIG • NEW SIG = OLDSIG – ARRAY[0] + ARRAY[5]
- 38. YOU HAVE TO ROLL WITH IT. • CAN SEARCH ENTIRE FILE WITH MINIMAL CALCULATIONS. IE FAST!
- 39. SO WHAT NOW? • CAN NOW SEARCH FILES QUICKLY FOR SIGNATURE MATCHES • MEANS WE CAN FIGURE OUT WHAT IS COMMON BETWEEN CLOUD AND LOCAL • CAN DOWNLOAD/UPLOAD ONLY THE DIFFERENCES.
- 40. PROVE IT!
- 41. FILE INTERNALS
- 43. LIES, MORE LIES AND STATISTICS • SMALL DB (14M). • CLEARED A SMALL TABLE. • UPDATE 340K • LARGE DB (555M). • CLEARED A SMALL TABLE • UPDATE 720K • VM (8G). • DELETED SOME FILES • UPDATE 800M
- 44. UPCOMING CHANGES • DEFRAG • DYNAMICALLY DETERMINE BLOCK SIZE • BETTER PARALLEL UPLOAD/DOWNLOAD • 32 BIT VERSION
- 45. LINKS • BLOG ON BLOBSYNC: • HTTPS://KPFAULKNER.WORDPRESS.COM/CATEGORY/BLOBSYNC/ • NUGET PACKAGE: • HTTPS://WWW.NUGET.ORG/PACKAGES/BLOBSYNC/ • GITHUB WITH SOURCE: • HTTPS://GITHUB.COM/KPFAULKNER/BLOBSYNC/