Your SlideShare is downloading. ×

Managing terabytes on aws

640

Published on

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. …

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky.
Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance.
Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.

Published in: Technology, Sports
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
640
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. RequestStorage File Listing ChecksumStorage
  • 2. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB request
  • 3. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB request
  • 4. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS request
  • 5. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS requestPUT request FS request DB + FS request
  • 6. EBS + LVM= WIN!
  • 7. /data250GB 250GB
  • 8. /data250GB 250GB 250GB 250GB
  • 9. /data250GB 250GB 250GB 250GB 250GB 250GB
  • 10. Backup files
  • 11. Backup filesBackup database
  • 12. Backup filesBackup databaseKeep them in sync!
  • 13. Backup filesBackup databaseKeep them in sync!
  • 14. Backup only deltas
  • 15. Backup only deltasManage threads toec2 efficiently
  • 16. Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • 17. Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • 18. CustomerEBSCustomerEBSCustomerEBS
  • 19. De-dupedEBSHA StorageCustomerEBSCustomerEBSCustomerEBS
  • 20. De-dupedGlacierDe-dupedEBSBackupHA StorageCustomerEBSCustomerEBSCustomerEBS
  • 21. Create empty Artifactory instance
  • 22. Create empty Artifactory instancePopulate from backstorage on demand
  • 23. Create empty Artifactory instancePopulate from backstorage on demandDrinkCosmopolitan
  • 24. Conclusions time1. Be lazy.
  • 25. Conclusions time1. Be lazy.2. Be smart.
  • 26. Conclusions time1. Be lazy.2. Be smart.3. Be agile.
  • 27. DrinkCosmopolitan

×