Managing terabytes on aws

  • 576 views
Uploaded on

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. …

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky.
Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance.
Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.

More in: Technology , Sports
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
576
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. RequestStorage File Listing ChecksumStorage
  • 2. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB request
  • 3. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB request
  • 4. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS request
  • 5. RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS requestPUT request FS request DB + FS request
  • 6. EBS + LVM= WIN!
  • 7. /data250GB 250GB
  • 8. /data250GB 250GB 250GB 250GB
  • 9. /data250GB 250GB 250GB 250GB 250GB 250GB
  • 10. Backup files
  • 11. Backup filesBackup database
  • 12. Backup filesBackup databaseKeep them in sync!
  • 13. Backup filesBackup databaseKeep them in sync!
  • 14. Backup only deltas
  • 15. Backup only deltasManage threads toec2 efficiently
  • 16. Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • 17. Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • 18. CustomerEBSCustomerEBSCustomerEBS
  • 19. De-dupedEBSHA StorageCustomerEBSCustomerEBSCustomerEBS
  • 20. De-dupedGlacierDe-dupedEBSBackupHA StorageCustomerEBSCustomerEBSCustomerEBS
  • 21. Create empty Artifactory instance
  • 22. Create empty Artifactory instancePopulate from backstorage on demand
  • 23. Create empty Artifactory instancePopulate from backstorage on demandDrinkCosmopolitan
  • 24. Conclusions time1. Be lazy.
  • 25. Conclusions time1. Be lazy.2. Be smart.
  • 26. Conclusions time1. Be lazy.2. Be smart.3. Be agile.
  • 27. DrinkCosmopolitan