Managing terabytes on aws
Upcoming SlideShare
Loading in...5
×
 

Managing terabytes on aws

on

  • 778 views

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. ...

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky.
Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance.
Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.

Statistics

Views

Total Views
778
Views on SlideShare
778
Embed Views
0

Actions

Likes
2
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Managing terabytes on aws Managing terabytes on aws Presentation Transcript

  • RequestStorage File Listing ChecksumStorage
  • RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB request
  • RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB request
  • RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS request
  • RequestStorage File Listing ChecksumStorageMetadatarequestsCache check, FSrequestDB requestHEAD requests Cache check, FSrequestDB requestGET request Cache check, FSrequestDB + FS requestPUT request FS request DB + FS request
  • EBS + LVM= WIN!
  • /data250GB 250GB
  • /data250GB 250GB 250GB 250GB
  • /data250GB 250GB 250GB 250GB 250GB 250GB
  • Backup files
  • Backup filesBackup database
  • Backup filesBackup databaseKeep them in sync!
  • Backup filesBackup databaseKeep them in sync!
  • Backup only deltas
  • Backup only deltasManage threads toec2 efficiently
  • Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • Backup only deltasManage threads toec2 efficientlyKeep the dataisolated
  • CustomerEBSCustomerEBSCustomerEBS
  • De-dupedEBSHA StorageCustomerEBSCustomerEBSCustomerEBS
  • De-dupedGlacierDe-dupedEBSBackupHA StorageCustomerEBSCustomerEBSCustomerEBS
  • Create empty Artifactory instance
  • Create empty Artifactory instancePopulate from backstorage on demand
  • Create empty Artifactory instancePopulate from backstorage on demandDrinkCosmopolitan
  • Conclusions time1. Be lazy.
  • Conclusions time1. Be lazy.2. Be smart.
  • Conclusions time1. Be lazy.2. Be smart.3. Be agile.
  • DrinkCosmopolitan