Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

One billion notes as 'Small Data' (Dave Engberg)

1,025 views

Published on

  • Be the first to comment

One billion notes as 'Small Data' (Dave Engberg)

  1. 1. One Billion Notes as ‘Small Data’ Dave Engberg, CTO
  2. 2. Evernote OverviewLaunched in 2008“Freemium” subscription serviceMobile and desktop33.3 million registered accounts1.12 billion notes1.6 billion unique attachments
  3. 3. Evernote Product Family Food Hello Penultimate Peek Skitch Clearly
  4. 4. Worldwide Reach
  5. 5. Last 30 days Accounts created 1.9 million Distinct users 11 million Active clients 1337 New notes 77 million Note edits 152 million HTTP(S) requests 14 billion Image OCR 90 million
  6. 6. Architecture philosophy Only optimize for your application’s top 1-2 challenges
  7. 7. Evernote CPU
  8. 8. Evernote service requirements Component Size Avg Load Peak Load CPU Low Medium Bandwidth Latency File storage Metadata
  9. 9. Network load Smooth traffic (no bursts) Nearly symmetric Sync clients: not latency-sensitive 100 TB in, 180 TB out per month
  10. 10. Evernote service requirements Component Size Avg Load Peak Load CPU Low Medium Bandwidth Medium Medium Latency Low Low File storage Metadata
  11. 11. File Storage2.8 billion attachments1.6 billion unique files374 TB, de-duplicatedEffectively permanentHigh redundancy required (stored 3x)
  12. 12. Evernote service requirements Component Size Avg Load Peak Load CPU Low Medium Bandwidth Medium Medium Latency Low Low File storage High Low Medium Metadata
  13. 13. MetadataStrong ACID transactional DB < 10 TB Peak riops: 350 Peak wiops: 50Near-realtime search < 10 TB Peak riops: 800 Peak wiops: 500
  14. 14. Evernote service requirements Component Size Avg Load Peak Load CPU Low Medium Bandwidth Medium Medium Latency Low Low File storage High Low Medium Metadata Low Medium High
  15. 15. Sharded architecture Software: Software: Tomcat Apache Java 6 mod_dav MySQL 5.1 Debian stable DRBD Xen Debian stable Hardware: Hardware: SuperMicro 1U SuperMicro 4U 2x L5630 CPU 1x L5630 CPU 96 GB RAM 12 GB RAM 6x 300GB Intel SSD 24x 3TB (Seagate) LSI RAID 5 (+spare) LSI RAID 6 (x3) ~$8,000 ~$12,000
  16. 16. 265 shards Around 400 Linux servers overall
  17. 17. Tiers of a cloudCloud Provider Strengths Evernote’s ServiceApplications with bursts: Consistent network usage- bandwidth- storage Consistent compute usage- compute File storage growsCPU-bound applications indefinitely: users * timeApplications with low or Random iops boundfixed storage,low-medium iopsFewer operations staff
  18. 18. CPU + Metadata comparisonEvernote shard AWS EC2 + EBS440 GB usable / VM High-Memory 2XL:sysbench: 5000 rwiops $300/month$8000 Provisioned IOPS EBSOver 4 years: volume (max 1000 iops):$166/month $155/month200,000 users/shard: Total: $455/month$0.01/user/year
  19. 19. File storageEvernote WebDAV AWS S3WebDAV server, 54 TB $99/TB/month$12,100Over 4 years:$250/month$4.70/TB/monthWith triple redundancy:$14/TB/month
  20. 20. Networking comparison AWS networking (out)Evernote managed ~$0.08/GB/month$5/Mbps/month 180TB/month~800Mbps peak $13,700/month$4,000/month

×