1. One Billion Notes as ‘Small Data’
Dave Engberg, CTO

2. Evernote Overview
Launched in 2008
“Freemium” subscription service
Mobile and desktop
33.3 million registered accounts
1.12 billion notes
1.6 billion unique attachments

3. Evernote Product Family
Food Hello
Penultimate Peek
Skitch Clearly

4. Worldwide Reach

5. Last 30 days
Accounts created 1.9 million
Distinct users 11 million
Active clients 1337
New notes 77 million
Note edits 152 million
HTTP(S) requests 14 billion
Image OCR 90 million

6. Architecture philosophy
Only optimize for your application’s
top 1-2 challenges

7. Evernote CPU

8. Evernote service requirements
Component Size Avg Load Peak Load
CPU Low Medium
Bandwidth
Latency
File storage
Metadata

9. Network load
Smooth traffic (no bursts)
Nearly symmetric
Sync clients: not latency-sensitive
100 TB in, 180 TB out per month

10. Evernote service requirements
Component Size Avg Load Peak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage
Metadata

11. File Storage
2.8 billion attachments
1.6 billion unique files
374 TB, de-duplicated
Effectively permanent
High redundancy required (stored 3x)

12. Evernote service requirements
Component Size Avg Load Peak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage High Low Medium
Metadata

13. Metadata
Strong ACID transactional DB
< 10 TB
Peak riops: 350
Peak wiops: 50
Near-realtime search
< 10 TB
Peak riops: 800
Peak wiops: 500

14. Evernote service requirements
Component Size Avg Load Peak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage High Low Medium
Metadata Low Medium High

15. Sharded architecture
Software: Software:
Tomcat Apache
Java 6 mod_dav
MySQL 5.1 Debian stable
DRBD
Xen
Debian stable
Hardware: Hardware:
SuperMicro 1U SuperMicro 4U
2x L5630 CPU 1x L5630 CPU
96 GB RAM 12 GB RAM
6x 300GB Intel SSD 24x 3TB (Seagate)
LSI RAID 5 (+spare) LSI RAID 6 (x3)
~$8,000 ~$12,000

17. 265 shards
Around 400 Linux servers overall

18. Tiers of a cloud
Cloud Provider Strengths Evernote’s Service
Applications with bursts: Consistent network usage
- bandwidth
- storage Consistent compute usage
- compute File storage grows
CPU-bound applications indefinitely: users * time
Applications with low or Random iops bound
fixed storage,
low-medium iops
Fewer operations staff

19. CPU + Metadata comparison
Evernote shard AWS EC2 + EBS
440 GB usable / VM High-Memory 2XL:
sysbench: 5000 rwiops $300/month
$8000
Provisioned IOPS EBS
Over 4 years: volume (max 1000 iops):
$166/month $155/month
200,000 users/shard: Total: $455/month
$0.01/user/year

20. File storage
Evernote WebDAV AWS S3
WebDAV server, 54 TB $99/TB/month
$12,100
Over 4 years:
$250/month
$4.70/TB/month
With triple redundancy:
$14/TB/month

21. Networking comparison
AWS networking (out)
Evernote managed
~$0.08/GB/month
$5/Mbps/month
180TB/month
~800Mbps peak
$13,700/month
$4,000/month