1.
Real Life Scaling:
A Tale of Two Websites
By Justin Carmony - Utah Open Source Conference 2009

2.
“It was the best of times, it was
the worst of times...”
- A Tale of Two Cities, Charles Dickens

3.
About Me
• Website Hobbyist since 1997
• Professional Since 2005
• Worked in Large Open Source & Proprietary Solutions
• Full Time Contractor & Consultant
• Sponsorship Manager for UTOSC
• Blog: http://www.justincarmony.com/blog/

4.
This Presentation
• Point You In The Right Direction
‣ Not a Substitute for Research & Homework
• Developers Perspective, Not So Much Sys Admin
• Q & A Session Afterwards
• Available for Questions Remainder of Conference
• Ask via Email: justin@justincarmony.com

5.
The Scaling Conundrum
Because No One Likes Premature Escalation

6.
The Scale of Scaling
Traffic

7.
The Scale of Scaling
# of Websites Traffic

8.
The Scale of Scaling
# of Websites Traffic

9.
The Problem:
Many Developers Skip “Medium”
and go Straight to “XXL”

10.
The Solution:
Implement Reasonable Solutions
For Your Website’s Size

11.
Scalability vs Performance
The Difference Is Important... Yet Useless

12.
Performance Before Scaling
Except With Limitations

13.
“Zero Theory”

14.
“Zero Theory”
200 Current Visitors
500 New Sign-Ups
300 Messages Sent
1,000 Users
10,000 Users
100,000 Users

15.
“Zero Theory”
200 Current Visitors 2,000 Current Visitors
500 New Sign-Ups 5,000 New Sign-Ups
300 Messages Sent 3,000 Messages Sent
1,000 Users 10,000 Users
10,000 Users 100,000 Users
100,000 Users 1,000,000 Users

16.
Tale #1: Cyber Evolution
Website: www.cevo.com
Online Gaming League
Co-Developed w/ Eric Ping
Communication Between Many
“Clients” & “Servers”
Great Growth Over 4 Years
Learning by Fire for Myself

17.
Tale #2: Dating DNA
Website: http://www.datingdna.com
Online Free Dating Service
#1 iPhone Dating App
Overnight 1,000% Growth
Continuing to Grow Rapidly
Unique Scaling Challenges

18.
#1 Challenge: The Database

19.
#1 Challenge: The Database
• 80% of Performance Issues were
Database Related

20.
#1 Challenge: The Database
• 80% of Performance Issues were
Database Related
• Issues Aren’t Noticeable Until
After Growth & High Load

21.
#1 Challenge: The Database
• 80% of Performance Issues were
Database Related
• Issues Aren’t Noticeable Until
After Growth & High Load
• Most Issues Were Due to Poor
Queries and/or Poor Indexes

22.
Database Abuse!
Just Because You Can, Doesn’t Mean You Should

23.
Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs

24.
Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs
Prevention: Learn About Database Design

25.
Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs
Prevention: Learn About Database Design
This is NOT for DBAs Only

26.
Example: Custom Forums
• In Our Defense, We Made This In
About 24 Hours
• Nested Queries = 1000+ Queries
Per Page - Not Joking
• 10 second Load Times
• End Result: Users Hated It

27.
Example: Standings Page
• Before:
‣ Nested Queries To Determine
Stats Each Row
‣ Regenerated Every Time Match
Updated
• After:
‣ Generate Every 30 Minutes
‣ Proper JOINs, Only One Query

28.
MySQL Locks
• Problem: Popular Tables Using MyISAM
• Solutions:
‣ Switched to InnoDB
‣ Optimized Logic to Reduce Writes
‣ Reduced JOINs in Queries

29.
Database Replication
Complicated, Major Trade-Offs, Requires Preparation

30.
Open APIs
• Can Get Heavily Abused
• Can Grow Unexpectedly
• Must Be Extremely Efficient
• CEVO’s APIs
‣ 100,000s of Player Lookups
‣ Thousands of CMN Players
‣ Match History Calls

31.
Identifying What’s Unique

32.
Identifying What’s Unique
• “What Unique Part of Your Site
Will be Complicated to Scale?”

33.
Identifying What’s Unique
• “What Unique Part of Your Site
Will be Complicated to Scale?”

34.
Identifying What’s Unique
• “What Unique Part of Your Site
Will be Complicated to Scale?”
• Important to Identify Early
• Allows you to Plan for the Future
• Many Solutions for the Common
Stuff to Scale

35.
X 2 -X
Dating DNA’s Compatibility Score Growth
X = # of Users

36.
# of Score Records

37.
# of Score Records
10 Users
200 Users
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users

38.
# of Score Records
10 Users 90 Records
200 Users
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users

39.
# of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users

40.
# of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users
300,000 Users
1,000,000 Users

41.
# of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users
1,000,000 Users

42.
# of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users 89,999,700,000 Records
1,000,000 Users

43.
# of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users 89,999,700,000 Records
1,000,000 Users One Trillion Records!

44.
How We Solved It
• Introduce Limits on Records per User
• Only Save Decent Scores
• Smart Logic to Predetermine Good Matches
• Shard Table Into Multiple Tables
• We’re Still Managing & Finding More Optimizations

45.
User Uploaded Content
• Multiple Aspects to Consider
‣ Storage (File Size)
‣ Serving (File Sizes, Bandwidth, Redundancy)
‣ Backups (Time, Speed)
• Challenges
‣ Content Outgrowing Server
‣ Serving Multiple Versions

46.
Memcached
• Scales Much Easier than Traditional Databases
• Reduced Load Off Databases by 70%
• Implement Quickly -- Its Awesome!
• Gave Presentation On @ UPHPU
‣ Check My Blog for Recording

47.
Hardware
• Be careful of “Throwing Hardware” at Problems
• Run on Realistic Hardware
• Developers, You Need To Be Cost Conscious
• Can You Afford to Scale Up? How About Scale Down?

48.
Other Challenges
• Apache Configurations
‣ MaxClient Limits Reached
• MySQL Configurations
‣ Max Connections Reached
• Network Communication
‣ Saturated Bandwidth Between
Servers

49.
Tools I’ve Used
• MySQL - JetProfiler (No FOSS, Hopefully One Day)
• Server Performance - top, htop, atop, iotop
• PHP - Zend Debugger & Profiler, xdebug
• FirePHP & FireBug

50.
So... Where’s The Scaling?

51.
Getting Ready To Scale

52.
Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”

53.
Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”

54.
Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications

55.
Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications
• Create Metrics to Monitor the Health of the
Components

56.
Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications
• Create Metrics to Monitor the Health of the
Components
• “Zero Theory” For Each Component

57.
Makes Components of
Dating DNA

58.
Makes Components of
Dating DNA
Dating DNA
Application

59.
Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Dating DNA Database
Web Servers
Application Cluster
Score Generation Memcache
System Cluster

60.
Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Dating DNA Database
Web Servers
Application Cluster
Score Generation Memcache
System Cluster

61.
Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Database
Web Servers Communication Cluster
Score Generation Memcache
System Cluster

62.
Scaling Components

63.
Scaling Components
Web Application
Server #1

64.
Scaling Components
Comp E1
Comp D1
Comp C1
Comp B1
Comp A1
Server #1

65.
Scaling Components
Comp E1
Comp D1
Comp C1
Comp B1
Comp A1
Server #1 Server #2

66.
Scaling Components
Comp C1
Comp B1 Comp E1
Comp A1 Comp D1
Server #1 Server #2

67.
Scaling Components
Comp C1 Comp A2
Comp B1 Comp E1
Comp A1 Comp D1
Server #1 Server #2

68.
Scaling Components
Comp C1 Comp A2
Comp B1 Comp E1
Comp A1 Comp D1
Server #1 Server #2 Server #3 Server #4

69.
Scaling Components
Comp C1 Comp A2 Comp C2
Comp B1 Comp E1 Comp B2
Comp A1 Comp D1 Comp A3
Server #1 Server #2 Server #3 Server #4

70.
Scaling Components
Comp C1 Comp A2 Comp C2
Comp B1 Comp E1 Comp B2 Comp B3
Comp A1 Comp D1 Comp A3 Comp A4
Server #1 Server #2 Server #3 Server #4

71.
Scaling Components
• Should Be Able to “Live” With Each Other
• Deployment & Management all Automated
‣ Not Necessarily “Automatic”
• Fault Tolerance
• Testing & Staging Environments

72.
Scaling Web Servers
• Challenges Load Balancer
‣ Routing
‣ Sessions
• Ideas Web Server Web Server
‣ Separate Dynamic &
Static
Cache DB
‣ Use Sub Domains

73.
Asynchronous Score
Generation
• Pass Message To “Score Server”
‣ Pass User ID Only
• Quickly Generate a Small Set of Scores
• Queue User for Full Process
‣ Spawn Small Generation
• Update “MEMORY” MySQL Table on Status

74.
The Open Source Advantage
• Choice & Options w/ Cost
• Solid Applications, Tested & Proven
• Great Communities
• Give Back, Karma++

75.
Last Thoughts
• Once again, K.I.S.S
• Proactive > Reactive
• Monitoring, Monitoring, Monitoring!
• The “Cloud”
• Get Advice from the Community

76.
Questions?

77.
Thank You

78.
Thank You
Website www.justincarmony.com
Email justin@justincarmony.com
Twitter JustinCarmony
Skype JustinCarmony
IRC irc.freenode.net #uphpu