Real Life Scaling: A Tale of Two Websites

Real Life Scaling: A Tale of Two Websites By Justin Carmony - Utah Open Source Conference 2009

“It was the best of times, it was the worst of times...” - A Tale of Two Cities, Charles Dickens

About Me • Website Hobbyist since 1997 • Professional Since 2005 • Worked in Large Open Source & Proprietary Solutions • Full Time Contractor & Consultant • Sponsorship Manager for UTOSC • Blog: http://www.justincarmony.com/blog/

This Presentation • Point You In The Right Direction ‣ Not a Substitute for Research & Homework • Developers Perspective, Not So Much Sys Admin • Q & A Session Afterwards • Available for Questions Remainder of Conference • Ask via Email: justin@justincarmony.com

The Scaling Conundrum Because No One Likes Premature Escalation

The Scale of Scaling Trafﬁc

The Scale of Scaling # of Websites Trafﬁc

The Problem: Many Developers Skip “Medium” and go Straight to “XXL”

The Solution: Implement Reasonable Solutions For Your Website’s Size

Scalability vs Performance The Difference Is Important... Yet Useless

Performance Before Scaling Except With Limitations

“Zero Theory”

“Zero Theory” 200 Current Visitors 500 New Sign-Ups 300 Messages Sent 1,000 Users 10,000 Users 100,000 Users

“Zero Theory” 200 Current Visitors 2,000 Current Visitors 500 New Sign-Ups 5,000 New Sign-Ups 300 Messages Sent 3,000 Messages Sent 1,000 Users 10,000 Users 10,000 Users 100,000 Users 100,000 Users 1,000,000 Users

Tale #1: Cyber Evolution Website: www.cevo.com Online Gaming League Co-Developed w/ Eric Ping Communication Between Many “Clients” & “Servers” Great Growth Over 4 Years Learning by Fire for Myself

Tale #2: Dating DNA Website: http://www.datingdna.com Online Free Dating Service #1 iPhone Dating App Overnight 1,000% Growth Continuing to Grow Rapidly Unique Scaling Challenges

#1 Challenge: The Database

#1 Challenge: The Database • 80% of Performance Issues were Database Related

#1 Challenge: The Database • 80% of Performance Issues were Database Related • Issues Aren’t Noticeable Until After Growth & High Load

#1 Challenge: The Database • 80% of Performance Issues were Database Related • Issues Aren’t Noticeable Until After Growth & High Load • Most Issues Were Due to Poor Queries and/or Poor Indexes

Database Abuse! Just Because You Can, Doesn’t Mean You Should

Database Abuse • Excessive Logging (instead of using rotated ﬁles, etc) • Excessive Writes (INSERT, UPDATE, REPLACE, DELETE) • Shear Volume of Repetitive Queries • Non-Indexed Searches • Sub-Queries Instead of JOINs

Database Abuse • Excessive Logging (instead of using rotated ﬁles, etc) • Excessive Writes (INSERT, UPDATE, REPLACE, DELETE) • Shear Volume of Repetitive Queries • Non-Indexed Searches • Sub-Queries Instead of JOINs Prevention: Learn About Database Design

Database Abuse • Excessive Logging (instead of using rotated ﬁles, etc) • Excessive Writes (INSERT, UPDATE, REPLACE, DELETE) • Shear Volume of Repetitive Queries • Non-Indexed Searches • Sub-Queries Instead of JOINs Prevention: Learn About Database Design This is NOT for DBAs Only

Example: Custom Forums • In Our Defense, We Made This In About 24 Hours • Nested Queries = 1000+ Queries Per Page - Not Joking • 10 second Load Times • End Result: Users Hated It

Example: Standings Page • Before: ‣ Nested Queries To Determine Stats Each Row ‣ Regenerated Every Time Match Updated • After: ‣ Generate Every 30 Minutes ‣ Proper JOINs, Only One Query

MySQL Locks • Problem: Popular Tables Using MyISAM • Solutions: ‣ Switched to InnoDB ‣ Optimized Logic to Reduce Writes ‣ Reduced JOINs in Queries

Database Replication Complicated, Major Trade-Offs, Requires Preparation

Open APIs • Can Get Heavily Abused • Can Grow Unexpectedly • Must Be Extremely Efﬁcient • CEVO’s APIs ‣ 100,000s of Player Lookups ‣ Thousands of CMN Players ‣ Match History Calls

Identifying What’s Unique

Identifying What’s Unique • “What Unique Part of Your Site Will be Complicated to Scale?”

Identifying What’s Unique • “What Unique Part of Your Site Will be Complicated to Scale?” • Important to Identify Early • Allows you to Plan for the Future • Many Solutions for the Common Stuff to Scale

X 2 -X Dating DNA’s Compatibility Score Growth X = # of Users

# of Score Records

# of Score Records 10 Users 200 Users 5,000 Users 25,000 Users 300,000 Users 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 5,000 Users 25,000 Users 300,000 Users 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 39,800 Records 5,000 Users 25,000 Users 300,000 Users 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 39,800 Records 5,000 Users 24,995,000 Records 25,000 Users 300,000 Users 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 39,800 Records 5,000 Users 24,995,000 Records 25,000 Users 624,975,000 Records 300,000 Users 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 39,800 Records 5,000 Users 24,995,000 Records 25,000 Users 624,975,000 Records 300,000 Users 89,999,700,000 Records 1,000,000 Users

# of Score Records 10 Users 90 Records 200 Users 39,800 Records 5,000 Users 24,995,000 Records 25,000 Users 624,975,000 Records 300,000 Users 89,999,700,000 Records 1,000,000 Users One Trillion Records!

How We Solved It • Introduce Limits on Records per User • Only Save Decent Scores • Smart Logic to Predetermine Good Matches • Shard Table Into Multiple Tables • We’re Still Managing & Finding More Optimizations

User Uploaded Content • Multiple Aspects to Consider ‣ Storage (File Size) ‣ Serving (File Sizes, Bandwidth, Redundancy) ‣ Backups (Time, Speed) • Challenges ‣ Content Outgrowing Server ‣ Serving Multiple Versions

Memcached • Scales Much Easier than Traditional Databases • Reduced Load Off Databases by 70% • Implement Quickly -- Its Awesome! • Gave Presentation On @ UPHPU ‣ Check My Blog for Recording

Hardware • Be careful of “Throwing Hardware” at Problems • Run on Realistic Hardware • Developers, You Need To Be Cost Conscious • Can You Afford to Scale Up? How About Scale Down?

Other Challenges • Apache Conﬁgurations ‣ MaxClient Limits Reached • MySQL Conﬁgurations ‣ Max Connections Reached • Network Communication ‣ Saturated Bandwidth Between Servers

Tools I’ve Used • MySQL - JetProﬁler (No FOSS, Hopefully One Day) • Server Performance - top, htop, atop, iotop • PHP - Zend Debugger & Proﬁler, xdebug • FirePHP & FireBug

So... Where’s The Scaling?

Getting Ready To Scale

Getting Ready To Scale • Compartmentalize “Parts” Into “Components”

Getting Ready To Scale • Compartmentalize “Parts” Into “Components” • Create “Scaling Strategy” For Each “Component”

Getting Ready To Scale • Compartmentalize “Parts” Into “Components” • Create “Scaling Strategy” For Each “Component” • Keep It Simple, Complexity == Complications

Getting Ready To Scale • Compartmentalize “Parts” Into “Components” • Create “Scaling Strategy” For Each “Component” • Keep It Simple, Complexity == Complications • Create Metrics to Monitor the Health of the Components

Getting Ready To Scale • Compartmentalize “Parts” Into “Components” • Create “Scaling Strategy” For Each “Component” • Keep It Simple, Complexity == Complications • Create Metrics to Monitor the Health of the Components • “Zero Theory” For Each Component

Makes Components of Dating DNA

Makes Components of Dating DNA Dating DNA Application

Makes Components of Dating DNA Jobs / Cron User Uploaded System Content Dating DNA Database Web Servers Application Cluster Score Generation Memcache System Cluster

Makes Components of Dating DNA Jobs / Cron User Uploaded System Content Database Web Servers Communication Cluster Score Generation Memcache System Cluster

Scaling Components

Scaling Components Web Application Server #1

Scaling Components Comp E1 Comp D1 Comp C1 Comp B1 Comp A1 Server #1

Scaling Components Comp E1 Comp D1 Comp C1 Comp B1 Comp A1 Server #1 Server #2

Scaling Components Comp C1 Comp B1 Comp E1 Comp A1 Comp D1 Server #1 Server #2

Scaling Components Comp C1 Comp A2 Comp B1 Comp E1 Comp A1 Comp D1 Server #1 Server #2

Scaling Components Comp C1 Comp A2 Comp B1 Comp E1 Comp A1 Comp D1 Server #1 Server #2 Server #3 Server #4

Scaling Components Comp C1 Comp A2 Comp C2 Comp B1 Comp E1 Comp B2 Comp A1 Comp D1 Comp A3 Server #1 Server #2 Server #3 Server #4

Scaling Components Comp C1 Comp A2 Comp C2 Comp B1 Comp E1 Comp B2 Comp B3 Comp A1 Comp D1 Comp A3 Comp A4 Server #1 Server #2 Server #3 Server #4

Scaling Components • Should Be Able to “Live” With Each Other • Deployment & Management all Automated ‣ Not Necessarily “Automatic” • Fault Tolerance • Testing & Staging Environments

Scaling Web Servers • Challenges Load Balancer ‣ Routing ‣ Sessions • Ideas Web Server Web Server ‣ Separate Dynamic & Static Cache DB ‣ Use Sub Domains

Asynchronous Score Generation • Pass Message To “Score Server” ‣ Pass User ID Only • Quickly Generate a Small Set of Scores • Queue User for Full Process ‣ Spawn Small Generation • Update “MEMORY” MySQL Table on Status

The Open Source Advantage • Choice & Options w/ Cost • Solid Applications, Tested & Proven • Great Communities • Give Back, Karma++

Last Thoughts • Once again, K.I.S.S • Proactive > Reactive • Monitoring, Monitoring, Monitoring! • The “Cloud” • Get Advice from the Community

Questions?

Thank You

Thank You Website www.justincarmony.com Email justin@justincarmony.com Twitter JustinCarmony Skype JustinCarmony IRC irc.freenode.net #uphpu

Real Life Scaling: A Tale of Two Websites

0 comments

Notes on slide 1

Favorites, Groups & Events