Scaling the LAMP Stack Future of Web Apps October 5, 2007

Introductions

Specific Problems, Challenges and Issues

About this workshop This is a broad topic Theory and application Real-world focus Interactive (please!)

This is a broad topic

Theory and application

Real-world focus

Interactive (please!)

About web apps and scaling Some different ways of looking at the problem…

Some different ways of looking at the problem…

Things to think about Multi-server: locking and concurrency Running many: keep in mind what’s expensive, sloppy or risky Code quality The law of truly large numbers

Multi-server: locking and concurrency

Running many: keep in mind what’s expensive, sloppy or risky

Code quality

The law of truly large numbers

Elements of Scaling Split up different tasks Use more hardware (intelligently) Partition Replicate Cache Optimize (code and hardware) Identify and fix weaknesses Manage

Split up different tasks

Use more hardware (intelligently)

Partition

Replicate

Cache

Optimize (code and hardware)

Identify and fix weaknesses

Manage

Tools and Components Apache + PHP MySQL File System (local) Networked File System Load Balancers memcached

Apache + PHP

MySQL

File System (local)

Networked File System

Load Balancers

memcached

Contemplating Scaling Understand what your app does (and how much) Identify the bottlenecks Solve near-term problems Design well, but don’t over-design

Understand what your app does (and how much)

Identify the bottlenecks

Solve near-term problems

Design well, but don’t over-design

Web apps do lots of things Different operations have different scaling issues.

Different operations have different scaling issues.

What does your app do? List the high level elements of what your application does. Separate out different functions that will have different scaling issues.

List the high level elements of what your application does. Separate out different functions that will have different scaling issues.

Common things that web apps do Manage connections/protocols Deliver static content Manage sessions Manage user data Render dynamic pages Access external APIs Process media

Manage connections/protocols

Deliver static content

Manage sessions

Manage user data

Render dynamic pages

Access external APIs

Process media

Update the list of things your app does Add anything you missed Note which items you do in quantity

Add anything you missed

Note which items you do in quantity

Easy vs. Difficult Scaling What happens when you add hardware? Does it work? Does more hardware = more performance?

What happens when you add hardware?

Does it work?

Does more hardware = more performance?

Things that break when you scale State that isn’t properly shared (especially sessions) Updates/refreshes (caching and replication issues)

State that isn’t properly shared (especially sessions)

Updates/refreshes (caching and replication issues)

Things that don’t improve when you add more servers Unpartitioned databases Anything that locks/blocks Inefficient code, especially big queries

Unpartitioned databases

Anything that locks/blocks

Inefficient code, especially big queries

Scaling Each Element (do easy separations first)

(do easy separations first)

Managing Connections/Protocols No problem putting on multiple servers Apache is good Not too far away out of the box Moderately tunable Linux tuning TCP stack (tune to handle unusual networking needs)

No problem putting on multiple servers

Apache is good

Not too far away out of the box

Moderately tunable

Linux tuning

TCP stack (tune to handle unusual networking needs)

Key Apache Configuration Issues MaxClients (and ServerLimit, ThreadLimit and ThreadsPerChild) Avoid using PHP (or other) handler unnecessarily Use the worker MPM Maybe MaxRequestsPerChild

MaxClients (and ServerLimit, ThreadLimit and ThreadsPerChild)

Avoid using PHP (or other) handler unnecessarily

Use the worker MPM

Maybe MaxRequestsPerChild

Delivering Static Content Don’t process it unnecessarily Either cache or use no Apache handlers Caching can let you treat semi-static content as static Multiple servers complicates updates, but is otherwise easy

Don’t process it unnecessarily

Either cache or use no Apache handlers

Caching can let you treat semi-static content as static

Multiple servers complicates updates, but is otherwise easy

General Discussion: Multi-server, state and sessions Rethinking state for multi-server environments What is state? Short-term state (sessions) Long-term state (application data) Managing state is usually the hardest part of scaling

Rethinking state for multi-server environments

What is state?

Short-term state (sessions)

Long-term state (application data)

Managing state is usually the hardest part of scaling

What happens with state Written (created/destroyed/changed) Read Stored

Written (created/destroyed/changed)

Read

Stored

Requirements for managing state Depend on what it is and how it is used Perfect coherence Performance of different operations

Depend on what it is and how it is used

Perfect coherence

Performance of different operations

Ways of scaling state Replication: make more copies Partitioning: split up the work Caching Should make different choices for different state/data elements

Replication: make more copies

Partitioning: split up the work

Caching

Should make different choices for different state/data elements

About Load Balancers What load balancers do Spread load Detect server failures Stickiness/persistence Acceleration (especially SSL) Fancy features (including good stickiness) are expensive

What load balancers do

Spread load

Detect server failures

Stickiness/persistence

Acceleration (especially SSL)

Fancy features (including good stickiness) are expensive

Why sticky sessions are not usually good in practice Servers fail Corner cases exist

Servers fail

Corner cases exist

Managing Sessions

Where session data can be stored Browser cookies Web server temporary files (not scalable) App server state Database Cache

Browser cookies

Web server temporary files (not scalable)

App server state

Database

Cache

PHP session management Default (files) method is not multi-server friendly, and thus not scalable (unless sticky) Can implement a different back-end easily

Default (files) method is not multi-server friendly, and thus not scalable (unless sticky)

Can implement a different back-end easily

Designing a session back-end Requirements Data storage options Cookies only (re-auth, let the browser take care of the logout – but less secure) Full-featured involves a combination of cookies and database and cache (discussion of session details)

Requirements

Data storage options

Cookies only (re-auth, let the browser take care of the logout – but less secure)

Full-featured involves a combination of cookies and database and cache

(discussion of session details)

Managing small user data Databases are more efficient, flexible and sharable than small files Frequently-read data should be cached

Databases are more efficient, flexible and sharable than small files

Frequently-read data should be cached

Managing large user data NFS has flaws but is almost inevitable Locking is usually not important, but can be Performance degradation can be sudden

NFS has flaws but is almost inevitable

Locking is usually not important, but can be

Performance degradation can be sudden

About NFS NFS is usually transparent to your app NFS is easy to implement gives you multiple-write access NFS locking is not to be trusted The Linux NFS client is slow for writes and can do bad things under stress

NFS is usually transparent to your app

NFS is easy to implement gives you multiple-write access

NFS locking is not to be trusted

The Linux NFS client is slow for writes and can do bad things under stress

User data and locking Names based on hashes often mean no locking is needed Databases do locking better than file systems do Locking requires housekeeping

Names based on hashes often mean no locking is needed

Databases do locking better than file systems do

Locking requires housekeeping

Disk Storage Hardware Disk performance can degrade suddenly If the ratio of access to storage is low, then even slow disk is usually fine Think about seek times and spindles

Disk performance can degrade suddenly

If the ratio of access to storage is low, then even slow disk is usually fine

Think about seek times and spindles

Rendering dynamic pages Depends heavily on application specifics (query, search, process, etc.) Watch out for: Onerous queries (create and watch slow query log) Locking of resources and/or incoherence if state changes Heavy CPU and memory usage Cache both elements and complete pages

Depends heavily on application specifics (query, search, process, etc.)

Watch out for:

Onerous queries (create and watch slow query log)

Locking of resources and/or incoherence if state changes

Heavy CPU and memory usage

Cache both elements and complete pages

Processing media CPU intensive May be memory intensive Might be spiky Might need its own server pool

CPU intensive

May be memory intensive

Might be spiky

Might need its own server pool

Hardware Start simple Observe performance and respond accordingly Get lots of memory

Start simple

Observe performance and respond accordingly

Get lots of memory

Hardware-driven behaviors Sudden degradation because demand exceeds supply (usually relieved unhappily) Get behind due to a spike, and recover Not enough resources for normal optimization

Sudden degradation because demand exceeds supply (usually relieved unhappily)

Get behind due to a spike, and recover

Not enough resources for normal optimization

Specific hardware issues Not enough memory Severe: paging/swapping Mild: poor automatic caching; slowness due to fragmentation Disk seek (very common) CPU (but might really be memory) Disk throughput (rare for web apps)

Not enough memory

Severe: paging/swapping

Mild: poor automatic caching; slowness due to fragmentation

Disk seek (very common)

CPU (but might really be memory)

Disk throughput (rare for web apps)

Hardware decisions SCSI/SAS vs. SATA Resource ratios Combining vs. splitting functions Big vs. little boxes

SCSI/SAS vs. SATA

Resource ratios

Combining vs. splitting functions

Big vs. little boxes

Techniques Caching Partitioning Replication Data management middleware Queuing

Caching

Partitioning

Replication

Data management middleware

Queuing

Caching Turn expensive operations in to cheap ones Reduce: Database reads Object and page calculation/rendering operations Cache objects and subobjects Add memory

Turn expensive operations in to cheap ones

Reduce:

Database reads

Object and page calculation/rendering operations

Cache objects and subobjects

Add memory

Apache Caching Can be done with zero application modifications Complete pages/HTTP requests only Must use Apache 2.2 Cache is not shared between servers

Can be done with zero application modifications

Complete pages/HTTP requests only

Must use Apache 2.2

Cache is not shared between servers

memcached Extremely useful Distributed caching system Requires new thinking and new coding Straightforward API

Extremely useful

Distributed caching system

Requires new thinking and new coding

Straightforward API

memcached URLs Home: http://www.danga.com/memcached/ Intro: http://www.majordojo.com/2007/03/memcached-howto.php PHP documentation: http://www.php.net/memcache

Home: http://www.danga.com/memcached/

Intro: http://www.majordojo.com/2007/03/memcached-howto.php

PHP documentation: http://www.php.net/memcache

Partitioning Mostly for data management Split load on to separate servers/pools Partition algorithm/mechanism must be lightweight Partition algorithm must anticipate the future

Mostly for data management

Split load on to separate servers/pools

Partition algorithm/mechanism must be lightweight

Partition algorithm must anticipate the future

File Storage Partitioning Index/database gives the most flexibility Hash-based is simplest

Index/database gives the most flexibility

Hash-based is simplest

Database partitioning You will need to do this, but perhaps later than you think Index vs. hash-based

You will need to do this, but perhaps later than you think

Index vs. hash-based

Replication Used where data is read far more than written Consider caching first Also used for failure recovery

Used where data is read far more than written

Consider caching first

Also used for failure recovery

Types of Replication Replication: sync vs. async Synchronous is not usually scalable Asynchronous only works with certain kinds of data and use cases, because of coherence issues

Replication: sync vs. async

Synchronous is not usually scalable

Asynchronous only works with certain kinds of data and use cases, because of coherence issues

Database Replication Simple but finicky Asynchronous (but not by much) Allows big queries and backups to be moved to separate servers

Simple but finicky

Asynchronous (but not by much)

Allows big queries and backups to be moved to separate servers

File System Replication Slow and very asynchronous Mostly for disaster recovery

Slow and very asynchronous

Mostly for disaster recovery

Data Management Middleware Mostly for databases Can handle partitioning and replication, and do it well Big investment in coding to the API Sometimes easier to add functionality to app

Mostly for databases

Can handle partitioning and replication, and do it well

Big investment in coding to the API

Sometimes easier to add functionality to app

Queuing Save work for later Useful for less urgent operations, especially messaging Can be used to wait for a pause, or to separate hardware

Save work for later

Useful for less urgent operations, especially messaging

Can be used to wait for a pause, or to separate hardware

Dealing with lots of hardware (operations) Automation Process

Automation

Process

Imaging/Provisioning Be consistent Use your distro’s automation (Kickstart, AutoYaST, etc.) Use boring, meaningful hostnames Make re-imaging easy

Be consistent

Use your distro’s automation (Kickstart, AutoYaST, etc.)

Use boring, meaningful hostnames

Make re-imaging easy

Deployment Systems Content and code replication Coherence/atomic updates Managing pieces and processes Simple scripts are fine Create audit trail Include back-out Think 3AM Do it!

Content and code replication

Coherence/atomic updates

Managing pieces and processes

Simple scripts are fine

Create audit trail

Include back-out

Think 3AM

Do it!

Monitoring systems A pain, but a lifesaver Start with built-in basics Add custom checks, especially end-to-end and communication between pieces Eliminate false alarms (ongoing) Nagios, usually

A pain, but a lifesaver

Start with built-in basics

Add custom checks, especially end-to-end and communication between pieces

Eliminate false alarms (ongoing)

Nagios, usually

Coping with hardware failure Have extra servers/capacity Load balancers handle stateless layers Replication prepares you to handle data layers manually Use middleware or app-level multiple writes to get true data layer redundancy

Have extra servers/capacity

Load balancers handle stateless layers

Replication prepares you to handle data layers manually

Use middleware or app-level multiple writes to get true data layer redundancy

Change management Part automation, part process Use version control on everything Stage changes with realistic data Know how to back out Consult the right people (internal and/or external)

Part automation, part process

Use version control on everything

Stage changes with realistic data

Know how to back out

Consult the right people (internal and/or external)

Efficiency Access the smallest amount (DB, FS, etc.) Don’t do complex stuff when simple will suffice

Access the smallest amount (DB, FS, etc.)

Don’t do complex stuff when simple will suffice

Using the database efficiently Keep it simple Know what queries you do Index every query key Cache to reduce demand Check slow query log Replicate if you need big queries

Keep it simple

Know what queries you do

Index every query key

Cache to reduce demand

Check slow query log

Replicate if you need big queries

The messy real world

Security and abuse Mostly same issues, just magnified You will be a target Spam (coming and going) Abuse of file storage

Mostly same issues, just magnified

You will be a target

Spam (coming and going)

Abuse of file storage

Corner Cases Murphy’s law enforcement Watch out for how different user activities relate Lock data, not functions Housekeeping

Murphy’s law enforcement

Watch out for how different user activities relate

Lock data, not functions

Housekeeping

Performance and tuning Observation and responsiveness is more important than pre-optimizing Redesign as needed Collect the data to be able to analyze (both resource utilization and end-user performance)

Observation and responsiveness is more important than pre-optimizing

Redesign as needed

Collect the data to be able to analyze (both resource utilization and end-user performance)

Miscellaneous Warnings

Files and directories Most default file system configurations get really slow with lots of files in one directory Numerical limits on files and subdirectories Some programs don’t like files over 2GB

Most default file system configurations get really slow with lots of files in one directory

Numerical limits on files and subdirectories

Some programs don’t like files over 2GB

AJAX Sequential round trips Make preloading invisible UI that waits for too many things

Sequential round trips

Make preloading invisible

UI that waits for too many things

Other topics Multiple sites CDNs

Multiple sites

CDNs

Scaling the LAMP Stack Future of Web Apps October, 2007 Daniel Lieberman [email_address]