Advanced Deployment
Scotland on Rails 2009
Jonathan Weiss, 28 March 2009
Peritor GmbH

Who am I?
Jonathan Weiss
• Consultant for Peritor GmbH in Berlin
• Specialized in Rails, Scaling, Deployment, and Code Review
• Webistrano - Rails deployment tool
• FreeBSD Rubygems and Ruby on Rails maintainer
http://www.peritor.com
http://blog.innerewut.de
2

Deployment
Deployment
Process Architecture
3

Deployment Process Requirements
Reproducible Accountable Notifications
Automatic
4

Deployment Tools
Several tools available
• Capistrano
• Webistrano
• Vlad
• Puppet
• Chef
The deployment process is usually not that complicated
5

Architecture
6

How deployment starts out …
7

… and how it ends
8

Agenda
Search
Background Processing
Scaling the database
Multiple Client Installations
Cloud Infrastructure
9

General Advice
-
Simple is better than complex
10

Search
11

Search
Full text search
Can become very slow on big data sets
12

Full Text Search Engine
Separate Service
• Creates full text index
• Application queries search daemon
• Index update through application or
database
Possible Engines
• Ferret
• Sphinx
• Solr
• Lucene
• …
13

Search Slave
Database replication slave
• Has complete dataset
• Migrates slow search queries from master
• Can use different database table engine
14

Database Index
PostgreSQL Tsearch2
• Core since 8.3
• Allows to create full text index on multiple columns
or arbitrary SQL expressions
MySQL MyISAM FULLTEXT index
• Only works with MySQL <= 5.0 and MyISAM tables
• Full text index on multiple columns
15

What to use?
Different characteristics
• Real-time updates and stale data
• Lost updates
• Performance
• Document content and format
• Complexity
16

Background Processing
17

Problem
Long running tasks
• Resizing uploaded images
• Mailing
• Computing an expensive operation
• Accessing slow back-ends
When running inside request-response-cycle
• Blocks user
• Blocks Rails instance
• Hard to monitor and debug
18

Solution
Asynchronous processing in the background
Message/Queue Scheduler
19

Background Processing
20

Options
Options for message bus: Options for background process:
• Database • (Ruby) Daemon
• Amazon SQS • Cron job with script/runner
• Drb • Forked process
• Memcache • Delayed Job / BJ / (Backgroundrb)
• ActiveMQ • run_later
• … • ….
21

Database/Ruby daemon example
22

Scaling the database
23

Scaling the database
One database for everything
• All domain data in one place
• The simplest solution
Problems at some point
• Number of read and write requests
• Data size
24

Scaling the database
Read Slave
• Slave replicates each SQL-statement
on the master
• Increase read performance by reading
from replicating slave
• Stale read problem
• Better used explicitly,
but then makes you think
Better use
memcached
25

Scaling the database
Master-Master
• Increase write and read performance
• Each server is a slave of the other
• Synchronization can be tricky
• Limited by database size
Better for HA than for
write performance
26

Data Partitioning
Partition on domain models
• Separate users and products
• Makes sense if JOINs are rare
• Scales reads/writes
• Reduces data size per database
• Depends on separate domains
Simple and
effective
27

Data Partitioning
Sharding
• Split data into shards
• All tables
• Only big ones like users
• Partition by id, hash function or lookup
• Complex and makes JOINs complicated
• Scales reads/writes
• Reduces data size per database
28

Data Partitioning
Sharding
• Split data into shards
• All tables
• Only big ones like users
• Partition by id, hash function or lookup
• Complex and makes JOINs complicated
• Scales reads/writes
• Reduces data size per database
Last resort
29

Alternatives
Data size is often the bigger problem
Archiving Reduce data size
30

Archiving
Get rid of (historical) data
• Delete old data
• Aggregate old data
• Partition old data
Have an archiving policy from the start
31

Reduce data size
Avoid exponential data growth
• Do not store data in database, move to
• File system
• S3
• SimpleDB
• Do not normalize data
• Duplicate data in order to remove JOINs (and JOIN tables)
• Combine indices
32

Multiple clients
33

Multiple Clients
NOT the same as multiple users
Client is more like a separate domain – i.e. expansion to another country
• Different settings
• Different themes
• Different features enabled
• Different language
• Different audience
How to combine in one app?
34

Multiple Clients
Questions to ask
• How many different clients?
• Is there shared state (users, settings, posts, …)?
• What is the expected data size and growth of each client?
35

Multiple Clients
The easy way to maintenance hell
• Fork the code
• One branch per client
• One install per client
36

Multiple Clients
Same code – same database
• Move different behavior into configuration
• Move configuration into database
• Scope data by DB-column
• Scope all data request in the code
37

Multiple Clients
Same code – partition the data
• Move different behavior into configuration
• Partition data by database
Hardcode database while booting
38

Multiple Clients
Same code – partition the data
• Move different behavior into configuration
• Partition data by database
Choose database dynamically
39

Multiple Clients
Generate local databases
• Import global content into master DB
• Push shared content in the correct
format to app DBs
• Build reverse channel if needed
40

Cloud Infrastructure
41

Cloud Infrastructure
Servers come and go
• You do not know your servers before deploying
• Restarting is the same as introducing a new machine
You can’t hardcode IPs
database.yml
42

Solution #1
Query and manually adjust
• Servers do not change that often
• New nodes probably need manual intervention
• Use AWS ElasticIPs to ease the pain
Set servers dynamically AWS Elastic IP
43

Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
44

Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
45

Central Directory
Different Implementations
• File on S3
• SimpleDB
• A complete service,
capable of monitoring and controlling your instances
46

Summary
Simple is better than complex
Carefully evaluate the different solutions
Only introduce a new component if you really need to
Everything has strings attached
Solving the data size problem often solves others too
47

Questions?
48

Peritor GmbH
Teutonenstraße 16
14129 Berlin
Telefon: +49 (0)30 69 20 09 84 0
Telefax: +49 (0)30 69 20 09 84 9
Internet: www.peritor.com
E-Mail: kontakt@peritor.com
49
Peritor GmbH - Alle Rechte vorbehalten 49