While it’s easy to get started developing applications using Ruby on Rails, the hard part is (as with most technologies) how to put it into production? There are a lot of options like Mongrel, nginx, Apache and JRuby, all of them difficult to evaulate without deeper knowledge of Rails and your application’s scaling needs. We will look at the available options to get a clearer picture how each setup is in different situations, and how you can utilize Capistrano for an easy deployment process in those situations. Simple as well as more complex setups will be discussed. We’ll try to include your problems as well as your experiences to discuss specific issues with deployment and scalability.

1. Deploying and Monitoring
Ruby on Rails
A practical guide
Mathias Meyer and Jonathan Weiss, 25.05.2009
Peritor GmbH

2. Who are we?
Jonathan Weiss
• Consultant for Peritor GmbH in Berlin
• Specialized in Rails, Scaling, Deployment, and Code Review
• Webistrano - Rails deployment tool
• FreeBSD Rubygems and Ruby on Rails maintainer
http://www.peritor.com
http://blog.innerewut.de
2

3. Who are we?
Mathias Meyer
• Consultant for Peritor GmbH in Berlin
• Specialized in Rails, Performance/Database Tuning,
Deployment, and Refactoring
• Maintainer of Macistrano –
Webistrano-Client for Mac OS X (and soon the iPhone),
run_later and acts_as_solr
http://www.paperplanes.de
3

4. Agenda
Infrastructure Deployment Practical Session Monitoring Q&A
4

5. Infrastructure
Infrastructure
5

6. Simple Rails Setup
One Rails instance handles all requests
Rails is single-threaded: There is only one concurrent request
6

7. Rails Setup
7

8. Rails Setup
8

9. Typical Rails Setup
• A load-balancer distributes the incoming requests
• Some load-balancers will deliver static requests themselves
• Several Rails instances handle all requests
• Number of concurrent requests equals number of Rails instances
9

10. Rails Setup Options
10

11. Deployment Questions
Apache? Pound?
mod_proxy_balancer?
FastCGI?
Mongrel?
Proxy? Load-balancer?
Nginx?
mod_rails?
Ebb?
HA-Proxy? Reverse Proxy?
Swiftiply?
Phusion Passenger?
Thin?
Pen?
Rails Application
Server? Lighttpd?
11

12. What we are going to cover today
Rails Application Server Proxy/Web Server
• FastCGI • Apache2
• Mongrel • Nginx
• mod_rails / Phussion Passenger • Lighttpd
• JRuby + Glassfish & Co. • HA-Proxy
12

13. FastCGI
13

14. FastCGI
• Protocol to communicate with long-running CGI applications
• Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd
• Proxy local and remote FastCGI instances
• Oldest way of deploying Rails
• Deprecated and unstable
• Hard to debug (FastCGI protocol)
14

15. FastCGI
15

16. FastCGI
16

17. 17

18. Mongrel
18

19. Mongrel
• Developed by Zed Shaw as an alternative to FastCGI
• Complete HTTP-Server that can load arbitrary Ruby-servlets
• Built-in Rails support
19

20. Mongrel
20

21. Mongrel
Apache 2.2 – mod_proxy_balancer
Lighttpd
Nginx
HA-Proxy
Pound
21

22. Mongrel Cluster
Utility to manage several Mongrel instances
22

23. Mongrel Cluster
Control Mongrels
23

24. Mongrel and Apache 2.2
Define Mongrel Cluster
in Apache
24

25. Simple Mongrel and Apache 2.2
Redirect all traffic to the Mongel cluster
25

26. A more complex example
• Redirect dynamic requests
• Serve static content
• Support cached pages
• Support maintenance page
• Enable client-side caching of
images, stylesheets, and JavaScript
• Compress output if supported
26

27. Mongrel
• Very robust
• Strict HTTP parser
• Easy to debug (HTTP!)
• Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer
• Can be a bit difficult to setup (mongrel_cluster, ports, Apache)
• Not so easy on mass/virtual hosting
27

28. mod_rails
28

29. mod_rails a.k.a Phusion Passenger
• Module for Apache 2.2 (and Nginx)
• Allows Apache to control Rails instances
• Apache starts and stops application instances depending on the application load
• Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.)
• Only manages Rails on one host - no remote instances
• Combine with HTTP-Proxy / balancing solution
29

30. Install Phusion Passenger
Install Apache module
Load and activate in Apache
30

31. Install Phusion Passenger
Install Passenger with nginx (installs custom nginx)
Configure nginx
31

32. Customized Phusion Passenger
Control Rails instance number
32

33. Control Phusion Passenger
Restart after deployment:
33

34. Phusion Passenger
One machine
34

35. Phusion Passenger
One machine Multiple machines
35

36. Phusion Passenger
• Ready for production
• Makes setup easier – on the single machine level
• Multiple servers still require load balancer
• Suitable for mass-hosting
• upcoming standard way of deploying Rails
36

37. JRuby
37

38. • Ruby Runtime on the Java Virtual Machine
• Implemented in Java and Ruby
• Compiles Ruby into Java-bytecode
• Integrates with Java code and libraries
• Java’s promises of native threads and JIT
• Allows for Ruby/Rails applications to be packaged as WAR files
• WAR files deployable on any J2EE-container:
Glassfish, JBoss, Tomcat, Jetty, …
38

39. JRuby on Rails
39

40. JRuby on Glassfish
One machine
40

41. JRuby on Glassfish
One machine Multiple machines
41

42. Setup JRuby on Glassfish
1. Download JRuby and Glassfish
2. From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.html
42

43. Warble Configuration
Define min/max
Rails runtimes
43

44. Glassfish for development
Shrunk-down version of Glassfish v3 - packaged as a gem
Can run any Rack-based application
Installation:
• $ jruby -S gem install glassfish
• $ glassfish
• There is no Step 3
44

45. Rails Setup
45

46. Proxy Options
46

47. Proxy Requirements
• Hide cluster backend from the user
• Load-balancer backend instances
• Recognize down hosts
• Fair scheduler
• (Deliver static content)
47

48. Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
48

49. Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
49

50. Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
50

51. Apache 2.2
Pro
• Stable, robust, and mature
• Many people know how to work with Apache
• Integrates well with other modules (SVN, DAV, Auth, …)
Con
• Apache can be complicated to configure
• The stock Apache is quite resource-hungry compared to pure proxy solutions
51

52. Nginx – From Russia with love
• Nginx - popular Russian webserver with good proxy support
• Can load-balance multiple backends and deliver static content
• Quite popular with Mongrel as the Rails backend
• Recent Passenger support
52

53. Nginx Configuration
Simple proxy example
Get complete version here:
http://brainspl.at/nginx.conf.txt
53

54. Nginx
Pro
• Stable, robust, and fast
• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files
• Simpler configuration file
• Can directly talk to memcached - SSI
Con
• More documentation would be nice
• No equivalent for many Apache modules
54

55. Lighttpd
• Lightweight and fast webserver
• Balancing proxy support
• Good FastCGI support
• Used to be popular – until Mongrel came around
55

56. Lighttpd Configuration
Simple proxy example
56

57. Lighttpd
Pro
• Fast and lightweight
• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files
• Simpler configuration file
Con
• Unstable for some people
• Slow development cycle
• More documentation would be nice
• Configuration file can be too simple (virtual host aliasing)
• No equivalent for many Apache modules
• (No Passenger support)
57

58. HA-Proxy
• HAProxy – reliable, high performance TCP/HTTP load balancer
• Proxying and content inspection
• No content serving, just a proxy
• Mature proxy module (fair scheduler)
• ACL support
See also similar Pound and Pen
58

59. HAProxy
Simple proxy example
59

60. HAProxy
Pro
• Mature, stable, robust, and fast
• TCP and HTTP balancing
Con
• Few Rails examples
• Usually not needed in a Rails setup
60

61. Recommended Setups
61

62. Small Site
Recommendation
Apache 2.2 with mod_rails /
Phusion Passenger
62

63. Medium Site
Recommendation
• Apache/Nginx as the frontend proxy
• Passenger/mod_rails as the backend
• Deliver static files with Apache
63

64. Large Rails Setup
Recommendation
• Redundant load-balancer
• Redundant proxy / web
• Passenger/mod_rails
64

65. Heavy Static Files
Recommendation
• Deliver static assets
through separate web
server farm
• Passenger/mod_rails
65

66. Java Shop
Recommendation
• Deliver a Rails-WAR file and you are
done
• Integrate with existing Java
landscape and infrastructure
66

67. 67

68. Application Server Handler
68

69. Application Server Handler
69

70. Application Server Handler
70

71. Rack
71

72. Rack Handler
72

73. Rack
73

74. Rack Middleware
74

75. Rack Middleware
• Authentication (HTTP, OpenID)
• Sessions
• Routing
• Caching
• Logging
• Debugging
75

76. Support
Frameworks Web Server
• Rails • CGI
• Merb • FastCGI
• Camping • Mongrel
• Sinatra • Phusion Passenger
• Mack • JRuby Servlets
• Maveric • WEBrick
• Halcyon • Ebb
• Ramaze • Thin
• Rum • LiteSpeed
• Vintage • Swiftcore
• Waves • Unicorn
76

77. Remarks
77

78. Ruby Enterprise Edition
• Copy-On-Write patches to Ruby 1.8
• Saves memory when spawning several Rails instances
• Used by Phusion Passenger if available
78

79. Thin, Ebb, Evented Mongrel & Co.
• Alternatives to Mongrel
• Claim to be faster, lighter, and what have you
• Rendering “Hello World” is usually not your bottleneck
Stick with stable and robust Mongrel, or better yet, with Passenger
79

80. Infrastructure
Deployment
80

81. 81

82. Deployment Options
82

83. Deployment Options
83

84. Deployment Options
84

85. Deployment Options
85

86. What does Capistrano do?
86

87. Capistrano Deployment Cycle
87

88. Requirements
88

89. What doesn’t Capistrano do?
• Plan your initial server setup
• Configure basic services
89

90. Basic Ingredients
• The cap command
• Variables
• Roles
• Tasks
• Namespaces
90

91. Basic Ingredients - cap
Your one-stop deployment shop
91

92. Basic Ingredients - Variables
• Configure basic project information
• Override Capistrano’s default assumptions
• Once set, variables are available globally
• Defined using the set method
92

93. Basic Ingredients - Roles
• Define types of servers
• Default roles
• :www
• :app
• :db
• All can point to the same server
• But all three must be defined
• At least one database server needs to be primary
93

94. Basic Ingredients - Roles
Define custom roles as you please
Can be reused when defining tasks
94

95. Basic Ingredients - Tasks
• Define an atomic set of actions for Capistrano
• Can be called from the command line
• Or other tasks
95

96. Basic Ingredients - Tasks
To find all the tasks available in your project, use
96

97. Basic Ingredients - Namespaces
Group tasks together logically
Namespaces and tasks are separated with “:”
97

98. Get Your Capistrano On
98

99. Get Your Capistrano On
Capfile, the place to include more recipes
99

100. Get Your Capistrano On
config/deploy.rb, application specific configuration
100

101. Capistrano’s Defaults
• Your SCM is Subversion
• Deployment directory is /u/apps/#{application_name}
• User for SCM and SSH is the currently logged-in user
• Commands are run with sudo
101

102. Get Your Capistrano On
102

103. Get Your Capistrano On
• Capistrano expects a directory structure
• Can be created with cap deploy:setup
103

104. The Deployment Lifecycle
104

105. The Deployment Lifecycle
Check the prerequisites:
105

106. The Deployment Lifecycle
Set up your application for the first time
106

107. The Deployment Lifecycle
The initial deployment
1. Checks the revision from the local machine
2. Checks out the code on the remote machines
3. Sets a link called current pointing to the lates release
4. Runs the migrations
5. Fires up application servers
107

108. The Deployment Lifecycle
108

109. The Deployment Lifecycle
Subsequent deployments
1. Checks the revision from the local machine
2. Checks out the code on the remote machines
3. Updates current link
4. Restarts application servers
109

110. The Deployment Lifecycle
110

111. Common Capistrano Tasks
Deploy and run migrations
Run only the migrations
Restart application servers
Rollback to the previous release
111

112. Have Your Shell, and Eat It Too
112

113. Invoking any Command
113

114. Deployment Strategies
114

115. Deployment Strategies
Direct checkout (from scratch) on the servers
115

116. Deployment Strategies
Keep a cached copy of the current SCM head
116

117. Deployment Strategies
Check out locally and transfer
117

118. Give it a little Spin
Capistrano expects a script called spin in script/process
For Passenger
118

119. Customizing Capistrano
119

120. Write your own Tasks
120

121. Write your own Tasks
121

122. Namespace your Tasks
122

123. Callbacks
Execute a task before another runs
Execute a task after another has finished
Callbacks are run in the order they’re defined
123

124. Useful Variables
124

125. Transactions
125

126. Transactions
126

127. Transactions
127

128. Transactions
128

129. The Rest
• Gem dependencies
• Support for deploying through gateway servers
• Parallel execution on multiple servers
• Server setup with deprec gem
129

130. Vlad the Deployer
130

131. Vlad the Deployer
• Simple, Rake-based deployment solution
• Comes with defaults for most popular SCM, app and web servers
131

132. Vlad the Deployer
Load
Configure
132

133. Vlad the Deployer
Deployment
133

134. Vlad the Deployer
• Simple, sometimes too simple
• No callback mechanism, instead the Rake syntax must be used
• Very non-verbose output, using --trace recommended
134

135. Chef
• System Integration Framework
• Written in Ruby – inspired by Puppet
• Manage multiple nodes
• Configuration
• Packages
• Custom scripts and commands
135

136. Architecture
136

137. Recipes
Install and upgrade packages
137

138. Recipes
Create and update remote files
138

139. Recipes
Create and update remote files
139

140. chef-deploy
Enable
Use in recipe
140

141. Chef and chef-deploy
• In heavy development: sometimes unstable and hard to debug
• Be prepared to get your hands dirty
• Better suited for configuration than deployment
• Community recipes at
http://github.com/opscode/cookbooks
141

142. One Click Deploy
142

143. Webistrano
143

144. Webistrano
• Web-UI to Capistrano
• Manages projects and their stages
• Alerting and Accounting
• Scriptable and extendable
• BSD License
http://labs.peritor.com/webistrano
144

145. Macistrano
• Mac-GUI to Webistrano
• Fire and monitor deployments from your desktop
http://github.com/mattmatt/macistrano
145

146. Infrastructure
Practical Session
146

147. Practical Capistrano
147

148. EC2 Instances
• Users
• root / railswaycon
• deploy / railswaycon
• MySQL
• root / ‘’
• Apache Configuration
• /etc/apache2/sites-enabled/default
• Rails
• /srv/www
148

149. Git Repository
• git://github.com/mattmatt/railscasts.git
• Setup the servers
• Deploy!
149

150. Tasks
• Make memcached restart with every deploy
• (Usually not a good idea, but why not?)
150

151. Tasks
• Make Apache recognize maintenance.html and display it if exists
151

152. Maintenance Page
• Create symbolic link to /etc/apache2/mods-available/rewrite.load in /etc/apache2/
mods-enabled
• Update virtual host configuration with rewrite rules
• Run cap deploy:web:disable
• Hook it into the deployment process
152

153. Tasks
• Deploy a Rack-based application
• Install Sinatra locally and on the server
• git://github.com/mattmatt/railswaycon.git
• Create deployment configuration
• Create Apache configuration
153

154. Infrastructure
Monitoring
154

155. The two questions of monitoring
155

156. 1. Is everything still running?
156

157. 2. What are the trends?
157

158. Monit
• Process-level monitoring
• Checks PID-files, ports, and permissions
• Reacts by executing a script and/or alerting
• Has a web GUI for administering nodes
158

159. Monit
MySQL
159

160. Monit
Apache
160

161. Monit
Mongrel
161

162. God
• Ruby-based configuration
• Easily extendible
• Can work with event- and poll-based conditions
• Comes with less recipes than Monit
162

163. God
Mongrel
163

164. Munin
• Host-level monitoring
• Master periodically asks nodes for local data
• Checks system resources and records historical data
• Allows to recognize trends and make predictions
• Alerting support
164

165. Munin
165

166. Munin
166

167. Munin
167

168. Exception Monitoring
• Monitor errors raised by your application
• Usually notifying you by email
• Good old exception_notification Plugin
• Exceptions as a Service
168

169. Hoptoad
169

170. Exceptional
170

171. New Relic
• Rails Performance monitoring
• Live view into your application
171

172. Other Tools
• Nagios
• Big Brother
• New Relic RPM
• FiveRuns
• JMX
172

173. Infrastructure
Advanced Deployment
173

174. Search
174

175. Search
Full text search
Can become very slow on big data sets
175

176. Full Text Search Engine
Separate Service
• Creates full text index
• Application queries search daemon
• Index update through application or
database
Possible Engines
• Ferret
• Sphinx
• Solr
• Lucene
• …
176

177. Search Slave
Database replication slave
• Has complete dataset
• Migrates slow search queries from master
• Can use different database table engine
177

178. Database Index
PostgreSQL Tsearch2
• Core since 8.3
• Allows to create full text index on multiple columns
or arbitrary SQL expressions
MySQL MyISAM FULLTEXT index
• Only works with MySQL <= 5.0 and MyISAM tables
• Full text index on multiple columns
178

179. What to use?
Different characteristics
• Real-time updates and stale data
• Lost updates
• Performance
• Document content and format
• Complexity
179

180. Background Processing
180

181. Problem
Long running tasks
• Resizing uploaded images
• Mailing
• Computing an expensive operation
• Accessing slow back-ends
When running inside request-response-cycle
• Blocks user
• Blocks Rails instance
• Hard to monitor and debug
181

182. Solution
Asynchronous processing in the background
Message/Queue Scheduler
182

183. Background Processing
183

184. Options
Options for message bus: Options for background process:
• Database • (Ruby) Daemon
• Amazon SQS • Cron job with script/runner
• Drb • Forked process
• Memcache • Delayed Job / BJ / (Backgroundrb)
• ActiveMQ • run_later
• … • ….
184

185. Database/Ruby daemon example
185

186. Cloud Infrastructure
186

187. Cloud Infrastructure
Servers come and go
• You do not know your servers before deploying
• Restarting is the same as introducing a new machine
You can’t hardcode IPs
database.yml
187

188. Solution #1
Query and manually adjust
• Servers do not change that often
• New nodes probably need manual intervention
• Use AWS ElasticIPs to ease the pain
Set servers dynamically AWS Elastic IP
188

189. Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
189

190. Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
190

191. Central Directory
Different Implementations
• File on S3
• SimpleDB
• A complete service,
capable of monitoring and controlling your instances
191

192. Q&A
192

193. Peritor GmbH
Teutonenstraße 16
14129 Berlin
Telefon: +49 (0)30 69 20 09 84 0
Telefax: +49 (0)30 69 20 09 84 9
Internet: www.peritor.com
E-Mail: kontakt@peritor.com
193
Peritor GmbH - Alle Rechte vorbehalten 193