Deploying And Monitoring Rails

Deploying and Monitoring Ruby on Rails A practical guide Mathias Meyer and Jonathan Weiss, 25.05.2009 Peritor GmbH

Who are we? Jonathan Weiss • Consultant for Peritor GmbH in Berlin • Specialized in Rails, Scaling, Deployment, and Code Review • Webistrano - Rails deployment tool • FreeBSD Rubygems and Ruby on Rails maintainer http://www.peritor.com http://blog.innerewut.de 2

Who are we? Mathias Meyer • Consultant for Peritor GmbH in Berlin • Specialized in Rails, Performance/Database Tuning, Deployment, and Refactoring • Maintainer of Macistrano – Webistrano-Client for Mac OS X (and soon the iPhone), run_later and acts_as_solr http://www.paperplanes.de 3

Agenda Infrastructure Deployment Practical Session Monitoring Q&A 4

Infrastructure Infrastructure 5

Simple Rails Setup One Rails instance handles all requests Rails is single-threaded: There is only one concurrent request 6

Rails Setup 7

Rails Setup 8

Typical Rails Setup • A load-balancer distributes the incoming requests • Some load-balancers will deliver static requests themselves • Several Rails instances handle all requests • Number of concurrent requests equals number of Rails instances 9

Rails Setup Options 10

Deployment Questions Apache? Pound? mod_proxy_balancer? FastCGI? Mongrel? Proxy? Load-balancer? Nginx? mod_rails? Ebb? HA-Proxy? Reverse Proxy? Swiftiply? Phusion Passenger? Thin? Pen? Rails Application Server? Lighttpd? 11

What we are going to cover today Rails Application Server Proxy/Web Server • FastCGI • Apache2 • Mongrel • Nginx • mod_rails / Phussion Passenger • Lighttpd • JRuby + Glassﬁsh & Co. • HA-Proxy 12

FastCGI 13

FastCGI • Protocol to communicate with long-running CGI applications • Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd • Proxy local and remote FastCGI instances • Oldest way of deploying Rails • Deprecated and unstable • Hard to debug (FastCGI protocol) 14

FastCGI 15

FastCGI 16

17

Mongrel 18

Mongrel • Developed by Zed Shaw as an alternative to FastCGI • Complete HTTP-Server that can load arbitrary Ruby-servlets • Built-in Rails support 19

Mongrel 20

Mongrel Apache 2.2 – mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound 21

Mongrel Cluster Utility to manage several Mongrel instances 22

Mongrel Cluster Control Mongrels 23

Mongrel and Apache 2.2 Deﬁne Mongrel Cluster in Apache 24

Simple Mongrel and Apache 2.2 Redirect all trafﬁc to the Mongel cluster 25

A more complex example • Redirect dynamic requests • Serve static content • Support cached pages • Support maintenance page • Enable client-side caching of images, stylesheets, and JavaScript • Compress output if supported 26

Mongrel • Very robust • Strict HTTP parser • Easy to debug (HTTP!) • Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer • Can be a bit difﬁcult to setup (mongrel_cluster, ports, Apache) • Not so easy on mass/virtual hosting 27

mod_rails 28

mod_rails a.k.a Phusion Passenger • Module for Apache 2.2 (and Nginx) • Allows Apache to control Rails instances • Apache starts and stops application instances depending on the application load • Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.) • Only manages Rails on one host - no remote instances • Combine with HTTP-Proxy / balancing solution 29

Install Phusion Passenger Install Apache module Load and activate in Apache 30

Install Phusion Passenger Install Passenger with nginx (installs custom nginx) Conﬁgure nginx 31

Customized Phusion Passenger Control Rails instance number 32

Control Phusion Passenger Restart after deployment: 33

Phusion Passenger One machine 34

Phusion Passenger One machine Multiple machines 35

Phusion Passenger • Ready for production • Makes setup easier – on the single machine level • Multiple servers still require load balancer • Suitable for mass-hosting • upcoming standard way of deploying Rails 36

JRuby 37

• Ruby Runtime on the Java Virtual Machine • Implemented in Java and Ruby • Compiles Ruby into Java-bytecode • Integrates with Java code and libraries • Java’s promises of native threads and JIT • Allows for Ruby/Rails applications to be packaged as WAR files • WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, Jetty, … 38

JRuby on Rails 39

JRuby on Glassﬁsh One machine 40

JRuby on Glassﬁsh One machine Multiple machines 41

Setup JRuby on Glassﬁsh 1. Download JRuby and Glassﬁsh 2. From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.html 42

Warble Conﬁguration Deﬁne min/max Rails runtimes 43

Glassfish for development Shrunk-down version of Glassfish v3 - packaged as a gem Can run any Rack-based application Installation: • $ jruby -S gem install glassfish • $ glassfish • There is no Step 3 44

Rails Setup 45

Proxy Options 46

Proxy Requirements • Hide cluster backend from the user • Load-balancer backend instances • Recognize down hosts • Fair scheduler • (Deliver static content) 47

Apache 2.2 • Apache 2.2 introduced mod_proxy_balancer • mod_proxy_balancer can speak to multiple backends and balance requests • Apache can acts as a pure proxy or can also serve static ﬁles 48

Apache 2.2 Pro • Stable, robust, and mature • Many people know how to work with Apache • Integrates well with other modules (SVN, DAV, Auth, …) Con • Apache can be complicated to conﬁgure • The stock Apache is quite resource-hungry compared to pure proxy solutions 51

Nginx – From Russia with love • Nginx - popular Russian webserver with good proxy support • Can load-balance multiple backends and deliver static content • Quite popular with Mongrel as the Rails backend • Recent Passenger support 52

Nginx Conﬁguration Simple proxy example Get complete version here: http://brainspl.at/nginx.conf.txt 53

Nginx Pro • Stable, robust, and fast • Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files • Simpler configuration file • Can directly talk to memcached - SSI Con • More documentation would be nice • No equivalent for many Apache modules 54

Lighttpd • Lightweight and fast webserver • Balancing proxy support • Good FastCGI support • Used to be popular – until Mongrel came around 55

Lighttpd Conﬁguration Simple proxy example 56

Lighttpd Pro • Fast and lightweight • Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files • Simpler configuration file Con • Unstable for some people • Slow development cycle • More documentation would be nice • Configuration file can be too simple (virtual host aliasing) • No equivalent for many Apache modules • (No Passenger support) 57

HA-Proxy • HAProxy – reliable, high performance TCP/HTTP load balancer • Proxying and content inspection • No content serving, just a proxy • Mature proxy module (fair scheduler) • ACL support See also similar Pound and Pen 58

HAProxy Simple proxy example 59

HAProxy Pro • Mature, stable, robust, and fast • TCP and HTTP balancing Con • Few Rails examples • Usually not needed in a Rails setup 60

Recommended Setups 61

Small Site Recommendation Apache 2.2 with mod_rails / Phusion Passenger 62

Medium Site Recommendation • Apache/Nginx as the frontend proxy • Passenger/mod_rails as the backend • Deliver static ﬁles with Apache 63

Large Rails Setup Recommendation • Redundant load-balancer • Redundant proxy / web • Passenger/mod_rails 64

Heavy Static Files Recommendation • Deliver static assets through separate web server farm • Passenger/mod_rails 65

Java Shop Recommendation • Deliver a Rails-WAR ﬁle and you are done • Integrate with existing Java landscape and infrastructure 66

67

Application Server Handler 68

Rack 71

Rack Handler 72

Rack 73

Rack Middleware 74

Rack Middleware • Authentication (HTTP, OpenID) • Sessions • Routing • Caching • Logging • Debugging 75

Support Frameworks Web Server • Rails • CGI • Merb • FastCGI • Camping • Mongrel • Sinatra • Phusion Passenger • Mack • JRuby Servlets • Maveric • WEBrick • Halcyon • Ebb • Ramaze • Thin • Rum • LiteSpeed • Vintage • Swiftcore • Waves • Unicorn 76

Remarks 77

Ruby Enterprise Edition • Copy-On-Write patches to Ruby 1.8 • Saves memory when spawning several Rails instances • Used by Phusion Passenger if available 78

Thin, Ebb, Evented Mongrel & Co. • Alternatives to Mongrel • Claim to be faster, lighter, and what have you • Rendering “Hello World” is usually not your bottleneck Stick with stable and robust Mongrel, or better yet, with Passenger 79

Infrastructure Deployment 80

81

Deployment Options 82

What does Capistrano do? 86

Capistrano Deployment Cycle 87

Requirements 88

What doesn’t Capistrano do? • Plan your initial server setup • Conﬁgure basic services 89

Basic Ingredients • The cap command • Variables • Roles • Tasks • Namespaces 90

Basic Ingredients - cap Your one-stop deployment shop 91

Basic Ingredients - Variables • Conﬁgure basic project information • Override Capistrano’s default assumptions • Once set, variables are available globally • Deﬁned using the set method 92

Basic Ingredients - Roles • Deﬁne types of servers • Default roles • :www • :app • :db • All can point to the same server • But all three must be deﬁned • At least one database server needs to be primary 93

Basic Ingredients - Roles Deﬁne custom roles as you please Can be reused when deﬁning tasks 94

Basic Ingredients - Tasks • Deﬁne an atomic set of actions for Capistrano • Can be called from the command line • Or other tasks 95

Basic Ingredients - Tasks To ﬁnd all the tasks available in your project, use 96

Basic Ingredients - Namespaces Group tasks together logically Namespaces and tasks are separated with “:” 97

Get Your Capistrano On 98

Get Your Capistrano On Capﬁle, the place to include more recipes 99

Get Your Capistrano On config/deploy.rb, application specific configuration 100

Capistrano’s Defaults • Your SCM is Subversion • Deployment directory is /u/apps/#{application_name} • User for SCM and SSH is the currently logged-in user • Commands are run with sudo 101

Get Your Capistrano On 102

Get Your Capistrano On • Capistrano expects a directory structure • Can be created with cap deploy:setup 103

The Deployment Lifecycle 104

The Deployment Lifecycle Check the prerequisites: 105

The Deployment Lifecycle Set up your application for the ﬁrst time 106

The Deployment Lifecycle The initial deployment 1. Checks the revision from the local machine 2. Checks out the code on the remote machines 3. Sets a link called current pointing to the lates release 4. Runs the migrations 5. Fires up application servers 107

The Deployment Lifecycle Subsequent deployments 1. Checks the revision from the local machine 2. Checks out the code on the remote machines 3. Updates current link 4. Restarts application servers 109

Common Capistrano Tasks Deploy and run migrations Run only the migrations Restart application servers Rollback to the previous release 111

Have Your Shell, and Eat It Too 112

Invoking any Command 113

Deployment Strategies 114

Deployment Strategies Direct checkout (from scratch) on the servers 115

Deployment Strategies Keep a cached copy of the current SCM head 116

Deployment Strategies Check out locally and transfer 117

Give it a little Spin Capistrano expects a script called spin in script/process For Passenger 118

Customizing Capistrano 119

Write your own Tasks 120

Write your own Tasks 121

Namespace your Tasks 122

Callbacks Execute a task before another runs Execute a task after another has ﬁnished Callbacks are run in the order they’re deﬁned 123

Useful Variables 124

Transactions 125

Transactions 126

Transactions 127

Transactions 128

The Rest • Gem dependencies • Support for deploying through gateway servers • Parallel execution on multiple servers • Server setup with deprec gem 129

Vlad the Deployer 130

Vlad the Deployer • Simple, Rake-based deployment solution • Comes with defaults for most popular SCM, app and web servers 131

Vlad the Deployer Load Conﬁgure 132

Vlad the Deployer Deployment 133

Vlad the Deployer • Simple, sometimes too simple • No callback mechanism, instead the Rake syntax must be used • Very non-verbose output, using --trace recommended 134

Chef • System Integration Framework • Written in Ruby – inspired by Puppet • Manage multiple nodes • Conﬁguration • Packages • Custom scripts and commands 135

Architecture 136

Recipes Install and upgrade packages 137

Recipes Create and update remote ﬁles 138

Recipes Create and update remote ﬁles 139

chef-deploy Enable Use in recipe 140

Chef and chef-deploy • In heavy development: sometimes unstable and hard to debug • Be prepared to get your hands dirty • Better suited for conﬁguration than deployment • Community recipes at http://github.com/opscode/cookbooks 141

One Click Deploy 142

Webistrano 143

Webistrano • Web-UI to Capistrano • Manages projects and their stages • Alerting and Accounting • Scriptable and extendable • BSD License http://labs.peritor.com/webistrano 144

Macistrano • Mac-GUI to Webistrano • Fire and monitor deployments from your desktop http://github.com/mattmatt/macistrano 145

Infrastructure Practical Session 146

Practical Capistrano 147

EC2 Instances • Users • root / railswaycon • deploy / railswaycon • MySQL • root / ‘’ • Apache Conﬁguration • /etc/apache2/sites-enabled/default • Rails • /srv/www 148

Git Repository • git://github.com/mattmatt/railscasts.git • Setup the servers • Deploy! 149

Tasks • Make memcached restart with every deploy • (Usually not a good idea, but why not?) 150

Tasks • Make Apache recognize maintenance.html and display it if exists 151

Maintenance Page • Create symbolic link to /etc/apache2/mods-available/rewrite.load in /etc/apache2/ mods-enabled • Update virtual host conﬁguration with rewrite rules • Run cap deploy:web:disable • Hook it into the deployment process 152

Tasks • Deploy a Rack-based application • Install Sinatra locally and on the server • git://github.com/mattmatt/railswaycon.git • Create deployment conﬁguration • Create Apache conﬁguration 153

Infrastructure Monitoring 154

The two questions of monitoring 155

1. Is everything still running? 156

2. What are the trends? 157

Monit • Process-level monitoring • Checks PID-ﬁles, ports, and permissions • Reacts by executing a script and/or alerting • Has a web GUI for administering nodes 158

Monit MySQL 159

Monit Apache 160

Monit Mongrel 161

God • Ruby-based conﬁguration • Easily extendible • Can work with event- and poll-based conditions • Comes with less recipes than Monit 162

God Mongrel 163

Munin • Host-level monitoring • Master periodically asks nodes for local data • Checks system resources and records historical data • Allows to recognize trends and make predictions • Alerting support 164

Munin 165

Munin 166

Munin 167

Exception Monitoring • Monitor errors raised by your application • Usually notifying you by email • Good old exception_notiﬁcation Plugin • Exceptions as a Service 168

Hoptoad 169

Exceptional 170

New Relic • Rails Performance monitoring • Live view into your application 171

Other Tools • Nagios • Big Brother • New Relic RPM • FiveRuns • JMX 172

Infrastructure Advanced Deployment 173

Search 174

Search Full text search Can become very slow on big data sets 175

Full Text Search Engine Separate Service • Creates full text index • Application queries search daemon • Index update through application or database Possible Engines • Ferret • Sphinx • Solr • Lucene • … 176

Search Slave Database replication slave • Has complete dataset • Migrates slow search queries from master • Can use different database table engine 177

Database Index PostgreSQL Tsearch2 • Core since 8.3 • Allows to create full text index on multiple columns or arbitrary SQL expressions MySQL MyISAM FULLTEXT index • Only works with MySQL <= 5.0 and MyISAM tables • Full text index on multiple columns 178

What to use? Different characteristics • Real-time updates and stale data • Lost updates • Performance • Document content and format • Complexity 179

Background Processing 180

Problem Long running tasks • Resizing uploaded images • Mailing • Computing an expensive operation • Accessing slow back-ends When running inside request-response-cycle • Blocks user • Blocks Rails instance • Hard to monitor and debug 181

Solution Asynchronous processing in the background Message/Queue Scheduler 182

Background Processing 183

Options Options for message bus: Options for background process: • Database • (Ruby) Daemon • Amazon SQS • Cron job with script/runner • Drb • Forked process • Memcache • Delayed Job / BJ / (Backgroundrb) • ActiveMQ • run_later • … • …. 184

Database/Ruby daemon example 185

Cloud Infrastructure 186

Cloud Infrastructure Servers come and go • You do not know your servers before deploying • Restarting is the same as introducing a new machine You can’t hardcode IPs database.yml 187

Solution #1 Query and manually adjust • Servers do not change that often • New nodes probably need manual intervention • Use AWS ElasticIPs to ease the pain Set servers dynamically AWS Elastic IP 188

Solution #2 Use a central directory service • A central place to manage your running instances • Instances query the directory and react 189

Solution #2 Use a central directory service • A central place to manage your running instances • Instances query the directory and react 190

Central Directory Different Implementations • File on S3 • SimpleDB • A complete service, capable of monitoring and controlling your instances 191

Q&A 192

Peritor GmbH Teutonenstraße 16 14129 Berlin Telefon: +49 (0)30 69 20 09 84 0 Telefax: +49 (0)30 69 20 09 84 9 Internet: www.peritor.com E-Mail: kontakt@peritor.com 193 Peritor GmbH - Alle Rechte vorbehalten 193

Deploying And Monitoring Rails

0 comments

Notes on slide 1

5 Favorites