Porting Rails Apps to High Availability Systems

4.116 Aufrufe
3.889 Aufrufe

Veröffentlicht am

Veröffentlicht in: Technologie
0 Kommentare
4 Gefällt mir
Statistik
Notizen
  • Als Erste(r) kommentieren

Keine Downloads
Aufrufe
Aufrufe insgesamt
4.116
Auf SlideShare
0
Aus Einbettungen
0
Anzahl an Einbettungen
18
Aktionen
Geteilt
0
Downloads
26
Kommentare
0
Gefällt mir
4
Einbettungen 0
Keine Einbettungen

Keine Notizen für die Folie

Porting Rails Apps to High Availability Systems

  1. 1. Porting Rails apps for High Availability Systems Marcelo Correia Pinheiro @salizzar
  2. 2. $ whoamiProgrammed with ASP, PHP, C#, Java, Python,Ruby, etc etcTwitter: @salizzarhttp://salizzar.net/ (always underconstruction)Locaweb PaaS/IaaS Engineering Team(Domain Registration)
  3. 3. TopicsCommon ArchitectureApplication ModularityUseful GemsDatabase ReplicationSession ManagementApp Distribution and DeploymentConfiguration ManagementLoad BalancingConclusion
  4. 4. Common ArchitectureOne machine running HTTP Server, App and Database(sometimes in a dedicated server)Production config files in application GIT repo (all of ususe git, alright?)Deploy via Capistrano / Vlad / * ?Sufficient for small apps with no perspective to growabsurdly or have monitored growingGrants HA? Hmmm... what do you do if your databasecorrupts journaling? (MongoDB REAL CASE)
  5. 5. Common ArchitectureThink about it and the possibility of anightmare due for a bad db query, zombieprocesses and unexpected errors that willfrozen your appWhat to do? Look at your app with aDistributed System perspective
  6. 6. Application ModularityRails apps, in most cases, can be splitted to twoor more small gems or apps (web services,checkout systems, asynch processing, etc)
  7. 7. Application ModularityYou write good code?If yes, creating gems with it is EASY. How to detect: first DRY while splitting a app How to create: $ bundle gem <asdfg> How to learn more: Creating Your Own Gem in www.rubygems.orgIf not, refactoring!
  8. 8. Application ModularityBreaking your Rails app in smaller apps: increase future improvements first step to make distribution via packaging more easy consequently, more scalableLook at ActiveController::Metal / sinatra /rack-api / goliath / cramp
  9. 9. Application ModularityBe careful: using multiple gems coupling Rails features inside your models (Rails.logger)
  10. 10. Useful GemsHypermediaAsynchronous Processing (resque)WebserversMonitoring
  11. 11. Useful GemsJSON for the winHypermedia-based REST API’s ROAR: https://github.com/apotonick/roar RestModel: https://github.com/rodrigues/ rest_model
  12. 12. Useful Gems# -*- encoding: UTF-8 -*-class Domain  attr_accessor :id, :creation_date, :expiration_dateendmodule DomainRepresenter  include Roar::Representer::JSON,          Roar::Representer::Feature::Hypermedia  property :id, :creation_date, :expiration_dateendrequest = RestClient.get http://an.app/domains/example.comdomain = Domain.new.extend DomainRepresenterdomain.from_json request.body
  13. 13. Useful Gems# -*- encoding: UTF-8 -*-class Domain < RestModel  property :id,               type: String  property :creation_date,    type: Date  property :expiration_date,  type: Date  property :provisioning_id,  type: Integerendrequest = RestClient.get http://an.app/domains/example.comdomain = Domain.from_source(request.body).first
  14. 14. Useful GemsAsynchronous Processing Resque resque-scheduler resque-retry resque-lock resque-logger
  15. 15. Useful Gemsresque-scheduler: https://github.com/bvandenbos/resque- scheduler cron-based Resque extension
  16. 16. Useful Gems# resque_scheduler.ymlappointment_notification_enqueuer:  cron:   "0 4 * * *"  class:  AppointmentNotificationEnqueuer  queue:  appointment_notification_enqueuer  args:    email:  true    sms:    true de cron Incluir config  description: Notify today appointments to customersfree_slot_monitor:  cron:   "0/15 * * * *"  class:  FreeSlotMonitor  queue:  free_slot_monitor  description:  Check for free slots in agenda
  17. 17. Useful Gems# -*- encoding: UTF-8 -*-class AppointmentNotificationEnqueuer  @queue = :appointment_notification_enqueuer  def self.perform(args)    appointments = Appointment.today    factory = NotificationFactory.new args    appointments.each do |appointment|      factory.enqueue appointment.id    end  endend
  18. 18. Useful Gemsresque-retry: https://github.com/lantins/resque-retry redis backed retry count times until reach a limit retry on all or specific exceptions retry functionality on Resque Web Server
  19. 19. Useful Gems# -*- encoding: UTF-8 -*-class EmailNotificationWorker  extend Resque::Plugins::Retry  @queue = :email_notification_enqueuer  @retry_limit = 4  @retry_delay = 300  def self.perform(args)    appointment = Appointment.find id: args[id]    mailer = MailNotification.new    mailer.notify_appointment appointment  endend
  20. 20. Useful Gems
  21. 21. Useful Gemsresque-lock: https://github.com/defunkt/resque-lock Grants only one instance of a job running in a time
  22. 22. Useful Gems# -*- encoding: UTF-8 -*-class SmsNotificationWorker  extend Resque::Plugins::Lock  def self.perform(args)    appointment = Appointment.find id: args[id]    sender = SmsNotification.new    sender.deliver_appointment appointment  endend
  23. 23. Useful Gemsresque-logger: https://github.com/salizzar/resque-logger Provides a logger for each Resque worker based on queue name Log Driven Development, please!
  24. 24. Useful Gems# config/initializers/resque.rblog_path = Rails.root.join logconfig = {  folder:     log_path,  class_name: Logger,  class_args: [ daily, 1.gigabyte ],  level:      Logger::INFO,  formatter:  Logger::Formatter.new,}Resque.logger = config
  25. 25. Useful Gems# -*- encoding: UTF-8 -*-class EmailNotificationWorker  extend Resque::Plugins::Retry, Resque::Plugins::Logger  @queue = :email_notification_enqueuer  @retry_limit = 4  @retry_delay = 300  def self.perform(args)    appointment = Appointment.find id: args[id]    logger.info(“trying to notify #{appointment.id}”)    mailer = MailNotification.new    mailer.notify_appointment appointment  endend
  26. 26. Useful GemsWeb Servers: Thin https://github.com/macournoyer/thin EventMachine powered Killer combo with Nginx Unicorn http://unicorn.bogomips.org Pre-fork based Great with low-latency network
  27. 27. Useful GemsMonitoring God https://github.com/mojombo/god Several memory leaks at beginning, very stable today Largely used to manage Resque workers and Thin clusters Bluepill https://github.com/arya/bluepill More simple than God, less features Some issues with signal handling
  28. 28. Useful GemsIf memory usage is a problem, use Monit. http://mmonit.com/monit VERY stable HIGH expansible EASY configuration
  29. 29. Database Replication Relational Database Replication is not trivial Be a friend of a sysadm CAP Theorem: YOU NEED TO KNOW No magic: look to your architecture Traditional replication strategies: master / slave for consistency master for write / slave for read
  30. 30. Database Replication My experience: Relational Databases: no choice MongoDB: replica set Redis: master / slave
  31. 31. Database Replication For MongoDB: MongoMapper https://github.com/jnunemaker/mongomapper Largely used, stable Mongoid https://github.com/mongoid/mongoid ActiveRecord-like
  32. 32. Database Replication brotip #1342: check if your MongoDB client detects when master server goes down automagically (MongoMapper NOT does it) Suggested keynote: MongoDB em Produção, dá pra confiar? http://www.slideshare.net/deferraz/ mongodb-em-producao
  33. 33. Session Management Given a problem, use the right tool PLEASE memcached http://memcached.org distributed memory object caching system key/value storage stores arbitrary data can be clustered
  34. 34. Session Management repcached http://repcached.lab.klab.org patch inside memcached to add data replication offers session sharing between memcached servers
  35. 35. Session Management available gems for memcache memcache-client https://github.com/mperham/memcache-client First Ruby client, written by Mike Perham memcached https://github.com/evan/memcached Ruby SWIG wrapper for libmemcached C library dalli https://github.com/mperham/dalli Major refactoring from memcache-client
  36. 36. App Distribution and Deployment Capistrano? Hnmm... talk with a sysadm about it. It works but lacks security severely (SSH, GCC in production is a invite for attack) Sysadms knows process monitoring tools, software distribution and server configuration better than us Be a friend of a sysadm Or LEARN A LOT to be one :)
  37. 37. App Distribution and Deployment Build self-packaged apps $ man dh_make (for Debian based distros) Hard or too boring? Maybe, but effort compensates $ dh_make --native --simple --indep -- packagename <yourapp> --copyright blank $ debuild # dpkg -i <yourapp>_<version>_<arch>.deb
  38. 38. App Distribution and Deployment Generating a distro-based package with your app grants fast update/rollback to sysadms # apt-get update && apt-get install <yourapp> # dpkg -i <your_app_package> If you want to automate builds: Setup a repository server Use Bricklayer and be happy
  39. 39. Configuration ManagementKeeping production configs in your app gitrepo is not always a good ideaGetting config files over control is VERYimportant to avoid a messy environmentStandardization is king in server configuration
  40. 40. Configuration ManagementAt this time, three choices: Puppet Chef cfengine
  41. 41. Configuration ManagementPuppet https://github.com/puppetlabs/puppet Written in Ruby Large community adoption Great number of recipes Works well
  42. 42. Configuration ManagementChef https://github.com/opscode/chef Written in Ruby Created as a Puppet alternative in order to solve some problems with it Large community adoption Works good
  43. 43. Configuration Managementcfengine http://cfengine.com Written in C Largely adopted by sysadms HARD to learn F****** ROBUST (a single master node can manage 4K servers)
  44. 44. Load BalancingWTF is Load Balancing? A network methodology to distribute workload across multiple computers in a transparent way for users Can be proved by /hard|soft/wareAlternatives: LVS with keepalived HAProxy with heartbeat
  45. 45. Load BalancingLVS (Linux Virtual Server) http://www.linuxvirtualserver.org Robust, high scalable solution Need to share HTTP sessions? It’s the choice Built-in Linux Kernel module since 2.4.28*, runs in layer-4 (transport layer)keepalived http://www.keepalived.org Largely used with LVS by sysadms Uses Virtual Router Redundancy Protocol (vrrp) - RFC
  46. 46. Load BalancingHAProxy http://haproxy.1wt.eu Generic solution Works if session sharing between servers is not required Runs in layer-7 (application layer), computationally more expensiveheartbeat http://www.linux-ha.org/wiki/Heartbeat Based on message exchange between clients in a cluster to detect failures
  47. 47. ConclusionPorting Rails apps to a HA environment is a not trivial challenge, butnothing insanely hard Forces you to think in application architecture, design, distribution, modularity brotip #915435: do it before coding Restricts you in order to make server correctly managed Implies you to be prudent on a deploy “it sucks at all” = bad excuse, my friendApp tunning later, without fear :)In fact, in HA deploys are more faster
  48. 48. Questions?No fear, answers are free as beer :)
  49. 49. Thank you!

×