Batch processing in rails

1,567 views

Published on

Slides for my talk on railsclub.ru 2011'04.

Published in: Self Improvement, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,567
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Batch processing in rails

    1. 1. Batch processing on RoR http://slidesha.re/k6zrSL@SergeyMoiseevmoiseev.sergey@gmail.com
    2. 2. О себе• http://moiseev--sergey.moikrug.ru/• пишу на Ruby с 2007 года.
    3. 3. Задача• Генерация контента (книг) для сайта.• Входной формат: PDF.• Выходной формат: SWF.• Генератор: print2flash.• Основной приоритет: скорость обработки (количество публикуемых книг в месяц).
    4. 4. Нюансы• Книги поступают разного качества, часть из них требует многократной переработки.• Print2flash может печатать не более одного задания за раз в пределах машины.• Требует windows окружения.
    5. 5. Архитектура• Множественные инсталляции Rails приложения.• Общая база.• Конечный автомат для контроля состояния задания.• Отдельные операции - rake таски запускаемые по крону или планировщику задач.
    6. 6. Конфигурация
    7. 7. https://github.com/rubyist/aasminclude AASMaasm_column :stateaasm_initial_state :loadedaasm_state :loaded, :enter => :reset_retry_count, :exit=>:apply_total_pagesaasm_state :in_abbyy, :enter => :copy_to_abbyyaasm_state :p2f_queue, :enter=>:apply_pages_countaasm_state :in_p2fline, :enter => :copy_to_p2flineaasm_state :p2f_printing, :enter => :p2f_printaasm_state :print_failed, :enter => :free_p2flineaasm_state :wait_for_confirm, :enter => :free_p2flineaasm_state :confirmedaasm_state :confirmed_non_fullaasm_state :on_serveraasm_state :on_server_non_fullaasm_state :publishedaasm_state :published_non_fullaasm_state :holdaasm_state :mycopaasm_state :restarting, :after_enter => :check_avail_pdfaasm_state :restarting_full, :exit=>:restart_full Состояния конечного автомата
    8. 8. Структура модельного слоя
    9. 9. Почему не очередь• Так казалось проще.• Для каждого задания можно одназначно сказать его статус.• Единый каталог со всем готовым на текущий момент контентом.
    10. 10. Гетерогенность• return unless RUBY_PLATFORM =~ /mswin/• Dir[File.join(work_catalog,"page*.pdf")]
    11. 11. Аудит https://gist.github.com/949049#file_book.rb https://github.com/collectiveidea/acts_as_auditedacts_as_audited :except => [:name, :file_name] https://gist.github.com/949049#file_production.rakedesc "Loads files into Production line"task :process_batch => :environment do Audit.as_user("rake:process_batch") do Book.getFromBatch unless running?("process_batch") endend https://gist.github.com/949049#file_books_controller.rbdef restart if @book.restartable? Audit.as_user current_user do @book.restart_production! end flash[:notice] = Обработка перезапущена. else flash[:error] = Перезапуск невозможен. end redirect_to request.refererend
    12. 12. http://munin-monitoring.org/Мониторинг
    13. 13. namespace :munin do task :config do puts <<-CONFIGgraph_title Produced pages in last 5 minutesgraph_args -l 0graph_vlabel pages amountgraph_category Appgraph_info This graph shows amount of pages producedtexts.label Text filesdocs.label Doc filesviews.label View filesquotes.label Quote filesCONFIG exit 0 end task :run => :environment do res = ActiveRecord::Base.connection.execute("select count(nullif(text_ready_time>(now()-5minutes::interval),false)) as c0,count(nullif(doc_ready_time>(now()-5 minutes::interval),false)) asc1,count(nullif(view_ready_time>(now()-5 minutes::interval),false)) asc2,count(nullif(quote_ready_time>(now()-5 minutes::interval),false)) as c3 from pages where updated_at >(now()-5 minutes::interval);") if res puts "texts.value #{res[0][c0]}" puts "docs.value #{res[0][c1]}" puts "views.value #{res[0][c2]}" puts "quotes.value #{res[0][c3]}" end exit 0 endend https://gist.github.com/949049#file_munin.rake
    14. 14. http://www.nagios.org/namespace :nagios do task :abbyy_activity, [:warn, :crit] => :environment do |t, args| if args.warn and args.crit states_list = ["loaded", "in_abbyy", "p2f_queue"] warn_level = args.warn.to_i crit_level = args.crit.to_i warn_events = Audit.find(:all, :conditions=>["auditable_type = Book and (username is null or usernameilike rake:%) and action = update and created_at > ?", warn_level.minutes.ago]) crit_events = Audit.find(:all, :conditions=>["auditable_type = Book and (username is null or usernameilike rake:%) and action = update and created_at > ?", crit_level.minutes.ago]) if (crit_count = crit_events.select{|a| a[changes][state] and (a[changes][state] &states_list).size > 1 }).empty? puts "CRITICAL No state changes at abbyy for #{crit_level} minutes" exit 2 end if (warn_count = warn_events.select{|a| a[changes][state] and (a[changes][state] &states_list).size > 1 }).empty? puts "WARNING No state changes at abbyy for #{warn_level} minutes" exit 1 end puts "OK Abbyy: state changes #{warn_count.size} for #{warn_level} minutes; state changes#{crit_count.size} for #{crit_level} minutes." exit 0 else exit 3 end end https://gist.github.com/949049#file_nagios.rake

    ×