Your SlideShare is downloading. ×
0
The Architecture of
PicCollage Server
http://godfat.org/slide/
2013-01-08-PicCollage.pdf
•PicCollage, Software and Services
Table of Contents
•PicCollage, Software and Services
•Ruby Codes
Table of Contents
•PicCollage, Software and Services
•Ruby Codes
•Miscellaneous Extras
Table of Contents
PicCollage
Software
Services
&
,
•Gemfile
•Procfile
•Rakefile
•rainbows.rb
•concurrency.rb
•config.ru
Ruby Codes
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
•image_util.rb
•run_later.rb
•/search
•/create_and_share
Ruby Codes
(Cont.) (Cont.)
•crazy inspect
•who cares backtrace for 404
•assets pipeline and
database connection
•fiber stack overflow
•fiber database co...
PicCollage
Software
Services
&
,
PicCollage
•Make collages of your photos
•Import photos from places
•Touch gestures to rotate, resize, delete
•Double-tap ...
PicCollage
Software
Services
&
,
Software
•Ruby 1.8.7 -> 1.9.2 -> 1.9.3
Software
•Ruby 1.8.7 -> 1.9.2 -> 1.9.3
•Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 ->
3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5...
Software
•Ruby 1.8.7 -> 1.9.2 -> 1.9.3
•Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 ->
3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5...
Software
•Ruby 1.8.7 -> 1.9.2 -> 1.9.3
•Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 ->
3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5...
Software
•Ruby 1.8.7 -> 1.9.2 -> 1.9.3
•Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 ->
3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5...
PicCollage
Software
Services
&
,
Services
•Github
Services
•Github
•Heroku Bamboo -> Cedar
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
•RedisToGo -> OpenRedis
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
•RedisToGo -> OpenRedis
•Memca...
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
•RedisToGo -> OpenRedis
•Memca...
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
•RedisToGo -> OpenRedis
•Memca...
Services
•Github
•Heroku Bamboo -> Cedar
•Heroku PostgreSQL 9.0 -> 9.2
•MongoHQ -> MongoLab
•RedisToGo -> OpenRedis
•Memca...
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
Gemfile
ruby '1.9.3'
source 'http://rubygems.org'
gem 'rails', '3.2.9'
gem 'jellyfish'
gem 'pg', :platform => :ruby
gem 're...
# service
gem 'newrelic_rpm', :require => false
gem 'exceptional'
gem 'apns', :github => 'jpoz/APNS'
# ^^^^ needs feedback...
# upload
gem 'paperclip' # needs various patches
gem 'aws-sdk'
# plist
gem 'nokogiri'
# pagination
gem 'kaminari' # needs ...
# make `rails server` work for zbatery
gem 'rack-handlers'
# respect rack logger
gem 'rack-rails-logger'
# redirect from w...
# api client
gem 'rest-core'
gem 'rest-more'
# bluebase
gem 'bb_more' , :path => 'bluebase'
gem 'bb_locales', :path => 'bl...
group :assets do
gem 'sass-rails'
gem 'coffee-rails'
gem 'uglifier'
end
group :cedar do
gem 'rib'
gem 'bond'
gem 'readline...
group :test do
gem 'minitest'
gem 'rr'
gem 'webmock'
gem 'kramdown'
end
group :test do
gem 'capybara'
gem 'rspec'
gem 'rsp...
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
Procfile
web: ruby -r bundler/setup -S zbatery
-c config/rainbows.rb
-p $PORT -E $RACK_ENV
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
task 'test:all' => ['test',
'test:bluebase']
task 'env:test' do
Rails.env = 'test'
end
Rakefile
task 'test:bluebase' => ['env:test'] do
# remove bundler shit
gemfile = ENV.delete('BUNDLE_GEMFILE')
sh './bin/bluebase-te...
task :test => ['test:api']
Rakefile (Cont.)
Rakefile
task 'test:api' => ['env:test',
'environment'] do
require './config/environments/test'
# force rails loaded, other...
# this is only available in 1.9.3+
if Test::Unit.const_defined?(:RunCount)
Test::Unit::RunCount.run_once{
status = Test::U...
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
worker_processes 2 # assume 2 CPU cores
preload_app true
l = ::Logger.new($stderr)
l.level = case ENV['RACK_ENV']
when 'pr...
rainbows.rb
Rainbows! do
use :EventMachine, :em_client_class =>
lambda{
require 'concurrency'
client = Concurrency.
eventm...
before_fork do |_, _|
# don't hold connections on master
unless defined?(Zbatery)
l.info("Discon' PSQL and Redis...")
AR::...
after_fork do |_, _|
# hold connections on workers
unless defined?(Zbatery)
l.info("Conn' PSQL and Redis...")
Redis.curren...
EM.error_handler do |e|
puts "Err: #{e.inspect}" 
" #{e.backtrace.inspect}"
begin
::Exceptional::Catcher.handle(e)
::NewRe...
Rainbows! EventMachine Thread Client
https://gist.github.com/4451322
https://github.com/godfat/rainbows/pull/2
rainbows.rb...
class ::RainbowsEMThreadPoolClient <
Rainbows::EventMachine::Client
def app_call input
@deferred = true
EM.defer do
# Call...
# Call the application here!
begin
@env.merge!(RACK_DEFAULTS)
status, headers, body = APP.call(@env)
@deferred = nil # EM....
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
concurrency.rb
module Concurrency
Model = Thread
module_function
def eventmachine_client
if Model == Thread
RainbowsEMThre...
concurrency.rb
module Concurrency
def wrap
if Model == Thread
EM.defer do
# ...
end
elsif Model == Fiber
Fiber.new{ yield ...
concurrency.rb
def wrap
begin
yield
rescue Exception => e
EM.instance_variable_get(
:@error_handler).call(e)
ensure
AR::Ba...
•Gemfile
•Procfile
•Rakefile
Ruby Codes
•rainbows.rb
•concurrency.rb
•config.ru
config.ru
use Api::HostDispatcher, Api::Server
map '/api' do
run Api::Server
end
map '/' do
run PicCollage::Application
end
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
Api::Server = Rack::Builder.new do
# Embrace Rack is the Jellyfish way
end
server.rb
use Rack::CommonLogger, debug_logger
use Rack::Deflater
use Rack::Static,
:urls => ['/images'], :root =>
"#{File.dirname(_...
use Api::MiddleError
use Api::MiddleDevice
use Api::MiddleAuth
use Api::MiddleCache
use Rack::Cache, {} # [snipped]
run Ap...
class Api::ServerCore
include Jellyfish
handle_exceptions false
def controller; Controller; end
class Controller < Api::Co...
class Api::ServerCore
post '/collages/create_and_share' do
render_collage create_collage{ |c|
transfer "/collages/#{c.id}/...
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
class Api::MiddleError
include Jellyfish
handle_exceptions true
def controller; Api::Controller; end
handle Api::Error do ...
error.rb (Cont.)
class Api::MiddleError
handle Jellyfish::NotFound do
render(error_custom(
Api::NotFound.new(
"Path not fo...
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
module Eagerload
module_function
def eagerload
if ENV['RAILS_GROUPS'] == 'assets'
$stderr.puts "Don't eagerload " 
"for as...
def eagerload
# ...
require 'rest-more'
RC.eagerload
require 'aws-sdk'
RC.eagerload(AWS::Core)
RC.eagerload(AWS::S3)
eager...
def eagerload
# ...
%w[app/models/**/*
app/controllers/**/*
app/mailers/**/*].
each{ |pattern|
require_all(Dir[pattern], 2...
def eagerload
# ...
require 'api/requirements'
require_all(Dir['lib/**/*'], 1)
$stderr.puts "Eagerly loaded."
true
end
eag...
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
ActiveSupport.on_load(:active_record) do
require 'newrelic_rpm'
NewRelic::Agent.after_fork(
:force_reconnect => true) if
d...
require 'shell'
Shell.module_eval do
include NewRelic::Agent::
MethodTracer
add_method_tracer :system_multi
end
newrelic.r...
Cocaine::CommandLine.module_eval do
include NewRelic::Agent::
MethodTracer
add_method_tracer :run
end
newrelic.rb (Cont.)
Paperclip::Attachment.module_eval do
include NewRelic::Agent::
MethodTracer
add_method_tracer :url
end
newrelic.rb (Cont.)
require 'redis'
Redis::Client.module_eval do
include NewRelic::Agent::
MethodTracer
add_method_tracer :process
end
newreli...
require 'rest-core/engine/future/future'
RC::Future.module_eval do
include NewRelic::Agent::MethodTracer
alias_method :yie...
(Cont.)newrelic.rb
def yield
# ...
metrics = ["External/#{path}/" 
"#{self.class.name}/#{method}",
"External/#{path}/all",...
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
shell.rb
require 'waiter'
require 'concurrency'
module Shell
module_function
# this is for cocaine
def call command, env={...
shell.rb
# e.g. Shell.system_multi(%w[identify
# image.png], %w[identify me.png])
def system_multi *cmds
if EM.reactor_run...
shell.rb
def system_multi *cmds
# ...
w = Waiter.new
r = {}
cmds.each.with_index do |cmd, idx|
system_async(*cmd) do |outp...
shell.rb
def system_async *args
tuple = system_spawn(args)
Thread.new do
begin
result = system_wait(*tuple)
rescue Excepti...
shell.rb
def system_async *args
# ...
if Concurrency::Model == Fiber
EM.next_tick{ yield(result) }
else
yield(result)
end
...
shell.rb
def system_spawn args
err = if %w[test production].
include?(Rails.env)
IO::NULL
else
Rails.logger.debug(
"Shell:...
shell.rb
def system_spawn args
# ...
rd, wr = IO.pipe
pid = Process.spawn(*args,
:out => wr, :err => err)
[pid, rd, wr]
en...
shell.rb
def system_wait pid, rd, wr
wr.close
result = rd.read
Process.waitpid(pid)
result
end
(Cont.)
•server.rb
•error.rb
•eagerload.rb
Ruby Codes
(Cont.)
•newrelic.rb
•shell.rb
•waiter.rb
waiter.rb
require 'thread'
require 'fiber'
class Waiter
def self.fiber_around?
RC::RootFiber != Fiber.current &&
Thread.ma...
waiter.rb
class Waiter
def initialize
if Waiter.fiber_around?
@fiber = Fiber.current
else
@condv = ConditionVariable.new
@...
waiter.rb
class Waiter
def resume value
if @fiber
@fiber.resume(value)
else
@value = value
@condv.broadcast
end
end
end
(C...
waiter.rb
class Waiter
def yield
if @fiber
Fiber.yield
else
@mutex.synchronize{
@condv.wait(@mutex) }
@value
end
end
end
(...
•image_util.rb
•run_later.rb
•/search
•/create_and_share
Ruby Codes
(Cont.) (Cont.)
collage_web.rb
class CollageWeb < Collage
extend ImageUtil
def self.create_from urls_or_url,
fallback_urls=[]
create(:imag...
image_util.rb
require 'tempfile'
require 'shell'
module ImageUtil
module_function
def urls2files urls_or_url,
fallback_url...
image_util.rb
module ImageUtil
module_function
def urls2images urls_or_url,
fallback_urls=[]
urls = [urls_or_url].flatten
...
image_util.rb
def urls2images ...
urls.zip([fallback_urls].flatten).
each.with_index do |(url, fb), idx|
ImageFile.from_ur...
image_util.rb
module ImageUtil
module_function
def merge_images images
# ...
end
ImageStruct =
Struct.new(:width, :height,...
image_util.rb
module ImageUtil
module_function
def wget?
@wget ||= !`which wget`.strip.empty?
end
end
(Cont
image_util.rb
module ImageUtil
module_function
def wget_command path, url
if wget?
['wget', '-O', path,
'--no-check-certif...
image_util.rb
def identify_command path
['identify', '-format', '%wx%h',
path]
end
def identify_result output
if m = outpu...
•image_util.rb
•run_later.rb
•/search
•/create_and_share
Ruby Codes
(Cont.) (Cont.)
run_later.rb
require 'concurrency'
module RunLater
module_function
def with
if EM.reactor_running?
EM.next_tick{
Concurren...
•image_util.rb
•run_later.rb
•/search
•/create_and_share
Ruby Codes
(Cont.) (Cont.)
/search
get '/search' do
params['q'].blank? &&
missing_arguments('q') # this raises
images, total = if limit == 0
[[], 0]
...
/search
case params['engine']
when 'pinterest'; sch_pinterest
when 'pinterest_api'; sch_pinterest_api
when 'bing' ; sch_bi...
/search
render('search' =>
with_paging('total' => total,
'limit' => limit,
'offset' => offset,
'q' => params[:q],
'engine'...
/search
def search_from_google offset, limit
google = RC::GoogleImage.new(
referer: request.url,
userip: request.ip)
searc...
•image_util.rb
•run_later.rb
•/search
•/create_and_share
Ruby Codes
(Cont.) (Cont.)
/create_and_share
post '/collages' do
render_collage create_collage
end
post '/collages/create_and_share' do
render_collag...
/create_and_share
def transfer path
status, headers, body =
jellyfish.call(
env.merge('PATH_INFO' => path))
self.status st...
/create_and_share
def create_collage &upload_callback
para = params
image = (para['image']||{})[:tempfile]
plist = case pa...
/create_and_share
if image
# ...
else
missing_arguments('image')
end
(Co
/create_and_share
c_para = params.merge('plist' => plist)
c_para.delete('image')
if rc_tumblr &&
for_auths_and_clients.emp...
/create_and_share
c = Collage.create_with_params(c_para)
# make paperclip happy
image.extend(OriginalFilename)
image.origi...
/create_and_share
def save_image img, &block
save_dimension img
process_thumbnails img, &block
self
end
(Co
(Co/create_and_share
def process_thumbnails img, &blk
if blk
RunLater.with do
process_thumbnails_inplace img
if blk.arity ...
•crazy inspect
Miscellaneous Extras
•crazy inspect
•who cares backtrace for 404
Miscellaneous Extras
•crazy inspect
•who cares backtrace for 404
•assets pipeline and
database connection
Miscellaneous Extras
•crazy inspect
•who cares backtrace for 404
•assets pipeline and
database connection
•fiber stack overflow
Miscellaneous Ext...
•crazy inspect
•who cares backtrace for 404
•assets pipeline and
database connection
•fiber stack overflow
•fiber database co...
•crazy inspect
•who cares backtrace for 404
•assets pipeline and
database connection
•fiber stack overflow
•fiber database co...
Q?
Upcoming SlideShare
Loading in...5
×

The Architecture of PicCollage Server

219

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
219
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "The Architecture of PicCollage Server"

  1. 1. The Architecture of PicCollage Server http://godfat.org/slide/ 2013-01-08-PicCollage.pdf
  2. 2. •PicCollage, Software and Services Table of Contents
  3. 3. •PicCollage, Software and Services •Ruby Codes Table of Contents
  4. 4. •PicCollage, Software and Services •Ruby Codes •Miscellaneous Extras Table of Contents
  5. 5. PicCollage Software Services & ,
  6. 6. •Gemfile •Procfile •Rakefile •rainbows.rb •concurrency.rb •config.ru Ruby Codes
  7. 7. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  8. 8. •image_util.rb •run_later.rb •/search •/create_and_share Ruby Codes (Cont.) (Cont.)
  9. 9. •crazy inspect •who cares backtrace for 404 •assets pipeline and database connection •fiber stack overflow •fiber database connection pool •thread-safe kaminari patch Miscellaneous Extras
  10. 10. PicCollage Software Services & ,
  11. 11. PicCollage •Make collages of your photos •Import photos from places •Touch gestures to rotate, resize, delete •Double-tap to edit, clip, etc •Clip photos by outlining the area •Lots of backgrounds and stickers
  12. 12. PicCollage Software Services & ,
  13. 13. Software •Ruby 1.8.7 -> 1.9.2 -> 1.9.3
  14. 14. Software •Ruby 1.8.7 -> 1.9.2 -> 1.9.3 •Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 -> 3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5 -> 3.2.6 -> 3.2.7 -> 3.2.8 -> 3.2.9 -> 3.2.10
  15. 15. Software •Ruby 1.8.7 -> 1.9.2 -> 1.9.3 •Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 -> 3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5 -> 3.2.6 -> 3.2.7 -> 3.2.8 -> 3.2.9 -> 3.2.10 •Sinatra -> Jellyfish
  16. 16. Software •Ruby 1.8.7 -> 1.9.2 -> 1.9.3 •Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 -> 3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5 -> 3.2.6 -> 3.2.7 -> 3.2.8 -> 3.2.9 -> 3.2.10 •Sinatra -> Jellyfish •Thin -> Zbatery
  17. 17. Software •Ruby 1.8.7 -> 1.9.2 -> 1.9.3 •Rails 3.1.0.rc4 -> 3.1.0.rc5 -> 3.1.0 -> 3.1.3 -> 3.2.1 -> 3.2.2 -> 3.2.3 -> 3.2.5 -> 3.2.6 -> 3.2.7 -> 3.2.8 -> 3.2.9 -> 3.2.10 •Sinatra -> Jellyfish •Thin -> Zbatery •Resque -> EM.next_tick or Thread.new -> Sidekiq (?)
  18. 18. PicCollage Software Services & ,
  19. 19. Services •Github
  20. 20. Services •Github •Heroku Bamboo -> Cedar
  21. 21. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2
  22. 22. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab
  23. 23. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab •RedisToGo -> OpenRedis
  24. 24. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab •RedisToGo -> OpenRedis •Memcache (?) -> Memcachier
  25. 25. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab •RedisToGo -> OpenRedis •Memcache (?) -> Memcachier •NewRelic
  26. 26. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab •RedisToGo -> OpenRedis •Memcache (?) -> Memcachier •NewRelic •Exceptional
  27. 27. Services •Github •Heroku Bamboo -> Cedar •Heroku PostgreSQL 9.0 -> 9.2 •MongoHQ -> MongoLab •RedisToGo -> OpenRedis •Memcache (?) -> Memcachier •NewRelic •Exceptional •SendGrid -> Mailgun
  28. 28. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  29. 29. Gemfile ruby '1.9.3' source 'http://rubygems.org' gem 'rails', '3.2.9' gem 'jellyfish' gem 'pg', :platform => :ruby gem 'redis-objects' gem 'dalli'
  30. 30. # service gem 'newrelic_rpm', :require => false gem 'exceptional' gem 'apns', :github => 'jpoz/APNS' # ^^^^ needs feedback patch # server platform :ruby do gem 'zbatery', :require => false gem 'em-http-request' gem 'yajl-ruby' end Gemfile (Cont.)
  31. 31. # upload gem 'paperclip' # needs various patches gem 'aws-sdk' # plist gem 'nokogiri' # pagination gem 'kaminari' # needs thread-safe patch # javascript gem 'jquery-rails' Gemfile (Cont.)
  32. 32. # make `rails server` work for zbatery gem 'rack-handlers' # respect rack logger gem 'rack-rails-logger' # redirect from www to root domain gem 'rack-rewrite' # cache for API gem 'rack-cache' Gemfile (Cont.)
  33. 33. # api client gem 'rest-core' gem 'rest-more' # bluebase gem 'bb_more' , :path => 'bluebase' gem 'bb_locales', :path => 'bluebase' gem 'bb_auth' , :path => 'bluebase' gem 'bb_redis' , :path => 'bluebase' Gemfile (Cont.)
  34. 34. group :assets do gem 'sass-rails' gem 'coffee-rails' gem 'uglifier' end group :cedar do gem 'rib' gem 'bond' gem 'readline_buffer', :platform => :ruby end Gemfile (Cont.)
  35. 35. group :test do gem 'minitest' gem 'rr' gem 'webmock' gem 'kramdown' end group :test do gem 'capybara' gem 'rspec' gem 'rspec-rails' end Gemfile (Cont.)
  36. 36. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  37. 37. Procfile web: ruby -r bundler/setup -S zbatery -c config/rainbows.rb -p $PORT -E $RACK_ENV
  38. 38. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  39. 39. task 'test:all' => ['test', 'test:bluebase'] task 'env:test' do Rails.env = 'test' end Rakefile
  40. 40. task 'test:bluebase' => ['env:test'] do # remove bundler shit gemfile = ENV.delete('BUNDLE_GEMFILE') sh './bin/bluebase-test' ENV['BUNDLE_GEMFILE'] = gemfile end Rakefile (Cont.)
  41. 41. task :test => ['test:api'] Rakefile (Cont.)
  42. 42. Rakefile task 'test:api' => ['env:test', 'environment'] do require './config/environments/test' # force rails loaded, otherwise it's # causing too much loading troubles, # by rails' autoloading mechanism Dir['./test/api/*.rb'].each do |test| require test end # ... end (Cont.)
  43. 43. # this is only available in 1.9.3+ if Test::Unit.const_defined?(:RunCount) Test::Unit::RunCount.run_once{ status = Test::Unit::Runner.new.run exit(status) if status != 0 } end Rakefile (Cont.)
  44. 44. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  45. 45. worker_processes 2 # assume 2 CPU cores preload_app true l = ::Logger.new($stderr) l.level = case ENV['RACK_ENV'] when 'production' ::Logger::WARN else ::Logger::DEBUG end logger l rainbows.rb
  46. 46. rainbows.rb Rainbows! do use :EventMachine, :em_client_class => lambda{ require 'concurrency' client = Concurrency. eventmachine_client l.info("Using #{client}") client } worker_connections 64 end (Cont.
  47. 47. before_fork do |_, _| # don't hold connections on master unless defined?(Zbatery) l.info("Discon' PSQL and Redis...") AR::Base.connection.disconnect! Redis.current.quit end end rainbows.rb (Cont.
  48. 48. after_fork do |_, _| # hold connections on workers unless defined?(Zbatery) l.info("Conn' PSQL and Redis...") Redis.current.connect AR::Base.connection. establish_connection end end rainbows.rb (Cont.
  49. 49. EM.error_handler do |e| puts "Err: #{e.inspect}" " #{e.backtrace.inspect}" begin ::Exceptional::Catcher.handle(e) ::NewRelic::Agent.instance. error_collector.notice_error(e) rescue Exception => e puts "Err: Exceptional/NewRelic:" " #{e.inspect}" " #{e.backtrace.inspect}" end rainbows.rb (Cont.
  50. 50. Rainbows! EventMachine Thread Client https://gist.github.com/4451322 https://github.com/godfat/rainbows/pull/2 rainbows.rb (Cont.
  51. 51. class ::RainbowsEMThreadPoolClient < Rainbows::EventMachine::Client def app_call input @deferred = true EM.defer do # Call the application here! end end end rainbows.rb (Cont.
  52. 52. # Call the application here! begin @env.merge!(RACK_DEFAULTS) status, headers, body = APP.call(@env) @deferred = nil # EM.next_tick? ev_write_response(status, headers, body, @hp.next?) rescue Exception => e EM.instance_variable_get( :@error_handler).call(e) handle_error(e) end rainbows.rb (Cont.
  53. 53. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  54. 54. concurrency.rb module Concurrency Model = Thread module_function def eventmachine_client if Model == Thread RainbowsEMThreadPoolClient elsif Model == Fiber RainbowsEMFiberSpawnClient else Rainbows::EventMachine::Client end end
  55. 55. concurrency.rb module Concurrency def wrap if Model == Thread EM.defer do # ... end elsif Model == Fiber Fiber.new{ yield }.resume else yield end end (Con
  56. 56. concurrency.rb def wrap begin yield rescue Exception => e EM.instance_variable_get( :@error_handler).call(e) ensure AR::Base.clear_active_connections! end (Con
  57. 57. •Gemfile •Procfile •Rakefile Ruby Codes •rainbows.rb •concurrency.rb •config.ru
  58. 58. config.ru use Api::HostDispatcher, Api::Server map '/api' do run Api::Server end map '/' do run PicCollage::Application end
  59. 59. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  60. 60. Api::Server = Rack::Builder.new do # Embrace Rack is the Jellyfish way end server.rb
  61. 61. use Rack::CommonLogger, debug_logger use Rack::Deflater use Rack::Static, :urls => ['/images'], :root => "#{File.dirname(__FILE__)}/public" use AR::ConnectionAdapters:: ConnectionManagement server.rb (Cont.)
  62. 62. use Api::MiddleError use Api::MiddleDevice use Api::MiddleAuth use Api::MiddleCache use Rack::Cache, {} # [snipped] run Api::ServerCore.new server.rb (Cont.)
  63. 63. class Api::ServerCore include Jellyfish handle_exceptions false def controller; Controller; end class Controller < Api::Controller include Api::ServerMethods include Jellyfish::NewRelic end end server.rb (Cont.)
  64. 64. class Api::ServerCore post '/collages/create_and_share' do render_collage create_collage{ |c| transfer "/collages/#{c.id}/share" } end end server.rb (Cont.)
  65. 65. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  66. 66. class Api::MiddleError include Jellyfish handle_exceptions true def controller; Api::Controller; end handle Api::Error do |error| render(error_custom(error)) end end error.rb
  67. 67. error.rb (Cont.) class Api::MiddleError handle Jellyfish::NotFound do render(error_custom( Api::NotFound.new( "Path not found."))) end handle Exception do |error| render(error_500(error)) end
  68. 68. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  69. 69. module Eagerload module_function def eagerload if ENV['RAILS_GROUPS'] == 'assets' $stderr.puts "Don't eagerload " "for assets precompilation." return end # ... eagerload.rb
  70. 70. def eagerload # ... require 'rest-more' RC.eagerload require 'aws-sdk' RC.eagerload(AWS::Core) RC.eagerload(AWS::S3) eagerload.rb (Cont
  71. 71. def eagerload # ... %w[app/models/**/* app/controllers/**/* app/mailers/**/*]. each{ |pattern| require_all(Dir[pattern], 2) } eagerload.rb (Cont
  72. 72. def eagerload # ... require 'api/requirements' require_all(Dir['lib/**/*'], 1) $stderr.puts "Eagerly loaded." true end eagerload.rb (Cont
  73. 73. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  74. 74. ActiveSupport.on_load(:active_record) do require 'newrelic_rpm' NewRelic::Agent.after_fork( :force_reconnect => true) if defined?(Unicorn) # ... newrelic.rb
  75. 75. require 'shell' Shell.module_eval do include NewRelic::Agent:: MethodTracer add_method_tracer :system_multi end newrelic.rb (Cont.)
  76. 76. Cocaine::CommandLine.module_eval do include NewRelic::Agent:: MethodTracer add_method_tracer :run end newrelic.rb (Cont.)
  77. 77. Paperclip::Attachment.module_eval do include NewRelic::Agent:: MethodTracer add_method_tracer :url end newrelic.rb (Cont.)
  78. 78. require 'redis' Redis::Client.module_eval do include NewRelic::Agent:: MethodTracer add_method_tracer :process end newrelic.rb (Cont.)
  79. 79. require 'rest-core/engine/future/future' RC::Future.module_eval do include NewRelic::Agent::MethodTracer alias_method :yield_original, :yield def yield method = env[RC::REQUEST_METHOD] path = env[RC::REQUEST_PATH] # ... newrelic.rb (Cont.)
  80. 80. (Cont.)newrelic.rb def yield # ... metrics = ["External/#{path}/" "#{self.class.name}/#{method}", "External/#{path}/all", "External/all", "External/allWeb"] self.class. trace_execution_scoped metrics do yield_original end end
  81. 81. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  82. 82. shell.rb require 'waiter' require 'concurrency' module Shell module_function # this is for cocaine def call command, env={} Shell.system(*command) end # e.g. Shell.system('identify', 'img') def system *args system_multi(args).first end
  83. 83. shell.rb # e.g. Shell.system_multi(%w[identify # image.png], %w[identify me.png]) def system_multi *cmds if EM.reactor_running? # ... else cmds.map{ |cmd| system_blocking(cmd) } end end (Cont.)
  84. 84. shell.rb def system_multi *cmds # ... w = Waiter.new r = {} cmds.each.with_index do |cmd, idx| system_async(*cmd) do |output| r[idx] = output w.resume(r.sort.map(&:last)) if r.size == cmds.size end end w.yield (Cont.)
  85. 85. shell.rb def system_async *args tuple = system_spawn(args) Thread.new do begin result = system_wait(*tuple) rescue Exception => e EM.instance_variable_get( :@error_handler).call(e) ensure # ... ennnd (Cont.)
  86. 86. shell.rb def system_async *args # ... if Concurrency::Model == Fiber EM.next_tick{ yield(result) } else yield(result) end (Cont.)
  87. 87. shell.rb def system_spawn args err = if %w[test production]. include?(Rails.env) IO::NULL else Rails.logger.debug( "Shell: #{args.inspect}") $stderr end # ... (Cont.)
  88. 88. shell.rb def system_spawn args # ... rd, wr = IO.pipe pid = Process.spawn(*args, :out => wr, :err => err) [pid, rd, wr] end (Cont.)
  89. 89. shell.rb def system_wait pid, rd, wr wr.close result = rd.read Process.waitpid(pid) result end (Cont.)
  90. 90. •server.rb •error.rb •eagerload.rb Ruby Codes (Cont.) •newrelic.rb •shell.rb •waiter.rb
  91. 91. waiter.rb require 'thread' require 'fiber' class Waiter def self.fiber_around? RC::RootFiber != Fiber.current && Thread.main == Thread.current end end
  92. 92. waiter.rb class Waiter def initialize if Waiter.fiber_around? @fiber = Fiber.current else @condv = ConditionVariable.new @mutex = Mutex.new end end end (Cont.)
  93. 93. waiter.rb class Waiter def resume value if @fiber @fiber.resume(value) else @value = value @condv.broadcast end end end (Cont.)
  94. 94. waiter.rb class Waiter def yield if @fiber Fiber.yield else @mutex.synchronize{ @condv.wait(@mutex) } @value end end end (Cont.)
  95. 95. •image_util.rb •run_later.rb •/search •/create_and_share Ruby Codes (Cont.) (Cont.)
  96. 96. collage_web.rb class CollageWeb < Collage extend ImageUtil def self.create_from urls_or_url, fallback_urls=[] create(:image => merge_images( urls2images(urls_or_url, fallback_urls))) end end
  97. 97. image_util.rb require 'tempfile' require 'shell' module ImageUtil module_function def urls2files urls_or_url, fallback_urls=[] urls2images(urls_or_url, fallback_urls).map(&:file) end end
  98. 98. image_util.rb module ImageUtil module_function def urls2images urls_or_url, fallback_urls=[] urls = [urls_or_url].flatten w = Waiter.new results = {} # ... w.yield.sort.map(&:last).compact end end (Cont
  99. 99. image_util.rb def urls2images ... urls.zip([fallback_urls].flatten). each.with_index do |(url, fb), idx| ImageFile.from_url(url, fb) do |img| results[idx] = img w.resume(results) if results.size == urls.size end end w.yield.sort.map(&:last).compact (Cont
  100. 100. image_util.rb module ImageUtil module_function def merge_images images # ... end ImageStruct = Struct.new(:width, :height, :file) class ImageFile < ImageStruct include ImageUtil # ... end (Cont
  101. 101. image_util.rb module ImageUtil module_function def wget? @wget ||= !`which wget`.strip.empty? end end (Cont
  102. 102. image_util.rb module ImageUtil module_function def wget_command path, url if wget? ['wget', '-O', path, '--no-check-certificate', url] else ['curl', '-o', path, '--insecure' , url] end end end (Cont
  103. 103. image_util.rb def identify_command path ['identify', '-format', '%wx%h', path] end def identify_result output if m = output.match(/(d+)x(d+)/) [m[1], m[2]].map(&:to_i) else nil end end (Cont
  104. 104. •image_util.rb •run_later.rb •/search •/create_and_share Ruby Codes (Cont.) (Cont.)
  105. 105. run_later.rb require 'concurrency' module RunLater module_function def with if EM.reactor_running? EM.next_tick{ Concurrency.wrap{ yield } } else yield end end end
  106. 106. •image_util.rb •run_later.rb •/search •/create_and_share Ruby Codes (Cont.) (Cont.)
  107. 107. /search get '/search' do params['q'].blank? && missing_arguments('q') # this raises images, total = if limit == 0 [[], 0] else # ... end # ... end
  108. 108. /search case params['engine'] when 'pinterest'; sch_pinterest when 'pinterest_api'; sch_pinterest_api when 'bing' ; sch_bing offset, limit when 'azure' ; sch_azure offset, limit when 'google'; sch_google offset, limit when 'flickr'; sch_flickr offset, limit else ; sch_google offset, limit # default google end (Cont.)
  109. 109. /search render('search' => with_paging('total' => total, 'limit' => limit, 'offset' => offset, 'q' => params[:q], 'engine' => engine, 'data' => images)) (Cont.)
  110. 110. /search def search_from_google offset, limit google = RC::GoogleImage.new( referer: request.url, userip: request.ip) search_image_and_flatten( google, offset, limit) do |q, n, i| google.search_image( q, rsz: n, start: i) end end (Cont.)
  111. 111. •image_util.rb •run_later.rb •/search •/create_and_share Ruby Codes (Cont.) (Cont.)
  112. 112. /create_and_share post '/collages' do render_collage create_collage end post '/collages/create_and_share' do render_collage create_collage{ |c| transfer "/collages/#{c.id}/share" } end
  113. 113. /create_and_share def transfer path status, headers, body = jellyfish.call( env.merge('PATH_INFO' => path)) self.status status self.headers headers self.body body end (Co
  114. 114. /create_and_share def create_collage &upload_callback para = params image = (para['image']||{})[:tempfile] plist = case para['plist'] when Hash; para['plist'][:tempfile] else ; para['plist'] end # ... (Co
  115. 115. /create_and_share if image # ... else missing_arguments('image') end (Co
  116. 116. /create_and_share c_para = params.merge('plist' => plist) c_para.delete('image') if rc_tumblr && for_auths_and_clients.empty? # no user for tumblr else # check user for any other cases c_para['user'] = current_user_with_check end (Co
  117. 117. /create_and_share c = Collage.create_with_params(c_para) # make paperclip happy image.extend(OriginalFilename) image.original_filename = params['image'][:filename] c.save_image(image, &upload_callback) (Co
  118. 118. /create_and_share def save_image img, &block save_dimension img process_thumbnails img, &block self end (Co
  119. 119. (Co/create_and_share def process_thumbnails img, &blk if blk RunLater.with do process_thumbnails_inplace img if blk.arity == 1; blk.call(self) else ; blk.call; end end else process_thumbnails_inplace img end end
  120. 120. •crazy inspect Miscellaneous Extras
  121. 121. •crazy inspect •who cares backtrace for 404 Miscellaneous Extras
  122. 122. •crazy inspect •who cares backtrace for 404 •assets pipeline and database connection Miscellaneous Extras
  123. 123. •crazy inspect •who cares backtrace for 404 •assets pipeline and database connection •fiber stack overflow Miscellaneous Extras
  124. 124. •crazy inspect •who cares backtrace for 404 •assets pipeline and database connection •fiber stack overflow •fiber database connection pool Miscellaneous Extras
  125. 125. •crazy inspect •who cares backtrace for 404 •assets pipeline and database connection •fiber stack overflow •fiber database connection pool •thread-safe kaminari patch Miscellaneous Extras
  126. 126. Q?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×