Service-Oriented Design and Implement with Rails3

Service-OrientedDesign and Implement with Rails 3 ihower @ Ruby Tuesday 2010/12/15

About Me• a.k.a. ihower • http://ihower.tw • http://twitter.com/ihower• Rails Developer since 2006• The Organizer of Ruby Taiwan Community • http://ruby.tw • http://rubyconf.tw

Agenda• What’s SOA• Why SOA• Considerations• The tool set overview• Service side implement• Client side implement• Library packaging• Caching

What’s SOA Service oriented architectures• “monolithic” approach is not enough• SOA is a way to design complex applications by splitting out major components into individual services and communicating via APIs.• a service is a vertical slice of functionality: database, application code and caching layer

a monolithic web app example request Load Balancer WebApps Database

a SOA example request Load request Balancer WebApp WebAppsfor Administration for User Services A Services B Database Database

Why SOA? Isolation• Shared Resources• Encapsulation• Scalability• Interoperability• Reuse• Testability• Reduce Local Complexity

Shared Resources• Different front-web website use the same resource.• SOA help you avoiding duplication databases and code.• Why not only shared database? • code is not DRY WebApp for Administration WebApps for User • caching will be problematic Database

Encapsulation• you can change underly implementation in services without affect other parts of system • upgrade library • upgrade to Ruby 1.9 • upgrade to Rails 3• you can provide API versioning

Scalability1: Partitioned Data Provides• Database is the ﬁrst bottleneck, a single DB server can not scale. SOA help you reduce database load• Anti-pattern: only split the database • model relationship is broken WebApps • referential integrity • increase code complexity Database A Database B• Myth: database replication can not help you speed and consistency

Scalability 2: Caching• SOA help you design caching system easier • Cache data at the right place and expire at the right times • Cache logical model, not physical • You do not need cache view everywhere

Scalability 3: Efﬁcient• Different components have different task loading, SOA can scale by service. WebApps Load Balancer Load Balancer Services A Services A Services B Services B Services B Services B

Security• Different services can be inside different ﬁrewall • You can only open public web and services, others are inside ﬁrewall.

Interoperability• HTTP is the most common interface, SOA help you integrate them: • Multiple languages • Internal system e.g. Full-text searching engine • Legacy database, system • External vendors

Reuse• Reuse across multiple applications• Reuse for public APIs• Example: Amazon Web Services (AWS)

Testability• Isolate problem• Mocking API calls • Reduce the time to run test suite

Reduce Local Complexity• Team modularity along the same module splits as your software• Understandability: The amount of code is minimized to a quantity understandable by a small team• Source code control

Design considerations• Partition into Separate Services• API Design• Which Protocol

How to partition into Separate Services• Partitioning on Logical Function• Partitioning on Read/Write Frequencies• Partitioning on Minimizing Joins• Partitioning on Iteration Speed

on Iteration Speed• Which parts of the app have clear deﬁned requirements and design?• Identify the parts of the application which are unlikely to change.• For example: The ﬁrst version data storage is using MySQL, but may change to NoSQL in the future without affecting front-app.

on Logical Function• Higher-level feature services • articles, photos, bookmarks...etc• Low-level infrastructure services • a shared key-value store, queue system

On Read/Write Frequencies• Ideally, a service will have to work only with a single data store• High read and low write: the service should optimize a caching strategy.• High write and low read: don’t bother with caching

On Join Frequency• Minimize cross-service joins.• But almost all data in an app is joined to something else. • How often particular joins occur? by read/ write frequency and logical separation. • Replicate data across services (For example: a activity stream by using messaging)

API Design Guideline• Send Everything you need • Unlike OOP has lots of ﬁnely grained method calls• Parallel HTTP requests • for multiple service requests• Send as Little as Possible • Avoid expensive XML

Versioning• Be able run multiple versions in parallel: Clients have time to upgrade rather than having to upgrade both client and server in locks step.• Ideally, you won’t have to run multiple versions for very long• Two solutions: • Including a Version in URIs • Using Accept Headers for Versioning (disadvantage: HTTP caching)

Physical Models & Logical Models• Physical models are mapped to database tables through ORM. (It’s 3NF)• Logical models are mapped to your business problem. (External API use it)• Logical models are mapped to physical models by you.

Logical Models• Not relational or normalized• Maintainability • can change with no change to data store • can stay the same while the data store changes• Better ﬁt for REST interfaces• Better caching

Which Protocol?• SOAP• XML-RPC• REST

RESTful Web services• Rails way• Easy to use and implement• REST is about resources • URI • HTTP Verbs: GET/PUT/POST/DELETE • Representations: HTML, XML, JSON...etc

The tool set• Web framework• XML Parser• JSON Parser• HTTP Client• Model library

Web framework• Ruby on Rails, but we don’t need afull features. (Rails3 can be customized because it’s lot more modular. We will discuss it later)• Sinatra: a lightweight framework• Rack: a minimal Ruby webserver interface library

ActiveResource• Mapping RESTful resources as models in a Rails application.• Use XML by default• But not useful in practice, why?

XML parser• http://nokogiri.org/• Nokogiri ( ) is an HTML, XML, SAX, and Reader parser. Among Nokogiri’s many features is the ability to search documents via XPath or CSS3 selectors.

JSON Parser• http://github.com/brianmario/yajl-ruby/• An extremely efﬁcient streaming JSON parsing and encoding library. Ruby C bindings to Yajl

HTTP Client• How to run requests in parallel? • Asynchronous I/O • Reactor pattern (EventMachine) • Multi-threading • JRuby

Typhoeus http://github.com/pauldix/typhoeus/• A Ruby library with native C extensions to libcurl and libcurl-multi.• Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic

Typhoeus: Quick exampleresponse = Typhoeus::Request.get("http://www.pauldix.net")response = Typhoeus::Request.head("http://www.pauldix.net")response = Typhoeus::Request.put("http://localhost:3000/posts/1", :body => "whoo, a body")response = Typhoeus::Request.post("http://localhost:3000/posts", :params => {:title => "test post", :content => "this is my test"})response = Typhoeus::Request.delete("http://localhost:3000/posts/1")

Hydra handles requests but not guaranteed to run in any particular order HYDRA = Typhoeus::HYDRA.new a = nil request1 = Typhoeus::Request.new("http://example1") request1.on_complete do |response| a = response.body end HYDRA.queue(request1) b = nil request2 = Typhoeus::Request.new("http://example1") request2.on_complete do |response| b = response.body end HYDRA.queue(request2) HYDRA.run # a, b are set from here

a asynchronous method def foo_asynchronously request = Typhoeus::Request.new( "http://example" ) request.on_complete do |response| result_value = ActiveSupport::JSON.decode(response.body) # do something yield result_value end self.hydra.queue(request) end

Usageresult = nilfoo_asynchronously do |i| result = iendfoo_asynchronously do |i| # Do something for iendHYDRA.run# Now you can use result1 and result2

a synchronous method def foo result = nil foo_asynchronously { |i| result = i } self.hydra.run result end

Physical Models mapping to database directly• ActiveRecord• DataMapper• MongoMapper, MongoId

Logical Models• ActiveModel: an interface and modules can be integrated with ActionPack helpers.• http://ihower.tw/blog/archives/4940

integrated with helper?• For example: • link_to post_path(@post) • form_for @post • @post.errors

A basic modelclass YourModel extend ActiveModel::Naming include ActiveModel::Conversion include ActiveModel::Validations def persisted? false endend

without validations class YourModel extend ActiveModel::Naming include ActiveModel::Conversion def persisted? false end def valid?() true end def errors @errors ||= ActiveModel::Errors.new(self) end end

Many useful modules• MassAssignmentSecurity• Serialization• Callback• AttributeMethods• Dirty• Observing• Translation

Serializersclass Person include ActiveModel::Serializers::JSON include ActiveModel::Serializers::Xml attr_accessor :name def attributes @attributes ||= {name => nil} endendperson = Person.newperson.serializable_hash # => {"name"=>nil}person.as_json # => {"name"=>nil}person.to_json # => "{"name":null}"person.to_xml # => "<?xml version="1.0" encoding="UTF-8"?>n<serial-person...

Mass Assignmentclass YourModel # ... def initialize(attributes = {}) if attributes.present? attributes.each { |k, v| send("#{k}=", v) if respond_to?("#{k}=") } end endendYourModel.new( :a => 1, :b => 2, :c => 3 )

MassAssignmentSecurityclass YourModel # ... include ActiveModel::MassAssignmentSecurity attr_accessible :first_name, :last_name def initialize(attributes = {}) if attributes.present? sanitize_for_mass_assignment(attributes).each { |k, v| send("#{k}=", v) ifrespond_to?("#{k}=") } end endend

Scenario we want to implement• an Users web service, which provide basic CRUD functions.• an web application with the Users client library

Service implement

Customized Rails3• We don’t need some components.• We can customize ActionController• Building a fast, lightweight REST service with Rails 3 http://pivotallabs.com/users/jdean/blog/articles/1419-building-a-fast- lightweight-rest-service-with-rails-3

# config/appliction.rb%w( active_record action_controller action_mailer).each do |framework| begin require "#{framework}/railtie" rescue LoadError endend

# config/application.rb[ Rack::Sendfile, ActionDispatch::Flash, ActionDispatch::Session::CookieStore, ActionDispatch::Cookies, ActionDispatch::BestStandardsSupport, Rack::MethodOverride, ActionDispatch::ShowExceptions, ActionDispatch::Static, ActionDispatch::RemoteIp, ActionDispatch::ParamsParser, Rack::Lock, ActionDispatch::Head].each do |klass| config.middleware.delete klassend# config/environments/production.rbconfig.middleware.deleteActiveRecord::ConnectionAdapters::ConnectionManagement

# /app/controllers/application_controller.rbclass ApplicationController < ActionController::Baseclass ApplicationController < ActionController::Metal include AbstractController::Logger include Rails.application.routes.url_helpers include ActionController::UrlFor include ActionController::Rendering include ActionController::Renderers::All include ActionController::MimeResponds if Rails.env.test? include ActionController::Testing # Rails 2.x compatibility include ActionController::Compatibility endend http://ihower.tw/blog/archives/4561

APIs design best practices (1) • Routing doesnt need Rails resources mechanism , but APIs design should follow RESTful. (This is because we dont have view in service and we dont need URL helpers. So use resources mechanism is too overkill) • RESTful APIs is stateless, each APIs should be independent. So, requests which have dependency relationship should be combined into one API request. (atomic)

APIs design best practices (2) • The best format in most case is JSON. ( one disadvantage is we can’t return binary data directly. ) • Use Yajl as parser. # config/application.rb ActiveSupport::JSON.backend = "Yajl" • Dont convert data to JSON in Model, the converting process to JSON should be place in Controller.

APIs design best practices (3)• I suggest it shouldnt include_root_in_json # config/application.rb ActiveRecord::Base.include_root_in_json = false• Please notice “the key is JSON must be string”. whether you use symbol or string in Ruby, after JSON encode should all be string.• related key format should be xxx_id or xxx_ids for example: { "user_id" => 4, "product_ids" => [1,2,5] }.to_json• return user_uri ﬁeld in addition to the user_id ﬁeld if need

a return data example model.to_json and model.to_xml is easy to use, but not useful in practice.# one record{ :name => "a" }.to_json# collection{ :collection => [ { :name => "a" } , { :name =>"b" } ], :total => 123 }.to_json If you want to have pagination, you need total number.

APIs design best practices (4) • except return collection, we can also provide Multi-Gets API. though params : ids. ex. /users?ids=2,5,11,23 • client should sort ID ﬁrst, so we can design cache mechanism much easier. • another topic need to concern is the URL length of GET. So this API can also use POST.

an error message return example{ :message => "faild", :error_codes => [1,2,3], :errors => ["k1" => "v1", "k2" => "v2" ] }.to_json

APIs design best practices (5) • error_codes & errors is optional, you can deﬁne it if you need. • errors is used to put models validation error : model.errors.to_json

HTTP status code We should return suitable HTTP status code• 200 OK• 201 Created ( add success)• 202 Accepted ( receive success but not process yet, in queue now )• 400 Bad Request ( ex. Model Validation Error or wrong parameters )• 401 Unauthorized

class PeopleController < ApplicationController def index @people = Person.paginate(:per_page => params[:per_page] || 20, :page => params[:page]) render :json => { :collection => @people, :total => @people.total_entries }.to_json end def show @person = Person.find( params[:id] ) render :json => @person.to_json end def create @person = Person.new( :name => params[:name], :bio => params[:bio], :user_id => params[:user_id] ) @person.save! render :json => { :id => @person.id }.to_json, :status => 201 end def update @person = user_Person.find( params[:id] ) @person.attributes = { :name => params[:name], :bio => params[:bio], :user_id => params[:user_id] } @person.save! render :status => 200, :text => "OK" end def destroy @person = Person.find( params[:id] ) @person.destroy render :status => 200, :text => "OK" endend

Client implement

Note• No active_record, we get data from service through HTTP client (typhoeus)• Model can include some ActiveModel, modules so we can develop more efﬁciently.• This model is logical model, mapping to the data from API, not database table. Its different to services physical model ( ORM- based)

# config/appliction.rb%w( action_controller action_mailer).each do |framework| begin require "#{framework}/railtie" rescue LoadError endend

Setup a global Hydry # config/initializers/setup_hydra.rb HYDRA = Typhoeus::Hydra.new

An example you can inherited from (1) class LogicalModel extend ActiveModel::Naming include ActiveModel::Conversion include ActiveModel::Serializers::JSON include ActiveModel::Validations include ActiveModel::MassAssignmentSecurity self.include_root_in_json = false # continued... end

class LogicalModel An example you can # continued... inherited from (2) def self.attribute_keys=(keys) @attribute_keys = keys attr_accessor *keys end def self.attribute_keys @attribute_keys end class << self attr_accessor :host, :hydra end def persisted? !!self.id end def initialize(attributes={}) self.attributes = attributes end def attributes self.class.attribute_keys.inject(ActiveSupport::HashWithIndifferentAccess.new) do |result, key| result[key] = read_attribute_for_validation(key) result end end def attributes=(attrs) sanitize_for_mass_assignment(attrs).each { |k, v| send("#{k}=", v) if respond_to?("#{k}=") } end

Model usage exampleclass Person < LogicalModel self.attribute_keys = [:id, :name, :bio, :user_id, :created_at, :updated_at] self.host = PEOPLE_SERVICE_HOST self.hydra = HYDRA validates_presence_of :title, :url, :user_id # ...end

class Person < LogicalModel # ... paginate def self.people_uri "http://#{self.host}/apis/v1/people.json" end def self.async_paginate(options={}) options[:page] ||= 1 options[:per_page] ||= 20 request = Typhoeus::Request.new(people_uri, :params => options) request.on_complete do |response| if response.code >= 200 && response.code < 400 log_ok(response) result_set = self.from_json(response.body) collection = result_set[:collection].paginate( :total_entries => result_set[:total] ) collection.current_page = options[:page] yield collection else log_failed(response) end end self.hydra.queue(request) end def self.paginate(options={}) result = nil async_paginate(options) { |i| result = i } self.hydra.run result endend

will_paginate hack! • in order to use will_paginates helper, we must set current_page manually, so we hack this way:# /config/initializers/hack_will_paginate.rb# This is because our search result via HTTP API is an array and need be paginated.# So we need assign current_page, unless it will be always 1.module WillPaginate class Collection def current_page=(s) @current_page = s.to_i end endend

from_json & loggingclass LogicalModel # ... def self.from_json(json_string) parsed = ActiveSupport::JSON.decode(json_string) collection = parsed["collection"].map { |i| self.new(i) } return { :collection => collection, :total => parsed["total"].to_i } end def self.log_ok(response) Rails.logger.info("#{response.code} #{response.request.url} in #{response.time}s") end def self.log_failed(response) msg = "#{response.code} #{response.request.url} in #{response.time}s FAILED: #{ActiveSupport::JSON.decode(response.body)["message"]}" Rails.logger.warn(msg) end def log_ok(response) self.class.log_ok(response) end def log_failed(response) self.class.log_failed(response) endend

class Person < LogicalModel # ... ﬁnd def self.person_uri(id) "http://#{self.host}/apis/v1/people/#{id}.json" end def self.async_find(id) request = Typhoeus::Request.new( person_uri(id) ) request.on_complete do |response| if response.code >= 200 && response.code < 400 log_ok(response) yield self.new.from_json(response.body) else log_failed(response) end end This from_json is deﬁned by ActiveModel::Serializers::JSON self.hydra.queue(request) end def self.find(id) result = nil async_find(id) { |i| result = i } self.hydra.run result endend

class Person < LogicalModel create&update # ... def create return false unless valid? response = Typhoeus::Request.post( self.class.people_uri, :params => self.attributes ) if response.code == 201 log_ok(response) self.id = ActiveSupport::JSON.decode(response.body)["id"] return self else log_failed(response) return nil end end def update(attributes) self.attributes = attributes return false unless valid? response = Typhoeus::Request.put( self.class.person_uri(id), :params => self.attributes ) if response.code == 200 log_ok(response) return self else log_failed(response) Normally data writes do not return nil need to occur in parallel end endend Or write to a messaging system asynchronously

delete&destroyclass Person < LogicalModel # ... def self.delete(id) response = Typhoeus::Request.delete( self.person_uri(id) ) if response.code == 200 log_ok(response) return self else log_failed(response) return nil end end def destroy self.class.delete(self.id) endend

Service client Library packaging• Write users.gemspec ﬁle• gem build users.gemspec• distribution • http://rubygems.org • build your local gem server• http://ihower.tw/blog/archives/4496

About caching• Internally • Memcached• Externally: HTTP Caching • Rack-Cache,Varnish, Squid

The End

References• Books&Articles: • Service-Oriented Design with Ruby and Rails, Paul Dix (Addison Wesley) • Enterprise Rails, Dan Chak (O’Reilly) • RESTful Web Services, Richardson&Ruby (O’Reilly) • RESTful WEb Services Cookbook, Allamaraju&Amundsen (O’Reilly)• Blog: • Rails3: ActiveModel http://ihower.tw/blog/archives/4940 • Rubygems http://ihower.tw/blog/archives/4496 • Rails3: Railtie Plugins http://ihower.tw/blog/archives/4873• Slides: • Distributed Ruby and Rails http://ihower.tw/blog/archives/3589

http://ihower.tw	1704
http://twitter.com	5
http://static.slidesharecdn.com	3
http://chinaonrails.com	3
http://feeds.feedburner.com	2
http://translate.googleusercontent.com	1
http://ubuntu.ihower.idv.tw	1
http://webcache.googleusercontent.com	1
http://seekr-artemis.heroku.com	1

Service-Oriented Design and Implement with Rails3

by Wen-Tien Chang

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 1,721

Statistics

Service-Oriented Design and Implement with Rails3 Presentation Transcript