From 1 To 30<br />How To Disassemble<br />One Monster App Into An Ecosystem Of 30<br />Jonathan Palley, CTO/COO<br />Guo L...
An<br />Experience<br />
A Tale of Two Buildings<br />
 Shanghai<br />Lotus Riverside Community<br />1909 Beijing<br />Forbidden City<br />
1<br />30<br />
What is one?<br />
The entire web application/system/platform runs as one single rails application<br />(We are talking about really large sy...
Problems<br />Confused new staff<br />Hard to test/extend/scale<br />
What is 30?<br />
A ecosystem of applications<br />
Independent<br />
Linked and Seamless<br />
Basic features of each app<br />Separate database<br />Runs independently (complete story) <br />Lightweight (single devel...
Developer autonomy
Technology (im)maturity safety</li></ul>APPEAL TO DEVELOPER LAZINESS<br />
What’s the mystery of the forbidden city?<br />
Consistent UI<br />Shared CSS/JS/Styleguide<br />Common Helpers in Shared Gem<br />Safely try new things<br />
interface<br />All applications use the same base CSS/JS<br />Keep all the application the same style<br /><%=idp_include_...
interface<br />CSS Framework<br />
interface<br />Abstract Common Helpers to Gem<br />Search function for models<br />
interface<br />Common Helpers: Combo search (cont)<br />View:<br /><%= search_form_for(HistoryRecord, :interaction_id, :re...
interface<br />Common Helpers: List table<br />well formatted<br />with pagination <br />sortable<br />customizable <br />
interface<br />Common Helpers: List table (cont)<br /><%=<br />idp_table_for(@history_records,:sortable=>true,:customize =...
interface<br />Development Lifecycle<br />Implement new View code/plugin in a second application<br />Abstract intoplugin ...
interface<br />data<br />
data<br />How do applications share data?<br />	(remember: each app has its own data)<br /><ul><li>“Read Only” Database Co...
 Services
 AJAX Loaded View Segments</li></li></ul><li>Business example<br />data<br />user<br />purchase<br />course<br />learning ...
Purchase App<br />data<br />Requirement: List course packages for user to select to purchase<br />but<br />The course pack...
Solution<br />data<br />readonly db connection<br />course<br />
Code<br />data<br />Model:<br />class CoursePackage< ActiveRecord::Base<br />acts_as_readonly:course<br />end<br />View:<b...
data<br />Why doesn’t this break the rule of loose coupling?<br />
data<br />acts_as_readonly in Depth<br />def acts_as_readonly(name, options = {})<br />config = CoreService.app(name).data...
data<br />acts_as_readonly in Depth<br />def acts_as_readonly(name, options = {})<br />config = CoreService.app(name).data...
data<br />Core service<br />classCoreService < ActiveResource::Base<br />self.site = :user<br />defself.app(app_name)<br /...
data<br />Centralized configuration<br />How does Core know all the configurations?<br />
data<br />Each app posts its configuration to core when it is started<br />
data<br />config/site_config.yml<br />app: course<br />api:<br />course_list: package/courses<br />config/initializers/idp...
data<br />core_service.rb in idp_lib<br />APP_CONFIG = YAML.load(Rails.root.join(“config/site_config.yml”))<br />defreset_...
data<br />
data<br />Again, implemented in gem<br />config/environment.rb<br />config.gem ‘idp_helpers’<br />config.gem ‘idp_lib’<br />
data<br />gems<br />
data<br />Web services for “write” interactions<br />class CoursePackageService< ActiveSupport::Base<br />self.site = :cou...
example<br />data<br />Roadmap needs to be generated after learner pays.<br />
codes<br />data<br />Course: app/controllers/roadmap_services_controller.rb<br />def  create<br />Roadmap.generate(params[...
data<br />AJAX Loaded Composite View<br /><div><br /><%=ajax_load(url_of(:course, :course_list)) %><br /></div><br />Ecosy...
interface<br />data<br />user<br />
Features of User Service<br />user<br />Registration/login<br />Profile management<br />Role Based Access Control<br />
Access Control<br />user<br />Each Controller is one Node<br />* Posted to user service when app starts<br />
Access Control<br />user<br />before_filter:check_access_right<br />defcheck_access_right<br />unlessxml_request?orinner_r...
user<br />How to share?<br />
Step 1: User Auth<br />user<br />
Step 1: User Auth<br />user<br />config/initializers/idp_initializer.rb<br />ActionController::Base.session_store = :activ...
Step 2: Access Control<br />user<br />Tell core its controllers structure<br />CoreService. reset_rights<br />defself.rese...
Step 2: Access Control<br />user<br />before_filter:check_access_right<br />defcheck_access_right<br />unlessxml_request?o...
Step 2: Access Control<br />user<br />has_page_right?<br />Readonly dbconn again<br />
Step 2: Access Control<br />user<br />class IdpRoleRight < ActiveRecord::Base<br />acts_as_readonly:user, :table_name=> "r...
user<br />Again, gems!<br />config/environment.rb<br />config.gem ‘idp_helpers’<br />config.gem ‘idp_lib’<br />config.gem ...
interface<br />service<br />data<br />user<br />
service<br />Support applications<br />File<br />Mail<br />Comet service<br />
File<br />class Article < ActiveRecord::Base<br />has_files<br />end <br />Specify Class that Has Files<br />Upload File i...
service<br />Comet<br />classChatRoom < ActiveRecord::Base<br />acts_as_realtime<br />end <br /><%= realtime_for(@chat_roo...
service<br />mail<br />Mail services<br />MailService.send(“test@idapted.com”, :welcome, <br />:user => “test”)<br />
Host all in one domain<br />Load each rails app into a subdir, we use Unicorn  unicorn_rails --path /user  unicorn_rails -...
Host all in one domain<br />use Nginx as a reverse proxylocation /user {    proxy_pass http://rails_app_user; } location /...
Host all in one domain<br />All you see is a uniform URL   www.eqenglish.com/user   www.eqenglish.com/studycenter   www.eq...
Pair-deploy<br />
How to split one into many?<br />
By Story<br />		Each App is one group of similar features.<br />By Data<br />		Each App writes to the same data<br />
Example<br />User Management<br />Course package<br />Purchase<br />Learning process<br />…<br />
Iteration<br />
Be adventurous at the beginning.<br />Split one into as many as you think is sensitive<br />
Then you may find<br />Some applications interact with each other frequently.<br />Lots of messy and low efficiency code t...
Merge them into one.<br />
Measurement<br />Critical and core task of single app should not call services of others.<br />One doesn’t need to know mu...
Upcoming SlideShare
Loading in …5
×

RailsConf 2010: From 1 to 30 - How to refactor one monolithic application into an application ecosystem

8,210 views

Published on

As your business grows bigger, you just can’t stop adding new models/controllers to your original rails application – resulting in a messy, unmaintainable and difficult to deploy monolithic application. Its time to refactor. This talk will share our experience, results and best practices in splitting a single rails “application-system” into 30 independently maintainable yet interconnected applications.

After two and a half years of development (starting in pre-Rails 1.0 days!), our live-trainer English learning system now supported multiple roles (learner/trainer/trainer supervisor/sales/materials creation/support/etc) and an exhaustive list of features to support our complex business processes. We set ourselves a year-long goal of splitting this monolithic system into small cooperating applications that could be developed independently by individual developers. At the same time, we could not lose the usability cohesiveness and data-interdependence that defined the power of our system.

Through numerous iterations, many mistakes and a bit of pure-luck we developed an optimized process for the refactor and best practices for making 30 independent rails apps behave as one. The results: lower development time, greater stability and scalability and much higher developer happiness.

We’ll talk about specific code, measurements, pitfalls, plugins, process and best practices to answer questions such as:

How to know where to split single applications into many. How to measure the result.
How the applications should interact with each other. How to reduce administration and DRY configuration applications.
How to share data among applications.
How to DRY for common logic.
How to make a consistent user experience.
How to interact with non-Ruby technology; in our case Erlang, FreeSWITCH (VoIP) and Flex

Published in: Technology, Business
1 Comment
11 Likes
Statistics
Notes
No Downloads
Views
Total views
8,210
On SlideShare
0
From Embeds
0
Number of Embeds
375
Actions
Shares
0
Downloads
57
Comments
1
Likes
11
Embeds 0
No embeds

No notes for slide
  • New staff “Agile web development with rails”
  • Make it easier to do something than not.
  • Services are too slow.Svn:externals
  • Use webservice. SLOW.Facebook stuff.Thrift
  • Discovery. HUGE PAIN: Configuring each app to know about each other app.
  • Interacting with Erlang
  • Easy.BackupLike “one application”. But shared between many.Polymorphic Connection!MySQL constraints!
  • Resources – Memory consumption. One app used a lot. Others not much. Memory is cheap.
  • RailsConf 2010: From 1 to 30 - How to refactor one monolithic application into an application ecosystem

    1. From 1 To 30<br />How To Disassemble<br />One Monster App Into An Ecosystem Of 30<br />Jonathan Palley, CTO/COO<br />Guo Lei, Chief Architect<br />© 2010 Idapted, Ltd.<br />
    2. An<br />Experience<br />
    3. A Tale of Two Buildings<br />
    4. Shanghai<br />Lotus Riverside Community<br />1909 Beijing<br />Forbidden City<br />
    5. 1<br />30<br />
    6. What is one?<br />
    7. The entire web application/system/platform runs as one single rails application<br />(We are talking about really large systems. <br /> Multiple different types of clients/functions)<br />
    8. Problems<br />Confused new staff<br />Hard to test/extend/scale<br />
    9. What is 30?<br />
    10. A ecosystem of applications<br />
    11. Independent<br />
    12. Linked and Seamless<br />
    13. Basic features of each app<br />Separate database<br />Runs independently (complete story) <br />Lightweight (single developer)<br />Tight internal cohesion and loose external coupling<br />Advantages<br /><ul><li>Independent Development Cycle
    14. Developer autonomy
    15. Technology (im)maturity safety</li></ul>APPEAL TO DEVELOPER LAZINESS<br />
    16. What’s the mystery of the forbidden city?<br />
    17. Consistent UI<br />Shared CSS/JS/Styleguide<br />Common Helpers in Shared Gem<br />Safely try new things<br />
    18. interface<br />All applications use the same base CSS/JS<br />Keep all the application the same style<br /><%=idp_include_js_css%><br /># =><br /><script src ="/assets/javascripts/frame.js" type="text/javascript"></script><br /><link href="/assets/stylesheets/frame.css" media="screen" rel="stylesheet" type="text/css" /><br />
    19. interface<br />CSS Framework<br />
    20. interface<br />Abstract Common Helpers to Gem<br />Search function for models<br />
    21. interface<br />Common Helpers: Combo search (cont)<br />View:<br /><%= search_form_for(HistoryRecord, :interaction_id, :released,[:rating, {:collection=>assess_ratings}],[:mark_spot_num,{:range=>true}], [:created_at, {:ampm=>true}]) <br />%><br />Controller:<br />@history_records= HistoryRecord.combo_search(params)<br />
    22. interface<br />Common Helpers: List table<br />well formatted<br />with pagination <br />sortable<br />customizable <br />
    23. interface<br />Common Helpers: List table (cont)<br /><%=<br />idp_table_for(@history_records,:sortable=>true,:customize => "history_records") do |item, col|<br />col.add:id, link_to(item.id, admin_history_record_path(item)),:order=>:id<br />col.build:duration, :waiting_time, :review_time<br />col.add:scenario, item.scenario_title, :order => :scenario_title<br />col.add:mark_spot_num<br /> end<br />%><br />
    24. interface<br />Development Lifecycle<br />Implement new View code/plugin in a second application<br />Abstract intoplugin using existing “idp” helpers<br />Put it into main view gem<br />
    25. interface<br />data<br />
    26. data<br />How do applications share data?<br /> (remember: each app has its own data)<br /><ul><li>“Read Only” Database Connections
    27. Services
    28. AJAX Loaded View Segments</li></li></ul><li>Business example<br />data<br />user<br />purchase<br />course<br />learning process<br />
    29. Purchase App<br />data<br />Requirement: List course packages for user to select to purchase<br />but<br />The course package data is stored in the “course” application<br />
    30. Solution<br />data<br />readonly db connection<br />course<br />
    31. Code<br />data<br />Model:<br />class CoursePackage< ActiveRecord::Base<br />acts_as_readonly:course<br />end<br />View:<br /><ul><br /><%CoursePackage.all.eachdo |package| %><br /><li><%= package.title %> <%= package.price %></li><br /><%end %><br /></ul><br />
    32. data<br />Why doesn’t this break the rule of loose coupling?<br />
    33. data<br />acts_as_readonly in Depth<br />def acts_as_readonly(name, options = {})<br />config = CoreService.app(name).database<br />establish_connectionconfig[Rails.env]<br />set_table_name(self.connection.current_database + (options[:table_name]||table_name).to_s)<br />end<br />
    34. data<br />acts_as_readonly in Depth<br />def acts_as_readonly(name, options = {})<br />config = CoreService.app(name).database<br />establish_connectionconfig[Rails.env]<br />set_table_name(self.connection.current_database + (options[:table_name]||table_name).to_s)<br />end<br />
    35. data<br />Core service<br />classCoreService < ActiveResource::Base<br />self.site = :user<br />defself.app(app_name)<br />CoreService.find(app_name)<br />end<br />end<br />
    36. data<br />Centralized configuration<br />How does Core know all the configurations?<br />
    37. data<br />Each app posts its configuration to core when it is started<br />
    38. data<br />config/site_config.yml<br />app: course<br />api:<br />course_list: package/courses<br />config/initializers/idp_initializer.rb<br />CoreService.reset_config<br />
    39. data<br />core_service.rb in idp_lib<br />APP_CONFIG = YAML.load(Rails.root.join(“config/site_config.yml”))<br />defreset_config<br />self.post(:reset_config, :app => {<br /> :name => APP_CONFIG["app"], <br />:settings => APP_CONFIG,<br /> :database => YAML.load_file(<br />Rails.root.join("config/database.yml"))})<br />end<br />
    40. data<br />
    41. data<br />Again, implemented in gem<br />config/environment.rb<br />config.gem ‘idp_helpers’<br />config.gem ‘idp_lib’<br />
    42. data<br />gems<br />
    43. data<br />Web services for “write” interactions<br />class CoursePackageService< ActiveSupport::Base<br />self.site = :course<br />end<br />
    44. example<br />data<br />Roadmap needs to be generated after learner pays.<br />
    45. codes<br />data<br />Course: app/controllers/roadmap_services_controller.rb<br />def create<br />Roadmap.generate(params[:user_id], params[:course_id])<br />end<br />Purchase: app/models/roadmap_service.rb<br />class RoadmapService < ActiveSupport::Base<br />self.site = :course<br />end<br />Purchase: app/models/order.rb<br />defactivate_roadmap<br />RoadmapService.create(self.user_id, self.course_id)<br />end<br />
    46. data<br />AJAX Loaded Composite View<br /><div><br /><%=ajax_load(url_of(:course, :course_list)) %><br /></div><br />Ecosystem url_for<br />Fetched from different applications<br />
    47. interface<br />data<br />user<br />
    48. Features of User Service<br />user<br />Registration/login<br />Profile management<br />Role Based Access Control<br />
    49. Access Control<br />user<br />Each Controller is one Node<br />* Posted to user service when app starts<br />
    50. Access Control<br />user<br />before_filter:check_access_right<br />defcheck_access_right<br />unlessxml_request?orinner_request?<br />access_deniedunlesshas_page_right?(params[:controller])<br />end<br />end<br />* Design your apps so access control can be by controller!<br />
    51. user<br />How to share?<br />
    52. Step 1: User Auth<br />user<br />
    53. Step 1: User Auth<br />user<br />config/initializers/idp_initializer.rb<br />ActionController::Base.session_store = :active_record_store<br />ActiveRecord::SessionStore::Session.acts_as_remote:user,<br /> :readonly=> false<br />
    54. Step 2: Access Control<br />user<br />Tell core its controllers structure<br />CoreService. reset_rights<br />defself.reset_rights<br />data = load_controller_structure<br />self.post(:reset_rights, :data =>data)<br />end<br />
    55. Step 2: Access Control<br />user<br />before_filter:check_access_right<br />defcheck_access_right<br />unlessxml_request?orinner_request?<br />access_deniedunlesshas_page_right?(params[:controller])<br />end<br />end<br />
    56. Step 2: Access Control<br />user<br />has_page_right?<br />Readonly dbconn again<br />
    57. Step 2: Access Control<br />user<br />class IdpRoleRight < ActiveRecord::Base<br />acts_as_readonly:user, :table_name=> "role_rights"<br />end<br />def has_page_right?(page)<br />roles = current_user.roles<br />roles_of_page = IdpRoleRight.all(:conditions => ["path = ?", page]).map(&:role_id)<br />(roles - (roles - roles_of_page)).size > 0<br />end<br />
    58. user<br />Again, gems!<br />config/environment.rb<br />config.gem ‘idp_helpers’<br />config.gem ‘idp_lib’<br />config.gem ‘idp_core’<br />
    59. interface<br />service<br />data<br />user<br />
    60. service<br />Support applications<br />File<br />Mail<br />Comet service<br />
    61. File<br />class Article < ActiveRecord::Base<br />has_files<br />end <br />Specify Class that Has Files<br />Upload File in Background to FileService<br />Idp_file_form<br />Store with app_name, model_name, model_id<br />Use readonly magic to easily display<br />@article.files.first.url<br />
    62. service<br />Comet<br />classChatRoom < ActiveRecord::Base<br />acts_as_realtime<br />end <br /><%= realtime_for(@chat_room, current_user.login) %><br /><%= realtime_data(dom_id, :add, :top) %><br />@chat_room.realtime_channel.broadcast(“hi world", current_user.login) <br />
    63. service<br />mail<br />Mail services<br />MailService.send(“test@idapted.com”, :welcome, <br />:user => “test”)<br />
    64. Host all in one domain<br />Load each rails app into a subdir, we use Unicorn  unicorn_rails --path /user  unicorn_rails --path /studycenter  unicorn_rails --path /scenario<br />
    65. Host all in one domain<br />use Nginx as a reverse proxylocation /user {    proxy_pass http://rails_app_user; } location /studycenter {    proxy_pass http://rails_app_studycenter; }<br />
    66. Host all in one domain<br />All you see is a uniform URL   www.eqenglish.com/user   www.eqenglish.com/studycenter   www.eqenglish.com/scenario<br />
    67. Pair-deploy<br />
    68. How to split one into many?<br />
    69. By Story<br /> Each App is one group of similar features.<br />By Data<br /> Each App writes to the same data<br />
    70. Example<br />User Management<br />Course package<br />Purchase<br />Learning process<br />…<br />
    71. Iteration<br />
    72. Be adventurous at the beginning.<br />Split one into as many as you think is sensitive<br />
    73. Then you may find<br />Some applications interact with each other frequently.<br />Lots of messy and low efficiency code to deal with interacting. <br />
    74. Merge them into one.<br />
    75. Measurement<br />Critical and core task of single app should not call services of others.<br />One doesn’t need to know much about others’ business to do one task (or develop).<br />Independent Stories<br />
    76. Pitfalls<br />Applications need to be on the same intranet. <br />No “right place” for certain cases.<br />
    77. Results: Higher Productivity<br /><ul><li>Faster build to deploy
    78. More developer autonomy
    79. Safer
    80. Scalable
    81. Easier to “jump in”
    82. Greater Happiness</li></li></ul><li>Thank you!<br />Q&A<br />http://www.idapted.com<br />http://developer.idapted.com<br />http://t.sina.com.cn/eqenglishhst<br />jpalley@idapted.com (@jpalley)<br />guolei@idapted.com (@fiyuer)<br />© 2010 Idapted, Ltd.<br />

    ×