AP4R on RubyConf2007

RubyConf 07 ( 2 Nov. 2007 ) AP4R Kiwamu Kato & Shun’ichi Shinohara Future Architect, Inc. 1

Introduction 2

kiwamu • Majored in Civil Engineering • Have enhanced and maintained messaging middle-ware in our company 3

kiwamu • Started Ruby last year • Leader of the AP4R team • http://d.hatena.ne.jp/kiwamu/ • The ﬁrst month of marriage 4

shino • Programmer, I love coding ! • The language of the year is C or Haskell 5

shino • Made a debut to the OSS world last year • http://d.hatena.ne.jp/ita-wasa • Sunday physicist, interested in the string theory 6

Corporate proﬁle • Future Architect, Inc. • http://www.future.co.jp/en/index.html • Aiming for optimization and synergy of business and IT • Headquarters: Tokyo, Japan • Foundation: November 28, 1989 • Number of employees: 594 (as of June 2007) • Revenue: ¥14,831M (2006) 7

Agenda 1. Overview ( the most important part. ) 2. Lightweight 3. Demo 4. Robust 5. Other features 6. Conclusion 8

1. Overview 9

1-1. What is AP4R ? 1-2. Typical use cases 1-3. What are advantages of AP4R ? 1-4. Case examples 10

What is AP4R ? 11

What is AP4R ? • A messaging library by Ruby • Asynchronous Processing for Ruby • Loose coupling by messaging 12

Why did we start AP4R ? • Developed a messaging middle-ware in house • Named “RtFA” 13

RtFA • Pure Java • Proprietary protocol and API (≠ JMS) • Time-proven on various systems 14

One of illustrative cases of “RtFA” Mission critical system in transportation business http://japan.internet.com/linuxtoday/20061228/5.html 15

Project summary • Over 100 million package data per day • Parallel processing • Load distribution 16

Project summary • Approx. 100 servers • Used “RtFA” 17

http://www.pragmaticprogrammer.com/titles/fr_j2r/ http://www.oreilly.co.jp/books/9784873113203/ 18

From RtFA to AP4R • Based on experiences of Java, carefully designed to keep good concepts and added “agility” • Focused on ease of use API, conﬁguration 19

Open source • Released as OSS last year • Take advantages of the community • For “Enterprise” 20

Typical use cases of AP4R • Quicker response to users • Load distribution 21

Quicker response to users 22

If quick response is not needed, execute it later 23

Synchronously Client Server Sequential tasks 24

Synchronously Client Server A B C D E Sequential tasks 24

Better solution Client Server Messaging Server 25

Better solution Client Server Messaging Server A B 25

Asynchronously Client Server Messaging Server A B 26

Asynchronously Client Server Messaging Server A B C D E 26

Typical cases • Logging • Sending mail • Creating heavy summary data 27

Advanced case • Intersystem connection 28

Moreover, load distribution 29

Easy to scale out Client Server Messaging Server 30

Easy to scale out Client Server Messaging Servers 31

There are some solutions that also use messaging service in the ( Ruby ) world. 32

What are AP4R’s advantages? 33

Lightweight and Robust 34

Lightweight ? 35

36

Ruby’s advantages • Dynamic • Succinct • Open class • Meta programming • etc.. 37

AP4R realized • Ease of use • Simple API / conﬁguration • Seamless collaboration with Rails • All-in-one support • Not only development • But also test and operation support 38

One of the advantages is Lightweight 39

Robust ? 40

AP4R guarantees a message delivery 41

If a message delivery is NOT guaranteed,... 42

it may cause message lost or message duplication 43

In other words... 44

•No items despite payment ! •Double withdrawals ! 45

Messages should be delivered in a reliable way, 46

No matter what happens • Database trouble • Network trouble • Server down • Busy and timeout • ... etc 47

The other advantage is Robust 48

AP4R is .. 49

Lightweight Robust Simple API Recovery Test support Flexible SAF system design 51

Case examples 52

•Online game (not opened yet to the public) • Interactivemediums.com 53

Online game 54

What is it ? • From England • Massively multiplayer web-based strategy game • Late alpha now; private beta testing in January 2008 55

They use AP4R for • Mail sending • Report generation • Data analysis routines • A variety of processor-intensive processes including database access 56

Scale • Two dedicated AP4R servers • Four Rails processes per AP4R • A few hundred messages an hour • MySQL to store messages 57

Interactivemediums.com 58

http://interactivemediums.com/ 59

What is it ? • From the U.S. • Provide mobile messaging solutions for clients, as well as internal products like TextMe for Business ( http://textmeforbusiness.com ) • In production since July 2007 60

http://textmeforbusiness.com/ 61

They use AP4R for • Intersystem connection • Inbound queue • Outbound queue • Routing queue • Text processing 62

Alert service on SMS AP4R queues acceptance inbound Text processing delivery delivery outbound delivery processing Third Party routing application 63

Scale • Three AP4R processes • Four mongrel clusters and so forth • Now at least a thousand messages a day ( Several thousand messages a day soon ) • MySQL to store messages 64

What did they choose AP4R ? 65

Questionnaire results • “Easy to distribute a kind of load across a number of servers” • “Easy to scale” • “Development is active” • “Development team had real-world experience with the technology” 66

Coffee break 67

Duke :Java => https://duke.dev.java.net/ 70

cute ! / cute ? 71

Mascot matters 72

:AP4R => ... 73

What / Why is that ? > “AP4R”.sub(/4/, ”A”) > “APAR”.downcase > “apar” • Three-banded armadillo Dictionary http://www.alc.co.jp/ Photo http://www.ﬂickr.com/photos/jeffclow/29738818/ 75

His name is Maro 78

Coffee break is done :-) 79

2. Lightweight 80

2-1. Simple API 2-2. Flexible system design 82

Simple API 83

API to put messages 84

ap4r.async_to({destination}, {data} [,{options}]) There’s also a block style c.f. RDoc: http://ap4r.rubyforge.org/doc/ 85

Class AsyncShopController < ApplicationController def order # synchronous part o = Order.new(:name => params[:name]) o.save! ap4r.async_to({:action => ‘payment’}, {:order_id => o.id}) redirect_to ... end def payment # asynchronous part Payment.new(:order_id => params[:order_id]).save! ... render :text => ‘true’ end end 86

Process ﬂow 87

User Apache Rails AP4R 88

User Apache Rails AP4R order message put 89

User Apache Rails AP4R dispatch 90

User Apache Rails AP4R payment 91

Flexible system design 92

Typical design 93

Load distribution for AP4R servers 95

Separate servers for sync/async 97

User Apache Rails AP4R sync async 98

Separate servers including reverse proxy 99

User Apache Rails AP4R URL rewrite 100

User Apache Rails AP4R URL rewrite HTTP 100

User Apache Rails AP4R URL rewrite HTTP 101

Dispatcher Producer Messaging Consumer AP4R put dispatch Generally put get 102

Protocols • Default is HTTP POST • Now supported • SOAP • XML-RPC • dRuby 103

Flow management 104

Two kinds of tasks processing number time Task “busy” many short Task “heavy” few very long 105

Conﬁguration dispatchers: - targets: queue.very_busy.* threads: 10 modify_rules: url: proc {|url| url.host = \"busy.async.host\"} - targets: queue.very_heavy.* threads: 1 modify_rules: url: proc {|url| url.host = \"heavy.async.host\"} 106

busy URL rewrite heavy 107

3. Demo 109

Outline • Shopping store application • Order and payment 110

Shopping store app. Client Server A B order payment 111

Better solution AP4R Client Rails Rails × 3 112

Synch’ly order AP4R Client Rails Rails × 3 A order 113

Async’ly payment AP4R Client Rails Rails × 3 A B order payment 114

Monitor message status AP4R Client Rails Rails × 3 100 75 50 25 0 115

Outline • Use mongrel cluster • Stress by ab • Monitor queue retaining status 116

117

118

Agenda 1. Overview ( the most important part. ) 2. Lightweight 3. Demo 4. Robust 5. Other features 6. Conclusion 119

4. Robust 120

Stored And Forward 122

Ensure Atomicity of application transaction and putting messages 123

Rails app-DB AP4R msg-DB CRUD Store SAF create commit Forward message put insert /commit SAF delete ✓ 124

Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe,Bobby Woolf Guaranteed Delivery [122] 125

Details 126

Typical ﬂow to create async. messages 127

start end 128

start CRUD end 128

start CRUD commit end 128

start CRUD commit message put / commit end 128

start CRUD commit message put / commit end data 128

start CRUD commit message put / commit end data messages 128

Ensure Atomicity of application transaction and putting messages 129

On updating database • If error occurs, • Any following messages must NOT be queued 130

On updating database • If everything succeeded, • All following messages must be queued 131

☺ Relief 132

Ap4r::Client#transaction Class ShopController < ApplicationController def order # synchronous side ap4r.transaction do o = Order.new(:name => params[:order_id]) o.save! ap4r.async_to({:action => ‘payment’}, {:order_id => o.id}) end redirect_to ... end end 134

Process ﬂow changes in queueing sequence 135

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database ap4r.async_to(...) end 136

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end 136

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end Commit database 136

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end Commit database Queue messages 136

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) Store end Commit database Queue messages 136

Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) Store end Commit database Queue messages Forward 136

Sequence in detail Without SAF 137

start CRUD commit message put / commit ☠ end data messages 139

start CRUD commit counterchange message put / commit end data messages 140

start CRUD message put / commit counterchange commit end data messages 141

start CRUD message put / commit commit end data messages 142

start CRUD message put / commit commit end data messages 143

start CRUD message put / commit commit ☠ end data messages 143

Better Safe than Sorry • Database trouble • Network trouble • Server down • busy and timeout • ... etc 144

☺ Embrace every exceptions ☠ 145

Sequence in detail With SAF 146

CRUD start SAF create commit message put / commit SAF update end data messages 147

CRUD start SAF create commit message put / commit SAF update ☺ end data messages 149

CRUD start SAF create commit message put / commit SAF update end data messages messages 151

CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 151

CRUD start SAF create commit message put / commit Recoverable SAF update ☺ end data messages messages 151

CRUD start SAF create commit message put / commit SAF update end data messages messages 153

CRUD start SAF create commit message put / commit Duplicated SAF update ☺ end data messages messages 153

Ensure Atomicity of application transaction and putting message 154

at-least-once between Producer and Channel 155

5. Other features 157

• Messages recovery when some troubles happen • Collaboration with Capistrano (2.x) • Test support 158

Test support 159

TDD / BDD 160

☺ It makes us feel relieved in developing an application 161

On async app. • Test matters also, but .. • Multiple processes communicating via N/W • take time • annoying to setup 162

Two ways of testing • Functional test • no N/W communication • fast but restricted • Async test • using N/W communication • slow but close to actual 163

Functional test • Provided a queue stub • Executed in memory, quickly 164

code sample test code here is against a ‘ﬁctional’ on-line shop application, which has an order action (sync) and payment action (async). 165

def test_order post :order, :order => {:item => \"introduction to AP4R\"} assert_response :redirect assert_redirected_to :action => 'index' messages = @controller.ap4r.queued_messages # ... (1) assert_equal 1, messages.keys.size, \"should have messages in just ONE queue\" assert messages.key?(\"queue.async_shop.payment\"), \"queue name is INCORRECT\" # ... (2) assert_equal 1, messages[\"queue.async_shop.payment\"].size, \"should have just ONE message for payment\" assert_match /order_id=\\d+/, messages[\"queue.async_shop.payment\"].first[:body], \"parameter order_id should be included with a numeric value\" # ... (3) end 1. get (stubbed)queued messages 2. assert the queue name 3. assert the message content 166

functional test rake .. CRUD message put Stub assert assert 167

Async test • Get on reality • Test everything, e.g serialization, cooperation. 168

code sample test code here is against a ‘ﬁctional’ application. 169

[RAILS_ROOT]/test/async/ap4r_test_helper.rb ENV[\"RAILS_ENV\"] = \"test\" require File.expand_path(File.dirname(__FILE__) + \"/../../config/environment\") require \"ap4r/service_handler\" ap4r_test_helper = Ap4r::ServiceHandler.new require 'test_help' class Test::Unit::TestCase self.use_transactional_fixtures = false self.use_instantiated_fixtures = false # Add more helper methods to be used by all tests here... cattr_accessor :ap4r_helper def ap4r_helper @@ap4r_helper end def with_services(&block) ap4r_helper.with_services(&block) end end Test::Unit::TestCase.ap4r_helper = ap4r_test_helper 170

[RAILS_ROOT]/test/async/async_shop_test.rb require \"#{File.dirname(__FILE__)}/ap4r_test_helper\" require 'net/http' class AsyncShopTest < Test::Unit::TestCase def test_http_dispatch ap4r_helper.stop_dispatchers # ... (1) assert_rows_added(Order, 1) { # ... (3) do_order # ... (2) } assert_rows_added(Payment, 1) { # ... (6) ap4r_helper.start_dispatchers # ... (4) ap4r_helper.wait_all_done # ... (5) } end private # Requests to <tt>async_shop/order</tt>. def do_order(item_name = \"test item\") Net::HTTP.start(\"localhost\", 3000, nil, nil) do |http| http.request_post(\"/async_shop/order\", \"order[item]=#{item_name}\") do |res| #nop end end end def assert_rows_added(model, rows) rows_before = model.count yield rows_after = model.count assert_equal rows, rows_after - rows_before, \"table '#{model.table_name}' should count up by #{rows}\" end end 171

stop dispatcher threads 1. request to Rails with HTTP POST 2. assert one row added to orders table 3. start dispatcher threads 4. wait for async action is completed 5. assert one row added to payments table 6. 172

async test rake .. service control HTTP CRUD message put assert HTTP CRUD assert 173

Summary • Two test methods. • Functional test • Async test • It makes us feel relieved to develop an application 174

6. Conclusion 175

AP4R is .. 176

• Messaging with simple API, simple conﬁguration • Seamless collaboration with Rails • Flexible system design 179

• Automatically invocation of async. processing by dispatcher • Guaranteed delivery by SAF • Test support • Recovery and deployment 180

Future Plans 181

Smooth operation ver.0.3.x ☑ Daemonize ☑ URL-rewrite ﬁlter ☑ DLQ / SAF recovery □ Protocols: Stomp, HTTP 182

Monitoring ver.0.4.x □ Coordinate with Cacti, ZABBIX, etc. □ Many AP4R processes □ Interactive management tool 183

Step to large scale ver.0.5.x □ Dynamic conﬁguration □ Automatic recovery □ Long run 24/7 184

Others □ Security ☑ Application testing support □ Blocking queues □ Ruby off Rails 185

AP4R@RubyForge • wiki: http://ap4r.rubyforge.org/wiki/wiki.pl?HomePage • source: svn co http://ap4r.rubyforge.org/svn/trunk/ap4r • ML: http://rubyforge.org/mailman/listinfo/ap4r-user Your patches are welcome! Thank you. by Kanican 186

• Future Architect logo belongs to Future Architect, Inc. Japan. All rights reserved. • Rails logo is trademarks of David Heinemeier Hansson. All rights reserved. • The Ruby logo is copyrighted (c) 2006, Yukihiro Matsumoto. It is released under the terms of the Creative Commons Attribution-ShareAlike 2.5 License. 187

AP4R on RubyConf2007

0 comments

Notes on slide 1

1 Favorite