AP4R on RubyKaigi2007 (English only)
Upcoming SlideShare
Loading in...5
×
 

AP4R on RubyKaigi2007 (English only)

on

  • 4,106 views

Rubykaigi2007 presentation

Rubykaigi2007 presentation

Statistics

Views

Total Views
4,106
Views on SlideShare
4,106
Embed Views
0

Actions

Likes
3
Downloads
42
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    AP4R on RubyKaigi2007 (English only) AP4R on RubyKaigi2007 (English only) Presentation Transcript

    • RubyKaigi 2007 AP4R Kiwamu Kato & Shun’ichi Shinohara Future Architect, Inc. 1
    • • Introduction • Demo • Real world examples • Technical topics 2
    • Introduction 3
    • Self Introduction 4
    • Kiwamu Kato • Majored in Civil Engineering • Has Enhanced and maintained messaging middleware in company 5
    • Kiwamu Kato • Started Ruby since last year • Leader of the AP4R team • http://d.hatena.ne.jp/kiwamu/ 6
    • shino • Programmer, I love coding ! • The language of the year is C or Haskell 7
    • shino • Last year, made a debut to the OSS world • http://d.hatena.ne.jp/ita-wasa/ 8
    • What is AP4R ? 9
    • Lightweight Messaging 10
    • What is AP4R ? • Asynchronous Processing for Ruby • Loose coupling by messaging 11
    • What can we do w/ AP4R ? • Quicker response to users • Load distribution 12
    • Press release 2006/09/05 http://itpro.nikkeibp.co.jp/article/NEWS/20060905/247218/ 13
    • Key points are .. 14
    • Lightweight and Robust 15
    • When is it useful ? 16
    • Quicker response to users 17
    • If NO speedy response is needed, execute it later
    • Before Client Server A B C D ... 19
    • After Client Server Messaging Server A B 20
    • After Client Server Messaging Server A B C D ... 21
    • Useful cases • Logging • Sending mail • Creating heavy summary data • Collaboration with other systems 22
    • Moreover, load distribution 23
    • Easy to scale out Client Server Messaging Server 24
    • Easy to scale out Client Server Messaging Servers 25
    • Only good things ? 26
    • Sounds easy ? 27
    • (Maybe) No 28
    • If message deliveries are NOT guaranteed,... 29
    • it may lead to message lost and message duplication 30
    • In other words... 31
    • •You can’t receive the goods despite payment ! • Double withdrawals ! 32
    • Messages should be delivered in a reliable way, 33
    • No matter what happens • Database trouble • Network trouble • Server down • Busy and timeout • ... etc 34
    • Lightweight and Robust 35
    • In Future Architect ... 36
    • We use messaging a lot • Developed own messaging library • Named “RtFA” 37
    • We use messaging a lot • Pure Java • Its own protocol, API (≠ JMS) • Time-proven on various systems 38
    • One of illustrative cases Mission critical system in transportation business http://japan.internet.com/linuxtoday/20061228/5.html 39
    • PJ summary • Over 100 million package data per day • Load distribution and parallel processing on 8 nodes of Oracle 10g RAC 40
    • PJ summary • Approx 100 servers • Used “RtFA”, an in-house middleware 41
    • Messaging is important
    • 43
    • Lightweight Messaging 44
    • http://www.pragmaticprogrammer.com/titles/fr_j2r/ http://www.oreilly.co.jp/books/9784873113203/ 45
    • From RtFA to AP4R • Based on implementation/experiences with Java, kept good parts, improved bad parts • Focus on ease of use API, configuration 46
    • Lightweight and Robust 47
    • Demo 48
    • Outline • Shopping store application • Order and payment 49
    • Shopping store app. Client Rails Order . Payment 50
    • Loose coupling AP4R Client Rails Rails × 3 Order 51
    • Loose coupling AP4R Client Rails Rails × 3 Payment Order 52
    • Loose coupling AP4R Client Rails Rails × 3 Payment Order Sleep 1 53
    • Outline • Use mongrel cluster • Stress by ab • Monitor queue retaining status 54
    • Real world examples 55
    • Two examples • Work With Rails • Our internal project 56
    • Working With Rails http://workingwithrails.com/ 57
    • WWR • is who’s who for Rails developers • listed 7,000 people from 104 countries 58
    • 59
    • WWR • aggregates feeds from user’s blog • has scaling problem in fetching them slow/stale 60
    • WWR solution • Queue Feeds • Fetch async’ly quot;Getting Distributed with Ruby on Railsquot; slide #44 (from slideshare) 61
    • WWR solution • Able to scale • Flexible to feed types 62
    • Internal project 63
    • Briefing • Work sheet management app. newly-building in Ruby • Labor cost accounting app. existing in Java 64
    • Requirements • Quick and rich UI • Replicate data between Java and Ruby 65
    • With AP4R • Separate loads from UI • Invoke Java from Ruby async’ly 66
    • Separate loads from UI Client Server Messaging Server 67
    • Separate loads from UI Client Server Messaging Server 67
    • Separate loads from UI Client Server Messaging Server 67
    • Separate loads from UI Client Server Messaging Server Create summary data, Send mails, etc. 68
    • Async’ly invoke Java Client Server Messaging Server 69
    • Async’ly invoke Java Client Server Messaging Server 69
    • Async’ly invoke Java Client Server Messaging Server data replication, etc. 69
    • Async’ly invoke Java Client Server Messaging Server data replication, etc. 69
    • Utilize Async! when • Needless proc. for responce data • Disjoined transactions are allowable • External calls (often slow, unstable) 70
    • Technical topics 71
    • Two topics here • API to put messages and process flow • Stored and Forward 72
    • API to put messages and process flow 73
    • ap4r.async_to({specify next}, {request data} [,{more opts}]) There’s also block style c.f. RDoc: http://ap4r.rubyforge.org/doc/ 74
    • Class ShopController < ApplicationController def order # synchronous side ... ap4r.async_to({:action => ‘payment’}, {:id => 1}) redirect_to ... end def payment # asynchronous side id = params[:id] ... render :text => “true” end end 75
    • Process flow 76
    • User Apache Rails AP4R 77
    • User Apache Rails AP4R order message put 78
    • User Apache Rails AP4R dispatch 79
    • User Apache Rails AP4R payment 80
    • Able to separate sync/async servers 81
    • User Apache Rails AP4R 82
    • Architecture AP4R HTTP AP4R DRB druby DRB reliable-msg SOAP XML -RPC AP4R MySQL mongrel HTTP 83
    • Thanks to DRb 4% 10% reliable-msg 20% mongrel 66% rails
    • Stored And Forward 85
    • Stored ? Forward ? 86
    • at-least-once between Producer and Channel 87
    • Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe,Bobby Woolf Guaranteed Delivery [122] 88
    • Typical flow to create async. messages 89
    • start end 90
    • start CRUD end 90
    • start CRUD commit end 90
    • start CRUD commit message put / commit end 90
    • start CRUD commit message put / commit end data 90
    • start CRUD commit message put / commit end data messages 90
    • Ensure (some) Atomicness of updating database and message creation 91
    • On updating database • Error occurs, • Any following messages must NOT be queued 92
    • On updating database • Everything succeeded, • All following messages must be queued 93
    • ☺ Relief 94
    • Lightweight and Robust 95
    • Ap4r::Client#transaction Class ShopController < ApplicationController def order # synchronous side ap4r.transaction do ... # CRUD on database ap4r.async_to ... end redirect_to ... end end 96
    • Process flow changes in queueing sequence 97
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database ap4r.async_to(...) end 98
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end 98
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end Commit database 98
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) end Commit database Queue messages 98
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) Store end Commit database Queue messages 98
    • Behind ap4r.transaction ap4r.transaction do ... # CRUD on database Insert into SAF table ap4r.async_to(...) Store end Commit database Queue messages Forward 98
    • Sequence in detail Without SAF 99
    • start CRUD commit message put / commit end data messages 100
    • start CRUD commit message put / commit end data messages 100
    • start CRUD commit message put / commit end data messages 101
    • start CRUD commit message put / commit end data messages 101
    • start CRUD commit message put / commit ☠ end data messages 101
    • start CRUD commit message put / commit ☠ end data messages 101
    • start CRUD commit message put / commit ☠ end data messages 101
    • start CRUD commit counterchange message put / commit end data messages 102
    • start CRUD message put / commit counterchange commit end data messages 103
    • start CRUD message put / commit commit end data messages 104
    • start CRUD message put / commit commit end data messages 104
    • start CRUD message put / commit commit end data messages 105
    • start CRUD message put / commit commit end data messages 105
    • start CRUD message put / commit commit ☠ end data messages 105
    • start CRUD message put / commit commit ☠ end data messages 105
    • start CRUD message put / commit commit ☠ end data messages 105
    • Better Safe than Sorry • Database trouble • Network trouble • Server down • busy and timeout • ... etc 106
    • ☺ Embrace every exceptions ☠ 107
    • Sequence in detail With SAF 108
    • CRUD start SAF create commit message put / commit SAF update end data messages 109
    • CRUD start SAF create commit message put / commit SAF update end data messages 110
    • CRUD start SAF create commit message put / commit SAF update end data messages 110
    • CRUD start SAF create commit message put / commit SAF update end data messages 111
    • CRUD start SAF create commit message put / commit SAF update end data messages 111
    • CRUD start SAF create commit message put / commit SAF update end data messages 111
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages 111
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages 111
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages 111
    • CRUD start SAF create commit message put / commit SAF update end data messages 112
    • CRUD start SAF create commit message put / commit SAF update end data messages 112
    • CRUD start SAF create commit message put / commit SAF update end data messages 113
    • CRUD start SAF create commit message put / commit SAF update end data messages 113
    • CRUD start SAF create commit message put / commit SAF update end data messages messages 113
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 113
    • CRUD start SAF create commit message put / commit Recoverable SAF update ☺ end data messages messages 113
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 113
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 113
    • CRUD start SAF create commit message put / commit SAF update end data messages 114
    • CRUD start SAF create commit message put / commit SAF update end data messages 114
    • CRUD start SAF create commit message put / commit SAF update end data messages 115
    • CRUD start SAF create commit message put / commit SAF update end data messages messages 115
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 115
    • CRUD start SAF create commit message put / commit Duplicated SAF update ☺ end data messages messages 115
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 115
    • CRUD start SAF create commit message put / commit SAF update ☺ end data messages messages 115
    • Ensure (some) Atomicness of data and messages 116
    • at-least-once between Producer and Channel 117
    • Summary 118
    • • Messaging with simple API, simple configuration. • Seamless collaboration with Rails 119
    • • Automatic invocation of async proc. • Guaranteed delivery by SAF 120
    • Lightweight and Robust 121
    • Future Plans 122
    • Smooth operation ver.0.3.x ☑ Daemonize ☑ URL-rewrite filter □ DLQ / SAF recovery □ Protocols: Stomp, HTTP 123
    • Monitoring ver.0.4.x □ Coordinate with Cacti, Nagios, etc. □ Many AP4R processes □ Interactive management tool 124
    • Step to large scale ver.0.5.x □ Dynamic configuration □ Automatic recovery □ Long running 125
    • Others □ Security □ Application testing support □ Blocking queues 126
    • AP4R@RubyForge • wiki: http://ap4r.rubyforge.org/wiki/wiki.pl?HomePage • source: svn co svn://rubyforge.org/var/svn/ap4r Your patches are welcome! Thank you. 127
    • • Future Architect logo belong to Future Architect, Inc. Japan. All rights reserved. • Rails logo are trademarks of David Heinemeier Hansson. All rights reserved. • The Ruby logo is copyright c 2006, Yukihiro Matsumoto. It is released under the terms of the Creative Commons Attribution-ShareAlike 2.5 License. 128