Ruby HTTP clients comparison

19,650 views

Published on

4 Comments
54 Likes
Statistics
Notes
No Downloads
Views
Total views
19,650
On SlideShare
0
From Embeds
0
Number of Embeds
190
Actions
Shares
0
Downloads
119
Comments
4
Likes
54
Embeds 0
No embeds

No notes for slide

Ruby HTTP clients comparison

  1. 1. Ruby HTTP clients comparison Hiroshi Nakamura nahi at Twitter, github Technical Architect at Appirio Japan CRuby and JRuby committer Asakusa.rb: http://qwik.jp/asakusarb/
  2. 2. Ruby HTTP Clients Matrixadvantages-and-disadvantages comparison ofHTTP client libraries http://bit.ly/RubyHTTPClients2012 Disclaimer: Im the author of "httpclient"
  3. 3. Agendanet/http16 libraries I pickedRuby HTTP Clients Matrix API style Compatibility Supported featuresPerformance ComparisonsMy Recommendations
  4. 4. net/http Net::HTTP::Proxy? net/https?
  5. 5. Ruby HTTP Client libraries
  6. 6. HTTP client libraries I didn’t evaluateCannot evaluate• activeresource (Rails specific)• http (under development)• http_request.rb (test doesnt pass)• nestful (no test)• typhoeus (under heavy rewrite)Obsolete• eventmachine (built-in client is obsolete)• right_http_connection (no update)• simplehttp (no update)• rfuzz (no update)
  7. 7. Evaluation AxisProject StatsAPI styleCompatibility: CRuby, JRuby, RubiniusSupported features Connection features Basic HTTP features Development support Advanced features http://bit.ly/RubyHTTPClientsFeatureTest (test/unit scripts)
  8. 8. Project Stats
  9. 9. Project Stats
  10. 10. API style
  11. 11. sync API (1/5) - Client instancenet/http, mechanize, httpclient, patron, curb, faraday client = Net::HTTP.new(host, port) p client.get(path).body client = Mechanize.new client = HTTPClient.new client = Patron::Session.new p client.get(url).body curl = Curl::Easy.new(url) curl.http_get p curl.body_str client = Faraday.new(:url => baseurl) p client.get(path).body
  12. 12. sync API (2/5) - Client class restfulie, excon, httpi p Restfulie.at(url).get!.body p Excon.get(url).body p HTTPI.get(url).body
  13. 13. sync API (3/5) - Resource rest-client, rufus-verbsrs = RestClient::Resource.new(http://example.com)rs[posts/1/comments].post Good article., :content_type => text/plainep = EndPoint.new( :host => "resta.farian.host", :port => 80, :resource => "inventory/tools")res = ep.get :id => 1res = ep.get :id => 2
  14. 14. sync API (4/5) - Include & Customize httparty, weary client = Class.new { include HTTParty } p client.get(url) class WearyClient < Weary::Client domain http://api.target.org/ get :retrieve, {path} end p WearyClient.new.fetch(:path => path).perform.body
  15. 15. sync API (5/5) - Others open-uri.rb, wrest# open-uri.rbp open(url) { |f| f.read }# wrestp http://www.google.co.jp/.to_uri.get.body
  16. 16. async API(1/2) - Callback em-http-request body = nil EM.run do req = EM::HttpRequest.new( http://www.google.com/).get req.callback do body = req.response EM.stop end req.errback do body = nil end end p body
  17. 17. async API(2/2) - Polling httpclient, weary, wrest client = HTTPClient.new conn = client.get_async(url) conn.finished? # => false # ... io = conn.pop.content while str = io.read(4096) p str end
  18. 18. parallel API - curbresponses = []m = Curl::Multi.newurls.each do |url| responses[url] = m.add(Curl::Easy.new(url) { |curl| curl.on_body { |data| responses[url] << data data.bytesize } })endm.performp responses.map { |e| e.bytesize }
  19. 19. API style
  20. 20. Compatibility
  21. 21. Compatibility
  22. 22. Connection features
  23. 23. 3 types of HTTP connections
  24. 24. Keep-Alive in em-http-request body = [] EM.run do conn = EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req1.callback { body << req1.response req2 = conn.get(:keepalive => true) req2.callback { body << req2.response req3 = conn.get(:keepalive => true) req3.callback { body << req3.response req4 = conn.get(:keepalive => true) req4.callback { body << req4.response EM.stop req4.errback { ... }} req3.errback { ... }} req2.errback { ... }} req1.errback { ... } end
  25. 25. Pipelining in em-http-request body = [] EM.run do conn = EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req2 = conn.get(:keepalive => true) req3 = conn.get(:keepalive => true) req4 = conn.get() req1.callback { body << req1.response } req2.callback { body << req2.response } req3.callback { body << req3.response } req4.callback { body << req4.response; EM.stop } req1.errback { ... } req2.errback { ... } req3.errback { ... } req4.errback { ... } end
  26. 26. NO verification by default?!if http.use_ssl? http.verify_mode = OpenSSL::SSL::VERIFY_NONE if options[:ssl_ca_file] http.ca_file = options[:ssl_ca_file] http.verify_mode = OpenSSL::SSL::VERIFY_PEER endendoptions[:ssl_ca_file] == nil => VERIFY_NONE
  27. 27. Connection features
  28. 28. Basic HTTP features
  29. 29. IRI: Internationalized Resource Identifierclient.get("http://www.ebooks.com/797059/some-kind-of-peace/grebe-camilla-träff-åsa-norlen-paul/") uri.rb doesnt support IRI addressable/uri does
  30. 30. "Cross-site Cooking" bug in httpclientSet-Cookie: TEST=test; path=/; domain=.com httpclient eats this cookie and send it to all *.com site Mechanize handles it properly like browsers http://en.wikipedia.org/wiki/Cross-site_cooking
  31. 31. Streaming upload/download# Chunked upload with Patronclient = Patron::Session.newres = client.request(:post, url, {}, :file => path_to_upload)# Chunked downloadclient.get_file(url, path_to_write)# Chunked upload with em-http-requestreq = EM::HttpRequest.new(url).post :file => path_to_upload# Chunked downloadreq = EM::HttpRequest.new(url).getreq.stream do |chunk| p chunkend
  32. 32. Basic HTTP features
  33. 33. Development Support
  34. 34. Response stubbing# Stubbing response bodyclient = HTTPClient.newclient.test_loopback_response << Hello!client.get(http://www.example.com/hello).body#=> "Hello!"# Stubbing HTTP responseclient.test_loopback_http_response << "HTTP/1.0 302 Found¥r¥nLocation: http://foo/¥r¥n¥r¥n" << "HTTP/1.0 200 OK¥r¥n¥r¥nHello!"client.post(http://www.example.com/todo, :follow_redirect => true, :body => {}).body#=> "Hello!"
  35. 35. IRB like shell rest-client, httpclient, wrest% restclient https://example.com user pass>> delete /private/resource% httpclient>> get "https://www.google.com", :q => :ruby% wrest>> http://www.google.com?q=ruby.to_uri.get
  36. 36. Replayable log rest-client% RESTCLIENT_LOG=/tmp/restclient.log restclient>> RestClient.get "https://www.google.com/"...% cat /tmp/restclient.logRestClient.get "https://www.google.com/", "Accept"=>"*/*;q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate"# => 200 OK | text/html 13354 bytes
  37. 37. Development Support
  38. 38. Advanced features
  39. 39. HTML form handling of Mechanize agent = Mechanize.new page = agent.get(url) form = page.form(login) form.email = nahi@ruby-lang.org form.password = jKH.P945wruV*qh3 page = agent.submit(form, form.button(submit))
  40. 40. Advanced features
  41. 41. Testing your clientwebmock by Bartosz Blimke (bblimke) Library for stubbing and setting expectations on HTTP requests in Ruby.vcr by Myron Marston (myronmarston) Record your test suites HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.
  42. 42. Performance ComparisonsServer Linode Xen VPS (Linode 512) at Fremont, CA Ubuntu 10.10 Apache 2.2, KeepAlive OnClient AWS EC2 (m1.small) at North Virginia (us-east-1b) Ubuntu 12.04 HTTP clients w/ CRuby 1.9.3p286Multiple downloads of 177B.html and 24MB.zipDont take it serious! http://bit.ly/RubyHTTPClientsBenchmarkScript
  43. 43. Multiple 177B downloads [sec]
  44. 44. [sec]
  45. 45. [sec]
  46. 46. My Recommendations• Speed is the king => em-http-request, curb w/ multi• HTML operation, Cookies => Mechanize• API client => Faraday and adapter based impls• SSL, Connectivity => httpclientCheck the matrix before you use the librariesPlease let me know when you find incorrect cell
  47. 47. Development Timeline

×