Ruby  application based on http www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Con...
Who am I? www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Servic...
www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb ...
Http Server www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Serv...
Http protocol www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Ruby API? Official? Community? www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Cons...
Get data from third-party website www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails C...
What if fetch contacts from gmail? What if fetch timelines from twitter? What if fetch friends from facebook? What if fetc...
What if fetch contacts from 163 mail? What if fetch weather from sina? What if fetch friends from kaixin001? What if fetch...
Act as  a http client (browser) <ul><li>Analyze the packets sent by http client (browser) in sniffer tools. </li></ul><ul>...
Sniffer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services...
Wireshark <ul><li>Detect any protocol (ssh, ftp, tcp, http, …) </li></ul><ul><li>Cross platform (Windows, Linux, Mac OS) <...
Wireshark www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Servic...
Wireshark www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Servic...
HTTP Analyzer <ul><li>Only detect http/https protocol </li></ul><ul><li>Concentrated on http protocol (request timing, coo...
HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Get session/new www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails ...
Get session/new response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting...
Post /session www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Post /session response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting -...
Get / www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services -...
Http protocol www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Resend same packets by Ruby www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consult...
Resend same packets by Ruby <ul><li>Send a GET request </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby...
Resend same packets by Ruby <ul><li>Parse GET response </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby...
Resend same packets by Ruby <ul><li>Send a POST request </li></ul>www.ekohe.com Web Development & Graphic Design China Rub...
Resend same packets by Ruby <ul><li>Parse POST response </li></ul>www.ekohe.com Web Development & Graphic Design China Rub...
Resend same packets by Ruby <ul><li>Send a GET request </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby...
Facility www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Service...
Mechanize <ul><li>A ruby library that makes automated web interaction easy.  </li></ul><ul><li>Automatically location redi...
Mechanize www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Servic...
Test www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - ...
Test <ul><li>Test http requests </li></ul><ul><li>Test http responses </li></ul>www.ekohe.com Web Development & Graphic De...
Test request www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Ser...
Test request www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Ser...
Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Fakeweb <ul><li>A test helper for faking responses to web requests. </li></ul><ul><li>No need to  modify existing code </l...
Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Test Process <ul><li>Do some stubs </li></ul><ul><li>Copy the http request/response as expected message from sniffer tools...
Performance www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Serv...
net-http-persistent <ul><li>a thread-safe wrapper for Net::HTTP that performs persistent connections for you </li></ul>www...
typhoeus <ul><li>Parallel http requests </li></ul><ul><li>based on libcurl and libcurl-multi </li></ul><ul><li>500ms respo...
Q&A Thank you www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Se...
Upcoming SlideShare
Loading in...5
×

Ruby application based on http

5,747

Published on

Do you have an experience to write a client application based on http? Such as fetching the contacts from email or writing a IM client. Some of the servers provide apis which make life better, but most of them not. So how can we get the data from these servers or communicating with these servers? This talk will teach you how to analysis the packets between client and server and share my experience about how to write the client application and how to do the test and refactor.

Published in: Technology
2 Comments
20 Likes
Statistics
Notes
No Downloads
Views
Total Views
5,747
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
107
Comments
2
Likes
20
Embeds 0
No embeds

No notes for slide

Ruby application based on http

  1. 1. Ruby application based on http www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  2. 2. Who am I? www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development <ul><li>Richard Huang a.k.a flyerhzm </li></ul><ul><li>Work at Ekohe </li></ul><ul><li>http://huangzhimin.com </li></ul><ul><li>@flyerhzm </li></ul><ul><li>http://github.com/flyerhzm </li></ul>
  3. 3. www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  4. 4. Http Server www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Oauth JSON XML Params Signature Web Service Restful Cloud Computing HTML XHTML
  5. 5. Http protocol www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development <ul><li>Request - Response </li></ul><ul><li>Header - Body </li></ul>Client (Browser) RubyConfChina Server Get /session/new Return session/new page Post /session Login success, redirect to / Get / Return home page
  6. 6. Ruby API? Official? Community? www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  7. 7. Get data from third-party website www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development if the website provides an api if the website provides an official/community ruby api use the ruby api else write the ruby api by your self end else write the ruby codes act as a http client (browser) end
  8. 8. What if fetch contacts from gmail? What if fetch timelines from twitter? What if fetch friends from facebook? What if fetch contacts from msn? What if fetch repositories from github? Lucky, there are ruby api to do theses! www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  9. 9. What if fetch contacts from 163 mail? What if fetch weather from sina? What if fetch friends from kaixin001? What if fetch contacts from fetion? T here are no ruby api to do theses! www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  10. 10. Act as a http client (browser) <ul><li>Analyze the packets sent by http client (browser) in sniffer tools. </li></ul><ul><li>Resend same packets by Ruby. </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  11. 11. Sniffer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  12. 12. Wireshark <ul><li>Detect any protocol (ssh, ftp, tcp, http, …) </li></ul><ul><li>Cross platform (Windows, Linux, Mac OS) </li></ul><ul><li>Both raw data and printable text </li></ul><ul><li>Not easy to view https encrypted data </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  13. 13. Wireshark www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development filter http one http request TCP/IP HTTP HTTP Header/Body Raw Data and Printable text
  14. 14. Wireshark www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development one http response
  15. 15. HTTP Analyzer <ul><li>Only detect http/https protocol </li></ul><ul><li>Concentrated on http protocol (request timing, cookie, …) </li></ul><ul><li>Easy to view https encrypted data </li></ul><ul><li>Windows version only </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  16. 16. HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development http request header http response header
  17. 17. HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development http response body
  18. 18. HTTP Analyzer www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development http request timing
  19. 19. Get session/new www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Get /session/new HTTP/1.1 Host: rubyconfchina.org Connect: keep-alive User-Agent: Mozilla/5.0 AppleWebKit/553.4 Accept: text/html,application/xhtml+xml,text/html;q=0.9 Accept-Encoding: gzip,deflate Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  20. 20. Get session/new response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development HTTP/1.1 200 OK Server: nginx/0.5.32 Date: Sun, 20 Jun 2010 09:25:08 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Cache-Control: private, max-age=0, must-revalidate Content-Encoding: gzip <!DOCTYPE html PUBLIC …> <html> …
  21. 21. Post /session www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Post /session HTTP/1.1 Host: rubyconfchina.org Connect: keep-alive User-Agent: Mozilla/5.0 AppleWebKit/553.4 Referer: http://rubyconfchina.org/session/new Content-Length: 106 Content-Type: application/x-www-form-urlencoded Accept: text/html,application/xhtml+xml,text/html;q=0.9 authenticity_token=…&login=flyerhzm&password=… &commit=Log+in
  22. 22. Post /session response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development HTTP/1.1 302 Moved Temporarily Server: nginx/0.5.32 Date: Sun, 20 Jun 2010 09:25:31 GMT Content-Type: text/html; charset=utf-8 Connect: keep-alive Location: http://rubyconfchina.org/ Cache-Control: no-cache Set-Cookie: auth_token=; _euruko_session=BAh7CTo Content-Length: 91 <html><body>You are being <a href=http://rubyconfchina.org/>redirected</a>. </body></html>
  23. 23. Get / www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Get / HTTP/1.1 Host: rubyconfchina.org Connect: keep-alive User-Agent: Mozilla/5.0 AppleWebKit/553.4 Accept: text/html,application/xhtml+xml,text/html;q=0.9 Accept-Encoding: gzip,deflate Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Cookie: auth_token=; _euruko_session=BAh7CTo
  24. 24. Http protocol www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development HTTP/1.1 200 OK Server: nginx/0.5.32 Date: Sun, 20 Jun 2010 09:25:08 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Cache-Control: private, max-age=0, must-revalidate Content-Encoding: gzip <!DOCTYPE html PUBLIC …> <html> …
  25. 25. Resend same packets by Ruby www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  26. 26. Resend same packets by Ruby <ul><li>Send a GET request </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development uri = URI.parse(&quot;http://rubyconfchina.org/session/new&quot;) http = Net::HTTP.new(uri.host, uri.port) headers = { &quot;User-Agent&quot; => &quot; Mozilla/5.0 AppleWebKit/553.4 &quot;, &quot;Accept&quot; => &quot;…&quot; } response = http.request_get(uri.request_uri, headers) puts response.body =~ /Login/ # 1942
  27. 27. Resend same packets by Ruby <ul><li>Parse GET response </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development raise Exception, &quot;get response should success&quot; unless Net::HTTPSuccess === repsonse if response.body =~ %r(<input name=&quot;authenticity_token&quot; .*?value=&quot;(.*>)&quot; />)m authenticity_token = $1 end
  28. 28. Resend same packets by Ruby <ul><li>Send a POST request </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development uri = URI.parse(&quot;http://rubyconfchina.org/session&quot;) http = Net::HTTP.new(uri.host, uri.port) body = &quot;authenticity_token=#{authenticity_token}& login=flyerhzm&password=…&quot; headers = { &quot;User-Agent&quot; => &quot; Mozilla/5.0 AppleWebKit/553.4 &quot;, &quot;Accept&quot; => &quot;…&quot; } response = http.request_post(uri.request_uri, body, headers)
  29. 29. Resend same packets by Ruby <ul><li>Parse POST response </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development raise Exception, &quot;get response should be redirect&quot; unless Net::HTTPRedirection === repsonse cookie = response[&quot;Set-Cookie&quot;] cookie = parse_cookie(cookie) Location = response[&quot;Location&quot;]
  30. 30. Resend same packets by Ruby <ul><li>Send a GET request </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development uri = URI.parse(Location) http = Net::HTTP.new(uri.host, uri.port) headers = { &quot;User-Agent&quot; => &quot; Mozilla/5.0 AppleWebKit/553.4 &quot;, &quot;Accept&quot; => &quot;…&quot;, &quot;Cookie&quot; => cookie } response = http.request_get(uri.request_uri, headers) raise Exception, &quot;get repsonse should success&quot; unless Net::HTTPSuccess === response puts response.body =~ /Login/ # nil
  31. 31. Facility www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  32. 32. Mechanize <ul><li>A ruby library that makes automated web interaction easy. </li></ul><ul><li>Automatically location redirection </li></ul><ul><li>Automatically cookies management </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  33. 33. Mechanize www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development a = Mechanize.new do |agent| agent.user_agent_alias = 'Linux Firefox' end a.get('http://rubyconfchina.org/session/new') do |page| puts page.body =~ /Login/ # 1942 home_page = page.form_with(:action => '/session') do |f| f.login = 'flyerhzm' f.password = '…' end.click_button puts home_page.body =~ /Login/ # nil end
  34. 34. Test www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  35. 35. Test <ul><li>Test http requests </li></ul><ul><li>Test http responses </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  36. 36. Test request www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Client Server GET /ht/sd.aspx?i=1 HTTP/1.1 Host: 221.176.31.39 … R fetion.com.cn SIP-C/4.0 F: 730020377 I: 1 Q: 1 R CN: 19D28D4978125CAA4F6E54277BA7D9EF CL: type=&quot;pc&quot; ,version=&quot;3.6.2020&quot; SIPP sid = 730020377 i = 0
  37. 37. Test request www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development @fetion.stubs(:sid).returns(730020377) @fetion.stubs(:i).returns(0) @fetion.stubs(:guid).returns(&quot;19D28D4978125CAA4F6E54277BA7D9EF&quot;) expected_sipc_message =<<-EOF R fetion.com.cn SIP-C/4.0 F: 730020377 I: 1 Q: 1 R CN: 19D28D4978125CAA4F6E54277BA7D9EF CL: type=&quot;pc&quot; ,version=&quot;3.6.2020&quot; SIPP EOF expected_sipc_message.gsub!(&quot; &quot;, &quot; &quot;).chomp! SipcMessage.register_first(@fetion).should == expected_sipc_message Copied from Sniffer tool
  38. 38. Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development Client Server HTTP/1.1 200 OK Server: nginx/0.5.32 … Set-Cookie: ssic=CBIOAAAm+FiuQgpcnFi; path=/ <?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?> <results status-code=&quot;200&quot;> <user uri=&quot;sip:730020377@fetion.com.cn;p=6907&quot; /> </results>
  39. 39. Fakeweb <ul><li>A test helper for faking responses to web requests. </li></ul><ul><li>No need to modify existing code </li></ul><ul><li>No need to write extensive stubs </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  40. 40. Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?><results status-code=&quot;200&quot;><user uri=&quot;sip:730020377@fetion.com.cn;p=6907&quot; /></results>|, :set_cookie => Q|ssic=CBIOAAAm+FiuQgpcnFi; path=/| ) @fetion.login @fetion.status_code.should == &quot;200&quot; @fetion.sid.should == &quot;730020377&quot; Copied from Sniffer tool
  41. 41. Test response www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?><results status-code=&quot;401&quot; desc=&quot;password error&quot; />|, :status => ['401', 'password error'] ) lambda { @fetion.login }.should raise_exception(Fetion::PasswordError) Copied from Sniffer tool
  42. 42. Test Process <ul><li>Do some stubs </li></ul><ul><li>Copy the http request/response as expected message from sniffer tools </li></ul><ul><li>Execute the method to generate http request/respones </li></ul><ul><li>Test with expected message </li></ul><ul><li>TDD (optional) </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  43. 43. Performance www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  44. 44. net-http-persistent <ul><li>a thread-safe wrapper for Net::HTTP that performs persistent connections for you </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development user system total real Net::HTTP 8.410000 2.400000 10.810000 ( 17.333671) Net::HTTP::Persistent 8,110000 0.880000 8.990000 ( 12.190094)
  45. 45. typhoeus <ul><li>Parallel http requests </li></ul><ul><li>based on libcurl and libcurl-multi </li></ul><ul><li>500ms response, call 20 times </li></ul>www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development user system total real Net::HTTP 0.030000 0.010000 0.040000 ( 10.054327) typhoeus 0.020000 0.070000 0.090000 ( 0.508817)
  46. 46. Q&A Thank you www.ekohe.com Web Development & Graphic Design China Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×