Dexador Rises

9,545 views

Published on

Talked at Lisp Meet Up #31

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,545
On SlideShare
0
From Embeds
0
Number of Embeds
5,722
Actions
Shares
0
Downloads
7
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Dexador Rises

  1. 1. Dexador Rises Lisp Meet Up #31 August 26, 2015 Eitaro Fukamachi Somewrite Co., Ltd.
  2. 2. I’m Eitaro Fukamachi @nitro_idiot fukamachi
  3. 3. My Products • Clack • Caveman2 • Woo • quickdocs.org • Qlot
  4. 4. Contributing • Roswell
  5. 5. We Work Remotely
  6. 6. We Work Remotely We’re hiring!
  7. 7. Today, let me talk about HTTP client.
  8. 8. HTTP client Client Server
  9. 9. HTTP client Client Server HTTP Request GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*
  10. 10. HTTP client Client Server HTTP Request HTTP Response GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */* HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive
  11. 11. HTTP client Client Server HTTP Request HTTP Response GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */* HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo
  12. 12. HTTP client libraries • Drakma • trivial-http • Carrier
  13. 13. HTTP client libraries • Drakma • De facto. Full-featured. • trivial-http • Simple. A few features. No SSL. • Carrier • Asynchronous.
  14. 14. HTTP client libraries • Drakma (usocket) • De facto. Full-featured. • trivial-http (usocket) • Simple. A few features. No SSL. • Carrier (cl-async/libuv) • Asynchronous.
  15. 15. HTTP client libraries • Drakma (usocket) • De facto. Full-featured. • trivial-http (usocket) • Simple. A few features. No SSL. • Carrier (cl-async/libuv) • Asynchronous. 75 LIBRARIES Required by 3 LIBRARIES None
  16. 16. Drakma • Ediware • Since 2006 • Still maintained at GitHub Mature??? Easy to use??? Fast???
  17. 17. No.
  18. 18. Pitfall 1: Force URL encoding
  19. 19. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” tag)) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => PURI:URI-PARSE-ERROR
  20. 20. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8))) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”
  21. 21. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8))) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp” tag = “AKB48” => OK tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"
  22. 22. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)) :preserve-uri t) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => OK tag = “AKB48” => OK tag = “乃木坂46” => OK
  23. 23. Pitfall 2: Poor language support
  24. 24. • Poor language support with flexi-streams Pitfall (2/3) of Drakma (drakma:http-request “http://www.hatena.ne.jp/”) (drakma:http-request “http://www.google.co.jp/”)
  25. 25. • Poor language support with flexi-streams Pitfall (2/3) of Drakma (drakma:http-request “http://www.hatena.ne.jp/”) ;; => body as UTF-8 string (drakma:http-request “http://www.google.co.jp/”) ;; => body as byte vector WARNING: Problems determining charset (falling back to binary): :SHIFT_JIS is not known to be a name for an external format.
  26. 26. Pitfall 3: Error handling
  27. 27. • Tend to forget error handling Pitfall (3/3) of Drakma (let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)
  28. 28. • Tend to forget error handling Pitfall (3/3) of Drakma (let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …) It fails if the HTTP response code is 4xx or 5xx.
  29. 29. • Tend to forget error handling Pitfall (3/3) of Drakma (multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …)) Raise an error unless the status is not 200
  30. 30. • Tend to forget error handling Pitfall (3/3) of Drakma (multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …)) Raise an error unless the status is not 200 Want to retry???
  31. 31. • Tend to forget error handling Pitfall (3/3) of Drakma (block nil (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (go retry)) (return body)))) With Auto-Retrying
  32. 32. • Tend to forget error handling Pitfall (3/3) of Drakma (block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (go retry)) (return body))))) Retry only 5 times
  33. 33. Many pitfalls.
  34. 34. Ridiculous.
  35. 35. We just wanted to send an HTTP request.
  36. 36. Dexador changes it.
  37. 37. • Full-featured. usocket based. • Use fast-http, QURI, Babel, cl-cookie Dexador: Another choice
  38. 38. Dexador: APIs (dex:get “http://lisp.org/“) (dex:post “http://lisp.org/“) (dex:head “http://lisp.org/“) (dex:put “http://lisp.org/“) (dex:delete “http://lisp.org/“)
  39. 39. Dexador: Language support ;; Shift_JIS (dex:get “http://www.google.co.jp/“) ;; EUC-JP (dex:get “https://mixi.jp/“)
  40. 40. Dexador: Error handling (handler-case (dex:get “http://cliki.net/“) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
  41. 41. Dexador: Error handling (handler-case (dex:get “http://cliki.net/“) (dex:http-request-forbidden () ;; for 403 forbidden ) (dex:http-request-service-unavailable () ;; for 503 service unavailable ) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
  42. 42. Dexador: Error handling ;; Ignore errors and continue (handler-bind ((dex:http-request-failed #'dex:ignore-and-continue)) (dex:get "http://lisp.org"))
  43. 43. Dexador: Auto-Retrying ;; Auto-retry on 503 error (handler-bind ((dex:http-request-service-unavailable #’dex:retry-request)) (dex:get "http://lisp.org"))
  44. 44. Dexador: Auto-Retrying ;; Retry only 5 times (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5))) (dex:get "http://lisp.org"))
  45. 45. Dexador: Auto-Retrying ;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))
  46. 46. Dexador: Auto-Retrying ;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org")) (block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (sleep 3) (go retry)) (return body))))) Dexador Drakma
  47. 47. You may ask…
  48. 48. You may ask… …is it fast?
  49. 49. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.024s
  50. 50. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s
  51. 51. Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s Benchmark x1.8 faster!
  52. 52. Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s Benchmark x1.8 faster!????
  53. 53. How about requesting over network?
  54. 54. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.505s
  55. 55. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s
  56. 56. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s x1.2 faster
  57. 57. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s 0.0036 sec/req
  58. 58. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s 0.0036 sec/req Too trivial improvement…
  59. 59. How come?
  60. 60. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s
  61. 61. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s Network Latency + Connection establishment Network Latency + Connection establishment
  62. 62. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s Network Latency + Connection establishment Network Latency + Connection establishment The largest bottleneck
  63. 63. Differences of Servers/Clients HTTP Server HTTP Client C S C C C S
  64. 64. Can’t help Network Latency and bandwidth.
  65. 65. Can skip connection establishment…?
  66. 66. • Reuse connections once established (Implicit) Connection-pooling
  67. 67. • Reuse connections once established (Implicit) Connection-pooling ;; Establish a new connection (dex:get “http://lisp.org/index.html“) ;; Reuse the above connection (dex:get “http://lisp.org/index.html“)
  68. 68. • Reuse connections once established (Implicit) Connection-pooling ;; Establish a new connection (dex:get “http://lisp.org/index.html“) ;; Reuse the above connection (dex:get “http://lisp.org/index.html“) 0.727 sec 0.380 sec
  69. 69. Benchmark (again)
  70. 70. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.013s 0.024s
  71. 71. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.005s 0.013s 0.024s
  72. 72. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.005s 0.013s 0.024s x4.8 faster!
  73. 73. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.396s 0.505s
  74. 74. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.219s 0.396s 0.505s
  75. 75. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.219s 0.396s 0.505s Still x2.3 faster!
  76. 76. • It’s pretty common to request to the same host multiple times • Can expect some performance improvements in real applications In real applications
  77. 77. Status
  78. 78. • Still BETA (v0.9.7) • Stabilizing. More performance improvements are secondary importance. • Bug reports are welcome • Tested with SBCL, CCL, ABCL and ECL
 on Travis CI Status
  79. 79. Try the new player and send me a feedback.
  80. 80. Thanks.
  81. 81. EITARO FUKAMACHI 8arrow.org @nitro_idiot fukamachi
  82. 82. See Also • Dexador: github.com/fukamachi/dexador

×