Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Dexador Rises
Lisp Meet Up #31
August 26, 2015
Eitaro Fukamachi
Somewrite Co., Ltd.
I’m Eitaro Fukamachi
@nitro_idiot fukamachi
My Products
• Clack
• Caveman2
• Woo
• quickdocs.org
• Qlot
Contributing
• Roswell
We Work Remotely
We Work Remotely
We’re hiring!
Today,
let me talk about HTTP client.
HTTP client
Client Server
HTTP client
Client Server
HTTP Request
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */*
HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */...
HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */...
HTTP client libraries
• Drakma
• trivial-http
• Carrier
HTTP client libraries
• Drakma
• De facto. Full-featured.
• trivial-http
• Simple. A few features. No SSL.
• Carrier
• Asy...
HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No ...
HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No ...
Drakma
• Ediware
• Since 2006
• Still maintained at GitHub
Mature??? Easy to use??? Fast???
No.
Pitfall 1:
Force URL encoding
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?...
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?...
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?...
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?...
Pitfall 2:
Poor language support
• Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”)
(drakma...
• Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”)
;; => b...
Pitfall 3:
Error handling
• Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-htm...
• Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-htm...
• Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status)
(drakma:http-request “http://cl...
• Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status)
(drakma:http-request “http://cl...
• Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil
(tagbody
retry
(multiple-value-bind (body status)
(drak...
• Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil
(let ((times 5))
(tagbody
retry
(multiple-value-bind (b...
Many pitfalls.
Ridiculous.
We just wanted to
send an HTTP request.
Dexador changes it.
• Full-featured. usocket based.
• Use fast-http, QURI, Babel, cl-cookie
Dexador: Another choice
Dexador: APIs
(dex:get “http://lisp.org/“)
(dex:post “http://lisp.org/“)
(dex:head “http://lisp.org/“)
(dex:put “http://li...
Dexador: Language support
;; Shift_JIS
(dex:get “http://www.google.co.jp/“)
;; EUC-JP
(dex:get “https://mixi.jp/“)
Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-failed (e)
(warn “An HTTP request fa...
Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-forbidden ()
;; for 403 forbidden
)
...
Dexador: Error handling
;; Ignore errors and continue
(handler-bind ((dex:http-request-failed
#'dex:ignore-and-continue))
...
Dexador: Auto-Retrying
;; Auto-retry on 503 error
(handler-bind ((dex:http-request-service-unavailable
#’dex:retry-request...
Dexador: Auto-Retrying
;; Retry only 5 times
(handler-bind ((dex:http-request-service-unavailable
(dex:retry-request 5)))
...
Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals
(handler-bind ((dex:http-request-service-unavailable
(d...
Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals
(handler-bind ((dex:http-request-service-unavailable
(d...
You may ask…
You may ask…
…is it fast?
Benchmark
Sending GET request 30 times to Local Server

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador

w/out ...
Benchmark
Sending GET request 30 times to Local Server

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador

w/out ...
Sending GET request 30 times to Local Server

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador

w/out conneciton...
Sending GET request 30 times to Local Server

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador

w/out conneciton...
How about requesting
over network?
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
How come?
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Differences of Servers/Clients
HTTP Server HTTP Client
C
S
C C
C
S
Can’t help
Network Latency and
bandwidth.
Can skip
connection establishment…?
• Reuse connections once established
(Implicit) Connection-pooling
• Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection
(dex:get “http://lisp.org...
• Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection
(dex:get “http://lisp.org...
Benchmark (again)
Benchmark
Sending GET request 30 times to Local Server

(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador

w/out ...
Benchmark
Sending GET request 30 times to Local Server

(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador

w/out ...
Benchmark
Sending GET request 30 times to Local Server

(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador

w/out ...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
Benchmark
Sending GET request 30 times to Remote Server

(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador

w/out conne...
• It’s pretty common to request to the same
host multiple times
• Can expect some performance
improvements in real applica...
Status
• Still BETA (v0.9.7)
• Stabilizing. More performance
improvements are secondary importance.
• Bug reports are welcome
• T...
Try the new player
and send me a feedback.
Thanks.
EITARO FUKAMACHI
8arrow.org
@nitro_idiot fukamachi
See Also
• Dexador: github.com/fukamachi/dexador
Dexador Rises
Upcoming SlideShare
Loading in …5
×

Dexador Rises

11,724 views

Published on

Talked at Lisp Meet Up #31

Published in: Technology
  • Be the first to comment

Dexador Rises

  1. 1. Dexador Rises Lisp Meet Up #31 August 26, 2015 Eitaro Fukamachi Somewrite Co., Ltd.
  2. 2. I’m Eitaro Fukamachi @nitro_idiot fukamachi
  3. 3. My Products • Clack • Caveman2 • Woo • quickdocs.org • Qlot
  4. 4. Contributing • Roswell
  5. 5. We Work Remotely
  6. 6. We Work Remotely We’re hiring!
  7. 7. Today, let me talk about HTTP client.
  8. 8. HTTP client Client Server
  9. 9. HTTP client Client Server HTTP Request GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*
  10. 10. HTTP client Client Server HTTP Request HTTP Response GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */* HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive
  11. 11. HTTP client Client Server HTTP Request HTTP Response GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */* HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo
  12. 12. HTTP client libraries • Drakma • trivial-http • Carrier
  13. 13. HTTP client libraries • Drakma • De facto. Full-featured. • trivial-http • Simple. A few features. No SSL. • Carrier • Asynchronous.
  14. 14. HTTP client libraries • Drakma (usocket) • De facto. Full-featured. • trivial-http (usocket) • Simple. A few features. No SSL. • Carrier (cl-async/libuv) • Asynchronous.
  15. 15. HTTP client libraries • Drakma (usocket) • De facto. Full-featured. • trivial-http (usocket) • Simple. A few features. No SSL. • Carrier (cl-async/libuv) • Asynchronous. 75 LIBRARIES Required by 3 LIBRARIES None
  16. 16. Drakma • Ediware • Since 2006 • Still maintained at GitHub Mature??? Easy to use??? Fast???
  17. 17. No.
  18. 18. Pitfall 1: Force URL encoding
  19. 19. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” tag)) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => PURI:URI-PARSE-ERROR
  20. 20. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8))) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”
  21. 21. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8))) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp” tag = “AKB48” => OK tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"
  22. 22. • Force URL encoding with PURI Pitfall (1/3) of Drakma (drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)) :preserve-uri t) tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => OK tag = “AKB48” => OK tag = “乃木坂46” => OK
  23. 23. Pitfall 2: Poor language support
  24. 24. • Poor language support with flexi-streams Pitfall (2/3) of Drakma (drakma:http-request “http://www.hatena.ne.jp/”) (drakma:http-request “http://www.google.co.jp/”)
  25. 25. • Poor language support with flexi-streams Pitfall (2/3) of Drakma (drakma:http-request “http://www.hatena.ne.jp/”) ;; => body as UTF-8 string (drakma:http-request “http://www.google.co.jp/”) ;; => body as byte vector WARNING: Problems determining charset (falling back to binary): :SHIFT_JIS is not known to be a name for an external format.
  26. 26. Pitfall 3: Error handling
  27. 27. • Tend to forget error handling Pitfall (3/3) of Drakma (let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)
  28. 28. • Tend to forget error handling Pitfall (3/3) of Drakma (let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …) It fails if the HTTP response code is 4xx or 5xx.
  29. 29. • Tend to forget error handling Pitfall (3/3) of Drakma (multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …)) Raise an error unless the status is not 200
  30. 30. • Tend to forget error handling Pitfall (3/3) of Drakma (multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …)) Raise an error unless the status is not 200 Want to retry???
  31. 31. • Tend to forget error handling Pitfall (3/3) of Drakma (block nil (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (go retry)) (return body)))) With Auto-Retrying
  32. 32. • Tend to forget error handling Pitfall (3/3) of Drakma (block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (go retry)) (return body))))) Retry only 5 times
  33. 33. Many pitfalls.
  34. 34. Ridiculous.
  35. 35. We just wanted to send an HTTP request.
  36. 36. Dexador changes it.
  37. 37. • Full-featured. usocket based. • Use fast-http, QURI, Babel, cl-cookie Dexador: Another choice
  38. 38. Dexador: APIs (dex:get “http://lisp.org/“) (dex:post “http://lisp.org/“) (dex:head “http://lisp.org/“) (dex:put “http://lisp.org/“) (dex:delete “http://lisp.org/“)
  39. 39. Dexador: Language support ;; Shift_JIS (dex:get “http://www.google.co.jp/“) ;; EUC-JP (dex:get “https://mixi.jp/“)
  40. 40. Dexador: Error handling (handler-case (dex:get “http://cliki.net/“) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
  41. 41. Dexador: Error handling (handler-case (dex:get “http://cliki.net/“) (dex:http-request-forbidden () ;; for 403 forbidden ) (dex:http-request-service-unavailable () ;; for 503 service unavailable ) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
  42. 42. Dexador: Error handling ;; Ignore errors and continue (handler-bind ((dex:http-request-failed #'dex:ignore-and-continue)) (dex:get "http://lisp.org"))
  43. 43. Dexador: Auto-Retrying ;; Auto-retry on 503 error (handler-bind ((dex:http-request-service-unavailable #’dex:retry-request)) (dex:get "http://lisp.org"))
  44. 44. Dexador: Auto-Retrying ;; Retry only 5 times (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5))) (dex:get "http://lisp.org"))
  45. 45. Dexador: Auto-Retrying ;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))
  46. 46. Dexador: Auto-Retrying ;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org")) (block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (sleep 3) (go retry)) (return body))))) Dexador Drakma
  47. 47. You may ask…
  48. 48. You may ask… …is it fast?
  49. 49. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.024s
  50. 50. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s
  51. 51. Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s Benchmark x1.8 faster!
  52. 52. Sending GET request 30 times to Local Server
 (lower is better) 0 0.009 0.018 0.026 0.035 Drakma Dexador
 w/out conneciton-pool 0.013s 0.024s Benchmark x1.8 faster!????
  53. 53. How about requesting over network?
  54. 54. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.505s
  55. 55. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s
  56. 56. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s x1.2 faster
  57. 57. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s 0.0036 sec/req
  58. 58. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s 0.0036 sec/req Too trivial improvement…
  59. 59. How come?
  60. 60. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s
  61. 61. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s Network Latency + Connection establishment Network Latency + Connection establishment
  62. 62. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool 0.396s 0.505s Network Latency + Connection establishment Network Latency + Connection establishment The largest bottleneck
  63. 63. Differences of Servers/Clients HTTP Server HTTP Client C S C C C S
  64. 64. Can’t help Network Latency and bandwidth.
  65. 65. Can skip connection establishment…?
  66. 66. • Reuse connections once established (Implicit) Connection-pooling
  67. 67. • Reuse connections once established (Implicit) Connection-pooling ;; Establish a new connection (dex:get “http://lisp.org/index.html“) ;; Reuse the above connection (dex:get “http://lisp.org/index.html“)
  68. 68. • Reuse connections once established (Implicit) Connection-pooling ;; Establish a new connection (dex:get “http://lisp.org/index.html“) ;; Reuse the above connection (dex:get “http://lisp.org/index.html“) 0.727 sec 0.380 sec
  69. 69. Benchmark (again)
  70. 70. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.013s 0.024s
  71. 71. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.005s 0.013s 0.024s
  72. 72. Benchmark Sending GET request 30 times to Local Server
 (lower is better) 0 0.006 0.012 0.018 0.024 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.005s 0.013s 0.024s x4.8 faster!
  73. 73. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.396s 0.505s
  74. 74. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.219s 0.396s 0.505s
  75. 75. Benchmark Sending GET request 30 times to Remote Server
 (lower is better) 0 0.15 0.3 0.45 0.6 Drakma Dexador
 w/out conneciton-pool Dexador
 w/ connection-pool 0.219s 0.396s 0.505s Still x2.3 faster!
  76. 76. • It’s pretty common to request to the same host multiple times • Can expect some performance improvements in real applications In real applications
  77. 77. Status
  78. 78. • Still BETA (v0.9.7) • Stabilizing. More performance improvements are secondary importance. • Bug reports are welcome • Tested with SBCL, CCL, ABCL and ECL
 on Travis CI Status
  79. 79. Try the new player and send me a feedback.
  80. 80. Thanks.
  81. 81. EITARO FUKAMACHI 8arrow.org @nitro_idiot fukamachi
  82. 82. See Also • Dexador: github.com/fukamachi/dexador

×