Talked at Lisp Meet Up #31

1. Dexador Rises
Lisp Meet Up #31
August 26, 2015
Eitaro Fukamachi
Somewrite Co., Ltd.

2. I’m Eitaro Fukamachi
@nitro_idiot fukamachi

3. My Products
• Clack
• Caveman2
• Woo
• quickdocs.org
• Qlot

4. Contributing
• Roswell

6. We Work Remotely

7. We Work Remotely
We’re hiring!

8. Today,
let me talk about HTTP client.

9. HTTP client
Client Server

10. HTTP client
Client Server
HTTP Request
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */*

11. HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */*
HTTP/1.1 200 OK
Date: Mon, 24 Aug 2015 00:41:36 GMT
Content-Type: text/html
Content-Length: 3771
Connection: keep-alive

12. HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */*
HTTP/1.1 200 OK
Date: Mon, 24 Aug 2015 00:41:36 GMT
Content-Type: text/html
Content-Length: 3771
Connection: keep-alive
ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo

13. HTTP client libraries
• Drakma
• trivial-http
• Carrier

14. HTTP client libraries
• Drakma
• De facto. Full-featured.
• trivial-http
• Simple. A few features. No SSL.
• Carrier
• Asynchronous.

15. HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No SSL.
• Carrier (cl-async/libuv)
• Asynchronous.

16. HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No SSL.
• Carrier (cl-async/libuv)
• Asynchronous.
75
LIBRARIES
Required by
3
LIBRARIES
None

17. Drakma
• Ediware
• Since 2006
• Still maintained at GitHub
Mature??? Easy to use??? Fast???

18. No.

19. Pitfall 1:
Force URL encoding

20. • Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?q=~A”
tag))
tag = “lisp” => OK
tag = “scheme” => OK
tag = “clojure” => OK
tag = “common lisp” => PURI:URI-PARSE-ERROR

21. • Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?q=~A”
(drakma:url-encode tag :utf-8)))
tag = “lisp” => OK
tag = “scheme” => OK
tag = “clojure” => OK
tag = “common lisp” => “common+lisp”

22. • Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?q=~A”
(drakma:url-encode tag :utf-8)))
tag = “lisp” => OK
tag = “scheme” => OK
tag = “clojure” => OK
tag = “common lisp” => “common+lisp”
tag = “AKB48” => OK
tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"

23. • Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?q=~A”
(drakma:url-encode tag :utf-8))
:preserve-uri t)
tag = “lisp” => OK
tag = “scheme” => OK
tag = “clojure” => OK
tag = “common lisp” => OK
tag = “AKB48” => OK
tag = “乃木坂46” => OK

24. Pitfall 2:
Poor language support

25. • Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”)
(drakma:http-request “http://www.google.co.jp/”)

26. • Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”)
;; => body as UTF-8 string
(drakma:http-request “http://www.google.co.jp/”)
;; => body as byte vector
WARNING: Problems determining charset (falling back to binary):
:SHIFT_JIS is not known to be a name for an external format.

27. Pitfall 3:
Error handling

28. • Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-html (plump:parse body)))
…)

29. • Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-html (plump:parse body)))
…)
It fails if the HTTP response code is 4xx or 5xx.

30. • Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status)
(drakma:http-request “http://cliki.net”)
(unless (= status 200)
(error “An HTTP request failed (Code=~D)”
status))
(let ((parsed-html (plump:parse body)))
…))
Raise an error unless the status is not 200

31. • Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status)
(drakma:http-request “http://cliki.net”)
(unless (= status 200)
(error “An HTTP request failed (Code=~D)”
status))
(let ((parsed-html (plump:parse body)))
…))
Raise an error unless the status is not 200
Want to retry???

32. • Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil
(tagbody
retry
(multiple-value-bind (body status)
(drakma:http-request "http://cliki.net/")
(unless (= status 200)
(go retry))
(return body))))
With Auto-Retrying

33. • Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil
(let ((times 5))
(tagbody
retry
(multiple-value-bind (body status)
(drakma:http-request "http://cliki.net/")
(unless (= status 200)
(when (= times 0)
(error "An HTTP request failed. (Code=~D)”
status))
(decf times)
(go retry))
(return body)))))
Retry only 5 times

34. Many pitfalls.

35. Ridiculous.

36. We just wanted to
send an HTTP request.

37. Dexador changes it.

38. • Full-featured. usocket based.
• Use fast-http, QURI, Babel, cl-cookie
Dexador: Another choice

39. Dexador: APIs
(dex:get “http://lisp.org/“)
(dex:post “http://lisp.org/“)
(dex:head “http://lisp.org/“)
(dex:put “http://lisp.org/“)
(dex:delete “http://lisp.org/“)

40. Dexador: Language support
;; Shift_JIS
(dex:get “http://www.google.co.jp/“)
;; EUC-JP
(dex:get “https://mixi.jp/“)

41. Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-failed (e)
(warn “An HTTP request failed (Code=~D)”
(dex:response-status e))))

42. Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-forbidden ()
;; for 403 forbidden
)
(dex:http-request-service-unavailable ()
;; for 503 service unavailable
)
(dex:http-request-failed (e)
(warn “An HTTP request failed (Code=~D)”
(dex:response-status e))))

43. Dexador: Error handling
;; Ignore errors and continue
(handler-bind ((dex:http-request-failed
#'dex:ignore-and-continue))
(dex:get "http://lisp.org"))

44. Dexador: Auto-Retrying
;; Auto-retry on 503 error
(handler-bind ((dex:http-request-service-unavailable
#’dex:retry-request))
(dex:get "http://lisp.org"))

45. Dexador: Auto-Retrying
;; Retry only 5 times
(handler-bind ((dex:http-request-service-unavailable
(dex:retry-request 5)))
(dex:get "http://lisp.org"))

46. Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals
(handler-bind ((dex:http-request-service-unavailable
(dex:retry-request 5 :interval 3)))
(dex:get "http://lisp.org"))

47. Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals
(handler-bind ((dex:http-request-service-unavailable
(dex:retry-request 5 :interval 3)))
(dex:get "http://lisp.org"))
(block nil
(let ((times 5))
(tagbody
retry
(multiple-value-bind (body status)
(drakma:http-request "http://cliki.net/")
(unless (= status 200)
(when (= times 0)
(error "An HTTP request failed. (Code=~D)”
status))
(decf times)
(sleep 3)
(go retry))
(return body)))))
Dexador
Drakma

48. You may ask…

49. You may ask…
…is it fast?

50. Benchmark
Sending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador
w/out conneciton-pool
0.024s

51. Benchmark
Sending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador
w/out conneciton-pool
0.013s
0.024s

52. Sending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador
w/out conneciton-pool
0.013s
0.024s
Benchmark
x1.8 faster!

53. Sending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador
w/out conneciton-pool
0.013s
0.024s
Benchmark
x1.8 faster!????

54. How about requesting
over network?

55. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.505s

56. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s

57. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s x1.2 faster

58. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s 0.0036 sec/req

59. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s 0.0036 sec/req
Too trivial improvement…

60. How come?

61. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s

62. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s
Network Latency
+
Connection
establishment
Network Latency
+
Connection
establishment

63. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
0.396s
0.505s
Network Latency
+
Connection
establishment
Network Latency
+
Connection
establishment
The largest bottleneck

64. Differences of Servers/Clients
HTTP Server HTTP Client
C
S
C C
C
S

65. Can’t help
Network Latency and
bandwidth.

66. Can skip
connection establishment…?

67. • Reuse connections once established
(Implicit) Connection-pooling

68. • Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection
(dex:get “http://lisp.org/index.html“)
;; Reuse the above connection
(dex:get “http://lisp.org/index.html“)

69. • Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection
(dex:get “http://lisp.org/index.html“)
;; Reuse the above connection
(dex:get “http://lisp.org/index.html“)
0.727 sec
0.380 sec

70. Benchmark (again)

71. Benchmark
Sending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.013s
0.024s

72. Benchmark
Sending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.005s
0.013s
0.024s

73. Benchmark
Sending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.005s
0.013s
0.024s
x4.8 faster!

74. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.396s
0.505s

75. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.219s
0.396s
0.505s

76. Benchmark
Sending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador
w/out conneciton-pool
Dexador
w/ connection-pool
0.219s
0.396s
0.505s Still x2.3 faster!

77. • It’s pretty common to request to the same
host multiple times
• Can expect some performance
improvements in real applications
In real applications

78. Status

79. • Still BETA (v0.9.7)
• Stabilizing. More performance
improvements are secondary importance.
• Bug reports are welcome
• Tested with SBCL, CCL, ABCL and ECL
on Travis CI
Status

80. Try the new player
and send me a feedback.

81. Thanks.

82. EITARO FUKAMACHI
8arrow.org
@nitro_idiot fukamachi

83. See Also
• Dexador: github.com/fukamachi/dexador