Dexador Rises

Dexador Rises
Lisp Meet Up #31
August 26, 2015
Eitaro Fukamachi
Somewrite Co., Ltd.

I’m Eitaro Fukamachi
@nitro_idiot fukamachi

My Products
• Clack
• Caveman2
• Woo
• quickdocs.org
• Qlot

We Work Remotely
We’re hiring!

Today,
let me talk about HTTP client.

HTTP client
Client Server
HTTP Request
GET / HTTP/1.1
Host: quickdocs.org
User-Agent: curl/7.43.0
Accept: */*

HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
Accept: */*
HTTP/1.1 200 OK
Date: Mon, 24 Aug 2015 00:41:36 GMT
Content-Type: text/html
Content-Length: 3771
Connection: keep-alive

HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1
Host: quickdocs.org
Accept: */*
HTTP/1.1 200 OK
Date: Mon, 24 Aug 2015 00:41:36 GMT
Content-Type: text/html
Content-Length: 3771
Connection: keep-alive
ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo

HTTP client libraries
• Drakma
• trivial-http
• Carrier

• Drakma
• De facto. Full-featured.
• trivial-http
• Simple. A few features. No SSL.
• Carrier
• Asynchronous.

• Drakma (usocket)
• trivial-http (usocket)
• Carrier (cl-async/libuv)
• Asynchronous.

• Drakma (usocket)
• trivial-http (usocket)
• Carrier (cl-async/libuv)
• Asynchronous.
75
LIBRARIES
Required by
3
LIBRARIES
None

Drakma
• Ediware
• Since 2006
• Still maintained at GitHub
Mature??? Easy to use??? Fast???

• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request
(format nil “http://b.hatena.ne.jp/search/tag?q=~A”
tag))
tag = “lisp” => OK
tag = “scheme” => OK
tag = “clojure” => OK
tag = “common lisp” => PURI:URI-PARSE-ERROR

(drakma:url-encode tag :utf-8)))
tag = “common lisp” => “common+lisp”

(drakma:url-encode tag :utf-8)))
tag = “common lisp” => “common+lisp”
tag = “AKB48” => OK
tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"

(drakma:url-encode tag :utf-8))
:preserve-uri t)
tag = “common lisp” => OK
tag = “AKB48” => OK
tag = “乃木坂46” => OK

Pitfall 2:
Poor language support

• Poor language support with ﬂexi-streams
(drakma:http-request “http://www.hatena.ne.jp/”)
(drakma:http-request “http://www.google.co.jp/”)

• Poor language support with ﬂexi-streams
(drakma:http-request “http://www.hatena.ne.jp/”)
;; => body as UTF-8 string
(drakma:http-request “http://www.google.co.jp/”)
;; => body as byte vector
WARNING: Problems determining charset (falling back to binary):
:SHIFT_JIS is not known to be a name for an external format.

• Tend to forget error handling
(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-html (plump:parse body)))
…)

(let* ((body (drakma:http-request “http://cliki.net”))
(parsed-html (plump:parse body)))
…)
It fails if the HTTP response code is 4xx or 5xx.

(multiple-value-bind (body status)
(drakma:http-request “http://cliki.net”)
(unless (= status 200)
(error “An HTTP request failed (Code=~D)”
status))
(let ((parsed-html (plump:parse body)))
…))
Raise an error unless the status is not 200

(drakma:http-request “http://cliki.net”)
(error “An HTTP request failed (Code=~D)”
status))
(let ((parsed-html (plump:parse body)))
…))
Raise an error unless the status is not 200
Want to retry???

(block nil
(tagbody
retry
(drakma:http-request "http://cliki.net/")
(go retry))
(return body))))
With Auto-Retrying

(block nil
(let ((times 5))
(tagbody
retry
(when (= times 0)
(error "An HTTP request failed. (Code=~D)”
status))
(decf times)
(go retry))
(return body)))))
Retry only 5 times

We just wanted to
send an HTTP request.

• Full-featured. usocket based.
• Use fast-http, QURI, Babel, cl-cookie
Dexador: Another choice

Dexador: APIs
(dex:get “http://lisp.org/“)
(dex:post “http://lisp.org/“)
(dex:head “http://lisp.org/“)
(dex:put “http://lisp.org/“)
(dex:delete “http://lisp.org/“)

Dexador: Language support
;; Shift_JIS
(dex:get “http://www.google.co.jp/“)
;; EUC-JP
(dex:get “https://mixi.jp/“)

Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-failed (e)
(warn “An HTTP request failed (Code=~D)”
(dex:response-status e))))

(handler-case (dex:get “http://cliki.net/“)
(dex:http-request-forbidden ()
;; for 403 forbidden
)
(dex:http-request-service-unavailable ()
;; for 503 service unavailable
)
(dex:http-request-failed (e)
(warn “An HTTP request failed (Code=~D)”
(dex:response-status e))))

;; Ignore errors and continue
(handler-bind ((dex:http-request-failed
#'dex:ignore-and-continue))
(dex:get "http://lisp.org"))

Dexador: Auto-Retrying
;; Auto-retry on 503 error
(handler-bind ((dex:http-request-service-unavailable
#’dex:retry-request))

;; Retry only 5 times
(dex:retry-request 5)))

;; Retry only 5 times at 3-second intervals
(dex:retry-request 5 :interval 3)))

;; Retry only 5 times at 3-second intervals
(dex:retry-request 5 :interval 3)))
(block nil
(let ((times 5))
(tagbody
retry
(when (= times 0)
(error "An HTTP request failed. (Code=~D)”
status))
(decf times)
(sleep 3)
(go retry))
(return body)))))
Dexador
Drakma

Benchmark
Sending GET request 30 times to Local Server 
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador 
w/out conneciton-pool
0.024s

Benchmark
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador 
0.013s
0.024s

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador 
0.013s
0.024s
Benchmark
x1.8 faster!

(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexador 
0.013s
0.024s
Benchmark
x1.8 faster!????

How about requesting
over network?

Benchmark
Sending GET request 30 times to Remote Server 
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.505s

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s x1.2 faster

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s 0.0036 sec/req

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s 0.0036 sec/req
Too trivial improvement…

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s
Network Latency
+
Connection
establishment
Network Latency
+
Connection
establishment

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
0.396s
0.505s
Network Latency
+
Connection
establishment
Network Latency
+
Connection
establishment
The largest bottleneck

Diﬀerences of Servers/Clients
HTTP Server HTTP Client
C
S
C C
C
S

Can’t help
Network Latency and
bandwidth.

Can skip
connection establishment…?

• Reuse connections once established
(Implicit) Connection-pooling

;; Establish a new connection
(dex:get “http://lisp.org/index.html“)
;; Reuse the above connection

;; Establish a new connection
;; Reuse the above connection
0.727 sec
0.380 sec

Benchmark
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador 
Dexador 
w/ connection-pool
0.013s
0.024s

Benchmark
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador 
Dexador 
w/ connection-pool
0.005s
0.013s
0.024s

Benchmark
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexador 
Dexador 
w/ connection-pool
0.005s
0.013s
0.024s
x4.8 faster!

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
Dexador 
w/ connection-pool
0.396s
0.505s

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
Dexador 
w/ connection-pool
0.219s
0.396s
0.505s

Benchmark
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexador 
Dexador 
w/ connection-pool
0.219s
0.396s
0.505s Still x2.3 faster!

• It’s pretty common to request to the same
host multiple times
• Can expect some performance
improvements in real applications
In real applications

• Still BETA (v0.9.7)
• Stabilizing. More performance
improvements are secondary importance.
• Bug reports are welcome
• Tested with SBCL, CCL, ABCL and ECL 
on Travis CI
Status

Try the new player
and send me a feedback.

EITARO FUKAMACHI
8arrow.org
@nitro_idiot fukamachi

See Also
• Dexador: github.com/fukamachi/dexador

Dexador Rises

More Related Content

What's hot

Similar to Dexador Rises

More from fukamachi

Recently uploaded

Dexador Rises