The Handlersocket plugin for MySQL provides a non-SQL interface to execute simple CRUD operations rapidly, aiming to enhance performance by omitting SQL parsing and managing multiple requests on the server-side. It supports various storage engines, offers a unique client-server protocol, and demonstrates significant performance improvements over traditional MySQL operations, particularly in handling read queries. Future developments are planned to include enhanced SQL support and atomic operations.

1. HandlerSocket plugin for
MySQL
Jun 29, 2010 DeNA Technology Seminar @ Yoyogi
IT Platform Dept., System Management Division
DeNA Co.,Ltd.
Akira Higuchi <higuchi dot akira at dena dot jp>

2. Who am I?
 Akira Higuchi, Ph.D. in science
 IT Platform Dept., DeNA Co.,Ltd.
 system-wide performance optimization
 middleware development
 The creator of HandlerSocket plugin
 Using GNU/Linux since 1993
 Fedora: yum install KoboDeluxe
 Debian: apt-get install kobodeluxe

3. About HandlerSocket plugin

4. What is HandlerSocket?
 Non-SQL interface for MySQL

5. What HandlerSocket aims
 Executes simple CRUD operations fast
 Omit SQL parsing
 Combine multiple requests on the server
side
 Allows SQL on the same database
 Only simple operations can be faster
 Seamless migration from SQL queries

6. HandlerSocket plugin
 Offers a direct and non-SQL interface to MySQL
storage engines
 Own TCP/IP listener
 Talks a text protocol
 There is a C++ and a Perl client libraries
 Only works with Linux
 The source code is here:
 https://github.com/ahiguti/HandlerSocket-Plugin-for-
MySQL
 More infos on the DeNA Tech Blog
 http://engineer.dena.jp/ (in Japanese)

7. Construction
Handler Interface
Innodb MyISAM Other storage engines …
SQL Layer Handlersocket Plugin
Listener for libmysql
libmysql libhsclient
Applications
mysqld
client app

8. Other NoSQL interfaces to
MySQL
 mycached
 http://developer.cybozu.co.jp/kazuho/2009
/08/mycached-memcac.html
 Works with any storage engines
 Talks the memcached protocol
 NDB API
 http://dev.mysql.com/doc/ndbapi/en/index
.html
 Dedicated for the ndbcluster engine

9. Performance

10. Performance
241009
159407
60191
15771
0 50000 100000 150000 200000 250000 300000
1 column
50 columns
(requests/sec)
handlersocket
libmysql
 Handlersocket executes simple read queries 4x
faster than mysqld/libmysql
 Very effective when many columns are retrieved
 The reason is described later

11. Commands supported by
HandlerSocket (for reading data)
 In pseudo-SQL...
SELECT f1, .. , fn FROM db.table
WHERE k1, ... , km = v1, ... , vm
ORDER BY index_i LIMIT offset, limit
 (k1, ... , km) are the key fields (or a
prefix) of the index_i
 =, >=, >, <=, and < can be used for a
comparator

12. Commands supported by
HandlerSocket (for modifying
data)
 UPDATE, DELETE, and INSERT
 Does not support transactions
 Modifications are recorded to the
binary log in the row-based format
 Modifications are durable

13. Command example
 create table db1.table1 (k int key, v char(20))
 insert into db1.table1 values (234, 'foo'), (678, ‘bar’)
$ telnet localhost 9998
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
P 0 db1 table1 PRIMARY k,v
0 1
0 = 1 234
0 2 234 foo
0 = 1 678
0 2 678 bar
opens the PK
find k = 234
find k = 678

14. Why fast?
 No SQL parsing
low CPU usage
 Executes multiple requests in bulk
low CPU/Disk usage
 Own client/server protocol
small network transmission size

15. Eliminating CPU consumption

16. oprofile results –
libmysql/mysqld
 Executes “SELECT v from table where k = ?” many
times
samples| %|
------------------
9669940 53.1574 mysqld
4438098 24.3970 vmlinux
1835976 10.0927 libpthread-2.5.so
1680656 9.2389 libc-2.5.so
397970 2.1877 e1000e
89136 0.4900 oprofiled
42881 0.2357 oprofile

17. oprofile results –
libmysql/mysqld
samples % symbol name
748022 7.7355 MYSQLparse(void*)
219702 2.2720 my_pthread_fastmutex_lock
205606 2.1262 make_join_statistics(JOIN*, TABLE_LIST*,
198234 2.0500 btr_search_guess_on_hash
180731 1.8690 JOIN::optimize()
177120 1.8317 row_search_for_mysql
171185 1.7703 lex_one_token(void*, void*)
162683 1.6824 alloc_root
131823 1.3632 read_view_open_now
122795 1.2699 mysql_select(THD*, Item***, TABLE_LIST*,
100276 1.0370 open_table(THD*, TABLE_LIST*, st_mem_root*,
99575 1.0297 mem_pool_fill_free_list
96434 0.9973 build_template(row_prebuilt_struct*, THD*,
86349 0.8930 get_hash_symbol(char const*, unsigned int,
 CPU usage inside mysqld

18. oprofile results –
libmysql/mysqld
samples % symbol name
204393 4.6054 schedule
118648 2.6734 tcp_sendmsg
115832 2.6099 tcp_recvmsg
106537 2.4005 tcp_v4_rcv
103915 2.3414 tcp_ack
103534 2.3328 system_call
93864 2.1150 dev_queue_xmit
86831 1.9565 __mod_timer
85891 1.9353 tcp_rcv_established
84083 1.8946 .text.task_rq_lock
 CPU usage inside the Linux kernel

19. oprofile results –
libmysql/mysqld
 libmysql/mysqld
 Much CPU time spent in mysqld
 Parsing SQL is slow
 schedule() is called frequently

20. oprofile results –
HandlerSocket
samples| %|
------------------
1919039 51.0453 vmlinux
811998 21.5987 mysqld
421215 11.2041 libpthread-2.5.so
207166 5.5105 e1000e
191566 5.0955 handlersocket.so
188618 5.0171 libc-2.5.so
13622 0.3623 oprofiled
5707 0.1518 oprofile
 CPU usage inside MySQL with HandlerSocket

21. oprofile results –
HandlerSocket
samples % symbol name
119684 14.7394 btr_search_guess_on_hash
58202 7.1678 row_search_for_mysql
46946 5.7815 mutex_delay
38617 4.7558 my_pthread_fastmutex_lock
37707 4.6437 buf_page_get_known_nowait
36528 4.4985 rec_get_offsets_func
34625 4.2642 build_template(row_prebuilt_struct*, THD*, TABLE*,
20024 2.4660 row_sel_store_mysql_rec
19347 2.3826 btr_cur_search_to_nth_level
16701 2.0568 row_sel_convert_mysql_key_to_innobase
13343 1.6432 cmp_dtuple_rec_with_match
11381 1.4016 ha_innobase::index_read(unsigned char*,
11176 1.3764 dict_index_copy_types
10762 1.3254 mtr_memo_slot_release
10734 1.3219 ha_innobase::init_table_handle_for_HANDLER()
 CPU consumption in mysqld

22. oprofile results –
HandlerSocket
samples % symbol name
129038 6.7241 tcp_sendmsg
80080 4.1729 tcp_v4_rcv
69658 3.6298 dev_queue_xmit
66171 3.4481 .text.skb_release_data
63316 3.2994 __qdisc_run
60279 3.1411 tcp_recvmsg
59703 3.1111 ip_output
58462 3.0464 .text.skb_release_head_state
48876 2.5469 tcp_ack
48733 2.5394 __alloc_skb
45660 2.3793 ip_queue_xmit
44671 2.3278 tcp_transmit_skb
 CPU consumption in the Linux kernel

23. oprofile results –
HandlerSocket
 HandlerSocket
 Most CPU time is consumed in the kernel
 schedule() is not called frequently
 Inside mysqld, innodb eats most CPU
time

24. Executing multiple requests
in bulk

25. Threading model
mysqld:
 Thread per connection (MySQL 5)
 Thread pooling (MySQL 6?)

26. Threading model
HandlerSocket:
 Small number of threads
 Many connections per thread
 Uses epoll()
 Virtually unlimited number of concurrent
connections
 Small memory footprint

27. HandlerSocket reader thread
reads requests from many clients
locks the DB, gets a read view
executes many requests
unlocks the DB
returns responses to clients
locks/unlocks
(1/#conns)
times per
request
handlersocket reader thread

28. HandlerSocket writer thread
reads requests from many clients
locks the DB, begins a transaction
executes multiple requests
commits, and unlocks the DB
returns responses to clients
handlersocket writer thread
executes multiple ops
in a single transaction

29. Write throughput
 Condition:
 Durable write
 sync_binlog = 1
 innodb_flush_log_at_trx_commit = 1
 innodb_support_xa = 1
 Write-back cache with BBU, or SSD
 Throughput:
 MySQL: up to 1000 qps
 HandlerSocket: up to 30000 qps

30. How HandlerSocket locks tables
 MyISAM:
 Shared-exclusive lock
 InnoDB:
 Reader threads don’t block
 Only one writer thread can be executed at the
same time
 HandlerSocket requests are deadlock-free
 Only simple operations are supported

31. Client/server protocol

32. MySQL C/S protocol
write(3, "L0003select column0,column1,column2,column3,column4 from
db_1.table_1 where k=15", 80) = 80
read(3,
"100100560023def4db_17table_17table_17column07column0fr
0<00037520000000060033def4db_17table_17table_17column
17column1fr0<00037520000000060043def4db_17table_17t
able_17column27column2fr0<00037520000000060053def4d
b_17table_17table_17column37column3fr0<0003752000000006
0063def4db_17table_17table_17column47column4fr0<000375
2000000500737600"0n0010001000110012001300145
00t37600"0", 16384) = 327
when the above query is executed...
SELECT column0, column1, column2, column3, column4
FROM db_1.table_1 where k = 15

33. HandlerSocket C/S protocol
write(3, "1t=t1t15n", 9) = 9
read(3, "0t5t0t1t2t3t4n", 8192) = 14
when an equivalent query is executed using
handlersocket...
libmysql handlersocket
request 80 bytes 9 bytes
response 327 bytes 14 bytes

34. MySQL C/S protocol
 The strace result shows that MySQL C/S
protocol is verbose
 Result-set metadata
http://forge.mysql.com/wiki/MySQL_Internals_ClientServe
r_Protocol#Field_Packet
 Result-set metadata become very large if a
result-set has many columns
 Neither a HANDLER statement nor a server-
side prepared statement does not help to
avoid this problem

35. Client libraries

36. libhsclient
 Client library for C++

37. Net::HandlerSocket
 Client library for Perl
 Invokes libhsclient via XS
my $cli = new Net::HandlerSocket(
{host => ‘localhost’, port => 9999});
$cli->open_index(1, ‘db1’, ‘table1’, ‘PRIMARY’, ‘k,v’);
my $res = $cli->exec_multi([
[ 1, ‘=‘, [ ’33’ ], 1, 0 ],
[ 1, ‘=‘, [ ’44’ ], 1, 0, ‘U’, [ ’44’, ‘hoge’ ] ],
[ 1, ‘>=‘, [ ’55’ ], 10, 20 ],
]);

38. Configuration hints

39. HandlerSocket configuration
options
 handlersocket_threads = 16
 Number of reader threads
 Recommended value is the number of logical
CPU
 handlersocket_thread_wr = 1
 Number of writer threads
 Recommended value is ... 1
 handlersocket_port = 9998
 Listening port for reader requests
 handlersocket_port_wr = 9999
 Listening port for writer requests

40. Other configuration options
 innodb_buffer_pool_size
 As large as possible
 innodb_log_file_size,
innodb_log_files_in_group
 As large as possible
 innodb_thread_concurrency = 0
 open_files_limit = 65535
 Number of file descriptors mysqld can open
 HandlerSocket can handle up to 65000
concurrent connections

41. Other configuration options
 innodb_adaptive_hash_index = 1
 Adaptive has index is fast, but consume
memory

42. Options related to durability
 sync_binlog = 1
 innodb_flush_log_at_trx_commit = 1
 innodb_support_xa = 1

43. Benchmark results

44. Benchmark
 Server:
 Core2Quad Q6600
 CentOS 5.4
 Single EXPI9301CT(e1000e)
 Single Intel X25-E (write-back cache disabled)
 Schema:
 CREATE TABLE table1 (k varchar(32) KEY, v varchar(32)) engine =
INNODB;
 Read benchmark:
 10000000 records
 SELECT v from table1 where k = ?
 Random access
 Write benchmark:
 10000000 records
 UPDATE table SET v = ? where k = ?
 Random access
 Durable write
 sync_binlog = 1
 innodb_flush_log_at_trx_commit = 1
 innodb_support_xa = 1

45. Throughput (reads)
0
50000
100000
150000
200000
250000
300000 1
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
# of concurrent connections
queries
per
sec
handlersocket read
mysql read
handlersocket write
mysql write

46. Throughput (writes)
1
10
100
1000
10000
100000
1
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
# of concurrent connections
queries
per
sec
handlersocket write
mysql write

47. Maximum response time
0
10
20
30
40
50
60
1
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
# of concurrent connections
max
response
time(sec)
handlersocket read
mysql read
handlersocket write
mysql write

48. Average response time
0
1
2
3
4
5
6
1
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
# of concurrent connections
average
response
time(sec)
handlersocket read
mysql read
handlersocket write
mysql write

49. Issues and future plans

50. Issues
 Difficult to build
 Requires the source of mysql
 MySQL binary compatibility?

51. Future plans
 ‘where’ clause
 Atomic read-modify-write operations
 SQL support?
 More language bindings