MySQL-Proxy

1. MySQL Proxy
Making MySQL more flexible
Jan Kneschke
jan@mysql.com

2. MySQL Proxy
●
proxy­servers forward requests to backends
and can
– transform, handle or block them
●
released under the GPL
– see http://forge.mysql.com/wiki/MySQL_Proxy
●
developed as part of the Enterprise Tools since
February 2007

3. Design Decisions
●
goal is to be transparent to the application layer
●
supports all platforms and languages
●
designed to handle thousands of parallel
connections (c10k)
●
uses a embedded scripting language for
customizations

4. Transparency
●
SHOW WARNINGS can be worked around with
Query Injection
●
SELECT USER() shows the connected user
(the proxy, not the client) which can be
corrected with result­set rewriting
●
host auth against the MySQL server

5. Latency
●
early tests via localhost
●
same script run directly and through the proxy
●
latency per mysql­packet: 0.4ms
●
ping RTT on 1Gbit: 0.1ms

6. Load Balancing
●
load balancing distributes the load across
several slaves
●
Shortest Queue First is default
– send new connections to the server with the least
number of open connections

7. Fail Over
●
dead host are detected
●
taking out of load balancing for 2min
●
uses custom load balancers to decide how to
handle a dead host
– hot + standby
– uses load balancing

8. Removing SPoF
●
one Proxy == Single Point of Failure
●
use external Heartbeat (linuxha.org) or
●
2 LB proxies + 1 Host Selecting Proxy per
application server

9. Failsafe Load Balancing

10. Flexibility
●
proxy embeds LUA
●
allows analyzing and manipulating packets
– Inspection
– Rewriting
– Blocking
– Injection

11. LUA
●
PiL http://lua.org/manual/5.1/
●
embedded, simple, efficient
●
can do OO­like programming
●
has scalars, tables, metatables and anonymous
functions

12. Query Rewriting
●
Macro Packages (ls, cd, who, ...)
●
tagging queries with SQL_CACHE
●
migrating table­names and SQL dialects
●
turn EXPLAIN UPDATE|DELETE into
equivalent EXPLAIN SELECT

13. Query Profiling
●
SHOW SESSION STATUS around a Query
Exec_time: 6749 us
.. Handler_read_rnd_next = 252
.. Handler_write = 252
.. Select_scan = 1

14. Query Statistics
●
Normalize Queries to track query usage
●
Count Table and Index usage
●
Optimize Query Cache Usage by injecting
SQL_CACHE in cachable queries
●
see lib/analyze­queries.lua

15. Auditing
●
Diagnostic Auditing
●
track which user+ip run which query or
accessed which objects when
●
assign query­costs
●
log gathered information in a central place
●
see lib/auditing.lua

16. Global Transaction ID
●
Inject a counter in all transactions
●
Answers questions like
– which slave is most current
– can I read from this slave, or do I have to read from
master
– you name it

17. Connection Pooling
●
reusing open connections between proxy and
server
●
reduces concurrency on the MySQL Server
●
external connection pool for PHP

18. Statement Routing
●
split the query stream into reading and writing
– READs go to the slaves
– WRITEs and transactions to the master
●
automatic scale­out
●
sharding

19. Tokenizer
●
turns a SQL query into a token stream
●
not a full parser, just a tokenizer for speed
reasons
●
understands KEYWORDS, /*comments*/,
“strings”, 123 and `literals`
●
later we'll add support for SQL modes

20. normalizing Queries
1: { TK_SQL_SELECT, select }
2: { TK_STAR, * }
3: { TK_SQL_FROM, from }
4: { TK_LITERAL, t1 }
5: { TK_SQL_WHERE, where }
6: { TK_LITERAL, id }
7: { TK_EQ, = }
8: { TK_INTEGER, 1 }
normalized query: SELECT * FROM `t1` WHERE
`id` = ?

21. Libraries
●
auto­config ●
parser
– SET GLOBAL ... – extract tablenames
●
balance ●
tokenizer
– load balancers – normalize()
●
commands – cleanup queries
– parse MySQL
Command Packets

22. Internals – LUA scripting
●
proxy.* is the namespace
●
proxy.connection.* is the current
connection
●
proxy.backends[...] are the backends
●
proxy.global.* is the global table
●
proxy.global.config.* is used for the
config

23. Internals ­ Scope
●
Each connection has its own script scope
●
proxy.global.* to share data between
connections
●
use local to make variables local to the
function
●
use package.seeall() to export functions
from modules

24. Internals ­ Threading
●
the global scope and threading don't play nice
by default
●
http://www.cs.princeton.edu/~diego/professional/lua
●
patches lua to apply mutexes around variable
access

25. Internals – Script Cache
●
0.6.0 we reload the script on each connection
start
●
adding a script cache with mtime check
●
lua_pushvalue(L, ­1) does the trick

26. Roadmap
●
to be released 0.6.0
– tokenizer
– read­write splitting
– Query Statistics
●
later
– parallel Queries
– proxy initiates connections

27. LUA ­ Gotchas
●
only false and nil are !true, 0 is true
●
to say “not equal” you use ~=
●
there are no shortcuts
– no a++, no a *= 4, ...
– no a > b ? a : b (there is “(a > b) and a or b)