MariaDB: in-depth (hands on training in Seoul)


Published on

MariaDB: in-depth is training that was conducted for partners selling/deploying MariaDB in Seoul. Its a practical hands-on introduction that can be completed in 1-day.

Published in: Technology

MariaDB: in-depth (hands on training in Seoul)

  1. 1. MariaDB: in-depth Colin Charles, Monty Program Ab | byte@bytebot.net | | @bytebot on Twitter Partner Training in Seoul, South Korea 19 March 2013 1
  2. 2. whoami• Work on MariaDB at Monty Program Ab• Previously from MySQL AB/Sun Microsystems• Have hacked on The Fedora Project and 2
  3. 3. who is he• Daniel Saito• APAC Sales Director, SkySQL• Former MySQL AB/Sun Microsystems/ Oracle• Former NTT Docomo 3
  4. 4. Agenda• 10.10am - we start• 12.30pm - 1.20pm - lunch• 1.30pm - we resume• 6pm - makgeoli time! (I’m not paying :P)• WiFi network is: 3Com (no password)• Please put mobiles on vibrate 4
  5. 5. We’re very social• Let’s use social media for the last time today (till the break!) - come say hi now!• facebook:• twitter: @MariaDB• g+: 5
  6. 6. What is MariaDB?• Community developed • MariaDB doesn’t depend on MySQL for • maria-captains: 50% MP, development - many features are developed 50% community including Sphinxsearch, Twitter, independently of MySQL SkySQL, Taobao, Facebook, Percona, Codership & • Backwards compatible with more MySQL• Feature enhanced • feature complete • better MySQL • replication supported for easy migration 6
  7. 7. Aims of MariaDB• Compatible, drop-in replacement to MySQL • your application shouldn’t care that its running MariaDB, easy upgrade (uninstall mysql, install mariadb, continue ops!)• Stable (bug-free) releases with no regressions• GPLv2 7
  8. 8. 3 years, many server releases• MariaDB 5.1, GA February 2010• MariaDB 5.2, GA November 2010• MariaDB 5.3, GA February 2012• MariaDB 5.5, GA April 2012• MariaDB Galera Cluster, GA March 2013• MariaDB 10.0, alpha November 2012, release aim April 2013 8
  9. 9. 9
  10. 10. MariaDB 5.6?• SkySQL is fully committed to supporting MySQL 5.6 and all future GA releases of the server• For customers, SkySQL may create a branch of MySQL 5.6 with appropriate fixes & features from MariaDB if Oracle does not provide timely fixes 10
  11. 11. MariaDB 10.0 11
  12. 12. The MariaDB Ecosystem 12
  13. 13. Importance of understanding MariaDB (and MySQL)• Generics are inefficient• Since you have chosen MariaDB: • maximise its strengths • minimise its weaknesses 13
  14. 14. Sample databases to play with• • sakila sample database, world database (used in MySQL training), menagerie database (used in book: Beginning MySQL), employees database (large dataset, comes with data - best to play with) 14
  15. 15. Picking hardware• Just use 64-bit hardware• VM’s are improving to use 64-bit OSes & software• Physical > virtual• Disk: battery backed storage, plan for RAID usage 15
  16. 16. Testing MySQL• Use MySQL Sandbox •• Express one-click MySQL installs• make_sandbox foo.tar.gz• Does not require root privileges• It is version aware: • mysqlsandbox-to-install-mariadb-10-0 16
  17. 17. Installation• Binaries (tarballs) are available at (source too) • Built, tested by MariaDB • Graphical installer & configuration for Microsoft Windows, with HeidiSQL GUI • Up-to-date monthly release schedule• RPM, DEB packages are provided, including YUM & APT repositories • Use the repository configuration tool• Inside Linux/*BSD distributions • Easy to install, basic defaults, may be older than upstream 17
  18. 18. Upgrades• Review changelogs carefully, even for minor versions• Make backups (using xtrabackup)• Don’t forget to run mysql_upgrade • by default with a distribution package• Replication can reduce downtime by upgrading the slave, promoting it, then upgrading the master 18
  19. 19. Packages (RPM example)• MariaDB-common • MariaDB-client • character sets • mysql client utilities• MariaDB-shared • MariaDB-test • latest libmysqlclient, some • test suite plugins, /etc/my.cnf.d/ • MariaDB-devel• MariaDB-compat • development headers enabling • compatible client libraries that you to build your own package replace mysql-libs • MariaDB-CassandraSE• MariaDB-server • plugin for CassandraSE • The actual server! 19
  20. 20. What is libmysqlclient?• Client libraries• Many applications are compiled against libmysqlclient• It is an Application Binary Interface (ABI)• When you write in Java & use Connector/J, Connector/J is compiled against libmysqlclient• Similarly with PHP & Connector/PHP 20
  21. 21. MySQL utilities• Why does MariaDB use all the same client utilities? • Because the aim is to be a 100% compatible drop-in replacement• There are very few non-MySQL utilities shipped: aria_chk, aria_dump_log, aria_ftdump, aria_pack, aria_read_log• xtstat (PBXT) is deprecated in MariaDB 5.5• mytop as it adds features not-present upstream 21
  22. 22. mysql• Command line tool, just shell, works over SSH• Allows execution of statements, keeps track of settings, etc. 22
  23. 23. mysqlmysql -uroot -pEnter password:Welcome to the MariaDB monitor. Commands end with ; or g.Your MariaDB connection id is 11Server version: 5.5.29-MariaDB Source distributionCopyright (c) 2000, 2012, Oracle, Monty Program Ab and others.Type help; or h for help. Type c to clear the current input statement.MariaDB [(none)]> show databases;+--------------------+| Database |+--------------------+| information_schema || mysql || performance_schema |+--------------------+3 rows in set (0.00 sec) 23
  24. 24. Finding help• How do you know what CLI options to use?• Use the man(ual) pages!• man mysql• man <command_name> usually works for any CLI command• HELP SELECT in mysql works too (help tables) - HELP <operator> 24
  25. 25. mysql_secure_insta llation• Now that you’ve tested MySQL works fine (by default, no root password), it’s time to ensure you secure the installation• Set root password• Remove anonymous users• Disallow remote root login• Remove test database• Reload privileges 25
  26. 26. mysqladmin• Manage user accounts, passwords, permissions• Display mysqld settings & status counters• Shutdown, create, drop databases PROCESSLIST SHOWshell> mysqladmin processlistshell> mysqladmin extended-status SHOW STATUSshell> mysqladmin -uroot -p create wordpress 26
  27. 27. mysqldump• Dump schema and/or data to .sql file, tab• Useful for backups, transferring data• Locks for consistency, so troublesomeshell> mysqldump --all-databases > backup.sqlshell> mysqldump --all-databases --single-transaction >backup.sql 27
  28. 28. Other tools• There are many other command line tools...• Front-ends • HeidiSQL • MySQL Workbench • Webyog’s SQLyog • Sequel Pro (OSX) 28
  29. 29. MariaDB Architecture 29
  30. 30. Making client connections• TCP/IP is available on all platforms • disable using --skip-networking• Socket files available on Unix • fastest communication path• MySQL connections are generally cheap • If you have many short running queries (typical web app), use MariaDB threadpool• Set connection limit max_connections=n in my.cnf 30
  31. 31. Connection Pool• mysqld’s main thread listens for connections• Each connection has a thread assigned to it• Threads can be: • created newly • reused from thread cache• User authentication processed based on hostname, username & password• Client specific buffers for session variables are allocated 31
  32. 32. Query Cache• Stores SELECT queries and their result sets• Subquery cache exists only in MariaDB• Frequently changed table data will cause queries to naturally be missed• Regularly replaced these days with memcached, et al. 32
  33. 33. SQL Parser• Lexical scanner & grammar rules• Parse SQL into tokens• Apply grammar rules to check statement validity• Construct a parse tree for Optimizer to use 33
  34. 34. Optimizer• Reads the parse tree and calculates the best query execution plan (QEP) to handle query • find indexes • determine JOIN order • eliminate unnecessary tables • etc.• The optimizer is smart, don’t try to force query plans 34
  35. 35. Pluggable Storage Engines• MariaDB can use many storage engines with different features• Installed/removed on fly with INSTALL/ UNINSTALL PLUGIN• Mixing & matching on same server, even on same query 35
  36. 36. Value proposition• No other database offers this capability• Unmatched flexibility + customisation potential• MEMORY engine for performance/routine lookup data• Right storage engine can improve performance in many applications• ARCHIVE compresses data, up to 80%• Partners & community benefit from this 36
  37. 37. What makes engines different?• Storage: how the data is stored on disk • Or in NDB (memory+disk), CassandraSE (access a Cassandra Cluster), SphinxSE (access the Sphinx daemon)• Indexes: improves search operations• Memory usage: improves data access for speed• Transactions: protects the integrity of your data (Atomic-Consistent-Isolated- Durable - ACID)• Locking level: MyISAM (table locks), InnoDB (row locks), old BDB (page locks)• Data types: Data types may be converted, MEMORY doesn’t support TEXT, etc.• Caching: InnoDB caches data & indexes, MyISAM caches indexes only (relying on OS disk cache for data)• Full-text search capability: MyISAM has this, InnoDB 5.6 just got this• GIS: MyISAM & Aria work (R-tree indexes exist), InnoDB 5.7 will probably have this 37
  38. 38. INFORMATION_SCHE MA• Holds metadata (data about the data) on all other databases & tables, exposed as regular tables• Generated on the fly• Has extensions in MariaDB as we expose more data than native MySQL 38
  39. 39. PERFORMANCE_SCH EMA• Allows for monitoring execution at a low level• This is a storage engine, monitoring server events (anything that takes time & can be instrumented)• Tables are views or temporary tables that use no on-disk storage• Such instrumentation isn’t free 39
  40. 40. MySQL 101• q / ctrl+D works • no grants, no access• if statement is not terminated by • ERROR 1046 (3D000): No “;” (or uncommonly, g) and you hit database selected <-- only in Enter, you get another line for input databases can you make tables -> • CREATE TABLE foo (name• G - vertical column output VARCHAR(20), sex CHAR(1), birth DATE);• SHOW is a MySQL-ism• CREATE DATABASE foo; • SHOW TABLES;• USE foo; • DESCRIBE foo;• GRANT ALL on test.* TO ‘dbuser’@‘localhost’; 40
  41. 41. MySQL 101• INSERT INTO foo • SELECT name FROM VALUES foo WHERE (‘fossasia’,’F’, sex=’f’; ‘2010-11-12’);• SELECT * FROM foo;• SELECT name FROM foo; 41
  42. 42. Use EXPLAIN 42
  43. 43. Ranges• SELECT * from room WHERE room_date between ‘2011-06-27’ AND ‘2011-07-02’;• must ensure that index is available on field being operated on by range operator (room_date)• too many records to return? Range optimisation won’t be used, and you get a index or full table scan 43
  44. 44. Scans & seeks• A seek, jumps into a random place (on disk/in memory) to fetch needed data• A scan, jumps to the start of data, and sequentially reads (on disk/in memory) until the end of needed data• Large amounts of data? Scan operations probably better than large amount of seeks 44
  45. 45. When do you get a full table scan• No WHERE condition• No index on any field with a WHERE condition• When range returns a large number of rows• SELECT * FROM 45
  46. 46. Primary keys• Always add one• When using InnoDB, use a small value (it is appended to every record in a secondary index)• Don’t add a PK? InnoDB adds one by default for you, and its a 6-byte integer 46
  47. 47. Schema/data types• MySQL has 9 to represent integers• Integer: TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT• Floating point: FLOAT, DOUBLE• Fixed point: DECIMAL• Other: BIT 47
  48. 48. Bad schema design• INT(1)• this is NOT 1 digit. This is 4-bytes in size. A TINYINT is 1 byte• TINYINT UNSIGNED stores 0-255 values• BIT? If values are 0 or 1, can be used 48
  49. 49. Bad schema design• BIGINT AUTO_INCREMENT• INT UNSIGNED stores 4.3 billion values!• If you have a billion rows, think about PARTITIONing already 49
  50. 50. Not using UNSIGNED• Don’t want negative values?• Use UNSIGNED• Adds level of data integrity for cases where negative values might be present 50
  51. 51. CHAR(n)• This is what VARCHAR was built for• VARCHAR(n) works• Imagine storing simply 10 bytes in CHAR(128)? Wasted space 51
  52. 52. ONLINE ALTER• In MySQL 5.6 (and upcoming MariaDB 10) • 5.5 have external tools: • html/oak-online-alter-table.html • • schema-change.html• So, be very careful when you use ENUM columns• Probably only useful for sex (‘M’, ‘F’) or status (‘A’, ‘I’), since “Active” and “Inactive” is something you do in the presentation layer• Internally stored as 0 or 1 (very efficient) 52
  53. 53. Storing IPv4/6 addresses• INT UNSIGNED works• VARCHAR(15) results in an average of 12 bytes being used• ... versus 4 bytes via INT UNSIGNED• IPv6 have data types - INET6_ATON(expr) & INET6_NTOA(expr) • MDEV-274 53
  54. 54. Using NOT NULL• Can save a byte per column per row data, depending on the storage engine being used• Always define columns as NOT NULL unless there is a reason not to 54
  55. 55. Character sets• Default is UTF-8• The web is UTF-8• You can define this at the database, schema, table or even column level• MySQL internal buffers are fixed width • VARCHAR(255) utf8 is 765 bytes just to store 1 byte! • Efficient disk usage with VARCHAR, but internal memory usage isn’t 55
  56. 56. Tuning tips• 1,000 small queries is worse than 1 slow query!• Bytes? We save space because you want things in contiguous amounts of memory 56
  57. 57. Let’s go back to storage engines for a moment 57
  58. 58. Using different engines• Create a table with a specified engine • CREATE TABLE t1 (..) ENGINE=XtraDB;• Changing existing tables • ALTER TABLE t1 ENGINE=Aria;• SHOW ENGINES; 58
  59. 59. Transactional vs. non- transactional• Transaction-safe tables • Update fails? Changes (InnoDB) have advantages reverted over non-transaction safe tables (MyISAM): • Concurrency - tables w/ many update + • server crash? Automatic concurrent reads recovery, or a backup +transaction log • Disadvantages in today’s environments (transaction • ROLLBACK can be overhead = slower), more executed to ignore disk space requirements, changes more memory to perform updates don’t seem like they apply any longer 59
  60. 60. Indexes• Tree Indexes • Hash Indexes (MEMORY, NDB, InnoDB) • B-Trees • If table fits entirely in • B+Trees (InnoDB) memory, fastest way to perform queries is a hash • T-Trees (NDB) index • Red-black binary trees • InnoDB has an internal (MEMORY) adaptive hash index. InnoDB monitors index searches, • R-Trees (MyISAM for spatial and if it notices that it will benefit from a hash index, indexes) InnoDB automatically builds one. (5.1.24 and greater) 60
  61. 61. Does the storage enginereally make a difference? MyISAM Inserts XtraDB Inserts ARCHIVE Inserts User Load Per Second Per Second Per Second 1 3,203.00 2,670.00 3,576.00 4 9,123.00 5,280.00 11,038.00 8 9,361.00 5,044.00 13,202.00 16 8,957.00 4,424.00 13,066.00 32 8,470.00 3,934.00 12,921.00 64 8,382.00 3,541.00 12,571.00 Using mysqlslap, against MariaDB 5.1.55, the ARCHIVE engine has 50% more INSERT throughput compared to MyISAM, and 255% more than XtraDB. 61
  62. 62. MyISAM• Pros? • no transactions • excellent INSERT • no foreign key performance support • small footprint • Typical uses • supports full-text • logging search (FTS) • auditing• Cons? • data warehousing 62
  63. 63. MyISAM II• In my.cnf, remember to • myisam_use_mmap set the key_buffer_size. enables MyISAM to use This is memory*0.40, as memory mapping (7-40% MyISAM uses the OS speed improvement) cache for tables • key_cache_segments = 1• Using DRBD (which enables segmented key requires transactional caches in MariaDB - engine) but still want ~250% improvements, as FTS? Create a FTS it mitigates thread “slave” MyISAM table contention for key cache lock 63
  64. 64. InnoDB• Maintains its own buffer pool (does aggressive memory caching)• Uses tablespaces (several files on disk, raw disk support)• Typically used for OLTP operations 64
  65. 65. InnoDB: Tuning• innodb_file_per_table - • innodb_flush_log_at_trx split InnoDB data into a _commit=1 - logs file per table, rather than flushed to disk at each one contiguous file transaction commit. Guarantees ACID. • allows optimize table `table` to run • innodb_log_file_size - keep it high• innodb_buffer_pool_size (64-512MB), though recovery time increases =(memory*0.8) (largest=4GB) 65
  66. 66. InnoDB: Tuning II• SHOW ENGINE INNODB STATUS;• InnoDB supports both row & statement based replication• Assign a primary key, otherwise InnoDB assigns one for you. Also keep it small - its appended to every record in secondary index 66
  67. 67. ARCHIVE• Store large amounts of data without indexes, in small disk footprint• SELECT and INSERT operations only• Good for data audit use• Uses AZIO (zlib) compression 67
  68. 68. FederatedX• Create logical pointers to tables that exist on other MySQL servers; these can then be linked together to form one logical database• A federated table pointing to an InnoDB table on another server, will have transaction support (in 5.1) • Capabilities limited to underlying engine on remote server• CREATE TABLE t1 (...) ENGINE=FEDERATED CONNECTION=mysql://username:pwd@myhost:3306/db_name/ tbl_name• Can also be used for synchronous replication • Federated table on master server pointing to slave; triggers on master table to write all changes to remote table once applied to the master 68
  69. 69. Memory• Previously known as HEAP tables• In-memory engine• Hash index used by default, B-Tree available too• Typical uses? • Lookup tables • Session data • Temporary tables • Calculation tables• Cons? • Server dies, all rows are lost 69
  70. 70. Aria• Based off the 5.1 code • 1.0 – crash-safe MyISAM, with cacheable row format • 1.5 – concurrent INSERT/SELECT • Soon to be merged into 6.0, then... • 2.0: transactional + ACID compliance • 3.0: high concurrency, online backup• Goal: ACID compliant, MVCC transactional storage engine, based on MyISAM• Target? Data warehousing• Uses big log files (1GB by default)• 8K pages used by default (MyISAM uses 1K pages)• Has group commit (MariaDB 5.2) to speed up inserts 70
  71. 71. PBXT (deprecated)• MVCC, transactional, ACID compliant, foreign key support• row-level locking for updates, so maximum concurrency• immediate notification if client processes are deadlocked• write-once, as it uses a log-based architecture (write data to DB without first writing to transaction log)• support for BLOB streaming with Blob Streaming engine 71
  72. 72. Storage Engine API• articles/creating-new-storage-engine.html• SHOW PLUGINS;• table• storage/example/ and storage/example/ha_example.h 72
  73. 73. Writing your own• Find the plugin path - show variables like “%plugins%”; +---------------+-----------------------------------+ | Variable_name | Value | +---------------+-----------------------------------+ | plugin_dir | /opt/maria/5.2.6/lib/mysql/plugin | +---------------+-----------------------------------+ • note that this is also where you store UDFs• Copy the relevant engine (eg.• INSTALL PLUGIN myengine SONAME;• Server registers plugin to mysql.plugin table, and now ENGINE=myengine will work 73
  74. 74. Things to think about• Backup is not engine-independent • MyISAM, InnoDB, PBXT • LVM/ZFS snapshots solve this• Different engines have different monitoring options• Mix and match; use summary tables 74
  75. 75. What others are using• Wordpress (blog): uses default engine, MyISAM is fine• MediaWiki (wiki): prefers InnoDB, except for “searchindex” table, which is MyISAM • phase3/maintenance/tables.sql?view=markup• vBulletin (forum): MyISAM• SugarCRM (CRM): MyISAM (with conversion script to InnoDB provided)• Zimbra Collaboration Suite: InnoDB 75
  76. 76. Now, let’s focus onMariaDB features 76
  77. 77. We start with...• What came in 5.1, 5.2, 5.3, 5.5 (jump around appropriately)• What comes in 10.0 series• I won’t talk about deprecated features like OQGRAPH, PBXT in-depth, or even the old MariaDB 5.1 pool of threads (5.5 threadpool is better) 77
  78. 78. XtraDB• A more performant InnoDB designed to scale on modern hardware• Less checkpointing (smoother), less flushing to disk 78
  79. 79. Switching between XtraDB & InnoDBmysqld --ignore-builtin-innodb --plugin_dir=/usr/local/mysql/lib/mysql/pluginOr in my.cnf[mysqld]ignore-builtin-innodbplugin-load=innodb=ha_innodb.soplugin_dir=/usr/local/mysql/lib/mysql/plugin 79
  80. 80. MariaDB 5.5: anopensource threadpool• Modified from 5.1 (libevent • thread_handling=po based), great for CPU ol-of-threads bound loads and short running queries • thread-pool-in-mariadb-55/• Windows (threadpool), Linux (epoll), Solaris (event ports), FreeBSD/OSX (kevents)• No minimization of concurrent transactions with dynamic pool size 80
  81. 81. Better for DBAs: non-blocking client library• start operation, do work • https:// in thread, operation processed, result travels about-non-blocking- back operation-in-the- client-library/ • use cases: multiple queries against single • fast node.js driver server (utilize more available: mariasql CPUs); queries against multiple servers (SHOW STATUS on • https:// many machines) mariasql-for-nodejs/ 81
  82. 82. LIMIT ROWS EXAMINED• The purpose of this optimization is to • SELECT * from t1, t2 LIMIT 10 provide the means to ROWS EXAMINED terminate the execution 1000; of SELECTstatements which examine too many rows, and thus use • en/limit-rows-examined/ too many resources. 82
  83. 83. SHOW STATUS• SHOW STATUS provides server status information. It is like mysqladmin extended-status.• SHOW STATUS LIKE ‘Key%’;•• variables/• MariaDB has opened_views, executed_triggers, executed_events, feature_* as new options 83
  84. 84. SQL Error Logging Plugin• Log errors sent to clients in a log file that can be analysed later. Log file can be rotated (recommended)• a MYSQL_AUDIT_PLUGINinstall plugin SQL_ERROR_LOGsoname; 84
  85. 85. Replication: selective skipping• All changes that are logged as events in the binlog are replicated to all slaves• However, sometimes you want all to be logged to binlog but skipped replication to slaves• @@skip_replication (session only)• replicate_events_marked_for_skip = replicate|filter_on_slave| filter_on_master (dynamic) 85
  86. 86. Replication: dynamic variables• The variables replicate_do_*, replicate_ignore_*, and replicate_wild_* have been made dynamic, so they can be changed without requiring a server restart.• replication-variables/ 86
  87. 87. Replication: Annotation of RBR events• MariaDB supports statement & row based replication (RBR)• In RBR, the binlog has no SQL statements, only events are logged (INSERT, DELETE, etc)• Option to include original SQL statement (default OFF)• annotate_rows_log_event/ 87
  88. 88. Replication: binlog event checksums• Backport from MySQL 5.6 (in MariaDB 5.3+)• binlog_checksum option• Slaves perform checksums on events received & will stop if there is corruption• checksums/ 88
  89. 89. Replication: groupcommit in the binary log• sync_binlog=1, • for-the-binary-log innodb_flush_log_at_trx_commi t=1 • SHOW STATUS LIKE binlog_%commits;• note_id=10150261692455933 89
  90. 90. Replication: START TRANSACTIONWITH CONSISTENT SNAPSHOT• Works with the binlog, possible to obtain the binlog position corresponding to a transactional snapshot of the database without blocking any other queries. • by-product of group commit in the binlog to view commit ordering• Used by the command mysqldump--single-transaction --master-data to do a fully non-blocking backup which can be used to provision a new slave• Works consistently between transactions involving more than one storage engine• with-consistent/ 90
  91. 91. GIS precise• MySQL has OpenGIS SFS (Simple feature access, SQL access method)• Now, SQL with full geometry types• ST_ prefix• in-533 91
  92. 92. Progress reporting• ALTER TABLE & LOAD DATA INFILEMariaDB [mail]> alter table mail engine = maria;Stage: 1 of 2 copy to tmp table 17.55% of stage doneMariaDB [mail]> select id, user, db, command, state,-> time_ms, progress from information_schema.processlist;+---------+-------------------+-----------+----------+| command | state | time_ms | progress |+---------+-------------------+-----------+----------+| Query | copy to tmp table | 23407.131 | 17.551 |+---------+-------------------+-----------+----------+1 row in set (0.47 sec) 92
  93. 93. TIME_MS in I_S.PROCESSLIST• Extra column TIME_MS has been added to the INFORMATION_SCHEMA.PROCESSLIST table• Units of milliseconds with microsecond precision (the unit and precision of the TIME column is one second). 93
  94. 94. New KILL syntax• HARD | SOFT & USER USERNAME are MariaDB-specific• SOFT ensures things that may leave a table in an inconsistent state aren’t interrupted (like REPAIR or INDEX creation for MyISAM or Aria)KILL [HARD | SOFT] [CONNECTION |QUERY] [thread_id | USER user_name] 94
  95. 95. SphinxSE 95
  96. 96. The old days• Download MySQL, including sources• Download SphinxSE for compiling• Download Sphinx to compile with MySQL support• Documented: sphinx-as-mysql-storage-engine-sphinxse 96
  97. 97. Today• Install sphinx from your distribution• Install MariaDB 5.5 from your distribution or from• Get started! 97
  98. 98. Getting startedmysql> INSTALL PLUGIN sphinxSONAME;Query OK, 0 rows affected(0.01 sec) 98
  99. 99. Another engine appears 99
  100. 100. What is SphinxSE?• SphinxSE is just the storage engine that still depends on the Sphinx daemon• It doesn’t store any data itself• Its just a built-in client to allow MariaDB to talk to Sphinx searchd, run queries, obtain results• Indexing, searching is performed on Sphinx 100
  101. 101. Configure sphinx!• /usr/local/sphinx/sphinx.conf• Source (multiple, include mysql, with connection info)• Setup indexer (esp. if its on localhost) - mem_limit, max_iops, max_iosize• Setup searchd (where to listen to, query log, etc.) 101
  102. 102. Use case scenarios• Already have an existing application that makes use of full-text-search in MyISAM? Porting should be easier• Have a programming language without a native API for Sphinx? Surely there’s a connector for MariaDB ;-) 102
  103. 103. Use case scenarios• Results from Sphinx itself almost always require additional work involving MariaDB • Say to pull out text column that Sphinx index doesn’t store • JOIN with another table (using a different engine) 103
  104. 104. An exampleCREATE TABLE t1( id INTEGER UNSIGNED NOT NULL, weight INTEGER NOT NULL, query VARCHAR(3072) NOT NULL, group_id INTEGER, INDEX(query)) ENGINE=SPHINX CONNECTION="sphinx://localhost:9312/test";SELECT * FROM t1 WHERE query=test it;mode=any; 104
  105. 105. Sphinx search tables• 1st column: INTEGER UNSIGNED or BIGINT (document ID)• 2nd column: match weight• 3rd column: VARCHAR or TEXT (your query)• Query column needs indexing, no other column needs to be 105
  106. 106. What actually happens• SELECT passes a Sphinx query as the query column in the WHERE clause• searchd returns the results• SphinxSE translates and returns the results to MariaDB 106
  107. 107. SHOW ENGINE SPHINX STATUS• Per-query & per-word statistics that searchd returns are accessible via SHOW STATUS mysql> SHOW ENGINE SPHINX STATUS; +--------+-------+-------------------------------------------------+ | Type | Name | Status | +--------+-------+-------------------------------------------------+ | SPHINX | stats | total: 25, total found: 25, time: 126, words: 2 | | SPHINX | words | sphinx:591:1256 soft:11076:15945 | +--------+-------+-------------------------------------------------+ 2 rows in set (0.00 sec) 107
  108. 108. What queries are supported?• Most of the Sphinx API is exposed to SphinxSE• query, mode, sort, offset, limit, index, minid, maxid, weights, filter, !filter, range, !range, maxmatches, groupby, groupsort, indexweights, comment, select• Sphinx search modes can also be supported via _sph attributes • obtain value of @groupby? use ‘_sph_groupby’ 108
  109. 109. Efficiency• Allow Sphinx to perform sorting, filtering, and slicing of result set • ... as opposed to using WHERE, ORDER BY, LIMIT clauses on MariaDB• Why? • Sphinx optimises and performs better on these tasks • Less data packed by searchd, and transferred and unpacked by SphinxSE 109
  110. 110. JOINs• Perform JOINs on a SphinxSE search table using tables from other engines SELECT content, date_added FROM test.documents docs -> JOIN t1 ON ( -> WHERE query="one document;mode=any"; +-------------------------------------+---------------------+ | content | docdate | +-------------------------------------+---------------------+ | this is my test document number two | 2006-06-17 14:04:28 | | this is my test document number one | 2006-06-17 14:04:28 | +-------------------------------------+---------------------+ 2 rows in set (0.00 sec) 110
  111. 111. User statistics• Understand server activity better, identify database loads user-statistics• Must be enabled first• /usr/local/Cellar/mariadb/5.2.7/bin/ mysqld_safe --datadir=/usr/local/var/mysql --userstat=1 111
  112. 112. userstats 112
  113. 113. Table Elimination• Resolve a query without accessing some tables query refers to • Great for querying highly normalised data• Basis of “anchor modelling” •• SQL Server 2005/2008, Oracle 11g have it 113
  114. 114. Virtual columns• Columns that are an expression that are calculated on retrieval• PERSISTENT or VIRTUAL• Similar to MS SQL or Oracle• 114
  115. 115. MyISAM segmented key caches• Mitigates thread contention for key cache lock, with notable performance improvements• Key caches divided into different segments, allowing for better key cache concurrency• 1-64 segments 115
  116. 116. Optimizer enchancements• Join additions • block nested loop joins for outer-joins, block hash joins, Batched Key Access (BKA)• Optimization for derived tables & views • mergeable derived tables processed like VIEWs + optimizer creates indexes over materialized derived tables• Disk access optimization • Index Condition Pushdown (ICP), Multi-Range Read (MRR) 116
  117. 117. Subquery optimizations• Semi-join optimization, • Goodbye rewriting as materialization for non- JOINs or separate correlated IN queries, queries subquery cache DBT-3, 60M rows, 29GB XtraDB 117
  118. 118. 118
  119. 119. EXPLAIN, DBT-3• INSTANT EXPLAIN• EXPLAIN Analyzer - explain_analyzer/ analyze/• Optimizer feature comparison matrix: en/optimizer-feature- comparison-matrix/ 119
  120. 120. Extended keys• Default is extended_keys=off• Extended Keys, introduced in MariaDB 5.5, is an optimization which makes use of existing components of InnoDB/XtraDB keys to generate more efficient execution plans. Using these components in many cases allows the server to generate execution plans which employ index-only look-ups. 120
  121. 121. NoSQL• Comes with HandlerSocket • direct access to XtraDB/InnoDB for CRUD operations • INSTALL PLUGIN handlersocket SONAME;• Dynamic columns • create columns with dynamic content • basically a blob with handling functions 121
  122. 122. Pluggable authentication• MariaDB & MySQL now uses password authentication via pluggable auth• Unix sockets• PAM• Active Directory (MySQL)• LDAP (interested?)• Kerberos (interested?) 122
  123. 123. Windows performanceOLTP read-only OLTP write-only 123
  124. 124. MariaDB 10.0 124
  125. 125. Why MariaDB 10.0?• The 5.5 merge took about a year (!)• We (MariaDB-5.5) have over 1.5 million lines of extra code with a ~61MB diff• We didn’t want to repeat this for 5.6• Also, MySQL 5.6 has a lot of re-factoring, thus loosing commit history 125
  126. 126. In a nutshell• Built on MariaDB 5.5• Backported features from MySQL 5.6• New features 126
  127. 127. What about tools?• SELECT VERSION() will • Use 9.0 for a name return 10.0.1-MariaDB • Lie to clients (no)• Oops, we found a bug in MySQL: https:// • Disallow replication (no) browse/MDEV-4088 & bug.php?id=68187 • Use handshake packet 5.5.30-mysql-10.0.2- MariaDB without• Still deciding: affecting VERSION() / @@global.version 127
  128. 128. What about tools II?• Tools really should recognise MariaDB version as there are already many new features that MySQL doesn’t have• eg. HeidiSQL supports virtual columns ( t=8671) 128
  129. 129. Versions Date Version Status12 Nov 10.0.0 Alpha 2012 6 Feb 10.0.1 Alpha 201322 Mar 10.0.2 Beta 2013 10.0.3 GA 129
  130. 130. Backported features (i.e. these are from MySQL 5.6) 130
  131. 131. InnoDB & XtraDB• MariaDB 10.0 ships InnoDB from MySQL 5.6• MariaDB 10.0.2/3 will probably have InnoDB Full Text Search enabled• Percona XtraDB ~ mid-2013 • no performance improvements expected, just functionality & features • bitmap changed page tracking so xtrabackup can do incremental backups without scanning all InnoDB files• XtraDB when GA can be used again 131
  132. 132. More from MySQL 5.6• PERFORMANCE_SCHEMA• InnoDB read-only transactions (TRANSACTION READ ONLY)• Optimizer: • EXISTS-TO-IN optimization • ORDER BY...LIMIT optimization (show only few rows of a result set)• CURRENT_TIMESTAMP as DEFAULT for DATETIME columns (this is a re-implementation in MariaDB) 132
  133. 133. Only in MariaDB 10.0 the new stuff! 133
  134. 134. Multi-source replication• Work from Taobao• Many users partition data across many masters... now you can replicate many masters to a single slave• Great for analytical queries, complete backups, etc.• replication/ 134
  135. 135. Only in 10.0• SHOW EXPLAIN for • Per-thread memory usage <thread_id> (https:// (Taobao) explain/) gets the query plan of a running statement • I_S.PROCESSLIST has MEMORY_USAGE & EXAMINED_ROWS• Faster ALTER TABLE with unique keys for Aria & MyISAM • SHOW STATUS has memory usage too • Segmented MyISAM keycaches (up to 64) since MariaDB 5.2 exist too 135
  136. 136. CassandraSE 136
  137. 137. CassandraSE• Integration with NoSQL/Big Data DB, Apache Cassandra cluster, seen as a storage engine to MariaDB• Combine (join) data between Cassandra & MariaDB• Write to Cassandra from SQL (SELECT, INSERT, UPDATE, DELETE)• CQL is great, but the goal is for you to just work with SQL, not switch between CQL & SQL• Data is mapped: rowkey, static columns, dynamic columns • super columns aren’t supported• No 1-1 direct map for data types (ref: cassandra-storage-engine/) 137
  138. 138. LevelDB Storage Engine• LevelDB is a key-value store, used even in the Chrome web browser• LevelDBSE supports single-statement transactions, secondary indexes, crash- proof slave replication, hot backups & more 138
  139. 139. Engine-independent persistent statistics• InnoDB has persistent statistics in MySQL 5.6; we have an engine-independent version• These statistics aren’t limited by the SE API, and are used by query optimizer to choose best execution plan for each statement• Statistics collected for non-indexed columns too (unlike InnoDB’s) 139
  140. 140. Dynamic columns• Store a different set of columns for every row in the table (kinda NoSQL-like)• Database interoperability is now a major feature; CassandraSE makes uses of this• Can now request a row in JSON format• And yes, you can name columns finally ;) 140
  141. 141. What about MySQL 5.6?• We love the fact that many features we’ve worked on for a long time are now in 5.6• Optimizer enhancements• Microseconds• Binary log annotations• Binary log group commit (10.0 has a newer faster version now)• Precise GIS• Threadpool 141
  142. 142. What are we missing from 5.6 currently?• EXPLAIN output in JSON• Online operations (ALTER TABLE)• InnoDB FTS• Global transaction IDs• InnoDB memcached interface• Many will be ready in final 10.0 release 142
  143. 143. Support• Five years from every release 143
  144. 144. Benchmarks• “Lies, damned lies, and statistics” - Mark Twain • mysql-5-6-vs-mariadb-10-0/ • mysql-performance-mysql-56-vs-mysql-55- vs-mariadb-55.html• Yes, we’ve gotten Oracle to notice MariaDB :-) 144
  145. 145. Continued commitments• Security • Since about a year now, we’re the go-to people for security - good track record• We don’t like regressions • mysql-5525 • partitioned-tables-mysql-5523• We care about backward compatibility & introduce features carefully • XtraDB innodb_adaptive_checkpoint=none|reflex|estimate| keep_average (no more reflex...) 145
  146. 146. What’s next• MariaDB Galera Cluster• Automated failover with MHA & MariaDB 146
  147. 147. What we didn’t discuss• Configuring MariaDB (MySQL-generics) • use a configurator tool• Proper backups• NDB Cluster• MySQL features like VIEWs, PARTITIONING, event scheduler, etc. Go buy a MySQL training class• Monitoring from SkySQL! :-)• Optimization 147
  148. 148. Q&A / Thanks / on twitter | 148