• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
common_schema 2.0: DBA's Framework for MySQL
 

common_schema 2.0: DBA's Framework for MySQL

on

  • 2,712 views

Slides from my presentation given at Percona Live 2013, Santa Clara

Slides from my presentation given at Percona Live 2013, Santa Clara

Statistics

Views

Total Views
2,712
Views on SlideShare
893
Embed Views
1,819

Actions

Likes
0
Downloads
9
Comments
0

11 Embeds 1,819

http://code.openark.org 1565
http://planetmariadb.org 148
http://sqlvn.blogspot.com 72
http://localhost 11
http://www.newsblur.com 9
http://www.feedspot.com 6
http://www.blogger.com 2
http://newsblur.com 2
http://abtasty.com 2
http://cloud.feedly.com 1
http://webcache.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    common_schema 2.0: DBA's Framework for MySQL common_schema 2.0: DBA's Framework for MySQL Presentation Transcript

    • common_schema2.0D B A s f r a m e w o r k f o r M y S Q LS h l o m i N o a c hh t t p : / / o p e n a r k . o r g
    • Copyright © 2013, Shlomi Noachcommon_schemaD B A s f r a m e w o r k f o r M y S Q Lcommon_schema2.0D B A s f r a m e w o r k f o r M y S Q L About Views & eval() Routines QueryScript Introducing rdebug
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachAbout common_schema common_schema is an open sourceproject, licensed under the GPL License. Authored by Shlomi Noachhttp://openark.org Major contributions by Roland Boumanhttp://rpbouman.blogspot.com Several contributions & suggestions by thecommunity(Thanks!)
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachAbout common_schema common_schema is a framework whichincludes:– Views: analytics, security, action-taking, ...– Routines: text, temporal, process, security, ...– QueryScript: an SQL oriented scriptinglanguage & interpreter– rdebug: debugger and debugging API forMySQL stored routines (alpha).
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachAbout common_schema It is a schema that lies next toINFORMATION_SCHEMA. It lies completely within the MySQL server,and does not require external packages ordependencies. No Perl scripts nor UDFs orplugins.
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachGetting & Installing common_schema distribution is a SQL file. Currently hosted on Google Code:http://code.google.com/p/common-schema/ Install by importing SQL file into MySQL:bash$ mysql < /tmp/common_schema-1.3.1.sqlcomplete- Base components: installed- InnoDB Plugin components: installed- Percona Server components: not installedInstallation complete. Thank you for usingcommon_schema!
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews Views providing non-trivial informationabout your table design, keys, grants,processes, transactions, locks, ... While INFORMATION_SCHEMA provideswith complete info, it is ofter difficult toaggregate. It is sometimes too normalized,and at other times too de-normalized. Some metadata is simply not available inINFORMATION_SCHEMA.
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews common_schemas views will present withvaluable knowledge And will typically offer recommendationsor possible actions based on thatknowledge. common_schema also provides themechanism to apply thoserecommendations.
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: redundant_keys Find duplicate/redundant keys in yourschema. Recommend DROP statements.http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/redundant_keys.html– Similar to pt-duplicate-key-checkermysql> select * from redundant_keyswhere table_schema=sakilaGtable_schema: sakilatable_name: rentalredundant_index_name: rental_date_2redundant_index_columns: rental_dateredundant_index_non_unique: 1dominant_index_name: rental_datedominant_index_columns: rental_date, inventory_id,customer_iddominant_index_non_unique: 0sql_drop_index: ALTER TABLE `sakila`.`rental`DROP INDEX `rental_date_2`
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: sql_range_partitions Analyze your range partitioned tables.Offers the ADD/REORGANIZE PARTITIONstatements to roll partitions:mysql> create table test.report … partition by range (…) ;mysql> select * from sql_range_partitions wheretable_name=report Gtable_schema: testtable_name: reportcount_partitions: 7sql_drop_first_partition: alter table `test`.`report`drop partition `p0`sql_add_next_partition: alter table `test`.`report`reorganize partition `p6` into (partition `p_20090701000000` values less than(1246395600) /* 2009-07-01 00:00:00 */ ,partition p_maxvalue values less than MAXVALUE)
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: sql grants Provide SQL based access to user accountsand their grants. Offers the relevant GRANT, REVOKE andDROP statements for such grants:GRANTEE: world_user@localhostuser: world_userhost: localhostpriv_level: `world`.*priv_level_name: schemacurrent_privileges: INSERT, SELECT, UPDATEsql_grant: GRANT INSERT, SELECT, UPDATE ON`world`.* TO world_user@%sql_revoke: REVOKE INSERT, SELECT, UPDATE ON`world`.* FROM world_user@%
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachSQL generation & eval() Views above present with “SQL columns”,offering a statement to execute. This is at the heart of common_schemasviews, and is part of the server-sidemechanism the framework stronglysupports. The eval() routine accepts such SQLcolumns and executes (evaluates) them.
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noacheval() Accepts a query returning a text column.Column data is expected to be SQLstatements to be executed.mysql> call eval("SELECT sql_add_next_partitionFROM sql_range_partitionsWHERE table_schema=webprod");Query OK, 0 rows affected-- A new partition has just been added on-- all range-partitioned tables in `webprod`
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: innodb_transactions Which transactions are running?– For how long?– How long are they being idle? Locked?– What queries are they issuing? Recommend KILL, apply with eval()mysql> call eval("SELECT sql_kill_queryFROM innodb_transactionsWHERE trx_idle_seconds >= 30");-- All idle transactions have just been killed
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: others processlist_grantees: identifies the userconnection with its associated account(missing info in MySQL) auto_increment_columns: find “free space”or “usage” for AUTO_INCREMENT values slave_status: get Seconds_behind_mastervia real SQL query last_query_profiling :aggregated profile forlast executed query
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: processlist_granteesmysql> select * from processlist_granteesGID: 41165491USER: dbuserHOST: web00.myapp:40049DB: profileCOMMAND: QueryTIME: 0STATE: updatingINFO: DELETE FROM locks WHERE id =helper AND dt < 2013-04-17 15:09:50GRANTEE: dbuser@%.myappgrantee_user: dbusergrantee_host: %.myappis_super: 0is_repl: 0is_current: 0sql_kill_query: KILL QUERY 41165491sql_kill_connection: KILL 41165491
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachViews: others sql_accounts: block/release accounts similar_grants: detect security roles global_status_diff_nonzero: detect statuschange innodb_locked_transactions: who lockswho, on which query? Recommend KILL more...http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_views.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/innodb_plugin_views.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_views.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/schema_analysis_views.html
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines common_schema offers more than 60useful stored routines, part of theframeworks function library. From text parsing & manipulation,through process diagnostics, queryanalysis & dynamic execution, to securityroutines, the function library extends andcomplements MySQLs own functions.http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/execution_routines.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/text_routines.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_routines.htmlhttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_routines.html
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines: security killall() kills connections by matching textwith user, host or grantee. security_audit() audits servers privilegestables and configuration to detect suchthreats as empty or duplicate passwords,excessive privileges, permissive hosts etc. duplicate_grantee(): duplicates an accountmysql> call killall(analytics);mysql> call killall(localhost);
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines: text extract_json_value(): Extract value fromJSON notation via Xpath. replace_sections(): Search and replace textappearing between section. get_num_tokens(): Return number oftokens in delimited text.mysql> select common_schema.get_num_tokens(the quick brown fox, );=> 4mysql> select common_schema.split_token(the quick brown fox, , 3);=> brown
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines: textmysql> set @json := {"menu": {"id": "file","value": "File","popup": {"menuitem": [{"value": "New", "onclick": "create()"},{"value": "Open", "onclick": "open()"},{"value": "Close", "onclick": "close()"}]}}};mysql> select extract_json_value(@json, //id) AS result;+--------+| result |+--------+| file |+--------+
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines: general query_checksum(): Checksum the resultset of a query. crc64(): Return a 64 bit CRC of given input,as unsigned big integer. random_hash(): Return a 64 bit CRC ofgiven input, as unsigned big integer.mysql> call query_checksum(select code, name from world.Country order by code);+----------------------------------+| checksum |+----------------------------------+| 79221e8d040e33cd3262f1680b4c8e54 |+----------------------------------+
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachRoutines: SQL & execution eval() evaluates the queries generated by agiven query. exec(), exec_file() dynamically executes agiven query or semicolon delimited list ofqueries. run(), run_file() execute QueryScript code.mysql> call exec(CREATE TABLE test.t(id INT);INSERT INTO test.t VALUES (2),(3),(5););
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript A SQL oriented scripting language, offeringtight integration with SQL commands, easyand familiar control flow syntax and highlevel abstraction of complex tasks. common_schema implements QueryScriptvia interpreter, based on stored routines. This makes QueryScript suitable foradministration and bulk tasks, not forOLTP tasks.http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/query_script.html
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachWhy QueryScript? Stored routine programming is a pain:– Requires one to actually store the routinewithin the schema: cant just run something.– Syntax is cumbersome (ANSI:SQL).– Does not offer deeper insight into MySQLslimitations and bottlenecks.– Does not provide with syntax for oh-so-common tasks– Verbose. Cant see the forest for the trees.
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachCompare: stored routineDELIMITER $$DROP PROCEDURE IF EXISTS some_proc $$CREATE PROCEDURE some_proc()READS SQL DATASQL SECURITY INVOKERbegindeclare some_id bigint unsigned default null;declare done tinyint default 0;declare my_cursor cursor for SELECT some_id FROM some_table;declare continue handler for NOT FOUND set done = 1;open my_cursor;read_loop: loopfetch my_cursor into some_id;if done thenleave read_loop;end if;-- do something with some_idend loop;close my_cursor;end $$DELIMITER ;
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachCompare: QueryScriptset @script := foreach ($some_id: select some_id from some_table){-- do something with $some_id};call run(@script); Significantly less overhead Familiar C-family syntax No need to store the code in schema Can execute script directly from filecall run(/path/to/script.qs);
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: variables Create variables which are known toinitialize as NULL and are known to cleanup as they exit scope. Can be expanded and used where MySQLdoes not allow variables.var $count := 20;var $pop := 1000000;var $tbl := City;select * from world.:${tbl}where population >= $pop limit :${count};
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: conditions Familiar if, while or loop-while statements. But conditions are also tightly integratedwith SQL, and so queries make for validconditions.if (@val > 3) {pass;}while (delete from world.Countrywhere Continent = Asia limit 10){throttle 2;}
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: foreach A sophisticated looping device, allowingiteration over queries, sets, numbersrange, databases, tables... But also acknowledges MySQL limitationsand provides with a safer access method. Table iteration uses INFORMATION_SCHEMAoptimizations to avoid excessive locks:foreach($tbl, $scm: table like wp_posts)alter table :${scm}.:${tbl} add columnpost_geo_location VARCHAR(128);
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: foreachforeach($year: 2001:2009)delete from sakila.rental whererental_date >= CONCAT($year, -07-01)and rental_date < CONCAT($year, -09-01);foreach($shard: {US, GB, Japan, FRA})create database dbshard_:${shard};foreach($scm: schema like wp%) {create table :$scm.wp_likes(id int, data VARCHAR(128));}foreach ($some_id: select some_idfrom some_table) {...}
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: split Automagically breaks a bulk operation intosmaller chunks. Supports DELETE, UPDATE,INSERT...SELECT, REPLACE...SELECT Supports single and multi table statementssplit (delete from sakila.rental whererental_date < NOW() - interval 5 year);split (insert into world.City_duplicateselect * from world.City) {throttle 1;}
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: splitsplit(update sakila.film_actorset last_update=now() where actor_id != 4){ select $split_columns as columns,$split_range_start as range_start,$split_range_end as range_end,$split_total_rowcount as total;}+----------------------+-------------+------------+-------+| columns | range_start | range_end | total |+----------------------+-------------+------------+-------+| `actor_id`,`film_id` | 1,1 | 39,293 | 978 |+----------------------+-------------+------------+-------++----------------------+-------------+------------+-------+| columns | range_start | range_end | total |+----------------------+-------------+------------+-------+| `actor_id`,`film_id` | 39,293 | 76,234 | 1978 |+----------------------+-------------+------------+-------++----------------------+-------------+-------------+-------+| columns | range_start | range_end | total |+----------------------+-------------+-------------+-------+| `actor_id`,`film_id` | 76,234 | 110,513 | 2978 |+----------------------+-------------+-------------+-------+
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: example Create shards & tables, and copy row data:-- iterate shardsforeach ($shard: {US, GB, Japan, FRA}) {create database dbshard_:${shard};-- iterate tablesforeach ($tbl: table in original_shard) {create table dbshard_:${shard}.:${tbl}like original_shard.:$tbl;-- copy table datasplit (insert into dbshard_:${shard}.:${tbl}select * from original_shard.:$tbl) {-- show progressselect $shard, $tbl, $split_total_rowcount;throttle 1;}}}
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachQueryScript: more goodies throttle statement controls scriptexecution time, reduces server load Local variables auto-cleanup, can be used(expanded) in statements where variablesnot allowed (table names, LIMIT value etc.) try-catch statement is available: easy errorhandling mechanism echo, eval, throw statements make foreasy development
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noachrdebug common_schema introduces rdebug:debugger and debugging API for MySQLstored routines. A server side solution, based in itself onstored routines. Provides routine API calls to managedebugging sessions of other routines. At this time (April 2013) in alpha stagehttp://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/rdebug.html
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noachrdebug Uses code injection; one must “compile”her routine with/out debug code.– Does not emulate code; retains logic of onesroutines, with some limitations. Supports:– Step into/over/out– Watch variables, modify variables– Set breakpoint, run to breakpoint– Watch stack state, statements
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noachrdebug common_schema provides with storedroutine API.– One may debug on command line usingmysql client– Or one may wrap with GUI tool to call uponroutine API Awaiting GUI client implementations!
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noachrdebugmysql [debugger]> call rdebug_step_into();+-------------+----------------+---------------+--------------+---------------------+| stack_level | routine_schema | routine_name | statement_id | entry_time |+-------------+----------------+---------------+--------------+---------------------+| 1 | test | review_colors | 145 | 2013-04-08 15:41:28 |+-------------+----------------+---------------+--------------+---------------------++----------------+---------------+---------------+---------------+----------------+| routine_schema | routine_name | variable_name | variable_type | variable_value |+----------------+---------------+---------------+---------------+----------------+| test | review_colors | current_color | local | white || test | review_colors | current_count | local | 10 || test | review_colors | done | local | 0 |+----------------+---------------+---------------+---------------+----------------++----------------+---------------+--------------+-----------------------------------------+| routine_schema | routine_name | statement_id | statement |+----------------+---------------+--------------+-----------------------------------------+| test | review_colors | 145 | call review_single_color(current_color) |+----------------+---------------+--------------+-----------------------------------------+mysql [debugger]> call rdebug_set_variable(current_color, orange);mysql [debugger]> call rdebug_step_over();
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi Noachcall for help() common_schema documentation isavailable in these formats:– Online documentation (directly in coderepository)– Bundled HTML download– Inline: Just call for help():call help(split);+--------------------------------------------------------------------------------+| help |+--------------------------------------------------------------------------------+| QueryScript Flow Control: split statement || || SYNOPSIS || || Single table operations, autodetect mode:
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachSupport common_schema Download and try it out Report issueshttp://code.google.com/p/common-schema/issues/list Happy to receive ideas and contributionshttp://bit.ly/UPC3vhhttp://code.google.com/p/common-schema/issues/entry?template=Request%20for%20new%20component Above all else: spread the word!
    • D B A s f r a m e w o r k f o r M y S Q Lcommon_schemaCopyright © 2012, Shlomi NoachThank you! Visit http://openark.org for news &updates. Other open source projects:– openark kithttp://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/introduction.html– mycheckpointhttp://code.openark.org/forge/mycheckpoint