MySQL Binary Log API Presentation - OSCON 2011
Upcoming SlideShare
Loading in...5
×
 

MySQL Binary Log API Presentation - OSCON 2011

on

  • 2,072 views

Presentation of the new MySQL binary log API from OSCON in Portland.

Presentation of the new MySQL binary log API from OSCON in Portland.

Statistics

Views

Total Views
2,072
Views on SlideShare
2,071
Embed Views
1

Actions

Likes
0
Downloads
38
Comments
0

1 Embed 1

https://si0.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    MySQL Binary Log API Presentation - OSCON 2011 MySQL Binary Log API Presentation - OSCON 2011 Presentation Transcript

    • <Insert Picture Here>An API for Reading the MySQL Binary LogMats Kindahl Lars ThalmannLead Software Engineer, MySQL Development Director, MySQLReplication & Utilities Replication, Backup & Connectors
    • The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may not be incorporated into anycontract. It is not a commitment to deliver anymaterial, code, or functionality, and should not berelied upon in making purchasing decisions.The development, release, and timing of anyfeatures or functionality described for Oracle’sproducts remains at the sole discretion of Oracle. 3
    • Outline•Replication Architecture•Binary logs•Binary log event•Reading binary log – Connecting to server – Reading from files•Reading events – Queries – Reading rows (row-based replication) – Other events 4
    • Replication Architecture Master Slave(s)Clients Changes 5
    • Replication Architecture Master Slave(s)Session Databa Databa Databa seSession Client Dump Dump se Dump seSession I/O SQL I/O SQL I/O SQL Binary Log 6
    • Replication Architecture Master Slave(s)Session Databa Database DatabaSession Client Dump Dump se Dump seSession I/O SQL I/O SQL I/O SQL Binary Log Relay Log 7
    • Replication to other systems Slave(s) Database Master I/O SQLSession Relay LogSession Client Dump Dump Dump Dump DataSession HBase Mining Full-text SOLR indexing Binary Log ? 8
    • Transforming events Subject of our presentationSe rv r e API Transformer SOLR le Fi “Change Data Capture” 9
    • Binlog API•Library to process replication events•API is ready for use•Goals: – Simple – Extensible – Efficient 10
    • <Insert Picture Here> Binlog API ● The replication listener How to capture events 11
    • First example#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “mysql://root@127.0.0.1:3360”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 12
    • Create network transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “mysql://root@127.0.0.1:3360”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 13
    • … or file transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 14
    • Connect the transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 15
    • Digression: set read position •Default: start at beginning <Insert Picture Here> •Set position explicitly:if (binlog.set_position(file, pos)){  /* Handle error */} 16
    • Read events#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) { Get event  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event­>get_type_code()         << endl;  }  return EXIT_SUCCESS;} 17
    • Steps summary•Create a transport – create_transport•Connect to server – connect•Set position – set_position•Start event loop – wait_for_next_event 18
    • <Insert Picture Here> Binlog API ● The replication listener Reading information in events 19
    • Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 20
    • Reading the header•Read common header <Insert Picture Here> – header()•Access fieldsswitch (event­>header()­>type_code) {case QUERY_EVENT:  …case USER_VAR_EVENT:  …case FORMAT_DESCRIPTION_EVENT:  …} 21
    • Binlog Event Common Header type_codetimestamp server_id • Data common to all events • Next Position 4 bytes – One-after-end of event • Timestamp – Statement start time • Flags – Binlog-in-use next_position – Thread-specific – Suppress “use” event_length – Artificial flags 19 Bytes – Relay-log event 22
    • Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 23
    • Query Eventthread_id exec_time •Most common event Common Header •Used for statements •Statement logged literally query – … in almost all cases std::vector<uint8_t> variables ed ecod to be d error_code ne eddb_name ca se: Spe cial 24
    • Reading event data•Cast to correct event type <Insert Picture Here>•Access fieldsswitch (event­>header()­>type_code) {case QUERY_EVENT:  Query_event *qev =    static_cast<Query_event*>(event);  cout << qev­>query << endl;  break;case USER_VAR_EVENT:  …case FORMAT_DESCRIPTION_EVENT:  …} 25
    • Event-driven API <Insert Picture Here> 26
    • Event-driven API• Content handlers <Insert Picture Here> wait_for_next_event 27
    • Saving user-defined variablesclass Save_handler  : public Content_handler{ … };Save_handler::Map vars;Save_handler save_vars(vars);binlog.content_handler_pipeline()  ­>push_back(&save_vars); 28
    • User-defined variablesclass Save_handler : public Content_handler {public: <Insert Picture Here>  typedef std::map<std::string, std::string> Map;  Save_handler(Map &container) : m_var(container)  { }  Binary_log_event *  process_event(User_var_event *event) {    m_var[event­>name] = event­>value;    return NULL;  }private:  Map &m_var;}; 29
    • Replace handlerclass Replace_vars :   public Content_handler{  Binary_log_event *  process_event(Query_log_event *event)  {    /* Code to replace variables */  } p cp}; 2. s ic- ba le: mp e xa Full 30
    • <Insert Picture Here> Binlog API ● The replication listener Example two: How to capture live row changes 31
    • Well cover this Row events in the binlog soon (trust me) Table map Header Write rowsTransaction Write rows Write rows Table mapTransaction Delete rows A bunch of rows Map table definition to table ID 32
    • Capturing row eventsclass Row_event_handler :   public Content_handler{public:  Binary_log_event *  process_event(Row_event *event)  {    switch(ev­>header()­>type_code)    {      case WRITE_ROWS_EVENT:      case UPDATE_ROWS_EVENT:      case DELETE_ROWS_EVENT:         ... 33
    • Capturing row events• The *_ROWS_EVENT Defined in the table map eventuint64_t table_id;uint16_t flags;uint64_t columns_len;uint32_t null_bits_len;vector<uint8_t> columns_before_image;vector<uint8_t> used_columns;vector<uint8_t> row; Raw row data 34
    • Reading rows•Wrap raw row data in Row_event_set•Iterate over rows using iteratorRow_event_set rows(row_event, table_map_event);Row_event_set::iterator it= rows.begin(); You need to have captured this before! 35
    • Reading fields of a row•Row_of_fields to iterate fields of a row – Turns row into row of fields sequenceRow_event_set rows(row_event, table_map_event); for (Row_event_set::iterator it = rows.begin() ;      it != rows.end() ;      ++it)   table_delete(os.str(), Row_of_fields(*it)); 36
    • Reading fields of a row• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) {  std::string str;  Converter().to(str, *it);  std::cout << id << "= " << str << std::endl; }} 37
    • Decoding a field• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) {  std::string str;  Converter().to(str, *it);  std::cout << id << "= " << str << std::endl; }} 38
    • Summary – whats it for?•Replicate to other systems – Hbase, SOLR, etc.•Triggering on specific events – Call the DBA when tables are dropped? – Monitor objects in database•Browsing binary logs – Extract subset of changes (by table, by execution time, etc.) – Statistics•Component in building other solutions – Point-in-time restore – Sharding / Load-balancing 39
    • Summary – what weve covered•Reading events•Creating content handlers•Processing queries•Processing rows•Reading fields•… but there is a lot more 40
    • • Available at labs – http://labs.mysql.com/• Source code available at launchpad – http://launchpad.net/mysql-replication-listener• MySQL High Availability Get it as free ebook: http://oreilly.com/go/ebookrequest Valid this week, mention event “MySQL Replication Update” 41