<Insert Picture Here>An API for Reading the MySQL Binary LogMats Kindahl                   Lars ThalmannLead Software Engi...
The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may no...
Outline•Replication Architecture•Binary logs•Binary log event•Reading binary log  – Connecting to server  – Reading from f...
Replication Architecture             Master              Slave(s)Clients                       Changes                    ...
Replication Architecture          Master           Slave(s)Session                                Databa                  ...
Replication Architecture          Master           Slave(s)Session                                  Databa                ...
Replication to other systems       Slave(s)                                         Database          Master              ...
Transforming events              Subject of our presentationSe   rv    r e            API                  Transformer    ...
Binlog API•Library to process replication events•API is ready for use•Goals: – Simple – Extensible – Efficient            ...
<Insert Picture Here>                                       Binlog API                        ●                          T...
First example#include <cstdlib>#include <iostream>#include <binlog_api.h>                                <Insert Picture H...
Create network transport#include <cstdlib>#include <iostream>#include <binlog_api.h>                                <Inser...
… or file transport#include <cstdlib>#include <iostream>#include <binlog_api.h>                                <Insert Pic...
Connect the transport#include <cstdlib>#include <iostream>#include <binlog_api.h>                                <Insert P...
Digression: set read position   •Default: start at beginning       <Insert Picture Here>   •Set position explicitly:if (bi...
Read events#include <cstdlib>#include <iostream>#include <binlog_api.h>                                <Insert Picture Her...
Steps summary•Create a transport  – create_transport•Connect to server  – connect•Set position  – set_position•Start event...
<Insert Picture Here>                                             Binlog API                              ●               ...
Binlog Event Structure                    • Common header   Common Header      • Generic data                      • Fixed...
Reading the header•Read common header                     <Insert Picture Here>  – header()•Access fieldsswitch (event­>he...
Binlog Event Common Header              type_codetimestamp             server_id    • Data common to all events           ...
Binlog Event Structure                    • Common header   Common Header      • Generic data                      • Fixed...
Query Eventthread_id        exec_time   •Most common event      Common Header                             •Used for statem...
Reading event data•Cast to correct event type             <Insert Picture Here>•Access fieldsswitch (event­>header()­>type...
Event-driven API                   <Insert Picture Here>                                     26
Event-driven API• Content handlers                  <Insert Picture Here>                     wait_for_next_event         ...
Saving user-defined variablesclass Save_handler  : public Content_handler{ … };Save_handler::Map vars;Save_handler save_va...
User-defined variablesclass Save_handler : public Content_handler {public:                                              <I...
Replace handlerclass Replace_vars :   public Content_handler{  Binary_log_event *  process_event(Query_log_event *event)  ...
<Insert Picture Here>                                               Binlog API                                ●           ...
Well cover this     Row events in the binlog                      soon (trust me)              Table map                  ...
Capturing row eventsclass Row_event_handler :   public Content_handler{public:  Binary_log_event *  process_event(Row_even...
Capturing row events• The *_ROWS_EVENT                         Defined in the                         table map eventuint6...
Reading rows•Wrap raw row data in Row_event_set•Iterate over rows using iteratorRow_event_set rows(row_event, table_map_ev...
Reading fields of a row•Row_of_fields to iterate fields of a row – Turns row into row of fields sequenceRow_event_set rows...
Reading fields of a row• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_...
Decoding a field• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_fields:...
Summary – whats it for?•Replicate to other systems  – Hbase, SOLR, etc.•Triggering on specific events  – Call the DBA when...
Summary – what weve covered•Reading events•Creating content handlers•Processing queries•Processing rows•Reading fields•… b...
• Available at labs   – http://labs.mysql.com/• Source code available at launchpad   – http://launchpad.net/mysql-replicat...
MySQL Binary Log API Presentation - OSCON 2011
MySQL Binary Log API Presentation - OSCON 2011
Upcoming SlideShare
Loading in...5
×

MySQL Binary Log API Presentation - OSCON 2011

2,112

Published on

Presentation of the new MySQL binary log API from OSCON in Portland.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,112
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
50
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MySQL Binary Log API Presentation - OSCON 2011

  1. 1. <Insert Picture Here>An API for Reading the MySQL Binary LogMats Kindahl Lars ThalmannLead Software Engineer, MySQL Development Director, MySQLReplication & Utilities Replication, Backup & Connectors
  2. 2. The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may not be incorporated into anycontract. It is not a commitment to deliver anymaterial, code, or functionality, and should not berelied upon in making purchasing decisions.The development, release, and timing of anyfeatures or functionality described for Oracle’sproducts remains at the sole discretion of Oracle. 3
  3. 3. Outline•Replication Architecture•Binary logs•Binary log event•Reading binary log – Connecting to server – Reading from files•Reading events – Queries – Reading rows (row-based replication) – Other events 4
  4. 4. Replication Architecture Master Slave(s)Clients Changes 5
  5. 5. Replication Architecture Master Slave(s)Session Databa Databa Databa seSession Client Dump Dump se Dump seSession I/O SQL I/O SQL I/O SQL Binary Log 6
  6. 6. Replication Architecture Master Slave(s)Session Databa Database DatabaSession Client Dump Dump se Dump seSession I/O SQL I/O SQL I/O SQL Binary Log Relay Log 7
  7. 7. Replication to other systems Slave(s) Database Master I/O SQLSession Relay LogSession Client Dump Dump Dump Dump DataSession HBase Mining Full-text SOLR indexing Binary Log ? 8
  8. 8. Transforming events Subject of our presentationSe rv r e API Transformer SOLR le Fi “Change Data Capture” 9
  9. 9. Binlog API•Library to process replication events•API is ready for use•Goals: – Simple – Extensible – Efficient 10
  10. 10. <Insert Picture Here> Binlog API ● The replication listener How to capture events 11
  11. 11. First example#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “mysql://root@127.0.0.1:3360”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 12
  12. 12. Create network transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “mysql://root@127.0.0.1:3360”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 13
  13. 13. … or file transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 14
  14. 14. Connect the transport#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) {  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event.get_type_code()         << endl;  }  return EXIT_SUCCESS;} 15
  15. 15. Digression: set read position •Default: start at beginning <Insert Picture Here> •Set position explicitly:if (binlog.set_position(file, pos)){  /* Handle error */} 16
  16. 16. Read events#include <cstdlib>#include <iostream>#include <binlog_api.h> <Insert Picture Here>int main(int argc, char *argv[]) { Get event  const char *url = “file:///tmp/binlog.0000001”;  Binary_log binlog(create_transport(url));  binlog.connect();  Binary_log_event *event;  while (true) {    int result = binlog.wait_for_next_event(&event);    if (result == ERR_EOF)      break;    cout << “ at “ << binlog.get_position()         << “ event type “ << event­>get_type_code()         << endl;  }  return EXIT_SUCCESS;} 17
  17. 17. Steps summary•Create a transport – create_transport•Connect to server – connect•Set position – set_position•Start event loop – wait_for_next_event 18
  18. 18. <Insert Picture Here> Binlog API ● The replication listener Reading information in events 19
  19. 19. Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 20
  20. 20. Reading the header•Read common header <Insert Picture Here> – header()•Access fieldsswitch (event­>header()­>type_code) {case QUERY_EVENT:  …case USER_VAR_EVENT:  …case FORMAT_DESCRIPTION_EVENT:  …} 21
  21. 21. Binlog Event Common Header type_codetimestamp server_id • Data common to all events • Next Position 4 bytes – One-after-end of event • Timestamp – Statement start time • Flags – Binlog-in-use next_position – Thread-specific – Suppress “use” event_length – Artificial flags 19 Bytes – Relay-log event 22
  22. 22. Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 23
  23. 23. Query Eventthread_id exec_time •Most common event Common Header •Used for statements •Statement logged literally query – … in almost all cases std::vector<uint8_t> variables ed ecod to be d error_code ne eddb_name ca se: Spe cial 24
  24. 24. Reading event data•Cast to correct event type <Insert Picture Here>•Access fieldsswitch (event­>header()­>type_code) {case QUERY_EVENT:  Query_event *qev =    static_cast<Query_event*>(event);  cout << qev­>query << endl;  break;case USER_VAR_EVENT:  …case FORMAT_DESCRIPTION_EVENT:  …} 25
  25. 25. Event-driven API <Insert Picture Here> 26
  26. 26. Event-driven API• Content handlers <Insert Picture Here> wait_for_next_event 27
  27. 27. Saving user-defined variablesclass Save_handler  : public Content_handler{ … };Save_handler::Map vars;Save_handler save_vars(vars);binlog.content_handler_pipeline()  ­>push_back(&save_vars); 28
  28. 28. User-defined variablesclass Save_handler : public Content_handler {public: <Insert Picture Here>  typedef std::map<std::string, std::string> Map;  Save_handler(Map &container) : m_var(container)  { }  Binary_log_event *  process_event(User_var_event *event) {    m_var[event­>name] = event­>value;    return NULL;  }private:  Map &m_var;}; 29
  29. 29. Replace handlerclass Replace_vars :   public Content_handler{  Binary_log_event *  process_event(Query_log_event *event)  {    /* Code to replace variables */  } p cp}; 2. s ic- ba le: mp e xa Full 30
  30. 30. <Insert Picture Here> Binlog API ● The replication listener Example two: How to capture live row changes 31
  31. 31. Well cover this Row events in the binlog soon (trust me) Table map Header Write rowsTransaction Write rows Write rows Table mapTransaction Delete rows A bunch of rows Map table definition to table ID 32
  32. 32. Capturing row eventsclass Row_event_handler :   public Content_handler{public:  Binary_log_event *  process_event(Row_event *event)  {    switch(ev­>header()­>type_code)    {      case WRITE_ROWS_EVENT:      case UPDATE_ROWS_EVENT:      case DELETE_ROWS_EVENT:         ... 33
  33. 33. Capturing row events• The *_ROWS_EVENT Defined in the table map eventuint64_t table_id;uint16_t flags;uint64_t columns_len;uint32_t null_bits_len;vector<uint8_t> columns_before_image;vector<uint8_t> used_columns;vector<uint8_t> row; Raw row data 34
  34. 34. Reading rows•Wrap raw row data in Row_event_set•Iterate over rows using iteratorRow_event_set rows(row_event, table_map_event);Row_event_set::iterator it= rows.begin(); You need to have captured this before! 35
  35. 35. Reading fields of a row•Row_of_fields to iterate fields of a row – Turns row into row of fields sequenceRow_event_set rows(row_event, table_map_event); for (Row_event_set::iterator it = rows.begin() ;      it != rows.end() ;      ++it)   table_delete(os.str(), Row_of_fields(*it)); 36
  36. 36. Reading fields of a row• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) {  std::string str;  Converter().to(str, *it);  std::cout << id << "= " << str << std::endl; }} 37
  37. 37. Decoding a field• Iterate over fields in Row_of_fieldsvoid table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) {  std::string str;  Converter().to(str, *it);  std::cout << id << "= " << str << std::endl; }} 38
  38. 38. Summary – whats it for?•Replicate to other systems – Hbase, SOLR, etc.•Triggering on specific events – Call the DBA when tables are dropped? – Monitor objects in database•Browsing binary logs – Extract subset of changes (by table, by execution time, etc.) – Statistics•Component in building other solutions – Point-in-time restore – Sharding / Load-balancing 39
  39. 39. Summary – what weve covered•Reading events•Creating content handlers•Processing queries•Processing rows•Reading fields•… but there is a lot more 40
  40. 40. • Available at labs – http://labs.mysql.com/• Source code available at launchpad – http://launchpad.net/mysql-replication-listener• MySQL High Availability Get it as free ebook: http://oreilly.com/go/ebookrequest Valid this week, mention event “MySQL Replication Update” 41
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×