Presentation of the new MySQL binary log API from OSCON in Portland.

2. <Insert Picture Here>
An API for Reading the MySQL Binary Log
Mats Kindahl Lars Thalmann
Lead Software Engineer, MySQL Development Director, MySQL
Replication & Utilities Replication, Backup & Connectors

3. The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
3

4. Outline
•Replication Architecture
•Binary logs
•Binary log event
•Reading binary log
– Connecting to server
– Reading from files
•Reading events
– Queries
– Reading rows (row-based replication)
– Other events
4

5. Replication Architecture
Master Slave(s)
Clients
Changes
5

6. Replication Architecture
Master Slave(s)
Session
Databa
Databa
Databa
se
Session
Client Dump
Dump se
Dump se
Session
I/O SQL
I/O SQL
I/O SQL
Binary Log
6

7. Replication Architecture
Master Slave(s)
Session
Databa
Database
Databa
Session
Client Dump
Dump se
Dump se
Session
I/O SQL
I/O SQL
I/O SQL
Binary Log Relay Log
7

8. Replication to other systems Slave(s)
Database
Master
I/O SQL
Session
Relay Log
Session
Client Dump
Dump
Dump
Dump
Data
Session HBase Mining
Full-text
SOLR indexing
Binary Log
?
8

9. Transforming events
Subject of our presentation
Se
rv
r e
API
Transformer SOLR
le
Fi
“Change Data Capture”
9

10. Binlog API
•Library to process replication events
•API is ready for use
•Goals:
– Simple
– Extensible
– Efficient
10

11. <Insert Picture Here>
Binlog API
●
The replication listener
How to capture events
11

12. First example
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “mysql://root@127.0.0.1:3360”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
12

13. Create network transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “mysql://root@127.0.0.1:3360”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
13

14. … or file transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
14

15. Connect the transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
15

16. Digression: set read position
•Default: start at beginning <Insert Picture Here>
•Set position explicitly:
if (binlog.set_position(file, pos))
{
/* Handle error */
}
16

17. Read events
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) { Get event
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event­>get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
17

18. Steps summary
•Create a transport
– create_transport
•Connect to server
– connect
•Set position
– set_position
•Start event loop
– wait_for_next_event
18

19. <Insert Picture Here>
Binlog API
●
The replication listener
Reading information in events
19

20. Binlog Event Structure
• Common header
Common Header • Generic data
• Fixed size
• Post-header
Post-header
• Event-specific data
• Fixed size
• Variable part
• Event-specific data
Variable Part
• Variable size
20

21. Reading the header
•Read common header <Insert Picture Here>
– header()
•Access fields
switch (event­>header()­>type_code) {
case QUERY_EVENT:
…
case USER_VAR_EVENT:
…
case FORMAT_DESCRIPTION_EVENT:
…
}
21

22. Binlog Event Common Header
type_code
timestamp server_id • Data common to all events
• Next Position
4 bytes – One-after-end of event
• Timestamp
– Statement start time
• Flags
– Binlog-in-use
next_position – Thread-specific
– Suppress “use”
event_length
– Artificial
flags
19 Bytes – Relay-log event
22

23. Binlog Event Structure
• Common header
Common Header • Generic data
• Fixed size
• Post-header
Post-header
• Event-specific data
• Fixed size
• Variable part
• Event-specific data
Variable Part
• Variable size
23

24. Query Event
thread_id exec_time •Most common event
Common Header
•Used for statements
•Statement logged
literally
query – … in almost all cases
std::vector<uint8_t> variables ed
ecod
to be d
error_code
ne ed
db_name ca se:
Spe cial
24

25. Reading event data
•Cast to correct event type <Insert Picture Here>
•Access fields
switch (event­>header()­>type_code) {
case QUERY_EVENT:
Query_event *qev =
static_cast<Query_event*>(event);
cout << qev­>query << endl;
break;
case USER_VAR_EVENT:
…
case FORMAT_DESCRIPTION_EVENT:
…
}
25

26. Event-driven API
<Insert Picture Here>
26

27. Event-driven API
• Content handlers <Insert Picture Here>
wait_for_next_event
27

28. Saving user-defined variables
class Save_handler
: public Content_handler
{ … };
Save_handler::Map vars;
Save_handler save_vars(vars);
binlog.content_handler_pipeline()
­>push_back(&save_vars);
28

29. User-defined variables
class Save_handler : public Content_handler {
public:
<Insert Picture Here>
typedef std::map<std::string, std::string> Map;
Save_handler(Map &container) : m_var(container)
{ }
Binary_log_event *
process_event(User_var_event *event) {
m_var[event­>name] = event­>value;
return NULL;
}
private:
Map &m_var;
};
29

30. Replace handler
class Replace_vars :
public Content_handler
{
Binary_log_event *
process_event(Query_log_event *event)
{
/* Code to replace variables */
} p
cp
}; 2.
s ic-
ba
le:
mp
e xa
Full
30

31. <Insert Picture Here>
Binlog API
●
The replication listener
Example two:
How to capture live row changes
31

32. We'll cover this
Row events in the binlog soon (trust me)
Table map
Header
Write rows
Transaction
Write rows
Write rows
Table map
Transaction
Delete rows A bunch of rows
Map table definition
to table ID
32

33. Capturing row events
class Row_event_handler :
public Content_handler
{
public:
Binary_log_event *
process_event(Row_event *event)
{
switch(ev­>header()­>type_code)
{
case WRITE_ROWS_EVENT:
case UPDATE_ROWS_EVENT:
case DELETE_ROWS_EVENT:
...
33

34. Capturing row events
• The *_ROWS_EVENT
Defined in the
table map event
uint64_t table_id;
uint16_t flags;
uint64_t columns_len;
uint32_t null_bits_len;
vector<uint8_t> columns_before_image;
vector<uint8_t> used_columns;
vector<uint8_t> row;
Raw row data
34

35. Reading rows
•Wrap raw row data in Row_event_set
•Iterate over rows using iterator
Row_event_set rows(row_event, table_map_event);
Row_event_set::iterator it= rows.begin();
You need to have
captured this before!
35

36. Reading fields of a row
•Row_of_fields to iterate fields of a row
– Turns row into row of fields sequence
Row_event_set rows(row_event, table_map_event);
for (Row_event_set::iterator it = rows.begin() ;
it != rows.end() ;
++it)
table_delete(os.str(), Row_of_fields(*it));
36

37. Reading fields of a row
• Iterate over fields in Row_of_fields
void table_delete (..., const Row_of_fields& fields)
{
Row_of_fields::iterator it= fields.begin();
for (int id = 0 ; it =! fields.end() ; ++it, ++id) {
std::string str;
Converter().to(str, *it);
std::cout << id << "= " << str << std::endl;
}
}
37

38. Decoding a field
• Iterate over fields in Row_of_fields
void table_delete (..., const Row_of_fields& fields)
{
Row_of_fields::iterator it= fields.begin();
for (int id = 0 ; it =! fields.end() ; ++it, ++id) {
std::string str;
Converter().to(str, *it);
std::cout << id << "= " << str << std::endl;
}
}
38

39. Summary – what's it for?
•Replicate to other systems
– Hbase, SOLR, etc.
•Triggering on specific events
– Call the DBA when tables are dropped?
– Monitor objects in database
•Browsing binary logs
– Extract subset of changes (by table, by execution time, etc.)
– Statistics
•Component in building other solutions
– Point-in-time restore
– Sharding / Load-balancing
39

40. Summary – what we've covered
•Reading events
•Creating content handlers
•Processing queries
•Processing rows
•Reading fields
•… but there is a lot more
40

41. • Available at labs
– http://labs.mysql.com/
• Source code available at launchpad
– http://launchpad.net/mysql-replication-listener
• MySQL High Availability
Get it as free ebook: http://oreilly.com/go/ebookrequest
Valid this week, mention event “MySQL Replication
Update”
41