HailDB A NoSQL API direct to InnoDB Stewart Smith
What is HailDB?
What is InnoDB?
ACID database enigne
Commonly found in MySQL
Originally was separate
Been in MySQL “forever”
HailDB and InnoDB have...
Features you'd expect
Exposed through SQL
Really an interface to an InnoDB API
InnoDB internal API
Complex and sometimes MySQL specific
HailDB history
Embedded InnoDB
A couple of releases...
Abandoned?
Enter HailDB
Completely free and open source project
http://www.haildb.com
Launchpad
$ bzr branch lp:haildb
Tarball releases
Packaged in Ubuntu Natty
Packaged in drizzle-developers PPA
Also RPMs
(Currently) no Win32 release
Source compatible with embedded_innodb
libhaildb.so
haildb.h
Pandora-build provides autoconf foo
What HailDB Provides
Cursor based API
Transactions
Key lookups
Index scans
DIY Joins
Updates
Inserts
Deletes
Use cases
Embedded, fast, concurrent,  relational database
Not arbitrary queries
Not SQL
High-Performance Storage Services Using HailDB and Java Ballroom E, 2:50pm Thursday
Engine for SQL DB
Using the API
ib_init()
ib_cfg_set_int()
ib_cfg_set_bool_on|off()
ib_cfg_set_text()
Set callbacks
ib_startup(“barracuda”)
ib_shutdown(flag)
DDL
ib_database_create() ib_database_drop()
ib_table_create()
ib_table_drop()
ib_table_truncate()
Transactions
ib_trx_begin()
ib_trx_commit() ib_trx_rollback()
ib_savepoint_take() ib_savepoint_release() ib_savepoint_rollback()
Cursors
ib_cursor_open_table()
ib_cursor_close()
ib_cursor_reset()
Position a cursor, perform row operations
Tuples
Represents columns from table record or secondary index
ib_tpl_t tpl = ib_clust_read_tuple_create(cursor)
ib_col_set_value()
ib_col_get_value()
Examples
Full Table Scan
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors  */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);
Insert a row
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);
Updating a row
Delete
ib_tuple_delete()
WHERE clause
Table statistics
HailDB
Questions?

HailDB: A NoSQL API Direct to InnoDB

Editor's Notes

  • #2 Talk to me!
  • #10 ACID compliance Transactions, savepoints, crash recovery, concurrency etc
  • #14 Row buffer format etc
  • #15 Row buffer format etc
  • #16 Clean, easy to use API, shared library. Based on early InnoDB Plugin
  • #17 Clean, easy to use API, shared library. Based on early InnoDB Plugin
  • #18 Oracle certainly doesn't like to announce things.
  • #19 So I created HailDB. Take existing Embedded InnoDB code. Fix bugs, add features. Update InnoDB based on.
  • #20 So I created HailDB. Take existing Embedded InnoDB code. Fix bugs, add features. Update InnoDB based on.
  • #26 Karmic, Lucid and Maverick
  • #27 I don't personally use RPM based distros... but there are people making RPMs
  • #28 Karmic, Lucid and Maverick
  • #29 Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
  • #30 Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
  • #31 Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
  • #32 Detect which HailDB version is present etc.
  • #43 Even where you may have connected to an external RDBMS. Need concurrency that sqlite doesn't provide Need less overhead than InnoDB in MySQL
  • #44 That's what SQL is for
  • #45 Use a SQL database. HailDB can be used to build a SQL database though.
  • #47 Like the HailDB engine in Drizzle. Our work-in-progress next gen way to interface between innobase (HailDB) and the DB kernel
  • #49 Call first. Sets up a few internal things.
  • #52 Used to set configuration options. HailDB does not provide config file parsing or command line option parsing.
  • #53 Log messages (defaults to printing to stderr) trx_is_interrupted (can interrupt threads stuck on locks) Panic handler (elegantly shut down the rest of your app on corruption)
  • #54 Starts InnoDB. Reads tablespaces, does recovery etc.
  • #55 The various innodb shutdown options NORMAL, NO_IBUF_MERGE, NO_BUFPOOL_FLUSH
  • #56 Schema lock. Shared for read, exclusive for write.
  • #57 Creates the directories for a database. Where file-per-table ibd files go. Otherwise, not really that useful :)
  • #58 Create a table. Done inside a transaction. Get DDL lock. Schema specified by API calls, not SQL. No foreign keys (yet)
  • #59 You can do several DDL operations in the same DDL transaction. You can even (within limits) do transactions on other tables. DFE in DDL txn
  • #60 Same as TRUNCATE TABLE in SQL
  • #61 Simple calls for begin/commit/rollback.
  • #62 Supports isolation levels
  • #63 Rather obvious. You can also query the state of the transaction, or in deadlock, ib_trx_release() if txn has been rolled back by HailDB
  • #64 Named savepoints. Maps exactly to what you'd expect from SQL usage. In future we'll have a lightweight “statement” savepoint
  • #65 Can have a cursor not yet attached to a transaction.
  • #66 There's also using_id() methods.
  • #67 When you're done
  • #68 Lets you reuse the data sturctures, increased perf
  • #69 Don't really need for INSERT. Do need for UPDATE and DELETE of course.
  • #71 So can lookup the primary key from a secondary index without having to read the clustered index
  • #72 To read/write row in clustered index. Other types for searching secondary index, redaing row from secondary index
  • #73 Used to set values. e.g. doing an UPDATE or INSERT row operation.
  • #74 A suite of calls both for specific data types (such as 64bit int) and pointer to it in memory (useful for BLOBs).
  • #75 A suite of calls both for specific data types (such as 64bit int) and pointer to it in memory (useful for BLOBs).
  • #93 A little more involved. Read a row, copy it to a new tuple, update that tuple, call ib_cursor_update_row()
  • #95 On a positioned cursor
  • #96 Search an index (or do a full table scan), examining rows until you find the ones you're looking for.
  • #97 Same ones used by MySQL and Drizzle optimisers. Likely not too useful in most HailDB using apps.
  • #98 And that's HailDB. A rather simple interface to an advanced transactional engine.