<jon@ivt.com.au>



Dealing drugs to Sakila
The problem
Applications are not static.

New versions mean schema changes.

App / schema mismatches are bad.

Schema chan...
Obvious solution



      Update scripts

Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Update scripts



        Run manually

Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Update scripts



Statically defined

Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
A better way?

Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Self-
               Healing
              Databases
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Reasons for change
New tables required.

New columns required.

Alterations to columns.

Alterations to contents of tables...
Failure modes
New tables required.
 “Unknown table”
New columns required.
 “ Unknown column”
Alterations to columns.
 ?
Al...
Reactive,
                    not
                 proactive
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Smart error trapping
1. Run queries blindly.
2. Detect failure conditions.
3. Fix them.
4. Profit!
Self-Healing Databases ...
But...
...if you don't have a db
abstraction layer you're


                         stuffed!
Self-Healing Databases      ...
Build, Borrow or Steal


      One central
     query executor
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
MySQL errors
MySQL has built-in error reporting: use it!

In PHP:
  $errno = mysql_errno($link);
  $error = mysql_error($l...
MySQL errors
Check for specific errors, such as:

  1146: Table doesn't exist
  1054: Unknown column
dev.mysql.com/doc/ref...
Missing table
● Store reference schemas in app
● Trap “1146” errors

● Examine error to determine table name

● Load refer...
Missing table
Embed reference schemas into your app.

[modulename]/sql/articles.sql:

CREATE TABLE `articles` (
`Serial` I...
Missing column
● Record schema changes in “alter” file
● Trap “unknown column” errors

● Load and execute alter file

● Re...
Missing column
Make execution of “alter” file idempotent.

[modulename]/sql/alter.php:

if (!$dbase->field_exists(“news”, ...
Missing trigger

   Missing triggers
   are a problem:
      silent death
Self-Healing Databases   Jonathan Oxer <jon@ivt....
Missing trigger

[modulename]/sql/alter.php:

if (!$dbase->trigger_exists(“UPDATESTOCK”))
{
    $s = “CREATE TRIGGER UPDAT...
Missing procedure
● Put procedure additions in “alter” file
● Trap “1106”, errors

● Load and execute alter file

● Rerun ...
Missing procedure

[modulename]/sql/alter.php:

if (!$dbase->procedure_exists(“DISCOUNTCALC”))
{
    $s = “CREATE PROCEDUR...
That's
  not all,
  folks!
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Problem:

 Multiple module
instances require
 data partitioning
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Solution:

     Three-tier
   dynamic table
  naming scheme
Self-Healing Databases   Jonathan Oxer <jon@ivt.com.au>
Dynamic table names
1: Module instance
2: Module name
3: Specific table
hotstuff_news_articles
hotstuff_news_comments
Self...
Benefits
Storage of schema with
module: error handler can
deduce path from table.
Upgrade of tables when you
don't know th...
Schema templates
Placeholders in reference schemas

[modulename]/sql/articles.sql:

CREATE TABLE <articles> (
`Serial` INT...
“Alter” templates
Make table names in “alter” file dynamic
[modulename]/sql/alter.php:
$articles = $instance.'_news_articl...
Benefits
Stop caring about:

● App / schema mismatches
● Knowing what tables are called

● Telling users to run upgrade sc...
Caveat:
this messes with

      triggers
           and

               procedures
Self-Healing Databases   Jonathan Oxer ...
Is This A Fairy Tale?




Self-Healing Databases     Jonathan Oxer <jon@ivt.com.au>
Is this a fairy tale?
Technique in production use in the
SiteBuilder web application framework
and modules for more than 7...
Is this a fairy tale?
Deployments include:

Siemens national intranet with over 5,000
dynamically managed tables
Shaver Sh...
Self-healing databases


Thankyou :-)
  These slides: jon.oxer.com.au/talks
   We're hiring: www.ivt.com.au/jobs
Flames: J...
Upcoming SlideShare
Loading in...5
×

Self-healing databases: managing schema updates in the field

4,573

Published on

Update database schemas for web applications on live production servers without your users noticing anything

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,573
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
60
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Self-healing databases: managing schema updates in the field

  1. 1. <jon@ivt.com.au> Dealing drugs to Sakila
  2. 2. The problem Applications are not static. New versions mean schema changes. App / schema mismatches are bad. Schema changes mean pain. Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  3. 3. Obvious solution Update scripts Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  4. 4. Update scripts Run manually Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  5. 5. Update scripts Statically defined Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  6. 6. A better way? Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  7. 7. Self- Healing Databases Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  8. 8. Reasons for change New tables required. New columns required. Alterations to columns. Alterations to contents of tables. Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  9. 9. Failure modes New tables required. “Unknown table” New columns required. “ Unknown column” Alterations to columns. ? Alterations to contents of tables. ? Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  10. 10. Reactive, not proactive Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  11. 11. Smart error trapping 1. Run queries blindly. 2. Detect failure conditions. 3. Fix them. 4. Profit! Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  12. 12. But... ...if you don't have a db abstraction layer you're stuffed! Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  13. 13. Build, Borrow or Steal One central query executor Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  14. 14. MySQL errors MySQL has built-in error reporting: use it! In PHP: $errno = mysql_errno($link); $error = mysql_error($link); Specify the link or you'll get the value from the last opened connection, not the last error from your connection. Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  15. 15. MySQL errors Check for specific errors, such as: 1146: Table doesn't exist 1054: Unknown column dev.mysql.com/doc/refman/5.0/en/error-handling.html Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  16. 16. Missing table ● Store reference schemas in app ● Trap “1146” errors ● Examine error to determine table name ● Load reference schema ● Create table ● Rerun original query ● Return result The user never even notices a glitch :-) Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  17. 17. Missing table Embed reference schemas into your app. [modulename]/sql/articles.sql: CREATE TABLE `articles` ( `Serial` INT NOT NULL AUTO_INCREMENT PRIMARY KEY , `Title` VARCHAR( 255 ) NOT NULL , `Article` TEXT NOT NULL ) ENGINE = MYISAM ; Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  18. 18. Missing column ● Record schema changes in “alter” file ● Trap “unknown column” errors ● Load and execute alter file ● Rerun original query ● Return result No harm, no foul. Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  19. 19. Missing column Make execution of “alter” file idempotent. [modulename]/sql/alter.php: if (!$dbase->field_exists(“news”, “Modified”)) { $s = “ALTER TABLE news ADD `Modified` TIMESTAMP NOT NULL”; $dbase->query($s); } Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  20. 20. Missing trigger Missing triggers are a problem: silent death Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  21. 21. Missing trigger [modulename]/sql/alter.php: if (!$dbase->trigger_exists(“UPDATESTOCK”)) { $s = “CREATE TRIGGER UPDATESTOCK ...”; $dbase->query($s); } Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  22. 22. Missing procedure ● Put procedure additions in “alter” file ● Trap “1106”, errors ● Load and execute alter file ● Rerun original query ● Return result (Note: trap “1107” and “1108” errors too, for handling altered procedures) Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  23. 23. Missing procedure [modulename]/sql/alter.php: if (!$dbase->procedure_exists(“DISCOUNTCALC”)) { $s = “CREATE PROCEDURE DISCOUNTCALC ...”; $dbase->query($s); } Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  24. 24. That's not all, folks! Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  25. 25. Problem: Multiple module instances require data partitioning Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  26. 26. Solution: Three-tier dynamic table naming scheme Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  27. 27. Dynamic table names 1: Module instance 2: Module name 3: Specific table hotstuff_news_articles hotstuff_news_comments Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  28. 28. Benefits Storage of schema with module: error handler can deduce path from table. Upgrade of tables when you don't know their name. Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  29. 29. Schema templates Placeholders in reference schemas [modulename]/sql/articles.sql: CREATE TABLE <articles> ( `Serial` INT NOT NULL AUTO_INCREMENT PRIMARY KEY , `Title` VARCHAR( 255 ) NOT NULL , `Article` TEXT NOT NULL ) ENGINE = MYISAM ; Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  30. 30. “Alter” templates Make table names in “alter” file dynamic [modulename]/sql/alter.php: $articles = $instance.'_news_articles'; if (!$dbase->field_exists($articles, “Modified”)) { $s = “ALTER TABLE $articles ADD `Modified` TIMESTAMP NOT NULL”; $dbase->query($s); } Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  31. 31. Benefits Stop caring about: ● App / schema mismatches ● Knowing what tables are called ● Telling users to run upgrade scripts Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  32. 32. Caveat: this messes with triggers and procedures Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  33. 33. Is This A Fairy Tale? Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  34. 34. Is this a fairy tale? Technique in production use in the SiteBuilder web application framework and modules for more than 7 years: ● 1.2 million lines of PHP ● 149 modules ● 11,779 SQL statements ● 1,247 embedded table schemas Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  35. 35. Is this a fairy tale? Deployments include: Siemens national intranet with over 5,000 dynamically managed tables Shaver Shop e-commerce system with tens of thousands of transactions / year Gift Store Online with over 800k users Brisbane Airport security credential system with 10k users / 30k cards Self-Healing Databases Jonathan Oxer <jon@ivt.com.au>
  36. 36. Self-healing databases Thankyou :-) These slides: jon.oxer.com.au/talks We're hiring: www.ivt.com.au/jobs Flames: Jonathan Oxer (jon@ivt.com.au)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×