Midas
On-the-fly-Schema-Migration Tool
For
Schema Migration Problems
Applications have to hand-roll their own schema
migration infrastructure or use some third-party...
Zero-downtime Deployment
Expansion Scripts
Apply changes to the documents safely that do not
break backwards compatibility...
The Mechanics
Run Expansion scripts before upgrading the application
Upgrade the cluster, a node at a time
Run Contraction...
Do we need DB rollback?
Short Answer
No
!

Long Answer
Reversing DB changes can lead to potential loss of
data or leave it...
How it works?
An Architectural Overview
Application
How it works?
An Architectural Overview
Application

Midas

Intercepts Responses at Protocol Level
Upgrades/Downgrades Doc...
Protocol Level brings
Transparency
From the App perspective
Midas is Agnostic of Language specific drivers and versions
wit...
Deep Dive

I’m just a decorating
proxy!

Configurer

Parser

Delta Files
(written in a DSL)

Res

Res

Transformer

Req

Re...
Delta scripts
Group ChangeSet as Directories
Group deltas as Expansion and/
or Contraction deltas as
Directories within Ch...
Delta Scripts - Convention
Must begin change set directories with a number and
subsequent change set directories are numbe...
Sample Delta Script
Each Delta script is written using a DSL
Very close to MongoDB lingo, virtually no learning
curve.
use...
Agile App Delivery & DevOps
Inject Midas into Architecture
Start or Middle of project

Supports Development of Application...
No Leaky Abstractions
Does not expect the Domain Model to be aware of
versioning.
Allows developers to focus on the domain...
Caveats
Never ever change a delta that has been applied to
production.
Always move forward in time.
Reverse a change by a ...
Midas Domain Model
Configuration
1
*

ResponseInterceptor
Node

*
1

Application

RequestInterceptor

1
1

Transformer
Pars...
Zero-Downtime Deployment
Configuration - 1
!

Injecting and Using Midas
Deployment config - 1
Site A
App v0
App v0
Load
Balancer

Site B
App v0
0

App v0

Offline does not mean
node is physically ...
Deployment config - 1
Site A
App v0

1. Inject Midas in to the architecture with
app configured in Expansion Mode
2. Start w...
Deployment config - 1
Site A

4. For all nodes in Site B - v1, Apply
new change set, say changeSet = 3

App v0
App v0
Load
...
Deployment config - 1
Site A
App v0
App v0
Load
Balancer

Midas
Site B
App v1

5. Flip to Site B

3

App v1

Offline does no...
Deployment config - 1
Site A
App v0
App v0
Load
Balancer

Midas
Site B
App v1

4

App v1

6. Confirm that App v1 is stable, ...
Deployment config - 1
Site A
App v0

8. Change Application mode to Contraction.

App v0
Load
Balancer

Midas
Site B
App v1
...
Deployment config - 1
Site A
App v0
App v0
Load
Balancer

Midas
Site B
App v1

6

App v1

9. Make Sure that Contraction of ...
Deployment config - 1
Site A
App v0

10. Prepare for next Release and Repeat
steps from #1.

App v0
Load
Balancer

Midas
Si...
Zero-Downtime Deployment
Configuration - 1
!

Node Removal at Runtime
2. Remove Node0 from
app.midas

!
Say, due to some problems, you want to
remove Node0 from service:

!

1. Remove Node0 fr...
2. Midas will sever all connections
with Node0

Site A
Load
Balancer

Node0

Midas

Node1

1

Offline does not mean
node is...
Zero-Downtime Deployment
Configuration - 1
!

Node Injection at Runtime
You want to inject new node for App
in service
1. Add Node2 to app.midas.

Site A
Load
Balancer

Node0
Node1
Node2

0

dem...
3. Add Node2 to LB.

Site A
Load
Balancer

Node0
Node1

Midas

Node2
Online
1

Offline
Zero-Downtime Deployment
Configuration - 1
!

App Removal at Runtime
Deployment config - 1
Site A

App1
Load
Balancer

2. Stop App2 LB so
that it stops receiving
requests

App2
Load
Balancer

...
1. Remove App2 from midas.config

!
Site A

App1
Load
Balancer

apps {!
demoApp1!
// demoApp2!
}!

!
Site B

Note: You don...
Zero-Downtime Deployment
Configuration - 1
!

App Injection at Runtime
Site A

App1
Load
Balancer

Site B

1. Say, you want to add App2
to Midas.

Midas
Site A

App2
Load
Balancer

Site B

2. S...
3. Create App2 dir within deltas dir
4. Set-up change sets with expansions
and contractions in App2 dir.

Site A

App1
Loa...
5. Add demoApp2.midas (with
mode and nodes configured)

Site A

App1
Load
Balancer

Site B

demoApp2Ver1_0 {!
mode = expans...
Site A

App1
Load
Balancer

9. Start the LB so that
App2 is live

App2
Load
Balancer

6. Add App2 to midas.config
apps {!
...
Zero-Downtime Deployment
Configuration - 2
Not Supported in this Release
Deployment config - 2
Node A
App v1
Load
Balancer

Node B
App v1
Node C
App v1

0

Online
Offline
Zero-Downtime Deployment
Configuration - 3
AWS
Not Supported in this Release
Deployment config - 3
Node A
App v1
Load
Balancer

Node B
App v1
Node C
App v1

0

Online
Offline
The Team
Brian Blignaut
bblignaut@equalexperts.com

Dhaval Dalal [@softwareartisan]
ddalal@equalexperts.com

Vivek Dhapola...
References

Owen Rogers
http://exortech.com/blog/2009/02/01/weeklyrelease-blog-11-zero-downtime-databasedeployment/
Thank-You
Upcoming SlideShare
Loading in …5
×

Midas - on-the-fly schema migration tool for MongoDB.

7,072 views

Published on

Midas is an on-the-fly schema migration for MongoDB.

Currently, applications have to hand-roll their own schema migration infrastructure or use some third-party tool.
It is difficult to migrate TBs of data without downtime (unacceptable from SLA stand-point!).
This is where Midas fills the gap.

It intercepts responses at MongoDB Protocol level and upgrades or downgrades document schema in-transit.
As Midas works at protocol level, it is agnostic of Language specific MongoDB drivers (Ruby, Python, C#
and Java drivers) and their versions within those languages.

Further, Midas is agnostic of the MongoDB configurations like Standalone, Replica Sets, Sharded environments.

Features Summary:
* Expansion operations - add, copy, merge, split, transform.
* Contraction operations - remove
* Schema migration for multiple applications simultaneously
* Support multi node configuration for application
* Without shutting down Midas, you can -
* Add or remove applications on-the-fly
* Add or remove nodes on-the-fly
* Add or remove deltas/changeset on-the-fly

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,072
On SlideShare
0
From Embeds
0
Number of Embeds
2,063
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Midas - on-the-fly schema migration tool for MongoDB.

  1. 1. Midas On-the-fly-Schema-Migration Tool For
  2. 2. Schema Migration Problems Applications have to hand-roll their own schema migration infrastructure or use some third-party tool ! Difficult to migrate TBs of data without downtime unacceptable from SLA stand-point! ! How about: on-the-fly schema migration - a Midas Touch?
  3. 3. Zero-downtime Deployment Expansion Scripts Apply changes to the documents safely that do not break backwards compatibility with existing version of the application. e.g Adding, copying, merging, splitting fields in a document. Contraction Scripts Clean up any database schema that is not needed after the upgrade. e.g. removing fields from a document.
  4. 4. The Mechanics Run Expansion scripts before upgrading the application Upgrade the cluster, a node at a time Run Contraction Scripts Once the system has been completely upgraded and deemed stable. Typically, contractions can be run, say days/weeks after complete validation.
  5. 5. Do we need DB rollback? Short Answer No ! Long Answer Reversing DB changes can lead to potential loss of data or leave it in an inconsistent state. Its safer to rollback application without needing to rollback DB changes as expansions are backward compatible.
  6. 6. How it works? An Architectural Overview Application
  7. 7. How it works? An Architectural Overview Application Midas Intercepts Responses at Protocol Level Upgrades/Downgrades Document schema in-transit
  8. 8. Protocol Level brings Transparency From the App perspective Midas is Agnostic of Language specific drivers and versions within those languages Works with versions of Ruby, Python, C# and Java drivers. From MongoDB perspective Midas is Agnostic of the MongoDB configurations Works with Standalone, Replica Sets, Sharded environments.
  9. 9. Deep Dive I’m just a decorating proxy! Configurer Parser Delta Files (written in a DSL) Res Res Transformer Req Req Application Midas
  10. 10. Delta scripts Group ChangeSet as Directories Group deltas as Expansion and/ or Contraction deltas as Directories within ChangeSet Write delta scripts using the .delta extension Midas relies on the order specified by you Ordering info is embedded by you within the change set directory name and delta files
  11. 11. Delta Scripts - Convention Must begin change set directories with a number and subsequent change set directories are numbered in ascending order. Example: use <changesetNumber>-<featureName> as convention for naming change set directory. Within a change set, expansion and contraction scripts are grouped by folders - expansions and contractions. Always begin delta scripts with a number and subsequent delta scripts are numbered in ascending order. Example: use <changeNumber>_<WhatTheChangeIs>.delta as convention for naming delta files
  12. 12. Sample Delta Script Each Delta script is written using a DSL Very close to MongoDB lingo, virtually no learning curve. use users db.customers.remove(‘["address.line1"]') db.customers.merge([“lname”, “fname”], “ “, “name”) ! ! use transactions db.orders.add(‘dispatch : { status: ‘NOT DELIVERED’}’)
  13. 13. Agile App Delivery & DevOps Inject Midas into Architecture Start or Middle of project Supports Development of Application in small-steps Add Application ChangeSets/Deltas on-the-fly Copes with Load Add/Remove Application Nodes on-the-fly Supports Multiple Applications Add/Remove Application(s) on-the-fly
  14. 14. No Leaky Abstractions Does not expect the Domain Model to be aware of versioning. Allows developers to focus on the domain while freeing them from versioning concerns. If you wish to take charge, Midas will not come in your way. Midas maintains versioning information within the document itself. _expansionVersion and _contractionVersion fields are part of the documents “touched” by Midas. Updates them during request and response.
  15. 15. Caveats Never ever change a delta that has been applied to production. Always move forward in time. Reverse a change by a counter-change. Force-update migration on documents that are not expanded by App demand, and then proceed to contraction. contracted by App demand, and then proceed to next Appupgrade cycle. Currently Midas supports Zero Downtime Configuration - 1 (see later slides), others will be supported in subsequent releases.
  16. 16. Midas Domain Model Configuration 1 * ResponseInterceptor Node * 1 Application RequestInterceptor 1 1 Transformer Parsed and Injected in <<Folder>> 1 ChangeSet * <<File>> Delta 1 * <<Verb>> Transformation
  17. 17. Zero-Downtime Deployment Configuration - 1 ! Injecting and Using Midas
  18. 18. Deployment config - 1 Site A App v0 App v0 Load Balancer Site B App v0 0 App v0 Offline does not mean node is physically taken down or is off Online Offline
  19. 19. Deployment config - 1 Site A App v0 1. Inject Midas in to the architecture with app configured in Expansion Mode 2. Start with all Sites, having changeSet = 0 (no change) for all nodes App v0 Load Balancer Midas Site B App v0 1 App v0 Offline does not mean node is physically taken down or is off Online Offline
  20. 20. Deployment config - 1 Site A 4. For all nodes in Site B - v1, Apply new change set, say changeSet = 3 App v0 App v0 Load Balancer Midas Site B App v1 2 App v1 3. Upgrade Site B to App v1. Offline does not mean node is physically taken down or is off Online Offline
  21. 21. Deployment config - 1 Site A App v0 App v0 Load Balancer Midas Site B App v1 5. Flip to Site B 3 App v1 Offline does not mean node is physically taken down or is off Online Offline
  22. 22. Deployment config - 1 Site A App v0 App v0 Load Balancer Midas Site B App v1 4 App v1 6. Confirm that App v1 is stable, in case it is not, bug fix and re-deploy it. 7. Once deemed stable, Make sure that expansion of all documents is complete. This can be currently achieved by writing a simple script, that does a find and save thru’ Midas. Online Offline
  23. 23. Deployment config - 1 Site A App v0 8. Change Application mode to Contraction. App v0 Load Balancer Midas Site B App v1 5 App v1 Online Offline
  24. 24. Deployment config - 1 Site A App v0 App v0 Load Balancer Midas Site B App v1 6 App v1 9. Make Sure that Contraction of all documents is complete. This can be achieved by writing a simple script that does find and save through Midas. Online Offline
  25. 25. Deployment config - 1 Site A App v0 10. Prepare for next Release and Repeat steps from #1. App v0 Load Balancer Midas Site B App v1 7 App v1 Offline does not mean node is physically taken down or is off Online Offline
  26. 26. Zero-Downtime Deployment Configuration - 1 ! Node Removal at Runtime
  27. 27. 2. Remove Node0 from app.midas ! Say, due to some problems, you want to remove Node0 from service: ! 1. Remove Node0 from LB. Site A Load Balancer Node0 demoAppV1 {! mode = contraction! // siteANode0 {! // ip = x.x.x.x! // changeSet = 3! // }! siteANode1 {! ip = y.y.y.y! changeSet = 3! }! } Midas Node1 Online 0 Offline
  28. 28. 2. Midas will sever all connections with Node0 Site A Load Balancer Node0 Midas Node1 1 Offline does not mean node is physically taken down or is off Online Offline
  29. 29. Zero-Downtime Deployment Configuration - 1 ! Node Injection at Runtime
  30. 30. You want to inject new node for App in service 1. Add Node2 to app.midas. Site A Load Balancer Node0 Node1 Node2 0 demoAppV1 {! mode = expansion! siteANode0 {! ip = x.x.x.x! changeSet = 3! }! siteANode1 {! ip = y.y.y.y! changeSet = 3! }! siteANode2 {! ip = z.z.z.z! changeSet = 3! }! } Midas 2. Midas is ready to receive connections from Node2 Offline does not mean node is physically taken down or is off Online Offline
  31. 31. 3. Add Node2 to LB. Site A Load Balancer Node0 Node1 Midas Node2 Online 1 Offline
  32. 32. Zero-Downtime Deployment Configuration - 1 ! App Removal at Runtime
  33. 33. Deployment config - 1 Site A App1 Load Balancer 2. Stop App2 LB so that it stops receiving requests App2 Load Balancer 0 Site B Midas Site A Site B 1. Say, you want to remove App2 from Midas. Online Offline
  34. 34. 1. Remove App2 from midas.config ! Site A App1 Load Balancer apps {! demoApp1! // demoApp2! }! ! Site B Note: You don’t have to remove demoApp2 dir or demoApp2.midas to remove the application from Midas (this allows you to re-inject App2 should the need be) Midas Site A App2 Load Balancer 2. Midas will sever all connections from App2 Site B 3. You can now take down App2 1 Online Offline
  35. 35. Zero-Downtime Deployment Configuration - 1 ! App Injection at Runtime
  36. 36. Site A App1 Load Balancer Site B 1. Say, you want to add App2 to Midas. Midas Site A App2 Load Balancer Site B 2. Setup the infrastructure for App2 (Sites and Nodes). Online 0 Offline
  37. 37. 3. Create App2 dir within deltas dir 4. Set-up change sets with expansions and contractions in App2 dir. Site A App1 Load Balancer Site B Midas Site A App2 Load Balancer Site B Online 1 Offline
  38. 38. 5. Add demoApp2.midas (with mode and nodes configured) Site A App1 Load Balancer Site B demoApp2Ver1_0 {! mode = expansion! siteANode1 {! ip = a.a.a.a! changeSet = 2! }! …! …! siteBNode1 {! ip = p.p.p.p! changeSet = 2! }! } Midas Site A App2 Load Balancer Site B Online 2 Offline
  39. 39. Site A App1 Load Balancer 9. Start the LB so that App2 is live App2 Load Balancer 6. Add App2 to midas.config apps {! demoApp1! demoApp2! }! ! Site B 7. Midas is ready to receive connections from App2 Midas Site A Site B 8. Start any Site that can go-live, say Site A here 2 Online Offline
  40. 40. Zero-Downtime Deployment Configuration - 2 Not Supported in this Release
  41. 41. Deployment config - 2 Node A App v1 Load Balancer Node B App v1 Node C App v1 0 Online Offline
  42. 42. Zero-Downtime Deployment Configuration - 3 AWS Not Supported in this Release
  43. 43. Deployment config - 3 Node A App v1 Load Balancer Node B App v1 Node C App v1 0 Online Offline
  44. 44. The Team Brian Blignaut bblignaut@equalexperts.com Dhaval Dalal [@softwareartisan] ddalal@equalexperts.com Vivek Dhapola vdhapola@equalexperts.com Komal Jain kjain@equalexperts.com
  45. 45. References Owen Rogers http://exortech.com/blog/2009/02/01/weeklyrelease-blog-11-zero-downtime-databasedeployment/
  46. 46. Thank-You

×