0
Migrating Legacy Data
Patrick Crowley
 Rob Kaufman
(We’re from SD Ruby)
Let’s talk about migrating
Suck => Less Suck
PHP => Rails
Perl => Rails
Java => Rails
Rails => Rails
Migrating = pain
• Develop new site
• Migrate data
• Deploy new site
Migrate data
• Dump database
• Drop some tables
• Rename attributes
• Tweak data by hand
• Import data into new app
• Pray things are o...
Sucks, right?
• Labor intensive
• Error prone
• One way trip
Trucker
Migrate legacy data
  (with less suck)
• Migrate data from day one
• Improve data over time
• Launch with confidence
rake +
legacy classes +
migration helper
Let’s get started
sudo gem install trucker
config.gem “trucker”
script/generate truck
$ script/generate truck
    create app/models/legacy
    create app/models/legacy/legacy_base.rb
    exists lib/tasks
    ...
config/environment.rb
Rails::Initializer.run do |config|
  config.load_paths += %W( #{RAILS_ROOT}/app/
models/legacy )
end
config/database.yml
legacy:
  adapter: mysql
  encoding: utf8
  database: trucker_legacy
  username: root
  password:
app/models/legacy/legacy_base.rb
class LegacyBase < ActiveRecord::Base
  self.abstract_class = true
  establish_connection...
app/models/legacy/legacy_post.rb
class LegacyPost < LegacyBase
  set_table_name "blog_posts"

  def map
    {
      :name ...
app/models/legacy/legacy_post.rb
class LegacyPost < LegacyBase
  set_table_name "blog_posts"

  def map
    {
      :name ...
lib/tasks/legacy.rake
namespace :db do
  namespace :migrate do

    desc 'Migrates posts'
    task :posts => :environment ...
Let’s do some migrating.
Don’t forget to import
your legacy database!
$ rake db:migrate:posts

Migrating all posts (1/10)
Migrating all posts (2/10)
Migrating all posts (3/10)
Migrating all po...
$ rake db:migrate:posts limit=5

Migrating 5 posts (1/10)
Migrating 5 posts (2/10)
Migrating 5 posts (3/10)
Migrating 5 po...
$ rake db:migrate:posts limit=5 offset=5

Migrating 5 posts after 5 (6/10)
Migrating 5 posts after 5 (7/10)
Migrating 5 po...
DEMO
Use helper method for
 custom migrations
namespace :db do
  namespace :migrate do

    desc 'Migrate pain_in_the_ass model'
    task :pain_in_the_ass => :environme...
What about?!?
Trucker helps you move.
But you still need to
pack your stuff up.
Some other things
  to think about
Encoding issues
• MySQL issues:
 “ALTER TABLE mytable
 CONVERT TO CHARACTER
 SET utf8;

• iconv
Java to the rescue?!?
JDBC Adapters vs
  ActiveRecord
• There are about 12 adapters in
  the wild for pure ActiveRecord

• The ActiveRecord adapter for
  sqlserver runs on Win/...
JDBC Adapters
ActiveRecord Tested:
 MySQL            Derby
 PostgreSQL       HSQLDB

 Oracle           H2

 Microsoft SQL    SQL...
MSSQL – An Example
• Download MSSQL driver
  (Google for it, they move it
  around a lot)

• Copy sqljdbc4.jar into
  RAILS_ROOT/lib

• Add r...
config/database.yml
legacy:
  adapter: jdbc
  username: USERNAME
  password: PASSWORD
  driver: com.microsoft.sqlserver.jdb...
Take it live
• Run in prod when you can
• Run locally when you can't
 (Heroku, VPNs, etc)
Resources
•   http://github.com/mokolabs/trucker

•   http://github.com/mokolabs/
    trucker_sample_app

•   http://spkr8.com/t/3749
Special thanks
to Dave Thomas
The End
Migrating Legacy Data (Ruby Midwest)
Migrating Legacy Data (Ruby Midwest)
Upcoming SlideShare
Loading in...5
×

Migrating Legacy Data (Ruby Midwest)

1,967

Published on

Slides from talk on legacy data migration. Includes introduction of Trucker gem and covers common migration issues.

This talk was given by Patrick Crowley and Rob Kaufman at RubyMidwest 2010 in Kansas City, MO.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,967
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Migrating Legacy Data (Ruby Midwest)"

  1. 1. Migrating Legacy Data
  2. 2. Patrick Crowley Rob Kaufman
  3. 3. (We’re from SD Ruby)
  4. 4. Let’s talk about migrating
  5. 5. Suck => Less Suck
  6. 6. PHP => Rails
  7. 7. Perl => Rails
  8. 8. Java => Rails
  9. 9. Rails => Rails
  10. 10. Migrating = pain
  11. 11. • Develop new site • Migrate data • Deploy new site
  12. 12. Migrate data
  13. 13. • Dump database • Drop some tables • Rename attributes • Tweak data by hand • Import data into new app • Pray things are okay
  14. 14. Sucks, right?
  15. 15. • Labor intensive • Error prone • One way trip
  16. 16. Trucker
  17. 17. Migrate legacy data (with less suck)
  18. 18. • Migrate data from day one • Improve data over time • Launch with confidence
  19. 19. rake + legacy classes + migration helper
  20. 20. Let’s get started
  21. 21. sudo gem install trucker
  22. 22. config.gem “trucker”
  23. 23. script/generate truck
  24. 24. $ script/generate truck create app/models/legacy create app/models/legacy/legacy_base.rb exists lib/tasks create lib/tasks/legacy.rake insert added legacy adapter to end of database.yml insert added new load path to environment.rb
  25. 25. config/environment.rb Rails::Initializer.run do |config| config.load_paths += %W( #{RAILS_ROOT}/app/ models/legacy ) end
  26. 26. config/database.yml legacy: adapter: mysql encoding: utf8 database: trucker_legacy username: root password:
  27. 27. app/models/legacy/legacy_base.rb class LegacyBase < ActiveRecord::Base self.abstract_class = true establish_connection "legacy" def migrate new_record = self.class.to_s.gsub(/ Legacy/,'::').constantize.new(map) new_record[:id] = self.id new_record.save end
  28. 28. app/models/legacy/legacy_post.rb class LegacyPost < LegacyBase set_table_name "blog_posts" def map { :name => self.headline.squish, :body => self.body.squish } end end
  29. 29. app/models/legacy/legacy_post.rb class LegacyPost < LegacyBase set_table_name "blog_posts" def map { :name => tweak(self.headline.squish), :body => self.body.squish } end def tweak(name) name.capitalize.gsub(/teh/, "the") end
  30. 30. lib/tasks/legacy.rake namespace :db do namespace :migrate do desc 'Migrates posts' task :posts => :environment do Trucker.migrate :posts end end end
  31. 31. Let’s do some migrating.
  32. 32. Don’t forget to import your legacy database!
  33. 33. $ rake db:migrate:posts Migrating all posts (1/10) Migrating all posts (2/10) Migrating all posts (3/10) Migrating all posts (4/10) Migrating all posts (5/10) Migrating all posts (6/10) Migrating all posts (7/10) Migrating all posts (8/10) Migrating all posts (9/10) Migrating all posts (10/10)
  34. 34. $ rake db:migrate:posts limit=5 Migrating 5 posts (1/10) Migrating 5 posts (2/10) Migrating 5 posts (3/10) Migrating 5 posts (4/10) Migrating 5 posts (5/10)
  35. 35. $ rake db:migrate:posts limit=5 offset=5 Migrating 5 posts after 5 (6/10) Migrating 5 posts after 5 (7/10) Migrating 5 posts after 5 (8/10) Migrating 5 posts after 5 (9/10) Migrating 5 posts after 5 (10/10)
  36. 36. DEMO
  37. 37. Use helper method for custom migrations
  38. 38. namespace :db do namespace :migrate do desc 'Migrate pain_in_the_ass model' task :pain_in_the_ass => :environment do Trucker.migrate :pain_in_the_ass, :helper => pain_in_the_ass_migration end end end def pain_in_the_ass_migration # Custom code goes here end
  39. 39. What about?!?
  40. 40. Trucker helps you move.
  41. 41. But you still need to pack your stuff up.
  42. 42. Some other things to think about
  43. 43. Encoding issues
  44. 44. • MySQL issues: “ALTER TABLE mytable CONVERT TO CHARACTER SET utf8; • iconv
  45. 45. Java to the rescue?!?
  46. 46. JDBC Adapters vs ActiveRecord
  47. 47. • There are about 12 adapters in the wild for pure ActiveRecord • The ActiveRecord adapter for sqlserver runs on Win/Unix, but Unix setup is complicated • Text adapter does CSV, Tab, and other plain text formats
  48. 48. JDBC Adapters
  49. 49. ActiveRecord Tested:  MySQL  Derby  PostgreSQL  HSQLDB  Oracle  H2  Microsoft SQL  SQLite3 Server  Informix  DB2  FireBird
  50. 50. MSSQL – An Example
  51. 51. • Download MSSQL driver (Google for it, they move it around a lot) • Copy sqljdbc4.jar into RAILS_ROOT/lib • Add require 'lib/sqljdbc4.jar' at the top of your environment.rb or application.rb
  52. 52. config/database.yml legacy: adapter: jdbc username: USERNAME password: PASSWORD driver: com.microsoft.sqlserver.jdbc.SQLServerDriver jdbc: sqlserver://63.134.199.59:1433
  53. 53. Take it live
  54. 54. • Run in prod when you can • Run locally when you can't (Heroku, VPNs, etc)
  55. 55. Resources
  56. 56. • http://github.com/mokolabs/trucker • http://github.com/mokolabs/ trucker_sample_app • http://spkr8.com/t/3749
  57. 57. Special thanks to Dave Thomas
  58. 58. The End
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×