Data Migration
in an Agile Open Source World


   Craig Smith
   Suncorp
Welcome…




Image http://www.neopoleon.com/blog/images/excel/7.jpg
According to Wikipedia:

                                                                                   Data migration...
“In” End
              Point


 Extract               Extract

Transform
  Load                           Transform


  (E...
File transfer




(Enterprise)                                                       Shared
                              ...
Data
 Migration
     +
Application
Integration
     =
  Similar
 Approach
              Image http://kansolutions.net/yaho...
Image http://blog.theworkinggroup.ca/wp-content/uploads/2009/04/agile-manifesto1.gif




                                 ...
Scrum Practices




Image http://blogs.conchango.com/Admin/ImageGallery/blogs.conchango.com/Colin.Bird/Scrum%20Overview%20...
Image Beck, Kent – Extreme Programming Explained: Embrace Change




                                                     ...
Development Practices
                            Fixing a                                                                ...
Image http://iqizone.com/images/paper%20pile.jpg




                Break It Down... Iteratively
Agile Builds Confidence




Image http://static.soxfirst.com/soxfirst.com/imgname--derivatives_time_bomb_for_big_banks---5...
Continuous (Inte)(Mi)gration
Throttle Data




Image http://i.ehow.com/images/GlobalPhoto/Articles/2289906/P1120995RPM_Full.jpg
Image http://www.mikkeman.nl/blog/wp-content/uploads/2009/10/priority-300x137.jpg http://ralphlosey.files.wordpress.com/20...
Volatility Categorisation




Image http://silverspaceship.com/static/shot_1.png
Image http://www.domainlanguage.com/_library/images_people_working/designing_at_whiteboard.jpg




       Analysis = BDUF ...
Implement “Big Bang”




Image http://silverspaceship.com/static/shot_1.png
Image http://sapplanning.files.wordpress.com/2008/07/integrate-technology.jpg




            Implement “Master + Deltas”
Data in non-binary format > diff




Image http://www.spycomponents.com/images/xml_at_work.gif
Image http://images.yogee.com.au/cat-dump-truck-609-2.jpg




                       Master Database Dumps
Activity Migration




Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg
Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg




                                                  ...
Open Source Continuous
 Integration / Build Tools

Continuous Integration Servers
 Hudson: https://hudson.dev.java.net
 ...
Open Source Developer Tools
IDE’s (multiple language support)
 Eclipse: http://www.eclipse.org
 NetBeans: http://www.net...
Open Source Testing Tools
Test Specification – xUnit Tools
 JUnit: http://www.junit.org
 PHPUnit: http://www.phpunit.de/...
Open Source ETL/ESB Tools
ETL
 Donkey:
     http://www.assembla.com/wiki/show/burro
 Pentaho (Kettle): http://kettle.pen...
Questions ??
Craig Smith
http://www.suncorp.com.au
http://www.agileacademy.com.au
craig.smith@suncorp.com.au
@smithcdau

 ...
Upcoming SlideShare
Loading in...5
×

Data Migration In An Agile Open Source World

4,461

Published on

Data Migration In An Agile Open Source World presented by Craig Smith at the Open Source Developers Conference (OSDC) 2009 in Brisbane.

Published in: Technology, Art & Photos
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,461
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
122
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Data Migration In An Agile Open Source World

  1. 1. Data Migration in an Agile Open Source World Craig Smith Suncorp
  2. 2. Welcome… Image http://www.neopoleon.com/blog/images/excel/7.jpg
  3. 3. According to Wikipedia: Data migration is the process of transferring data between ... computer systems ... is usually performed programmatically to achieve an automated migration ... required when organizations ... change ... or upgrade to new systems, or when systems merge... Image: http://i.ehow.com/images/GlobalPhoto/Articles/2065416/dictionary_Full.jpg
  4. 4. “In” End Point Extract Extract Transform Load Transform (ETL) Load Pattern “Out” End Point
  5. 5. File transfer (Enterprise) Shared database Application Integration Remote Patterns Procedure Invocation Messaging Image http://www.eaipatterns.com/eaipatterns.html
  6. 6. Data Migration + Application Integration = Similar Approach Image http://kansolutions.net/yahoo_site_admin/assets/images/integration.302161703_std.jpg
  7. 7. Image http://blog.theworkinggroup.ca/wp-content/uploads/2009/04/agile-manifesto1.gif Agile Manifesto
  8. 8. Scrum Practices Image http://blogs.conchango.com/Admin/ImageGallery/blogs.conchango.com/Colin.Bird/Scrum%20Overview%20Diagram.png
  9. 9. Image Beck, Kent – Extreme Programming Explained: Embrace Change XP Practices
  10. 10. Development Practices Fixing a Fast (<3 Use It 100% Nulls broken build mins) Or Lose Code Are Evil is always the Builds It highest Coverage priority Test Changing The Code Is Following Things Prod If In Doubt, Code Is A Use Is A Team Code Spike It Liability Outlandish Out (On A Decision... Class Branch) Statics Names Are Evil Molecular No Code Over Ownership Atomic All Prod Create Use Tests Code Small Delegation Paired (Or Commit Classes And Over Reviewed) Often Methods Inheritance Source http://www.assembla.com/wiki/show/burro/Development_Principles
  11. 11. Image http://iqizone.com/images/paper%20pile.jpg Break It Down... Iteratively
  12. 12. Agile Builds Confidence Image http://static.soxfirst.com/soxfirst.com/imgname--derivatives_time_bomb_for_big_banks---50226711--bomb.jpg
  13. 13. Continuous (Inte)(Mi)gration
  14. 14. Throttle Data Image http://i.ehow.com/images/GlobalPhoto/Articles/2289906/P1120995RPM_Full.jpg
  15. 15. Image http://www.mikkeman.nl/blog/wp-content/uploads/2009/10/priority-300x137.jpg http://ralphlosey.files.wordpress.com/2009/10/playing-cards.jpg Prioritisation & Categorisation
  16. 16. Volatility Categorisation Image http://silverspaceship.com/static/shot_1.png
  17. 17. Image http://www.domainlanguage.com/_library/images_people_working/designing_at_whiteboard.jpg Analysis = BDUF NDUF SDUF
  18. 18. Implement “Big Bang” Image http://silverspaceship.com/static/shot_1.png
  19. 19. Image http://sapplanning.files.wordpress.com/2008/07/integrate-technology.jpg Implement “Master + Deltas”
  20. 20. Data in non-binary format > diff Image http://www.spycomponents.com/images/xml_at_work.gif
  21. 21. Image http://images.yogee.com.au/cat-dump-truck-609-2.jpg Master Database Dumps
  22. 22. Activity Migration Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg
  23. 23. Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg Reconciliation
  24. 24. Open Source Continuous Integration / Build Tools Continuous Integration Servers  Hudson: https://hudson.dev.java.net  Cruise Control: http://cruisecontrol.sourceforge.net Build Tools  Ant: http://ant.apache.org  Maven: http://maven.apache.org  Gradle: http://www.gradle.org  SCons: http://www.scons.org  Phing: http://phing.info  Rake: http://rake.rubyforge.org/
  25. 25. Open Source Developer Tools IDE’s (multiple language support)  Eclipse: http://www.eclipse.org  NetBeans: http://www.netbeans.org  RadRails: http://www.aptana.com/rails Database Clients  DBVisualizer: http://www.dbvis.com/  SQuirreL: http://squirrel-sql.sourceforge.net Web Service / SOAP Tools  soapUI: http://www.soapui.org Version Control Systems  Subversion: http://subversion.tigris.org  Git: http://git-scm.com Database Versioning  Liquibase: http://www.liquibase.org/
  26. 26. Open Source Testing Tools Test Specification – xUnit Tools  JUnit: http://www.junit.org  PHPUnit: http://www.phpunit.de/ Test Specification Tools – Expressive/BDD  FitNesse: http://fitnesse.org  Concordion: http://www.concordion.org  Cucumber: http://cukes.info  easyb: http://www.easyb.org Test Execution / Record / Log / UI  Selenium: http://seleniumhq.org  Watir: http://wtr.rubyforge.org Performance Test  JMeter: http://jakarta.apache.org/jmeter/  Grinder: http://grinder.sourceforge.net/
  27. 27. Open Source ETL/ESB Tools ETL  Donkey: http://www.assembla.com/wiki/show/burro  Pentaho (Kettle): http://kettle.pentaho.org/  Talend: http://www.talend.com  Clover: http://www.cloveretl.com/  Scriptella: http://scriptella.javaforge.com/ ESB  Mule: http://www.mulesoft.org  Apache ServiceMix: http://servicemix.apache.org  Apache Camel: http://camel.apache.org/
  28. 28. Questions ?? Craig Smith http://www.suncorp.com.au http://www.agileacademy.com.au craig.smith@suncorp.com.au @smithcdau Suncorp is one of Australia and New Zealand's largest diversified financial services providers, supplying banking, insurance and wealth management products to around 7 million customers through well-established and recognised brands such as AAMI, Australian Pensioners Insurance Agency, Shannons, Vero, Asteron and Tyndall, as well as Suncorp and GIO. Today, Suncorp is Australia's fifth largest bank and second largest domestic general insurance group, with over 16,000 staff. Suncorp has representation in 450 offices, branches and agencies throughout Australia and New Zealand.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×