Your SlideShare is downloading. ×
Evolutionary db development
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Evolutionary db development

893
views

Published on

话题来自OpenParty “清雨榕香”活动

话题来自OpenParty “清雨榕香”活动


0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
893
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Changing from waterfall to Agile At an early stage identify requirements , reconcile , design , begin coding minimize changes due extensive preliminary work
  • automation
  • explain
  • Classify and Prioritize data Client list the most important data Try to figure out Data pattern( Data cleaning) Only working on meaningful data Verify data by system logics ( data referencing, records number…) Check it randomly
  • Transcript

    • 1. Evolutionary Database Development
    • 2. Software Development Process is Changing © ThoughtWorks 2009 Fix / Integrate $ Test Code Design Specifications Use Cases / Functional Specs Requirements Gathering Project Plan/Estimation $ $ $ $ Level Stories Vision & High- Iteration Release
    • 3.  
    • 4. Team Collaboration DB Apps BackEnd Traditional Team In Agile team, DBA
      • Role!= Person
      • knowledge of the functionality
      • Acknowledge team in interaction
      DEVS I’m responsible for Web design DBA I’m responsible for Database Domain expert I’m responsible for backend apps DBA Our responsible for business value Our responsible for business value It’s our responsibility DEVS Domain Expert Agile Team
    • 5.
      • DBA sit close to all the roles
      • Help make a decision
      • Educating Developers write better SQL
      • How to make the DBA redundant
      Best Practices
    • 6. Refactoring “ A disciplined way to make change to your source code to improve its design, making it easier to work with” Martin Folwer
    • 7. Database Refactoring Before: After: Behavioral semantics
      • Change structure
      A small change to Database schema that improves its design while retaining both its behavioral and informational semantics
      • Data Migration
      • Verify data quality
      Informational semantics Customer … Balance Account AccountID(PK) Customer … Account AccountID(PK) Balance
    • 8.
      • Structural
      • Data Quality
      • Referential Integrity
      • Architectural
      • Method
      • Non-Refactoring Transformation
      What will be changed in DB Refactoring?
      • Small Steps
      • Frequent Changes
      • Test First
      Principles
    • 9.
      • Test your Database Schema
      • Test the way Apps uses schema
      • Validate your data migration
      • Test external program code
      • Check data quality
      Test-Driven is also important for DB design TEST Fail Fast Fail Often
    • 10.
      • Not trying to "Get it Right up Front“
      • Build the simplest thing that can possibly work
      • Treat changes as database refactoring ... Every DAY
      • functionality added in increments
      Evolutionary Modeling Design
    • 11. A Story
    • 12. Background
      • CRM system
        • Transactional operations
        • Reporting/ Statistics functions for manager
        • Based on Legacy system
        • 24*7 , a very busy system
      • Client’s expectation
        • Improvement of Usability and performance
      • Legacy Database
        • Sql server 2005
        • 200G
      • Team
        • Distributed Agile team. (Beijing + Hongkong)
    • 13. Refactoring database schema Introduce transition period for SAFTY
    • 14. Never get rid of old schema immediately Data sync in real time
    • 15. What we did in this project Legacy DB NEW DB FlatFile Triger SystemNameID Newvalue OldValue columnName Read Data Near real Time Read Data systemName SystemNameID(PK) SystemLogin … systemName_log SystemNameID(PK) OperDatetime Account AccountID(PK) AccountName SystemNameID Isdelete
    • 16. Onetime Migration
      • Need to figure out logical mapping
      • Problem:
      • There are over 400 procs, over 1000 tables, over 150 views
      • Data Concept/logics changed
      • How we resolve:
      • Communication is very important
      • Split a function into small parts, more smaller more easy more safe
      Person Organisation Person User contact
    • 17.   The process of database refactoring
    • 18.
        • Verify data quality by domain logics after one time migration
      1.One user Should not have 2 office telephone Select user From user a left join telephone b On a.userid=b.userid Group by a.user Having count(*)>=2 2. A telephone should be assigned to a user Select telephone From telephone Where userid is null 3. A telehpone shoulde not be assigned to 2 or more user Select officeTele From telephone a left join user Group by officeTele Having count(userid)>=2 … adding more verification sql User UserID(PK) FirstName LastName … Telephone TeleID(PK) OfficeTele HomeTele UserID(FK)
    • 19. Sanity Check
      • Brainstorming
      • Feedback from CI/Production
      Don’t make an issue to happen for two times
    • 20. Put test scripts into CI Put verification scripts into CI Run verification scripts in production often, be aware of production data
    • 21. Our workflow Production App Pre-Production App Highly iterative Development System and Acceptance Testing Operations and support Devs’ App DB App Bug Reports Bug Reports Bug Reports Project-level Integration Testing Frequent Deployment Frequent Deployment Highly Controled Deployment CVS
      • Tests
      • SanityCheck
      Brainstorming Production Users QA Devs Devs
    • 22. The strategy for migration performance
      • Online system don’t allow that migration spend too long time
      Migration Update contactNote a Set ContactNote=c1,ContactId=c2 From ContactInfo where a.messid=b.Messid Too long ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId
    • 23. Delcare row_num number:=0; Begin for c_CN in (select MessId,C1,C2 from contactInfo a left join CI_Mig_log b on a.Messid=b.messid where flag=0) update contactNote set contactNote=c_CN.c1, contactId=c_CN.c2 where Messid=c_CN.Messid update CI_MIG_log f set f.flag=1 where messid=c_Cn.MessId row_num:=row_num+1; if mod(row_num,1000)=0 then commit; end if; End loop; Cancel it if there is no enough resource for migration Migration ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId CI_Mig_log MessID(PK) Flag
    • 24. Versioning Database
      • All database schemas can be thought of as DB refactoring
      • As updates are applied to a database, the changes will be recorded in a table similar to the following:
      Change Date 1_Create_Customer_Table.sql 4-15-07 2_Add_e-mail_address_column.sql 4-17-07 3_Add_fax_number_column.sql 4-18-07 4_Add_transaction_table.sql 4-21-07 5_Add_transaction_status_column.sql 4-24-07 6_Add_customer-transaction_view.sql 4-27-07
    • 25. Put them under configuration management control CREATE TABLE money ( eek NUMBER ); //Test for… DB DDL Insert into AA(mydata) Values(11); Meta Data Delete from .. DML Create index optimization Merge into Data Migration Tests Installation scripts
    • 26. Configuration of DB project Database project under version control Tiny db backup Deltascripts of dbdeploy For defining db objects which depends on schema For data onetime migration For data sync For checking dirty data Tool exclusively for database project other scripts…
    • 27. Database Deployment An automated process is needed to make the process of upgrading out-of-date databases
    • 28. Our Practices
      • Nothing is used only once
      • Automate tasks such as
        • Physical table deployment
        • Usage statistics
        • Schema verification
        • Data migration verification
      • Introduce tools ,like
        • Ant, dbdeploy
    • 29. Management DB deploymnet DB Deploy - http://dbdeploy.com/
    • 30. DBDeploy http://dbdeploy.com/documentation/getting-started/rules-for-using-dbdeploy/
      • N aming convention for delta scripts :
        • NUMBER COMMENT.SQL
        • e.g. 1_Create_Customer_Table.sql
      • Undo section – marked by comments:
      CREATE TABLE FOO ( FOO_ID INTEGER NOT NULL, FOO_VALUE VARCHAR(30) ); ALTER TABLE FOO ADD CONSTRAINT PK_FOO PRIMARY KEY (FOO_ID); --//@UNDO DROP TABLE FOO;
    • 31.
      • <target name=&quot;gen-and-exec-delta-script&quot;>
      • <dbdeploy
        • driver=&quot;oracle.jdbc.OracleDriver&quot;
        • url=&quot;jdbc:oracle:thin:@localhost:1521:XE&quot;
        • userid=&quot;dylan&quot;
        • password=&quot;nalyd&quot;
        • dir=&quot;./sql/deltas/&quot;
        • outputfile=&quot;./build_output/db-deltas-hsql.sql&quot;
        • dbms=&quot;ora&quot;/>
        • <sql
        • driver=&quot; oracle.jdbc.OracleDriver&quot;
        • url=&quot; jdbc:oracle:thin:@localhost:1521:XE &quot;
        • userid=&quot;dylan&quot;
        • password=&quot;nalyd&quot;
        • src=&quot;./build_output/db-deltas.sql&quot;
        • onerror=&quot;abort&quot;/>
      • </target>
      Ant
    • 32. DBDeploy
      • Go to directory with SQL files:
        • “ 1 create_customer_table.sql”
        • “ 2 add_customer_id_constraint.sql”
      • Run “ant”
      Output: gen-and-exec-delta-script: [dbdeploy] dbdeploy v2.11 [dbdeploy] Reading change scripts from directory C:Projectsdbdeploy-demosqldeltas... [dbdeploy] Changes currently applied to database: [dbdeploy] 1, 2 [dbdeploy] Scripts available: [dbdeploy] 1, 2, 3, 4 [dbdeploy] To be applied: [dbdeploy] 3, 4 [sql] Executing file: C:Projectsdbdeploy-demouild_outputdb-deltas.sql [sql] 8 of 8 SQL statements executed successfully
    • 33. Automate Tasks
      • CreateNewTestDB
      • upgradeDB
      <target name= “-parseDbScxripts” >…</target> … <target name=&quot;-upgradeDB&quot; depends=&quot;-parseDbScripts, -dbdeploy, -runDeltaScript, -dropDbLogic, -createDbLogic&quot; description=&quot;Upgrade specified database to latest version&quot; /> <target name=&quot;rebuildDB&quot; depends=&quot; -parseDbScripts, -dropDb, -createDb, -initialiseDb, -dbdeploy, -runDeltaScript, -createDbLogic&quot; description=&quot;drop, recreate and initialise the Connect database&quot; />
    • 34. Batch file: library ant ant.exe –buildfile:evovle.build –D:rebuildDB -logfile:build.txt Shared to devs Do what we want by just One Command
    • 35.
      • Reference:
        • Evolutionary Database Design
        • http://martinfowler.com/articles/evodb.html
        • Refactoring Databases: Evolutionary Database Design
    • 36. Q&A