Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

KYSUC - Keep Your Schema Under Control

4,899 views

Published on

Abstract:
A apresentação centra-se na temática de ter forma de controlar, versionar e actualizar toda a parte de Base de Dados de um projecto. Estamos a falar, desde a produção de modelos ER, a versionamento de scripts, passando pelo deploy dos mesmos e terminado na documentação. A apresentação conta ainda com uma breve demonstração do uso da ferramenta Flyway para versionar e controlar a execução de scripts nos diversos ambientes de um projecto.

Sobre o Nuno Alves:
Chamo-me Nuno Alves nascido em Coimbra, Portugal e vivi maioritariamente em Leiria. Licenciado em Engenharia Informática na ESTG-IPLeiria (Escola Superior de Tecnologia e Gestão) onde o gosto por dados e bases de dados se começou a desenvolver. Daí, profissionalmente a minha área de actuação ser em torno de bases de dados e infra-estruturas. Tenho cerca de 10 anos de experiência repartidos pelas áreas Financeira, Seguros, Governo, Militar em tecnologias que vão desde Oracle, PostgreSQL, MSSQLServer a DB2.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

KYSUC - Keep Your Schema Under Control

  1. 1. KYSUC Keep Your Schema Under Control 04.FEV.2016 Nuno Alexandre Alves
  2. 2. Speaker • Nuno Alves • Born at Coimbra / Lives in Leiria • Degree in Informatics Engineering (ESTG-IPLeiria) • Some experience with databases • Nowadays working in Critical Software, SA nuno.alexandre.alves@gmail.com nuno.alexandre.alves
  3. 3. Disclaimer This presentation is: • From my personal experience • My own suggestions This presentation is NOT to: • Pointing fingers • Enumerate errors • This is better than that • Do not use this
  4. 4. Just another day in “paradise” • How do you develop your scripts ? • Do you have a ER model of your Database(s) ? • Do you have your Database documented ? • How do you control your scripts versions ? • Do you have your test data under scripts ? • How do you test your scripts ? • How do you deploy your application and database to a specific version ?
  5. 5. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
  6. 6. Goals for Today • Updated ER model • Scripts versioned • Easy Deploy • Easy Documentation • Schema Under Control !
  7. 7. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
  8. 8. ER Model (1/4) Start New Project (from requirements) • Build your Conceptual Model • First Physical Model • First Scripts (version 1.0.0) • Print (paper not PDF) your Physical Model and put it in a Wall next to the team (old-school but it works) Assumptions: • High Level Architecture designed • RBDMS almost decided (for the first scripts)
  9. 9. ER Model (2/4) Start New Project (already in PROD) / Maintenance • Ask for actual documentation (get lucky ) • DDL of actual production database • Import DDL in your development servers • Reverse Engineering • Build Conceptual Model (from Physical) if necessary • Script Physical Model with actual application version (ex: v3.5.2 – it’s your baseline)
  10. 10. ER Model (3/4) Development / Maintenance • Keep your database changes updated in your ER Model • Use Reverse Engineering to do checkpoints (compare)
  11. 11. ER Model (4/4) Tools • Use a Case Tool • PowerDesigner • Enterprise Architect • …
  12. 12. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
  13. 13. Scripts (1/10) Scripts Development (projects reality) • Manually • I don’t have scripts. My ORM tool does it for me !?! • Database is something to store some data on it. I don’t care.
  14. 14. Scripts (2/10) Scripts Development (projects reality) • WTF ?! • Your medical records relies on a database system • Your bank account relies on a database system • Your salary relies on a database system • … • Your life relies on too many database systems • Take your own conclusions
  15. 15. Scripts (3/10) Then someday • How do I execute the database scripts ? Order By ? • Which development database is updated ? • How do you develop scripts ? • One script vs multiple scripts ? • Full Scripts vs Partial Scripts ? • It’s your ORM your Database developer ? • Naming Conventions for objects (where is the guide?) • I have a app log with this error “constraint SYS_098374 violated”. What is that constraint ? • The column CGTRTHY stores what ?
  16. 16. Scripts (4/10) Database Scripts Under Control • Why ? • Is your source code under control ? • Does your source code have versions ? • Database scripts = source code ? • I think Yes ! • Table definition • Table initial data • Function / Stored Procedure / Package
  17. 17. Scripts (5/10) Decide your Database Storage Strategy • Check production for rolling projects • Decide according to hosting supplier or client needs • Tablespaces • Data files • File groups My Suggestion (minimum) • One Tablespace for Data • One Tablespace for Indexes • One Tablespace for LOBs
  18. 18. Scripts (6/10) Decide your Database / Schema(s) architecture • One schema, multiple users with diff grants • Multiple schemas, one/multiple users My Suggestion • Keep it simple • One schema/database, multiple users (grants/views) • FKs between multiple schemas not supported in all RDBMS • Distributed Transactions (do you really want ?!) • Multiple queries (joins in app server) • Are you creating your own RDBMS ?!
  19. 19. Scripts (7/10) Decide your Database Naming Conventions • Easy identify a database object • Name every object in database • Don’t let default names under RDBMS
  20. 20. Scripts (8/10) Script like source code • Use SQL formatting (for human readable) • Comment your SQL • Define objects language: PT; EN; FR … (don’t mix) Script for documentation • Comment every object (mandatory: wait and see) • Keep documentation in database dictionary • Useful for Reverse Engineering • Useful for Automatic Documentation • COMMENT ON TABLE • COMMENT ON COLUMN TABLE_NAME.COLUMN_NAME
  21. 21. Scripts (9/10) Tools • Manually (Notepad, VI, … is your friend!) • Liquibase • mybatis • MIGRATEdb • migrate4j • Flyway • …
  22. 22. Scripts (10/10) Scripts Under Control ?
  23. 23. Flyway (1/6) Overview • Scripts: SQL, Java • RDBMS Agnostic (JDBC driver) • Inside your source code • Placeholders (variables) • Multiple Execution Modes • Java API • Maven / Ant • Command line • Ex: V2_0_0_1_0001__script_name.sql
  24. 24. Flyway (2/6) Decide your Script Naming Convention • V<JIRA_FIX_VERSION>_1_<4_digits>__<JIRA_KEY>.sql • <4_digitos> • Execution order
  25. 25. Flyway (3/6) Put some code in your Script • Add an header to your SQL file • Add JIRA_KEY or JIRA URL as a comment • Commit
  26. 26. Flyway (4/6) Decide your Execution Mode • Command line (java -jar …) • Maven/ANT • configure inside your IDE or in CLI • Use Maven profiles is necessary (useful in multiple databases/environments)
  27. 27. Flyway (5/6) Flyway in action • flyway:info • Check schema version already installed • You need flyway:baseline for the first time
  28. 28. Flyway (6/6) Flyway in action • flyway:migration • Script execution • With/without target release
  29. 29. Flyway In Action
  30. 30. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
  31. 31. Deploy (1/7) Deploy Database Scripts • Do you test your scripts ? (how many times) • Do you re-create your schema from scratch ? • Do you have automatic deployment for your app ? • And your database ? • Do you have night builds ? • They include database too ? and automated tests ? • Do you simulate your client releases in development environments ? • Do you have confidence in your own scripts ?
  32. 32. Deploy (2/7) Deploy Database (full / upgrade) • Use your Continuous Integration • Jenkins • Team City • Bamboo • Other …
  33. 33. Deploy (3/7) Jenkins + Flyway = easy database deploy • Why Jenkins ? • Easy to configure • Parametrized builds • Many plugins (CVS, SVN, Maven, JIRA, Email, …) • Scheduler • Live Build Logs / Console output directly in browser • Why not ?!
  34. 34. Deploy (4/7) Jenkins + Flyway = easy database deploy • Configure a Jenkins Job • Checkout script from VCS • Put some parameters • if necessary for customs build • for multiple versions • for multiple databases • Execute Flyway • via Maven • via custom Shell Script / Windows batch
  35. 35. Jenkins + Flyway In Action
  36. 36. Deploy (5/7) Continuous Monitor your database • Oracle Enterprise Manager (Oracle) • pgwatch (PostgreSQL) • db2top (DB2) • SQL Server Management Studio (SQLServer) • MySQL Enterprise Monitor (MySQL) • … • Check administrator views (available in every RDBMS)
  37. 37. Deploy (6/7) Continuous Monitor your database • Check some random SQL Statements • CPU • I/O • Get query plan for these statements • Help development team write better SQL • Create indexes according to business needs • Monitor sessions • Check Connection Pooling parameters • Min / Max Pool sizes (warn: multiple Connections Pools)
  38. 38. Deploy (7/7) Help Development Team to help yourself • Write SQL Statements to log (debug mode) • Ask for log4jdbc / P6Spy / … (with parameters) • Write query times to log • Analyze top time consuming queries • Analyze top running queries • Ask for application times vs database times (in log) • Most of times the application consumes more time (yes it’s true) • Try to analyze how many queries are made for a business operation • Check the number of columns retrieved in every query (it’s really necessary 80 columns ?)
  39. 39. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
  40. 40. Documentation (1/3) How do you produce your database documentation ? • Manually • I don’t have database documentation • “My client doesn’t need this documentation” • And your team ? • How do you maintain a project without documentation? • Most (~All) of your production tickets are resolved by SQL statements ! • Client business data is stored in “your” database (really?!)
  41. 41. Documentation (2/3) Automatic Documentation • Use RDBMS data dictionary • information_schema.tables, … • user_tables, … • syscat.tables, … • Remember COMMENT ON … from scripts ? • “Export” to something useful • Word Document/PDF ? • Who reads a 800 pages document ?
  42. 42. Documentation (3/3) Automatic Documentation • Perl script (reads data dictionary, outputs One HTML file) • Send to your client as a deliverable • Publish in your wiki • Open locally in your browser • Produce custom CSS: 30m - 1h • Update Documentation: 1 minute • Remember COMMENT ON 
  43. 43. Summary (from initial goals) (1/2) ER Model • Use a Case Tool • Keep your model updated Scripts versioned • Use Flyway / Liquibase / Other • Script everything (data tests included) • Review your scripts • Naming Conventions • Comments
  44. 44. Summary (from initial goals) (2/2) Easy Deploy • Use Jenkins / Team City / Bamboo / Other • Keep your database under Continuous Integration • Schedule your database builds • Parametrize your database deploy • Monitor your database close • Release Matrix isn’t nice to have (is mandatory) Easy Documentation • Comment your objects in data dictionary
  45. 45. Main Goal Updated ER Model + Scripts versioned + Easy Deploy + Easy Documentation = KYSUC
  46. 46. Extra Credits Define / ask for backup/restore plan • Test your backups (restore is your friend) • Check backup/restore times • Check database growth Compare data • Compare your production data with development data: reference data • Try to equal PKs IDs across environments (for referencial data)
  47. 47. That’s It! Thank you for your time.

×