Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL

This case study describes migrating the most used application for primary schools in the Netherlands from Oracle to PostgreSQL. The application uses a multi-tenant, single schema database (i.e. 6000 schools in a single database) and runs using a typical Java EE frontend.

You will learn about our application architecture, hardware platform, reasons for switching, migration strategies considered and the results of our migration.

Since the CFP closes one week before our actual migration we can't reveal the results in this abstract, but the presentation will capture all the things that went wrong and well

  • Login to see the comments

  • Be the first to like this

Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL

  1. 1. Converting 85% of Dutch Primary Schools 
 from Oracle to PostgreSQL Martijn Dashorst topicus.nl
  2. 2. Martijn Dashorst @dashorst martijndashorst.com github.com/dashorst
  3. 3. “I’d rather not work with databases” Martijn Dashorst (1997, first job interview)
  4. 4. Big thank you to Klaasjan Brand
  5. 5. Please Give Feedback at
 https://2019.pgconf.eu/f
  6. 6. What is ParnasSys? Why migrate? Planning Stages Conclusions & Questions
  7. 7. Dutch Education System AGES 4-12 13-16/18 16-23 Primary Education Secondary Education Vocational/University Education pre-vocational (4 yrs), pre-applied science (5 yrs), pre-research university (6 yrs) vocational (4 yrs), applied science (4 yrs, bachelor), pre-research university (bachelor/master) EDUCATION TYPE
  8. 8. Primary Education (4-12) 1.5M students 141k employees 6973 schoolsages 4-12, 2019-2020 2018 2018 Source: https://www.onderwijsincijfers.nl
  9. 9. Financing Primary Schools € government school teachers buildings materials € taxes lumpsum number of
 students
 at 1 October
  10. 10. Financing Primary Schools € government school teachers buildings materials € taxes lumpsum number of
 students
 at 1 October administration
  11. 11. ParnasSys ~ noun. 
 1. A SaaS to run primary schools by keeping student records for school financing, student counseling and guidance, attendance keeping and communicating with parents 

  12. 12. ParnasSys ~ noun. 
 1. A SaaS to run primary schools by keeping student records for school financing, student counseling and guidance, attendance keeping and communicating with parents 2. The only pinkmagenta administration 
 system in the world
  13. 13. Java Web App Hibernate (ORM) Business Objects (SAP) MAXDB/SAPDB Single database Single schema Multi-tenant Development started October 2003 Production in 
 April 2004
  14. 14. First Database Migration in 2005 SAPDB/MAXDB ORACLE User base growing fast Unpredictable query performance in MaxDB "Nobody got ever fired for buying Oracle"
  15. 15. 85% of dutch primary schools use ParnasSys ~5M student dossiers in our database 1,270,000 students in 2019-2020 2019
  16. 16. 85% of dutch primary schools use ParnasSys ~5M student dossiers in our database 1,270,000 students in 2019-2020 2019
  17. 17. 800+ pages 360 tables data & indices ~ 2TB user files ~ 5TB
  18. 18. What is ParnasSys? Why migrate? Planning Stages Conclusions & Questions
  19. 19. Mismatch Between Future of Oracle and our Product
  20. 20. Cost Structure of Running Oracle Not Compatible with our Profitability
  21. 21. The Oracle Migration Assistant Paid Us a Visit TM
  22. 22. The stranglehold
 of the software giants [...] Especially Oracle would force unfair contracts upon clients, and claim many millions per year of late payments. [...] http://fd.nl/weekend/1318745/de-wurggreep-van-de-softwarereuzen
  23. 23. Developer Happiness Or How Do I Run My Database on My Local Machine
  24. 24. DevOps Happiness on Oracle is Incompatible with our Profitability
  25. 25. What is ParnasSys? Why migrate? Planning Stages Conclusions & Questions
  26. 26. Inventory ✅ Database contents ✅ Java Application Code ✅ Business Objects Universe ✅ Business Objects Reports ✅ Backup & Restore ✅ Monitoring ✅ Training 🚫 No stored procedures 🚫 No custom functions 🚫 No triggers
  27. 27. Seasonality of Education
  28. 28. 2 Moments in a Year to Migrate: X-Mas & Summer Vacation
  29. 29. X-Mass 2 weeks 23 December 2019–5 January 2020
  30. 30. Summer vacation(s) North 15 juli 2019–23 august 2019 Middle 22 juli 2019–30 august 2019 South 8 juli 2019–16 august 2019
  31. 31. Ideal migration moment:
 start of summer vacation
  32. 32. Stage 1: make it run
 incorrectly, correctly, fast Stage 2: migrate production
 in accordance to customer expectations
  33. 33. 1 Year Migration
  34. 34. 1 Year Migration 3 Year Migration
  35. 35. What is ParnasSys? Why migrate? Planning Stages Conclusions & Questions
  36. 36. Stage 1: make it run
  37. 37. Free Works™ Great to find issues in your application
  38. 38. Perl
 “The only language that looks the same before and after RSA encryption” Direct datatype mapping
 E.G. number(1,0) does not map to boolean Slow
 One import takes whole day Stable or fast (pick one)
 Single-threaded: stable but slow
 Multi-threaded: fast but unstable One-way, big bang only
  39. 39. Differences between Oracle and PostgreSQL “The good thing about standards is that there are so many to choose from.” ― Andrew S. Tanenbaum
  40. 40. Mapping datatypes Oracle PostgreSQL number(1,0) number(1,0) boolean Java int boolean enum Sex {
 Female, Male
 } Business study year (1-8) present (yes/no) sex (female/male) number(1,0) number(1,0) number(1,0) varchar2(6) varchar2(6)
  41. 41. ORM is 100% portable @Basic @Temporal(TemporalType.DATE) private Date birthdate; @Formula("add_months(birthdate, (to_char(sysdate, 'yyyy') to_char(birth @Temporal(TemporalType.DATE) private Date birthday; CREATE OR REPLACE FUNCTION birthday(birthdate IN DATE) RETURN D IS BEGIN RETURN add_months(birthdate-1, 
 (to_char(sysdate, 'yyyy') - to_char(birthdate, 'yyyy')) * 12)+1; END;
  42. 42. Oracle speaks in tongues select 1 from dual select nvl(null, 'n/a') from dual select * from (select x,y from ...) 
 where rownum between 100 and 110 sysdate trunc
  43. 43. Performance ID ORGANIZATION NAME BIRTHDATE ...
 14342 1231 JOHN SMITH 2010-09-23 14345 1231 MARTIJN DASHORST 2011-03-12 ... ID ORGANIZATION NAME
 151231 1231 Group 1A 151232 1231 Group 1B ... tenant identifier
  44. 44. Oracle is smarter... UPDATE testresult tr SET tr.endresult = TRUE WHERE tr.organisation = ? AND tr.testpart IN ( SELECT part1.id FROM testpart part1 WHERE part1.preference = ( SELECT MIN(part2.preference) FROM testresult tr3 INNER JOIN testpart part2 ON tr3.testpart = part2.id WHERE part2.preference > 0 AND tr3.score IS NOT NULL AND tr3.test = tr.test AND ((part1.usemodulenorm IS NULL AND part2.usemodulenorm IS NULL) OR part1.usemodulenorm = part2.usemodulenorm)))
  45. 45. Oracle is smarter...UPDATE testresult tr SET tr.endresult = TRUE WHERE tr.organisation = ? AND tr.testpart IN ( SELECT part1.id FROM testpart part1 WHERE part1.preference = ( SELECT MIN(part2.preference) FROM testresult tr3 INNER JOIN testpart part2 ON tr3.testpart = part2.id WHERE part2.preference > 0 AND tr3.test = tr.test UPDATE testresult tr SET tr.endresult = TRUE WHERE tr.organisation = ? AND tr.testpart IN ( SELECT part1.id FROM testpart part1 WHERE part1.preference = ( SELECT MIN(part2.preference) FROM testresult tr3 INNER JOIN testpart part2 ON tr3.testpart = part2.id WHERE part2.preference > 0 AND tr3.test = tr.test AND tr3.organisation = tr.organisation)) 100x
improvement
  46. 46. Java Web App Business Objects (SAP) PDF Universe Report definition Parameters PDF
  47. 47. Teachers Generate a lot of PDFs
 but that's OK, because 
 Business Objects generates a lot of SQL
  48. 48. Business Objects 
 can be a bit nostalgic
  49. 49. Quizz! SELECT * FROM A, B WHERE B.column(+) = A.column SELECT * FROM A RIGHT OUTER JOIN B ON B.column = A.column SELECT * FROM B LEFT OUTER JOIN A ON B.column = A.column SELECT * FROM A LEFT OUTER JOIN B ON B.column = A.column SELECT * FROM B RIGHT OUTER JOIN A ON B.column = A.column 1 2 3 4
  50. 50. Quizz! SELECT * FROM A, B WHERE B.column(+) = A.column SELECT * FROM A RIGHT OUTER JOIN B ON B.column = A.column SELECT * FROM B LEFT OUTER JOIN A ON B.column = A.column 1 2
  51. 51. Production ready?
  52. 52. Application works Business Objects works GO!
  53. 53. Application works Business Objects works GO!
  54. 54. How long takes migration? Behavior of application under actual load?
 - Heavy use of application side caching in Java process
 - Modification of data
 - Synthetic tests lie Can we fall back to Oracle?
  55. 55. Stage 2: Migrate Production
  56. 56. Need help...
 1-800-KILLORACLE
  57. 57. Why don't you use HVR? –Splendid Data
  58. 58. Cross-database replication using redo logs
  59. 59. Remote UNIX
 Machine Remote XYZ
 Machine Hub Machine PC HVR HVR Scheduler HVR HVR HVR GUI Database Location Hub DB Database Location IntegrateCapture Inetd Router Transaction Files Inetd HVR Listener Service
  60. 60. Lots of databases supported
  61. 61. Lots of databases supported
  62. 62. Why HVR? • Speed • Real-time sync with low latency • Options to test prior to actual migration • Safe • Fallback • Expensive tool, but Special Price for Special Friend
  63. 63. Noteworthy issues • Use primary key on each table • Fix datatype incompatibilities because of differences between databases
 E.G. time in date field ✅ Oracle 🚫 PostgreSQL • Disable constraint checking in target database • Don't perform DDL changes • Don't update too many rows in a table too often
  64. 64. Oracle based deployment DB1 PROD failsafe DB0 PROD TEST APP PROD Business Objects APP TEST ORACLEORACLE ORACLE APP PROD APP PROD Fail safe server Offline Archive Log shipping
  65. 65. DB0 DB1 DB2 PROD PROD TEST TEST failsafe TEST PROD failsafe PROD
 standby APP PROD Business Objects APP TEST ORACLEORACLE ORACLE ORACLE PSQL PSQL PSQL HVR migration deployment
  66. 66. DB0 DB1 DB2 PROD PROD TEST TEST failsafe TEST PROD failsafe PROD
 standby APP PROD Business Objects APP TEST ORACLEORACLE ORACLE ORACLE PSQL PSQL PSQL Prod Testing Business Objects
  67. 67. PostgreSQL based deployment DB1 DB2 PROD failsafe DB1 PROD TEST APP PROD Business Objects APP TEST PSQLPSQL PSQL APP PROD APP PROD DB0 PROD ORACLE
  68. 68. D-Dayde-oracle day 22 Juli 2019
  69. 69. What is ParnasSys? Why migrate? Planning Stages Conclusions & Questions
  70. 70. Make a *really* good inventory of things that have to be migrated Pick a strategy that works for you Plan around your business to minimise risk and maximise opportunity A great way to find bugs in your system Don't be afraid to pay for tools and help
  71. 71. “Nobody got fired from migrating away from Oracle” “Yet”
  72. 72. Migrate our Secondary Education Application from Oracle to PostgreSQL
  73. 73. Feedback
 https://2019.pgconf.eu/f
  74. 74. Questions?
  75. 75. THANK YOU! @dashorst martijndashorst.com github.com/dashorst

×