1. Technical Tooling for a
Major KFS Upgrade
Shyam Gedela (gedela@msu.edu)
Harsha Ummerpillai (harsha07@msu.edu)
Michigan State University
11/11/2014
1

2. Kuali Days 2014
Indianapolis
Overview
• Upgraded from KFS 3/R1 to KFS5/R2
• Overall Goal
– Vanilla
– No functional impact - technical upgrade
– Shared Rice so KC can integrate
• Emphasis on tooling, not upgrade
process
2

3. Kuali Days 2014
Indianapolis
Background
• Foundation
• Liquibase scripts were not enough
• Scripts to update status code not fast
• Rice maintenance document update too simplistic
• No easy way to take changes like RC Code
• MSU
• No central issue tracking
• Non standard commit comments
• 3.0/R1 to 5.0/R2
• Most functionality will go forward
3

4. Kuali Days 2014
Indianapolis 4

5. Understand the problem.
Analysis
5

6. Kuali Days 2014
Indianapolis
Collect data, correlate and make it searchable.
1. Functionality changes – Use Revision Tool
1. Application Enhancement Requests (AER) list
2. Defect tracking tool (DTT) list
3. Application Support Request (ASR) list
4. Code changes (Fisheye)
Analysis
6

7. Kuali Days 2014
Indianapolis
Searchable database using code and issue ID -
AER, ASR, DTT, Fisheye
1. Identify issues with code impact (F)
2. Correlate items from different lists (F)
3. Search unit of work using issue ID or code file name (T)
4. Categorized each issue into a decision matrix (F)
1. MSU Only (e.g. Interfaces, Conversions, some Features)
2. Contribute
3. Already in foundation
4. Remove from production
• Change in business process
F – Functional, T - Technical
Analysis – Revision Tool
7

8. 8
Revision Tool

9. Kuali Days 2014
Indianapolis
1. Code upgrade.
2. Database upgrade
1. Metadata upgrade.
2. Data split and
upgrade.
3. KIM and Workflow
upgrade.
3. Split Rice
4. Automate
5. Practice makes perfect.
Overall Technical approach
9

10. Customizations created difference between the foundation and
MSU version.
Code Upgrade
10

11. Kuali Days 2014
Indianapolis
Code Upgrade Approach
• Transfer all MSU code from 3 to 5
– Tool to compare deltas before and after
• Technical review to ensure complete transfer
• Remove code based on functional approval
• Git tooling for mixing bowl approach
11

12. Kuali Days 2014
Indianapolis
Code Upgrade
KFS 3.0 to 5.0
12

13. Database upgrade involved metadata and data upgrade.
Database Upgrade
13

14. Kuali Days 2014
Indianapolis
Metadata Approach
14

15. Kuali Days 2014
Indianapolis
Metadata Upgrade
• Submit all changes to functional review.
• Blanket Rules
– Accept new columns.
– Accept columns made null.
– Accept column size increase
– Accept column data type changes.
– Accept new tables and renames.
– Accept new and modified Foreign Keys.
– Accept new and modified Primary Keys.
– Ignore dropped columns.
– Ignore columns made not-null.
– Ignore column size decrease.
– Ignore FKs removed.
– Accept bootstrap/mixed data changes
15

16. Data Approach
16

17. Kuali Days 2014
Indianapolis
Database Upgrade KIM
• Primary Keys changed
• Complexities with uniquely identifying and
comparing data.
• Special JDBC programs to print diff reports.
• Needed functional review.
• Custom/manual SQL script generation.
• Review and apply against production data.
• Run custom comparators to review impact after
script is applied.
17

18. Kuali Days 2014
Indianapolis
Database Upgrade KEW
• Exported production version of KEW XML files.
• Exported vanilla version of KEW XML files.
• Compare XML files using “WinMerge”.
• Functional review of differences.
• Manual edit of production XML files.
• Combine, sort and merge all XML files into a
single DocTypes.xml per application.
• Ingest into upgraded database to verify the
impact.
18

19. Phase 1 is done! Onto
Phase 2
19

20. Kuali Days 2014
Indianapolis
Split Rice • Create new Rice environment
• Split Rice data from applications
• Workflow doc IDs didn’t collide
20

21. Production Move
21

22. Kuali Days 2014
Indianapolis
Upgrade Automation
22

23. Kuali Days 2014
Indianapolis
Production Move Plan & Execution
23
ID Pred Task Owner
Planned
Start Start Time Duration Prod Notes
1
Verify PROD shared rice setup (DS xml files, Lucene
and configuration properties) MW 3/6/2014 8am 30 mins
2
Copy rice root war to rice prod server to check URL
is working MW 3/6/2014 8am 15 mins FN-D will provide root war to MW
3
On all application servers (6 servers total) change
log level for org.apache.cxf to ERROR in jboss-
log4j.xml located at /home2/jboss/server/ais/conf MW 3/6/2014 8am 30 mins
applied change to prod rice servers but
needs to wait for Jboss conversion on kfs
to make those changes. Will be finished
in step 6
4 Create release tag and reconfigure bamboo plans FN-D 3/6/2014 9am 30 mins
5 Post Service Alert - Finance & Data Services
Integratio
n 3/10/2014 12pm 15 mins
6 Prep for Jboss upgrade MW 3/12/2014 12pm 2 hrs
Tar up new version and other
preparation
7
Verify KFS DBA login is working in FINPRD as
designed - remove after use DBA 3/13/2014 9:00am 15 mins
8
Backup (level 0) FINPRD and MSUDIM and
MSUEDW DBA 3/13/2014 11am 6 hrs
9
Ensure times across prod servers are synced
(including Rice) VSS 3/13/2014 12pm 30 mins
automate this by creating daemon
process
10 1 Verify new sentinel app id RICE Security 3/13/2014 1pm 30 mins
11 10
Verify that authorize existing application (KFS,
KMM, OOI) users to Shared Rice (RICE) Security 3/13/2014 1:30pm 30 mins
12
Remove D6501.IDM.DAILY.PF from Saturday night’s
schedule Prod Svcs 3/13/2014 11am 15 mins
13 Prep for bamboo plans FN-D 3/13/2014 9am 1 hr
14
disable UC4 jobs -
N81.DBA.ORACLE.STORES.TO.MSUEDW.COPY and
N81.DBA.ORACLE.KFS.TABLES.COPY Prod Svcs 3/14/2014 1pm 15 mins disable sat. & sun.
15 Create EDW and DIM restore point for ETL and BI DS 3/14/2014 8am 4 hrs
16 Run N81.FP.PCARD.NIGHTLY.REVIEW.PF at noon Prod Svcs 3/14/2014 12pm 15 mins
17 16 Functional review of Pcard File FN-F 3/14/2014 12:15pm 15 mins
18 17 Run N81.FP.PCARD.NIGHTLY.DOCUMENT.PF Prod Svcs 3/14/2014 12:30pm 30 mins
19
Lock Finance App logins except list of validation
NetIDs MW 3/14/2014 7pm 30 mins script
20 Shut down Paybase Servers (including PBPRD) VSS/DBA 3/14/2014 7pm 10 mins
21 19
Run KFS 3 Nightly Batch (with N81.CAB.EXTRACT Job
disabled) Prod Svcs 3/14/2014 7:30pm 2 hrs
22 21
Verify N81.Kuali.Nightly.PF (KFS nightly batch) and
D6501.KFS.Apply.Updates (from IDM) are complete.
Stop Queue.N81.KFS. Prod Svcs 3/14/2014 9:30pm 15 mins
23 22 Generate CAB Reconciliation List in Production FN-D 3/14/2014 9:45pm 15 mins
24 22 Check that all Log Activity stopped FN-D 3/14/2014 9:45pm 30 mins
25 24 Post KFS Redirect page (update F5) MW 3/14/2014 10:15pm 15 mins
26 24 Stop XI Communications N82.BC.X.STOP.CC.PF Prod Svcs 3/14/2014 10:15pm 5 mins BASIS backup may be needed
27 25 Disable WebMon MW 3/14/2014 10:30pm 10 mins
28 26 Run N81.SYS.STOP.ALL.SERVERS.PF Prod Svcs 3/14/2014 10:20pm 15 mins
29 28
Backup (level 1) FINPRD and MSUDIM and
MSUEDW DBA 3/14/2014 10:35pm 30 mins plus Sergey
30 29
Pre-snapshot DBA task (stop OEM, check/update
autostart, disable backup, change process
parameter - 1200) DBA 3/14/2014 11:05pm 15 mins
31 30 Perform snapshot VSS 3/14/2014 11:20pm 30 mins
Shutdown DB VMs except ebsprd100d1
and app VMs
32 31 Run EDW/DIM Finance/OOI Nightly Batch Prod Svcs 3/14/2014 11:50pm 5 hrs
33 31 Jboss Upgrade and File Configuration MW 3/14/2014 11:50pm 3 hrs
clean out all files in ais-deploy directory,
copy new wars from jboss-deployment,
keep logs directory, rename latest
server.log. /home2/n81/jboss-
deployment -location to copy wars
from, copy out opt files rice-config and
rice keystore to each production
34 32
Check if EDW/DIM-FIN Batch run is complete. Stop
Queue.N85.EDW.Finance Prod Svcs 3/15/2014 5am 15 mins
35 32 Build shared Rice schema (All except 4 tables) DBA 3/15/2014 6am 2 hrs
36 34 Create EDW/DIM restore point DBA 3/15/2014 8am 15 mins

24. Kuali Days 2014
Indianapolis
Resources
• https://tools.test.ebsp.msu.edu/
24