Download to read offline
![32© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | 11898
Appendix B – Tomcat TNSNAME Configuraiton
Changes made to tomcat.sh and tomcat_liferay.sh
In order to allow servlets to utilize the TNSNAME for our Oracle instance we modified the start scripts for the CAS, Portal,
and Admin Tomcat instances by adjusting the variable JAVA_OPTS as follows:
…
OUR_TNS_ADMIN=“/path/to/oracle/client/network/admin”
if [ -d $OUR_TNS_ADMIN ]; then
JAVA_OPTS=“$JAVA_OPTS –Doracle.net.tns_admin=$OUR_TNS_ADMIN”
else
echo “Could not set oracle.net.tns_admin, directory missing.”
fi
…](https://image.slidesharecdn.com/v511898afullyredundantluminis5installation-151216215016/85/A-Fully-Redundant-Luminis-5-Installation-32-320.jpg)
Uploaded byWilliam Moore
A Fully Redundant Luminis 5 Installation
The University of Manitoba upgraded its Luminis system in 2013 to achieve full redundancy across hardware, operating systems, databases, and applications. Key improvements included a scalable virtualized server environment, active/passive database configuration, and updated portal software to Luminis 5. The resulting system enhanced user experience, stability, and supportability while addressing previous challenges with outages and limited future integration capabilities.
More Related Content
Similar to A Fully Redundant Luminis 5 Installation
A Fully Redundant Luminis 5 Installation
- 1. 1© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 A Fully Redundant Luminis 5 Installation William Moore University of Manitoba April 14, 2015 11898
- 2. 2© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Session rules of etiquette • Please turn off your cell phone/pager • If you must leave the session early, please do so as discreetly as possible • Please avoid side conversation during the session Thank you for your cooperation!
- 3. 3© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Introduction In 2013 the University of Manitoba decided to upgrade its version of Luminis. The objective of this upgrade was to have a fully redundant installation throughout the whole stack. This presentation will explain what was done to make the hardware, OS, Database, and Application layers redundant to meet this objective.
- 4. 4© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 1 Who are we? 2 Our History with Luminis 3 What have we done? 4 Resulting System 5 Questions & Answers Agenda
- 5. 5© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Who are we?
- 6. 6© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 About the University of Manitoba • Staff: 8,716 (4,754 Academic; 3,962 Support) • Students: 29,759 (25,363 Undergrad; 3,748 Graduate) • Distributed campuses
- 7. 7© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 William Moore • 15 years at the University of Manitoba in IT now Solution Architect • Past Projects: • Identity Management System • Portal (“JUMP”) system • Learning Management Systems (LMS)
- 8. 8© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Our History with Luminis
- 9. 9© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 2004 Portal (“JUMP”) Implementation Using Luminis 3 Single Sign On • LMS • Purchasing • Travel & Expense • SSB • HR • Research Tools Authentication • LMS • Wiki • Co- curricular transcripts • Lab Animal Tracking Communication
- 10. 10© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Poorly Positioned for the Future • Suffered a number of outages • Off support software across the whole stack (O/S, DB, Portal) • Failed migration attempts to Luminis 4 • Limited ability to add redundancy to address outages • Lack of business engagement • Technically driven vision drove organic evolution
- 11. 11© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 What have we done?
- 12. 12© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Business Direction 1. Engage business 2. Consider mobile accessibility 3. Employed outside expertise to help plan the technical solution and educate the technical staff 4. Plan to resolve operational issues with an upgraded system 5. Position for future application integration with the portal
- 13. 13© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Key elements of the technical solution 1. Scalable virtualized x86 Server environment 2. Active/Passive Database (DB) configuration 3. Redundant LDAP 4. Redundant CAS 5. Upgraded the Portal software to Luminis 5 LoadBalancer CAS Admin Portal App Server DBLayer LDAP Virtual Servers/Storage
- 14. 14© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 (1) Implemented Server Architecture Portal 1 Portal 2CAS1 CAS2 LoadBalancer LDAP1 LDAP2 Admin Active Passive Portal 3 Portal 4 LoadBalancer CAS Admin Portal App Server DBLayer LDAP Virtual Servers/Storage
- 15. 15© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 (2) Database Configuration • Have a site license for Oracle • First system to use Oracle on virtual x86 hardware • Use two Oracle databases in an Active/Passive configuration utilizing DataGuard to synchronize the data • Use TNSNAMEs entry in the software layer to handle switching between database nodes Active DB Passive DB SAN Layer LoadBalancer CAS Admin Portal App Server DBLayer LDAP Virtual Servers/Storage
- 16. 16© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 (3) Redundant LDAP SAN Layer LDAP1 LDAP2 LoadBalancer • Consulted with Ellucian on how to enable LDAP replication • Had to learn how to enable LDAP replication by reading OpenDJ documentation • Prototyped the implementation in our staging environment LoadBalancer CAS Admin Portal App Server DBLayer LDAP Virtual Servers/Storage
- 17. 17© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 (4) CAS Configuration DBLayer LDAP SAN Layer CAS1 CAS2 LoadBalancer• Consulted with Ellucian on how to enable CAS redundancy • Had to learn how to enable storing the CAS session information in a database by going to JASIG • Prototyped the implementation in our development environment LoadBalancer CAS Admin Portal App Server DBLayer LDAP Virtual Servers/Storage
- 18. 18© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 (5) Upgrade to Luminis 5 • Fresh install of Luminis 5, no migration from Luminis 3 • Applied a mobile theme
- 19. 19© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Resulting System
- 20. 20© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Streamlined Desktop User Interface
- 21. 21© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 New Mobile Interface
- 22. 22© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Well Positioned for the Future • Business engagement • Established architecture • Stable platform • Scalable up and down depending on capacity requirements • Increased performance • Better supportability
- 23. 23© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Successes Since launching (Nov 24, 2014) we have had the following occur with out causing a disruption to our users: • 1 network file storage move (this took 6 hours) • 2 O/S patching operations (of all nodes) • A virtual environment issue causing a number of VMs to fail
- 24. 24© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Lessons Learned • Benefit from many small prototypes • Thorough testing required when using cloning • Better education for application installation and configuration • Required knowledge of the underlying products such as CAS, Liferay, OpenDJ, Tomcat • Required a better understanding of the load balancer’s capabilities • Data Guard instructor led training would have been more effective
- 25. 25© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Questions & Answers
- 26. 26© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Thank you! William Moore William.moore@umanitoba.ca Please complete the online session evaluation form. Session ID 11898
- 27. 27© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Additional Information Slides
- 28. 28© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Load Balancer Configuration • Adjust LDAP idle timeout to 1,800 from 300 • Removed web caching from the HTTP pools • Configured LDAP health monitor • 5 second interval • Look for ou=site under ou=Luminis Configuration,o=cp • Configured HTTP health monitor • Poll a static image rather then the root document LoadBalancer LDAP CAS
- 29. 29© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Portal/Admin Tomcat tweaks • Some portlets did not behave well with terminating SSL at the load balancer • Configure Tomcat to enable TNSNAMES See Appendix B • Configured the Tomcat instances to masquerade as having received a secure request LoadBalancer User SecureRequestSecureResponse Tomcat Non secure request Non secure response Servlet ProcessRequest HTML Response Direction GetConnection Info server.xml … <Connector port=“80” proxyPort=“443” secure=“true” scheme=“https” …
- 30. 30© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Fault Tolerance Tests • Developed test cases to ensure our redundancy worked • Had all members in one room to work through the tests Portal Environment Component/Service Failover / HA Test Case Validation Steps Expected Results Tester(s) Test Date Pass/Fail Comments LDAP Uncontolled Shutdown of Server Add user See user in remaining LDAP. Successful login as new user. Brian, Margaret July 9, 2014 Passed Restart 2nd LDAP verify changes are replicated to restarted server. Successful login as modified user. Brian, Margaret July 9, 2014 Passed Shutdown 1st LDAP Add a user Brian, Margaret July 9, 2014 Passed Restart 1st LDAP User added in previous step is replicated to 1st LDAP Brian, Margaret July 9, 2014 Passed Controlled Shutdown of Server while users are being added Add 1000 users during shutdown if started just before shutdown initiated, some users should appear in both LDAP instances while some will only get added to the remaining available LDAP Margaret Bill (user data file) Sample Test Plan
- 31. 31© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Appendix A – LDAP Replication commands LDAP 1 Commands • dsreplication enable –-host1 ldap1 –-port1 4444 –-bindDN1 “cn=Directory Manager” –-bindPassword “pwd” –-replicationPort1 1388 – -host2 ldap2 –-port2 4444 –bindDN2 “cn=Directory Manager” –bindPassword2 “pwd” –replicationPort2 1388 –adminUID admin – adminPassword “pwd” –baseDN “o=cp” –baseDN “o=messaging” –trustAll LDAP 2 Commands • dsreplication initialize –-baseDN “o=cp” –-baseDN “o=messaging” –-adminUID admin –-adminPassword “pswd” –-hostSource ldap1 –-portSource 4444 –hostDestination ldap2 –portDestination 4444 --trustAll
- 32. 32© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Appendix B – Tomcat TNSNAME Configuraiton Changes made to tomcat.sh and tomcat_liferay.sh In order to allow servlets to utilize the TNSNAME for our Oracle instance we modified the start scripts for the CAS, Portal, and Admin Tomcat instances by adjusting the variable JAVA_OPTS as follows: … OUR_TNS_ADMIN=“/path/to/oracle/client/network/admin” if [ -d $OUR_TNS_ADMIN ]; then JAVA_OPTS=“$JAVA_OPTS –Doracle.net.tns_admin=$OUR_TNS_ADMIN” else echo “Could not set oracle.net.tns_admin, directory missing.” fi …
- 33. 33© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Appendix C – CAS Database Configuration Changes made to ticketRegistry.xml 1. Added the http://www.springframework.org/schema/tx namespace to the beans node as part of the xmlns attribute 2. Added the following schemas to the beans node as part of the xsi-schemaLocation attribute o http://www.springframework.org/schema/tx o http://www.springframework.org/schema/tx/spring-tx-3.1.xsd 3. Added the following bean o <bean class=“org.springframework.orm.jpa.support.PersistenceAnnoutationBeanPostProcessor”/> 4. Made the following changes to bean ticketRegistryCleaner o added attribute p:lock-ref=“cleanerLock” o changed p:repeatInterval to 1800000 5. Added the following bean o <bean id=“cleanerLock” class=“org.jasig.cas.ticket.registry.support.JpaLockingStrategy” p:uniqueId=“${host.name}” p:applicationId=“cas-ticket-registry-cleaner”/>
- 34. 34© 2015 ELLUCIAN.CONFIDENTIAL & PROPRIETARY | 11898 Appendix C – CAS Database Configuration Changes made to pom.xml Added the following dependencies within the <dependencies> node <dependency> <groupId>c3p0</groupId> <artifactId>c3p0</artifactId> <version>0.9.1.2</version> <scope>runtime</scope> </dependency> <dependency> <groupId>org.hibernate</groupId> <artifactId>hibernate-core</artifactId> <version>4.1.0</version> <scope>runtime</scope> </dependency> <dependency> <groupId>org.hibernate</groupId> <artifactId>hibernate-entitymanager</artifactId> <version>4.1.0</version> <scope>runtime</scope> </dependency>
Editor's Notes
- #4 Stack = System How many people are running LP5? How many of you are in a HA availability How many are planning to perform HA?
- #7 https://goo.gl/maps/2kKba The University of Manitoba is Western Canada’s first university, founded in 1877. The University of Manitoba is the only medical doctoral University in the province of Manitoba. The main Fort Garry Campus is a 274-hectare complex where more than 60 major buildings support the teaching and research programs of 23 faculties. The University’s operating budget is typically over $380 million. It is one of Manitoba’s largest employers with over 5,000 full- and part-time academic and support staff. The University is home to nearly 28,000 students, including international students representing 144 countries, enrolled in undergraduate, graduate and professional degree programs. The University invests heavily in research and currently holds 47 Canada Research Chairs and has generated over $11.6 million in technology commercialization royalty revenues over the past five years. Located at the Bannatyne campus, the Faculty of Health Sciences has established links to the major hospitals in Winnipeg. Bannatyne Campus is a complex of 10 buildings located in central Winnipeg connected to the Health Sciences Centre. Other satellite locations include the William Norrie Centre for social work education for inner city residents; University of Manitoba Downtown Aboriginal Education Centre offering degree and certificate programs in partnership with Aboriginal, First Nations, and Métis communities; agricultural research farms at Glenlea and Carman; and field stations at Delta Marsh and Star Lake. Building, equipment and library holdings for all University locations at replacement value are worth more than $2 billion.
- #8 For the last 15 years I've worked at the University of Manitoba in central IT. Recently I've taken the role of solution architect which had allowed me to plan /design a solution utilizing relevant areas of IT to achieve that design.
- #10 I would like to first talk a little bit about our experience with Luminis. We first installed Luminis 3 on a single server back in 2004. Our goal at the time was to replace two homegrown Web applications that provided student data from our mainframe. We had a number of services which we did not wish to manage passwords in so where possible we utilized SSO requiring the user to first log into the portal. For those systems where an SSO was not possible we would connect them to the LDAP server provided by Luminis to make use of the same credentials. Beyond authentication the portal also became a communications platform for all members of the community, making use of campus and personal announcements as well as using the communication options within groups.
- #11 We had huge catastrophic failures cuaisng no access for days position to positive side Though we started with business engagement at the start of the project, it began to wane shortly after going live. As such what occurred over the next 10 years was more of an organic approach with no specific road map. This impacted our ability to move to Luminis 4. As the 3 to 4 migration was not free from issues any interest we had from the business to move forward was eroded as the efforts kept failing and other critical needs were created. As we stayed on Luminis 3 for almost 10 years we then ran into supportability issues with all aspects of the technical solution. Over the last two years we began to suffer from hardware failures. The first major failure caused the portal service to be out of commission for a couple of days waiting for replacements. During that time students were unable to log into our LMS to access their course notes to prepare for final exams. In addition certain university administrative functions (purchasing and travel) were unavailable. Unfortunately our portal architecture was such that the only thing we could do was have the LMS use a second LDAP server to restore access. Purchasing and travel had to wait for the new hardware. Talk about why we were still on Luminis 3 Solaris 9 Luminis 3.3.3.64 Oracle 10G Major hardware failure in July 2013 Impact to LMS, Purchasing, and Travel What was done to help the LMS
- #13 how did we sell it to the business? engaging the business was fundamental to our success everyone has a mobile device so being able to present our solution to mobile devices wanted to be part of the organization
- #14 these were the key elements of our technical scalability, redundant, virtualization we will explore each area on the next number of slides
- #15 Using virtual hardware so that at essentially no cost we can scale our QA environment up to mirror production environment for not only functional testing but non functional testing this also keeps the physical footprint down We took redundancy to a new level by using multiple SANs Virtual nodes are split across two SANs to ensure no single point of failure to the whole environment due to storage going out Database nodes on virtual hardware as well Physical F5 load balancer in an Active/Standby pair SSL termination at the load balancer for all Portal tiers meant to save SSL management Each tier has at least one spare node for redundancy Each tier sized to accommodate historical volumes 25 nodes across 3 environments Speak to evolution of environments
- #16 site license kept costs in check allowing us to remain with Oracle rather than switch to MySQL
- #17 may not find the expertise out there but utilize prototypes mention some F5 tweaks
- #18 Speak to evolution of dev to next to prod again Define JASIG
- #19 3 to 5 no migration
- #23 Business engagement we understand the priorities better funding not an issue roll outs are better Established architecture Stable platform Scalable up and down depending on capacity requirements Increased performance Better supportability support by vendor better testing in other environments better local work time commitments By building a highly available and distributed Luminis platform it has allowed us to: Perform maintenance of the database and O/S during a normal work day It has ensured rapid response to users Allows us to quickly grow horizontally as the usage increases
- #24 d d Putting in a new blade server to the VM environment caused a SAN related outage on one of the VM hosts affecting a portion (portal 4 and cas1) of our production environment
- #29 The load balancer by default would close an LDAP connection if it had been idle for 300 seconds, which did not work with the CAS connection pool that would keep a connection for 1,800 seconds