University of Alberta went from a pilot project of 13 courses in January 2011 to running all centrally supported courses (3600+) in Moodle in September 2012. Our central Moodle instance has seen more than 500,000 page loads and 24,000 unique visitors per day. This is a technical discussion of the challenges involved with providing this solution in large enterprise environment. We will discuss the architecture decisions, processes required to support this project, major incidents, resolutions and workarounds that were encountered during the first year and a half of the life of this project.

1. MoodleMoot 2013
Grzegorz Dostatni
0 to 100 in 18 months

2. Introduction
Incidents
Questions
Architecture and Setup
Organization
What is next?
Results
Agenda

3. Introduction
In 2010 University of Alberta was
looking for a new LMS.
Objectives:
- Reduce licensing costs
- Create a service people want to
use
- Improve service reliability
- Collaborate with other institutions
across the province and beyond

4. Organization
Vice-Provost Information
Technology
- Sponsor
Centre for Teaching and
Learning
- Application Support and Development
Academic Information and
Communication Technologies
- System Administration, Database
Administration, Networking
And Many more...

5. Developers
• New features
• Bug fixes
• Frequent Updates
(See their talk at 3:30
Collaboration without
Compromising)
Organization
VS.
System Admins
• Stability
• Security
• Redundancy

6. Results
Uptime 99.992%
On Oct 1, 2012:
508,681 page
views
3.8 million apache
hits
0.332 s average
page return time

7. Results
Daily Page Views
Total Number
of page views
65 million

8. Results
Daily Unique Visitors
Total number
of unique
visitors who
have logged
in at least
once:
54,844

9. Architecture
Decisions
Infrastructure Hardware considerations
- 6 physical hosts
- VMWare Cluster
- Application cluster behind Hardware
Loadbalancers
- Failover systems in another data centre
- BigIP F5 loadbalancers
- EMC CX-4 fiber attached storage

10. Architecture
Decisions
Software Software Decisions
- Ubuntu LTS
- Postgresql 9.0
- Database in a VM
- FIleserver replication using LVM and
DRBD
- Hourly (nearly) backups with 30 day
retention
- Backups happen on Standby servers
- eAcceletor
- "make everything as simple as possible,
but not simpler"

11. Architecture
All Environments
3 production
Environments:
• Main
Production
• Archive (old
content)
• CPD (non
credit)

12. Architecture
Production
Defined
scalability paths
• Adding nodes
to cluster
• Increasing
performance of
DB

13. Architecture
Backups
Consistent
backups require
a snaphot in time
of both DB and
FS.
Using Database
Point in Time
recovery to
achieve
synchronization

14. Architecture
Monitoring
Catch problems
before they turn
into outages
All machines are monitored for
- CPU load
- Disk
- Free Memory
Database
- Postgresql errors
- Long running processes
Fileserver
- DRBD Mirror Status
Application
- Number of apache processes

15. Incidents
Outages
1 hour outage
when our UPS
failed
0.5 hour outage
cause unknown
(disk
contention?)

16. Incidents
Problems
Missing Database Index
after upgrade to 2.2.3
DRBD Replication failure
Moodle Cron issues

17. What is next?
Suggestions for improvements
- moodle cron.php
- Application functional testing
- Application monitoring
- create our own Ubuntu repository
- scaling up and clustering Postgresql

18. Questions?
More information, including scripts,
documentation, Disaster Recovery
procedures, installation instructions,
please go to
http://www.ualberta.ca/~dostatni/
moodlemoot2013