Your SlideShare is downloading. ×
0
Moodle is dead…
The Problem
• On the 30th of July 2013 the VLE, DBA and Network teams of Information
services were invited to a meeting wh...
Team brief
• The team are allowed access to any of the offsite backup systems
(VMs, Data and Databases).
• No University h...
Incident Procedure
Incident Manager
Coordinates
troubleshooting/Information
gathering
IT Incident Duty Manager
(Technical ...
Who was involved?
Head Of
Applications
DBA?
VLE/WEB
Data Centre
&
Operations
Edinburgh Napier’s
Moodle Infrastructure
Some stats…
What Solution? (Approx. 2hrs)
• The following decisions were made during the initial Emergency
Incident meeting.
1. We wou...
Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No acce...
 Register for Amazon Web Services (AWS) account
a. Purchased AWS Business support.
 Setup AWS (Approx. 30mins)
a. Create...
Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No acce...
Tasks involved
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Creat...
Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No acce...
Tasks involved
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Creat...
Tasks involved continued...
 Installation & Configuration of Services (Approx. 2hrs)
a. Apache, PHP & MySQL Installed, se...
Tasks involved continued...
 Moodle Administration (Approx. 1hr)
a. Recreated Moodle admin account
b. Switch Moodle user ...
Moodle in the Cloud!
Lessons Learned
• Backups were only stored on high availability disks across two campuses
• Recommend
• Backup to tape eve...
Lessons Learned continued…
• SMS students about the downtime of the system
• Recommend
• Storing of student SMS/phone deta...
Going forward
• Have a cloud Active Directory solution available.
• Have a parked AWS instance available when required.
• ...
Useful links
• Amazon Web Services - http://aws.amazon.com/
• Windows Azure (Cloud AD) - http://www.windowsazure.com/en-us...
Upcoming SlideShare
Loading in...5
×

Moodle is dead... Iain Bruce, James Blair, Michael O'Loughlin

297

Published on

Moodle is dead... Iain Bruce, James Blair, Michael O'Loughlin
Presented at Edinburgh Moodlemoot 2014
www.moodlemoot.ie

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
297
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Moodle is dead... Iain Bruce, James Blair, Michael O'Loughlin"

  1. 1. Moodle is dead…
  2. 2. The Problem • On the 30th of July 2013 the VLE, DBA and Network teams of Information services were invited to a meeting which was to test our Business continuity with our VLE environment (Moodle). The teams were given the following scenario. • All University systems have been shut down due to a full power failure which has affected both Craiglockhart and Merchiston with no other current services at our Sighthill campus; this also means that there is no external internet access from inside the University. • Moodle as being the most critical system at this time of the year is essential and has to be back on-line as quickly as possible to allow students access to their current course work.
  3. 3. Team brief • The team are allowed access to any of the offsite backup systems (VMs, Data and Databases). • No University hardware can be used for the process. • Staff were allowed to use their on-call laptops. • Staff have access to budget to gain resources if needed.
  4. 4. Incident Procedure Incident Manager Coordinates troubleshooting/Information gathering IT Incident Duty Manager (Technical Services) Takes ownership of Incident through to resolution Technical Expert(s) Update call notes with progress Call Management system records the various stages of the Incident.
  5. 5. Who was involved? Head Of Applications DBA? VLE/WEB Data Centre & Operations
  6. 6. Edinburgh Napier’s Moodle Infrastructure
  7. 7. Some stats…
  8. 8. What Solution? (Approx. 2hrs) • The following decisions were made during the initial Emergency Incident meeting. 1. We would use Amazon Web Services and create a virtual machine in the cloud.  AWS provided scalable solutions, import/export options for database 2. We would look to get database and files backup from tape 3. Switch user accounts to manual 4. We discussed how to communicate out to students.
  9. 9. Tasks involved  Obtain a database backup (Approx. 4hrs) a. Installed Zmanda Community Edition to recover MySQL b. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of MySQL), meant a Hot backup was obtained. c. Truncated logs and statistics to speed up the import d. Import Database into AWS MySQL database  Obtain Moodle user files from backup (Approx. 5hrs) a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.
  10. 10.  Register for Amazon Web Services (AWS) account a. Purchased AWS Business support.  Setup AWS (Approx. 30mins) a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space. b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance. c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.  Transfer Moodle user files to Amazon (Approx. 7hrs) a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup files. b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs. Tasks involved
  11. 11. Tasks involved  Obtain a database backup (Approx. 4hrs) a. Installed Zmanda Community Edition to recover MySQL b. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of mysql), meant a Hot backup was obtained. c. Truncated logs and statistics to speed up the import d. Import Database into AWS MySQL database  Obtain Moodle user files from backup (Approx. 5hrs) a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.  Register for Amazon Web Services (AWS) account a. Purchased AWS Business support.  Setup AWS (Approx. 30mins) a. Launch Amazon console in correct region. b. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space. c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.  Transfer Moodle user files to Amazon (Approx. 7hrs) a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup files. b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.
  12. 12. Tasks involved  Setup AWS (Approx. 30mins) a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space. b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance. c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.  Transfer Moodle user files to Amazon (Approx. 7hrs) a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup files. b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.
  13. 13. Tasks involved  Obtain a database backup (Approx. 4hrs) a. Installed Zmanda Community Edition to recover MySQL b. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of MySQL), meant a Hot backup was obtained. c. Truncated logs and statistics to speed up the import d. Import Database into AWS MySQL database  Obtain Moodle user files from backup (Approx. 5hrs) a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.  Register for Amazon Web Services (AWS) account a. Purchased AWS Business support.  Setup AWS (Approx. 30mins) a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space. b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance. c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.  Transfer Moodle user files to Amazon (Approx. 7hrs) a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup files. b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.
  14. 14. Tasks involved  Setup AWS (Approx. 30mins) a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space. b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance. c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.  Transfer Moodle user files to Amazon (Approx. 7hrs) a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup files. b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.
  15. 15. Tasks involved continued...  Installation & Configuration of Services (Approx. 2hrs) a. Apache, PHP & MySQL Installed, services started. b. Installation of GIT c. Cloned our Moodle code from Git Hub where all our commits are backed up automatically. d. Backed up ignored files separately, for example config.php e. Ensure Apache user has correct permissions on directory. f. Alter Moodle config.php with new database credentials.
  16. 16. Tasks involved continued...  Moodle Administration (Approx. 1hr) a. Recreated Moodle admin account b. Switch Moodle user accounts to ‘Manual’ and regenerate passwords c. Successfully access Moodle user accounts and data on AWS, various admin/teacher/student accounts accessed and tested.
  17. 17. Moodle in the Cloud!
  18. 18. Lessons Learned • Backups were only stored on high availability disks across two campuses • Recommend • Backup to tape every 2 weeks. • Passwords for a number of the services were only with the DBAs • Recommend • Storing of passwords in the same way across the teams. • Access to all passwords for the teams. • Further Backups • Recommend • Additional Emergency MySQL dump • Filter out redundant files from backup to reduce time • Migrate only necessary tables • Search and replace text inside sql file before import (mainly for hardcoded urls) • Replicate data up to Cloud
  19. 19. Lessons Learned continued… • SMS students about the downtime of the system • Recommend • Storing of student SMS/phone details off campus or on on-call laptops • Moodle • Recommend • Create basic scripts for the processes against the database on user accounts • Active directory is one of the main services to allow our system to work • Recommend • Store an active directory server in the cloud or at another universities network
  20. 20. Going forward • Have a cloud Active Directory solution available. • Have a parked AWS instance available when required. • Use Vagrant/PuPHPet GUI to have consistent setup and manage virtual machines
  21. 21. Useful links • Amazon Web Services - http://aws.amazon.com/ • Windows Azure (Cloud AD) - http://www.windowsazure.com/en-us/ • Vagrant/PuPHPet - https://puphpet.com/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×