This document discusses the evolution of Continuous Integration practices at Moodle HQ. It describes how Moodle moved from a process in 2001-2010 with few automated tests and inconsistencies to a modern CI/CD process. This process includes pre-integration and post-integration checks run on Jenkins, including unit testing, code quality checks, database compatibility tests, and other validation tests. It also discusses expanding test coverage through automated UI tests with Selenium and later Behat scenarios written by developers.

1. Continuous Integration:
Improving Quality in Moodle
Dan Poltawski
Integrator
Moodle HQ
@dan_p
the world’s open source learning platform
#mootieuk15

2. the world’s open source learning platform
Wide topic, today focusing on:
• “Each integration is verified by an automated build
(including test) to detect integration errors as quickly as
possible.” - Martin Fowler
http://www.martinfowler.com/articles/continuousIntegration.html
Continuous Integration

3. the world’s open source learning platform
Integration Process: 2001 - 2010
Numerous problems:
• Single developer may be only person aware of a
change
• Inconsistency was rife
• Developers often unaware of the consequences of
some changes
• Testing was not enforced
• Unit tests were limited and frequently broken
New Development
Release

4. the world’s open source learning platform
Moved to git and brought about a change in
development process
• Enforced code review
• Enforced ‘manual testing’
• Integration team introduced to move
towards ensuring more consistency
• Introduction of CI systems
Integration Process: Today
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release

5. the world’s open source learning platform
• Needed automated tests to supplement the
manual reviews done by the Integration
team
• CI Server was setup - Jenkins
• Open source/extensible
• Self hosted
• http://jenkins-ci.org
Post-integration checker
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release

6. the world’s open source learning platform
Post-integration checker
Jenkins polls for changes in
integration.gitCode pushed to integration.git
Issue status is set (e.g. failed)
Detects code changes and starts
running checked jobs
Jenkins notifies status changes
Integration team Jenkins

7. the world’s open source learning platform
Post-Integration Checks

8. the world’s open source learning platform
Post-integration checks: whitespace
• First job to be created (the whitespace police
started it all!)
• Detect incorrectly spacing added
• Frequent error:
• Easy to miss setting up your editor for
Moodle
• Not spotted easily by reviewers

9. the world’s open source learning platform
Post-integration checks: compare upgrade & install
• Detects DB inconsistencies
between install and upgrade code:
1. Runs the installer
2. Run upgrade from a previous
version
3. Compares the database
schema from both
4. Fails in the case of any
differences
• Finds schema inconsistencies
previously frequently undetected -
(common source of number of 1.9 -
> 2.x upgrade bugs)
• Also brings fringe benefit of testing
upgrade code (though limited by
lack of data)

10. the world’s open source learning platform
• Initially support for Moodle 2.0-2.2 with
SimpleTest
• Custom CLI generated test data, ran
tests and output results as JUnit xml file
• Culturally testing was not embedded
• Tests rarely run or added by developers
• Had to mock everything - no way to
setup test data
Post-integration checks: Unit testing

11. the world’s open source learning platform
• Moodle 2.3 added support for PHPUnit
• Data generators significantly improved unit testing
experience
• Helped move cultural change towards automated tests
• Challenges:
• Keeping the test time down
• ‘Random’ fails
Post-integration checks: Unit testing

12. Moodle 2.3 Moodle 2.6 Moodle 2.9
24,273 44,491 58,011
the world’s open source learning platform
Unit test assertions

13. the world’s open source learning platform
• Many created a result of mistakes slipping through:
• Incorrect upgrade savepoints
• version.php file check
• Check files built correctly (less, shifter, grunt)
• php syntax check (php -l)
• Unresolved merge conflicts
Post-integration checks: Others

14. the world’s open source learning platform
• Desirable to check before integration
• Can help reduce delays
(timezones) by providing more rapid
feedback
• Better suited to more pedantic
checks
• Provide supplemental information
rather than just pass/fail
Pre-integration checker
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release
Inline
comments must end
in full-stops, exclamation
marks, or question
marks.

15. the world’s open source learning platform
Pre-integration checker
Merges branch with moodle.git
Run pre check jobs against
merged branch
Produce ‘smurffile’ of changes,
filtered by lines affected by patch
jshint
phpcs
csslint
phpdoc
php -l
commit
savepoint
thirdparty
Report summary back to tracker
checkstyle.xml

16. the world’s open source learning platform
• Not possible to run all our testing scenarios on a
per-commit basis due to time to run
• Nightly server does these long-running tasks
• Behat (Acceptance tests)
• Multi-platform tests
• phpunit: pgsql, mariadb, mysql, mssql,
oracle
• behat: firefox, chrome, phantomjs
Nightly checks
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release

17. the world’s open source learning platform
//Teacher creates a discussion
@Test
public void teacherCreateDiscussion() throws Exception {
//Adding a discussion
forum.clickForumLink(this.properties.get("nameOfForum"));
forum.clickAddNewDiscussionTopicButton();
forumPosts.enterSubjectField(this.properties.get("teacherSubject"));
forumPosts.enterMessage(this.properties.get("teacherMessage"));
forumPosts.clickPostToForum();
}
Automated UI tests: Selenium Native
• Java-based selenium test suite
• Not embedded in development process
• Tests written by test manager, not developers
• Maintained in separate repository
Scenarios in 2.3
28

18. the world’s open source learning platform
Given I add a "Forum" to section "1" and I fill the form with:
| Forum name | Test forum name |
| Forum type | Standard forum for general use |
| Description | Test forum description |
| Subscription mode | Optional subscription |
And I add a new discussion to "Test forum name" forum with:
| Subject | Test post subject |
| Message | Test post message |
Automated UI tests: Behat
• Improving from pure selenium implementation
• No java :-) Human-friendly ‘Gherkin’ language
• PHP integration code
• Tests required to land with features by developers
• Culturally embedded Scenarios in 2.9
412

19. the world’s open source learning platform
• Many pre-checker ideas
• Run phpunit/behat/multi-
platform tests in pre-checker
(split by component)
• Intelligent re-checking when
branches get updated
• Backup/restore checks
• Custom jira fields for prechecker
• Automated rejections
• Webservices testing
• Run all post-check’s in pre-checker
• Others as we detect them
Future

20. Questions?
@dan_p
the world’s open source learning platform
https://github.com/moodlehq/moodle-local_ci