The document discusses automated system testing for a learning management system (LMS). It introduces the motivation for test automation due to shifts in agile software development. The focus of automation is on regression testing core functionality through the system testing level. Over 30 test cases were identified for the LMS by interviewing support staff and researching common user cases. Both synthetic and real test data are used, including course materials, user accounts, and submissions. The infrastructure implements page object modeling and stores test data in JSON files to provide data independently to test cases.

1. WISSEN
TECHNIK
LEIDENSCHAFT
Automated system testing for
a learning management
system
31.10.2019
Lukas Krisper

2. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
2

3. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
3

4. A shift towards Agile Software Development
• Over the last two decades …
• Emerge of agile methodologies in software development
• Acceleration of software delivery and feedback cycles
• Automation of manual processes
• Quality Assurance?
• Only 20% of IT organisations have automated more than half of their
test cases (State of Testing Report, 2018)
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
4
Motivation

5. New Releases in Summer 2019
• TeachCenter 3.0
• Learning Management System (LMS)
• Migration of whole LMS to new Moodle Version (3.5)
• New GUI for all users
• Set of Unit-Tests existed
• Sparely documented manual tests
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
5
Motivation

6. Research Questions
• (RQ1) How can the existing core functionality of a large software system
be assured automatically in new versions of the system?
• (RQ2) Which test cases and test suites need to be designed to test a
learning management system effectively?
• (RQ3) Which test data is needed to test a learning management system
and how can the data be provided to automated test cases?
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
6
Motivation

7. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
7

8. Focus of Automation Activities
• Software Quality Characteristic*: Functional Suitability
• Testing Level: System Testing
• Testing Type: Regression Tests
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
8
Test Automation
*According to ISO-25010

9. Software Quality Characteristic: Functional Suitability
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
9
Test Automation
• Does the software product do what it is supposed to do?
• Functional testing as basis of all testing activities
• A software product that does not provide functional suitability is rather
worthless
• E.g. calculator that calculates very fast
and has a nice UI, but the calculations are wrong
2 + 2 = 5

10. Testing Level: System Testing
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
10
Test Automation
• Software product is considered in its entirety
• Test environment has close resemblance to production environment
• Tests conducted from user’s perspective
• Typically using the systems GUI for interaction
• Automated system tests as second line of defense
• If tests fail at this level → malfunctioning system + wrong or missing
test at lower testing level

11. Testing Type: Regression Tests
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
11
Test Automation
• Is already existing functionality affected by the changes applied to a software?
• The first type of tests to be considered when automating tests
• Quickly run a set of tests after applying changes:
• Extensions to a feature
• Bug fixes
• Refactorings
• …
• Especially useful when changes happen late in the project and time for
manual testing is short

12. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
12

13. What Test Cases should be automated?
• Important consideration!
• Interview with first level support of TeachCenter 2.0
• Research in literature on usages of LMS
• Results:
• Knowledge on most important use cases for students and teachers
E.g. reading the course’s contents, communicating with other
participants, submitting material to the course, …
• > 30 test cases combined to different test suites
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
13
Test Cases for a LMS

14. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
14

15. Real Data vs Synthetic Data
• Real Data
• Transfer data from production environment to test environment
• Data has to be edited (according to laws on data protection/privacy*)
• Important: Keep consistency in data set!
• Synthetic Data
• Newly created data for the purpose of testing
• Always GDPR compliant
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
15
Test Data for a LMS
* e.g. General Data Protection Regulation (GDPR)

16. Data for a LMS
• A LMS typically contains data on
• Courses (course descriptions, learning material, …)
• Data on people involved in the courses (names, email-addresses, …)
• Data from interaction of people with the courses (submissions, …)
• TeachCenter 3.0 is tested with both synthetic and real data
• Data used for automated tests is labeled as such
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
16
Test Data for a LMS

17. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
17

18. Overview on Infrastructure
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
18
Implementation and Infrastructure

19. Test Suite 1
Test Case 1
Test Case n
Test Case 2
…
Test Suite 1
Test Case 1
Test Case n
Test Case 2
…
Overview on Implementation
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
19
Implementation and Infrastructure
Test Suite 1
Test Case 1
Test Case n
Test Case 2
…
Test Data
Manager
Test Driver
Utility
e.g. initialise headless Chrome
browser with cleared cache
e.g. get PDF files for
submissions, references to
test data stored in JSON
files, …
Page Object 1Page Object 1Page Object 1
TestNG
Page Object 1Page Object 1
Helper Classes
e.g. taking screenshots on errors
See following slides

20. PageObject Design Pattern
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
20
Implementation and Infrastructure
• Components of a website are modelled into
reusable objects → “Page Objects”
• Page Objects offer functionality to interact with
website (click links, enter input fields, …)
• Tests use Page Objects to interact with website
(perform action → assert result of action)

21. Identification of Objects in UI
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
21
Implementation and Infrastructure
• Tests should be executable regardless …
• screen resolution
• window sizes
• language settings
• Identification of objects using stable features (IDs or XPath expressions)
DOM Element <button type= "button" id= "btn1" data-role= "next "> Next </button >
ID btn1
XPath expression //button[@data-role=’next’]

22. Agenda
• Motivation
• Test Automation
• Test Cases for an LMS
• Test Data for an LMS
• Implementation and Infrastructure
• Conclusion
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
22

23. (RQ1) How can the existing core functionality of a large software
system be assured automatically in new versions of the system?
• Regression Tests for „Functional Suitabilty“
• System Testing Level
• Large software system = many integrated components that interact
with each other → “high” testing level to ensure stability and detect
side effects
• Continuous Integration Environment (Automation Server)
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
23
Conclusion

24. (RQ2) Which test cases and test suites need to be designed to test
a learning management system effectively?
• Identification of use cases
• Interview with 1st level support + literature research
• Prioritisation of use cases
• Implementation of test cases based on use case
• Execution of suites of test cases according to prioritisation of use cases
• Multiple times a day ~ once a week
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
24
Conclusion

25. (RQ3) Which test data is needed to test a learning management
system and how can the data be provided to automated test cases?
• Synthetic and real test data
• 1 configured course with various activities and resources
• 1 student‘s account (enrolled to course)
• 1 teacher‘s account (enrolled to course)
• 1 PDF file
• Test data is persisted on test system (references in JSON-files) …
• Or provided to the test cases on demand
• Separation of test data and automated test cases
• No hardcoded test data
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
25
Conclusion

26. Thank you for your attention!
Lukas Krisper, 31.10.2019
Automated system testing for a learning management system
26
Conclusion