Qt test framework

QtTest
Unit Testing Framework
Justin Noel
Senior Consulting Engineer
ICS, Inc.

Qt is the Kitchen Sink Toolkit
• Qt comes with just about everything you need
• SQL, Sockets, XML, GUI, etc
• Often one of the challenges of using Qt is knowing all the
modules
• And what they do!
• This is ICS performs these webinars!
• Qt comes with it’s own Unit Testing Framework
• QtTest

Types of Testing
• Unit Testing
• Test one discrete area of the software
• Usually on a class basis. Think “Test Piston Rod”
• Tests can be written and performed as code is developed
• White box tests. Often written by developers
• “Does the software perform as the developer intended?”
• Functional Testing
• Tests subsystems of the whole software system
• Usually a group of classes. Think “Test Engine”
• Tests can be preformed once subsystem is wired together
• Tests interaction between specific groups of units
• “Does the software work together?”

Types of Testing
• System Testing
• End to end testing of the whole software system
• Sometimes includes actual hardware. Think “Test Car”.
• Tests are written late in project
• Because of the end to end nature.
• Black box tests. Often written by separate SQA team
• “Does the software do right thing?”

Test Frameworks for Qt
• QTest / QTest-QML
• Unit testing framework that comes with Qt
• Qt specific functionality
• Can test blind signals. Can simulate mouse and key events.
• Qt specific features can be used outside of QTest framework
• Google Test
• Unit testing framework from Google
• Unique ability to do “Isolated Testing” via Google Mock
• Requires “Inversion of Control” pattern
• Squish
• Playback / Record frame work for “end to end” system testing.
• Used for System Testing

Benefits of Unit Testing
• Test Early!
• Find errors while code is actively being developed
• Rather than long investigations during system testing
• Test Often!
• Run unit tests per commit. Or even pre-commit!
• Any failures must have been introduced by this 200 lines of code!
• Test for Regressions!
• Did new functionality or bug fixing break existing functionality?
• What tests failed?
• Did a code change re-introduce a bug?
• Every bug can have it’s own set of unit tests.

Design for Testability
• Your code must be testable!
• There are design techniques to help make your code more testable
• Try not to add functions to production code just for tests
• Subclassing for tests is acceptable (.ui file members)
• Isolation of units is your primary goal
• Can I test just this class?
• Without re-testing lower level classes?
• Also think about error conditions
• Some may be hard to produce in production environments
• Can I stimulate a bad MD5 error?
• Socket disconnects?
• Default switch cases?

Build Application as Libraries
• Building the bulk of your application as a set of libraries
increases testability
AppBackend.dll
App.exe
(main.cpp)
AppUi.dll
BackendTests.exe UiTests.exe
C++ Test Fixtures QML Test Fixtures

Layered Design
• A layered design is more testable
• Easy to isolate lower layers for testing
• Test Communications without Data, Presentation or Visualization
• No upwards dependencies
• Hard downward dependencies
Visualization Layer (QML)
Presentation Layer (QObjects)
Data Layer (Local Data Storage)
Communications Layer (TCP)
Calls
Down
Signals
Up

Break Downward Dependencies
• Dependency injection works well
• An Inversion of Control Pattern
• Classes take their dependencies as constructor arguments
• Classes do not construct any members themselves
• Loose coupling with signals and slots also works well
• Classes interface to each other via signals and slots only
• 3rd class wires say the UI classes to the Backend classes
• Lets you substitute a test fixture object for a dependency
object
• Instead of an actual production object.

Why is isolation important?
• Avoids testing the same code over and over
• You have already tested the data model
• Why do the UI tests need to also test the model?
• This will make your tests run very quickly
• If there is a failure in a low level class it won’t trigger errors in higher
level tests
• Avoid testing side effects
• Tests can be implementation agnostic
• DB, Files, Cloud. Shouldn’t matter for UI tests.
• Tests should only depend on interfaces, not behavior

EE Analogy: Motor Controller Test
Motor Controller
7408 AND
Momentary
Switch
Safety
Interlock
Motor
1
2
3
Stimulus: Apply PowerVerify: High or Low

QtTest is a unit testing framework
• QtTest comes with classes to build test fixtures for units
• Has some “special sauce” for Qt
• Test signals without needing to write slots (QSignalSpy)
• Simulate user events (MousePress, KeyPress, FocusIn, etc)
• QtTest is not an isolation framework
• You will have to make replacement classes for dependencies
yourself
• These are usually referred to as Mock Classes.
• Next month we will talk about Google Mock which is can help a lot
with isolation framework.

Creating a Test Fixture
• Can be implemented inside a source file
• No need for a header file
• Inherit from QObject
• Qt uses introspection to deduce test functions
• Private slots are test functions
• There are some reserved function names
• Use Q_TEST_MAIN macro to generate a main()
• Or create you own main.cpp to run multiple test fixtures

Creating a Test Fixture
#include <QtTest>
class TestQString: public QObject
{
Q_OBJECT
private slots:
void toUpper() {
QString str = "Hello"; // Prepare
str = str.toUpper(); // Stimulate
QCOMPARE(str, QString("HELLO")); // Verify
}
};
QTEST_MAIN(TestQString)
#include “testqstring.moc”

Test Fixture Project
QT += testlib # Plus libs your uses.
CONFIG += testcase # Creates make check target
SOURCES += TestQString.cpp
qmake
make
make check
********* Start testing of TestQString *********
Config: Using QtTest library %VERSION%, Qt %VERSION%
PASS : TestQString::initTestCase()
PASS : TestQString::toUpper()
PASS : TestQString::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of TestQString *********

Running Multiple Tests
#include “QTestString.h” // Easier if you use a header file
int main(int argc, char** argc)
{
bool success = true;
r &= QTest::qExec(TestQString(), argc, argv);
...
return success;
}

Initializing and Cleaning Up
class TestQString: public QObject
{
Q_OBJECT
private slots:
void initTestCase() { // Called before ALL tests. }
void cleanupTestCase() { // Called after ALL tests}
void init() { // Called before each test. }
void cleanup() { // Called after each test }
void toUpper();
void toLower();
};

Writing Good Tests
• Test one thing at a time
• It’s tempting to get as much done in one function as you can.
• This makes one verification depend on another verification
• Tests will be brittle and changes to code could break lots of
verifications
• Do not have your tests depend on each other
• Order should not matter!
• Not depending on other tests will help you when you remove
functionality.
• You do not want to have to fix a cascading waterfall of tests!

Use New Object For Each Test
• Use init()
• Create your Object Under Test
• Plus whatever dependencies it has
• Production dependencies
• Or Test dependencies like Mock Classes
• Use cleanup()
• Do not be sloppy and leak memory in you tests.
• Your tests should run clean under Valgrind
• Aside for some suppressions for FreeType and FontConfig

Use New Object For Each Test
class MotorControllerTests: public QObject
{
Q_OBJECT
private slots:
void init() { controller = new MotorController; }
void cleanup() { delete controller; }
void motorRunsIfInterlockAndSwitchAreTrue() {
controller.setInterLock(true);
controller.setSwitch(true);
QCOMPARE(controller.motorOn(), true)
}
void motorDoesNotRunIfInterlockIsFalse() { ... }
private:
MotorController* controller;
};

Data Driven Tests
• Sometimes you need to test many inputs to a function
• NULL, Same Value Twice, Negative, Positive, NAN, etc
• This isn’t great test code. Violates Test 1 Thing
QCOMPARE(QString("hello").toUpper(), QString("HELLO"));
QCOMPARE(QString("Hello").toUpper(), QString("HELLO"));
QCOMPARE(QString("HellO").toUpper(), QString("HELLO"));
QCOMPARE(QString("HELLO").toUpper(), QString("HELLO"));
• Or worse this. Lots of error prone typing if not trivial.
void testhelloToUpper() {...}
void testHELLOToUpper() {...}
void testHeLOToUpper() {...}

Data Driven Tests
private slots:
void toUpper_data() {
QTest::addColumn<QString>("string");
QTest::addColumn<QString>("result");
//We can name the sub tests
QTest::newRow("all lower") << "hello" << "HELLO";
QTest::newRow("mixed") << "Hello" << "HELLO";
QTest::newRow("all upper") << "HELLO" << "HELLO";
}
void TestQString::toUpper() { // Called 3 Times
QFETCH(QString, string);
QFETCH(QString, result);
QCOMPARE(string.toUpper(), result);
}
};

Testing Events and Signals
• Use QSignalSpy to attach to any signal
• Signal parameters need to be QVariant compatible
• May require Q_DECLARE_METATYPE()
• May require qRegisterMetaType<>()
• Verify with count() and arguments()
• Use QtTest static methods to stimulate events
• void mousePress(widget, button, modifiers, pos)
• void keyClicks(widget, string, modifiers)
• Etc, etc

Testing Events and Signals
class QPushButtonTests: public QObject
{
Q_OBJECT
private slots:
void init() { //Create, Show, Wait For Window Shown }
void emitsClickedWhenLeftMouseButtonIsClicked() {
QSignalSpy spy(button, &QPushButton::clicked);
QTest::mouseClick(button, Qt::LeftButton)
QCOMPARE(spy.count(), 1);
}
private:
QPushButton* button;
};

Stimulating Signals
• Often your object under test is depending on signals from
other objects.
• For example QPushButton clicked()
• Don’t re-test the button
• Directly call QPushButton clicked()
• signals access specifier is actually a #define public

Stimulating Signals
private slots:
void init() {
settings = new Settings;
settingsDialog = new SettingsDialog(settings);
}
void LanguageComboBoxUpdatesOnSettingsLanguageChanged() {
QComboBox* combo = settings->ui()->langCombo;
settings->lanuageChanged(“en_US”);
QCOMPARE(combo->currentText(), “English”);
}
private:
Settings* settings;
TestableSettingsDialog* button; // Exposes ui member via function
};

Testing QML with QTest
• Writing tests in QML is very similar to C++
• Create a test fixture class
• Implement specially named functions
• Prepare, Stimulate, Verify.
• KISS – Keep It Short and Simple
• There are QML equivalents to
• compare, verify, wait
• SignalSpy
• mouseClick(...)
• TestCase
• Specific test fixture class. C++ just uses QObject.

UI Control Test
• UI Controls are easy to test as they have no dependencies
TestCase {
name: "Button“
Button { id: button }
SignalSpy { id: clickedSpy; target: button; signal: ”clicked” }
function test_mouseClickEmitClicked() //Tests are run in alpha order
{
mouseClick(button)
compare(clickedSpy.count, 1)
}
}

New Objects For Each Test
TestCase {
id: testCase
property Button button: null
name: "Button“
Component { id: buttonFactory; Button{} }
function init() { button = buttonFacotry.createObject(testCase) }
function cleanup() { button.destroy() }
function test_mouseClickEmitClicked()
{
mouseClick(button)
compare(clickedSpy.count, 1)
}
}

Running QML Tests
• qmltestrunner
• Prebuilt test runner.
• Takes a filename or directory of files to run
• Many command line options
• See Qt Documentation
• Can run many test by itself.
• All UI controls should be testable this way
• Custom main.cpp runner
• Reuse the guts of qmltestrunner
• Add injected type information or other setup
• As required by your app

qmltestrunner
$QTDIR/bin/qmltestrunner –input TestDir

Custom main.cpp
#include <QtQuickTest/quicktest.h>
#include <QtCore/qstring.h>
#ifdef QT_OPENGL_LIB
#include <QtOpenGL/qgl.h>
#endif
#define TEST_DIR “UiTests” // Pass this as DEFINE in project file
int main(int argc, char **argv)
{
//Register Types Here
return quick_test_main(argc, argv, “UiTests", TEST_DIR);
}

Screen Tests
• Screens are more interesting
• They depend on Controllers and Models from C++
• QML and C++ is natively a layered architecture
• Most common ways to communicate with C++ is injection
• Of C++ instances or C++ types
• However we can Mock Controllers and Models
• Using QML instead of C++

C++ Integration Mechanisms
• Context Properties – Injects global pointers at root context
• view.rootContext()->setContextProperty(“coffeeMaker”, &maker);
• QML Singletons – Inject a singleton factory for import
• qmlRegisterSingletonType<CoffeeMaker>(“MrCoffee", 1, 0,
“CoffeeMaker",
factoryFunction);
• Registered Types – Add new types for import
• qmlRegisterType<CoffeeMaker>(“MrCoffee“, 1, 0, “CoffeeMaker“);

Mocking Context Properties
• Create a Mock[ClassName].qml file
• Create the equivalent property, signal and function API
• The TestCase item is the root item for your test
• It literally is the rootContext()
• Simply add properties to the TestCase with the same names
• They will have global scope
• Tests can be run through the standard qmltestrunner

Mocking Singletons
• Create a custom QML Test main.cpp
• Call qmlRegisterSingletonType with URL to file name of mock
• qmlRegisterSingletonType(QUrl("file:///path/MockCoffeeMaker.qml"),
“MrCoffee", 1, 0,
“CoffeeMaker");
• Run tests as usual.
• Objects will use mock versions of C++ classes

Mocking Registered Types
• Inside a directory structure similar to production
• Mocks/com/ics/qtest/examples/
• Create a qmldir manifest file listing your mock types
• CoffeeMaker 1.0 MockCoffeeMaker.qml
• Manipulate QML2_IMPORT_PATH to pull in testing
module rather than production module

CoffeeMaker.h
class CoffeeMaker : public QObject
{
Q_OBJECT
Q_PROPERTY(int temp READ temp
NOTIFY tempChanged)
Q_PROPERTY(int targetTemp READ targetTemp
WRITE setTargetTemp
NOTIFY targetTempChanged)
public:
enum Strength {
Bold=0, Medium, Light
} Q_ENUM(Strength)
CoffeeMaker(IPump& pump, IHeater& heater);
...
Q_INVOKABLE void brew(Strength strength);
};

main.cpp
int main(int argc, char** argv)
{
QGuiApplication app(argc, argv);
...
CoffeeMaker coffeeMaker(pump, heater);
qmlRegisterUncreatableType<CoffeeMaker>(“MrCoffee”,
1, 0,
“CoffeeMaker”, “”);
QQuickView view;
view.rootContext()->setContextProperty(“coffeeMaker”, & coffeeMaker);
view.setSource(“qrc:/Main.qml”);
view.show();
return app.exec();
}

CoffeeScreen.qml
import QtQuick 2.5
import MyComponents 1.0
import MrCoffee 1.0
Screen {
...
SpinBox {
value: coffeeMaker.targetTemp
onValueChanged: coffeeMaker.targetTemp = value
Connections {
target: coffeeMaker
onTargetTempChanged: value = coffeeMaker.targetTemp
}
}
Button {
text: qsTr(“Brew”)
onClicked: coffeeMaker.brew(CoffeeMaker.Medium)
}
}

CoffeeScreen.qml With Hooks
Screen {
property alias _spinBox: spinBox // No QML protected specifier 
property alias _button: button
...
SpinBox {
id: spinBox
value: coffeeMaker.targetTemp
onValueChanged: coffeeMaker.targetTemp = value
Connections {
target: coffeeMaker
onTargetTempChanged: value = coffeeMaker.targetTemp
}
}
Button {
id: button
text: qsTr(“Brew”)
onClicked: coffeeMaker.brew(CoffeeMaker.Medium)
}
}

Only Test CoffeeScreen
• CoffeeMaker is tested separately
• Is a discrete unit
• Has it’s own unit tests.
• A broken CoffeeMaker object shouldn’t fail CoffeeScreen tests
• UI Controls are tested separately
• SpinBox and Button are also discrete units
• Have their own tests
• Changing UI control APIs could cause tests not to run
• Controls are leaf nodes in the software design
• Easy to test by themselves.
• Should stabilize quickly
• When was the last time the API for SpinBox changed?

CoffeeScreen Text Fixture
CoffeeScreen
SpinBox
PushButton
MockCoffeeMaker
Stimulus: Set Values / EmitVerify: Was Function Called?

MockCoffeeMaker.qml
QtObject {
property int temp: 0
property int targetTemp: 0
property int brewInvokations: 0
property var brewStrengthParameter: []
function brew(strength) {
brewStrengthParameter[brewInvokations] = strength
brewInvokations++
}
function reset() {
brewStrengthParameter = []
brewInvokations = 0
}
}

TestCase with Mocks
UiTestCase.qml
---------------------------------------------------------------------
TestCase {
when: windowShown
property MockCoffeeMaker coffeeMaker: MockCoffeeMaker {}
function resetAllMocks() {
coffeeMaker.reset();
}
}

CoffeeScreenTest.qml
import “../qmlSource”
UiTestCase {
id: testCase
name: "CoffeeScreen“
property CoffeeScreen coffeeScreen: null
Component { id: factory; CoffeeScreen {} }
function init() {
resetAllMocks();
coffeeScreen = factory.createObject(testCase)
}
function cleanup() {
coffeeScreen.destroy()
coffeeScreen = null
}
...

...
function test_spinBoxValueIsBoundToCoffeeMakerTemp() {
coffeeMaker.temp = 50
compare(coffeeScreen._spinBox.value, 50)
}
}
// Bug #532
function test_settingSpinBoxValueDoesNotBreakBinding() {
coffeeScreen._spinBox.value = 99
coffeeMaker.temp = 20
compare(coffeeScreen._spinBox.value, 20)
}
}

...
function test_buttonClickedCallsBrew() {
coffeeScreen._button.clicked()
compare(coffeeMaker.brewInvokations, 1)
}
function test_buttonClickedCallsBrewWithMediumStregth() {
coffeeScreen._button.clicked()
compare(coffeeMaker.brewStrengthArgument[0], CoffeeMaker.Medium)
}

Custom Test Runner
#include <QtQuickTest/quicktest.h>
#include <QtCore/qstring.h>
#ifdef QT_OPENGL_LIB
#include <QtOpenGL/qgl.h>
#endif
#include “CoffeeMaker.h”
int main(int argc, char **argv)
{
qmlRegisterUncreatableType<CoffeeMaker>(“MrCoffee”, 1, 0,
“CoffeeMaker”, “”);
return quick_test_main(argc, argv, “UiTests", “.”);
}

Thank You!
Justin Noel
Senior Consulting Engineer
ICS, Inc.
www.ics.com

Qt test framework

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Qt test framework

Similar to Qt test framework (20)

More from ICS

More from ICS (20)

Recently uploaded

Recently uploaded (20)

Qt test framework