The document summarizes the current state of quality assurance testing at Samsung Open Source Group and proposes next steps. It describes their use of compile-time testing, unit testing, and Jenkins for continuous integration. However, graphical testing is missing and unit test coverage is only 39.2%. The document recommends prioritizing graphical testing using Excatness, increasing unit test coverage 5-10% per release, adding pre-push continuous integration, and expanding the test matrix to cover more configurations. Overall progress on testing has been slow and motivation for writing new unit tests is needed.

1. 1Samsung Open Source Group
Stefan Schmidt
Samsung Open Source Group
stefan@osg.samsung.com
QA Status Report
EFL Developer Day EU 2015

2. 2Samsung Open Source Group
Agenda
●
QA Summary
●
Missing Parts &
Proposed changes
●
Summary

3. 3Samsung Open Source Group
QA Summary

4. 4Samsung Open Source Group
Current: Compile Time Testing
●
Developers machine (One arch, make,
hopefully make check and distcheck)
●
Machines of other developers
●
Jenkins covers different architectures,
different compilers, different
dependencies and versions

5. 5Samsung Open Source Group
Current: Unit Testing
●
Test cases collected in test suite and run
with make check
●
Should be run before committing
●
Also run by Jenkins
●
While unit testing is the main idea it also
had catched segfaults due to basic run
time testing of new code

6. 6Samsung Open Source Group
Current: Miscellaneous
●
Static Analysis
●
Different EFL profiles
●
Different configuration options
(Wayland, XCB)
●
Different make targets (all, check,
distcheck, doc)

7. 7Samsung Open Source Group
Missing Parts &
Proposed Changes

8. 8Samsung Open Source Group
Missing: Graphical Tests
●
No usage of graphical test cases
●
Tooling is there: Excatness
●
Almost no developer uses it, regularly
●
No automation

9. 9Samsung Open Source Group
Steps Forward: Graphical Testing
●
Identify and fix problems with exactness on Jenkins
– Done ✔
●
Run exactness during the nightlies
– Screenshots and test cases have been updated by
Kabeer ✔
– Stopped working three months ago when Kabeer left ✘
●
So far screenshots for elementary_tests and some edje
examples
– Build job for elementary_test and edje ✔/✘

10. 10Samsung Open Source Group
Missing: Unit Testing
● Only 39.2% of our code base covered
● Newly added API's should be mandatory
to come with unit tests
● My goal would be that we increase
coverage 5-10% per release
● How to motivate unit test
writing?

11. 11Samsung Open Source Group
Steps Forward: Unit Testing
●
Proposal: Make test cases for public API additions (EAPI)
mandatory from today
– We agreed on it to be mandatory ✔
– Left hard to test parts out of this agreement (edje, gl, ...) ✘
– From the 72 new API's in 1.15 (54) and 1.16 (18) only 35 had
tests ✘
●
Aim for 5% coverage increase in every upcoming release
– Still at the same level as 6 months ago (39.2% vs. 39.6%) ✘
●
Aim for 60 - 70% coverage at the end of 2015
– Barely keeping up with new code, not increasing coverage ✘

12. 12Samsung Open Source Group
Missing: Pre-push CI
● CI infrastructure only available after a push to master
● The various tooling only runs on code committed
to the master branch of our repos
● Testing the code before this stage could be
beneficial
● Testing developer branches
on demand and hooking it
up with Phab to run it on
incoming submissions

13. 13Samsung Open Source Group
Steps Forward: Pre-push CI
●
Two entry points we should cover
●
Developers: Allow dev branches to be run through Jenkins
– Not tackled ✘
●
Contributors: Run incoming Phab submissions through
Jenkins
– Phab plugin installed in Jenkins to handle parts of it ✔
– No actual build job on Jenkins for it ✘
– No feedback of success / failure to differential ✘

14. 14Samsung Open Source Group
Code Path Complexity
●
More valid combinations for test matrix
– New jobs for xcb and wayland have been added
●
What other configuration options should be
officially supported?
– Harfbuzz, systemd, web-p and xinput2.2 have
been mentioned
– Builds jobs with these options enabled are
pending ✘

15. 15Samsung Open Source Group
API's and Breaks
●
Run ABI/API checker every night
– Setup a Jenkins job to run the ABI/API checker on EFL ✔
– Manually run right now ✘
●
Catch breaks
●
List with new additions for
– summary mail to list to highlight additions and maybe raise
discussion if needed. (Shortly after the addition and not weeks later) ✘
– summary of new additions for release NEWS ✘
– notifications for manual bindings to catchup ✘
– check if new APIs have documentation ✘

16. 16Samsung Open Source Group
Summary

17. 17Samsung Open Source Group
Summary
●
Slow progress on all fronts
●
We had various problems with Jenkins which
slowed done the addition of new jobs and Phab
integration
●
Coverage is not increasing at all
– How to motivate this further?
●
Exactness is stuck
– Someone willing to take over?

18. 18Samsung Open Source Group
Thank you.

19. 19Samsung Open Source Group
Attributions
● Brunoschalch:
https://www.flickr.com/photos/bsgphotos/7594903774/in/photostream/
● Katja Schulz: https://www.flickr.com/photos/treegrow/14404553548
● Fernando Flores: https://www.flickr.com/photos/ferjflores/8697860914
● David Brossard: https://www.flickr.com/photos/string_bass_dave/8152835443
● Anne Reeves: https://www.flickr.com/photos/charlock/10677411025
● Andrew: https://www.flickr.com/photos/polandeze/741311758
● Michael Hodge:
https://www.flickr.com/photos/mhodge/1216047199/in/photostream/
– https://www.flickr.com/photos/mhodge/1216909436/
Nick Harris: https://www.flickr.com/photos/nickharris1/5585837979/
● Dawn Houghtaling-Montague:
https://www.flickr.com/photos/dawnsviews/3803444495/
● USGS Bee Inventory and Monitoring Lab: https://www.flickr.com/photos/usgsbiml/11
675053206/