The document discusses the current and missing parts of Samsung's open source QA process. It covers their use of compile-time testing, unit testing, and static analysis currently. Areas missing include runtime testing, graphical unit tests, code coverage, and pre-push continuous integration. The presentation proposes steps like making tests mandatory for new APIs, increasing test coverage 5-10% per release, improving graphical testing, and adding pre-push continuous integration through Jenkins and Phabricator hooks. It ends with questions about these proposals and announcements of job opportunities.

1. 1Samsung Open Source Group
Stefan Schmidt
Samsung Open Source Group
s.schmidt@samsung.com
QA: Where We Are And Where We
Should Go
EFL Developer Day US 2015

2. 2Samsung Open Source Group
Agenda
●
Current QA Summary
●
Missing QA Parts
●
Steps Forward
●
Q&A Topics

3. 3Samsung Open Source Group
Current QA Summary

4. 4Samsung Open Source Group
Current: Compile Time Testing
●
Developers machine (One arch, make,
hopefully make check and distcheck)
●
Machines of other developers
●
Jenkins covers different architectures,
different compilers, different
dependencies and versions

5. 5Samsung Open Source Group
Current: Unit Testing
●
Test cases collected in test suite and
run with make check
●
Should be run before committing
●
Also run by Jenkins
●
While unit testing is the main idea it also
had catched segfaults due to basic run
time testing of new code

6. 6Samsung Open Source Group
Current: Miscellaneous
●
Static Analysis
●
Different EFL profiles
●
Different configuration options
(Wayland, XCB)
●
Different make targets (all, check,
distcheck, doc)

7. 7Samsung Open Source Group
Missing QA Parts

8. 8Samsung Open Source Group
Missing: Runtime Testing
●
We rely on the developer testing his code
before committing
●
We rely on the community of developers
and users to run the latest code and
report bugs
●
No automation

9. 9Samsung Open Source Group
Missing: Graphical Unit Tests
●
No usage of graphical test cases
●
Tooling is there: Excatness
●
Almost no developer uses it, regularly
●
Automation is highly needed

10. 10Samsung Open Source Group
Missing: Code Coverage
●
Only 39.2% of our code base covered by u
nit tests
●
Newly added API's should be
mandantory to come with unit tests
●
My goal would be that we increase coverag
e 5-10% per release
●
How to motivate unit test writing?

11. 11Samsung Open Source Group
Missing: Pre-push CI
●
CI infrastructure only available post-push to master
●
The various tooling only runs on code committed
to the master branch of our repos
●
Testing the code before this stage could be
beneficial
●
Testing developer branches on demand and
hooking it up with Phab to run it on incoming
submissions

12. 12Samsung Open Source Group
Missing: Code Complexity
●
More valid combinations for test matrix
– New jobs for xcb and wayland have
just been added
●
What other configuration options should
be officially supported?

13. 13Samsung Open Source Group
Steps Forward

14. 14Samsung Open Source Group
Steps Forward: Unit Testing
●
Proposal: Make test cases for public API
additions (EAPI) mandatory from today
●
Aim for 5% coverage increase in every
upcoming release
●
Aim for 60 - 70% coverage at the end of
2015
●
Not only NULL checks, but valid test cases

15. 15Samsung Open Source Group
Steps Forward: Graphical Testing
●
Identify the problems with exactness on
Jenkins
●
Find a way to fix or work around them
●
Run exactness during the nightlies
●
So far screenshots for elementary_tests
and some edje examples.
●
What else should be tested?

16. 16Samsung Open Source Group
Steps Forward: API's and Breaks
●
Run ABI/API checker every night
●
Catch breaks
●
Catch new additions
– Summary mail to list to highlight additions and maybe
raise discussion if needed. (Shortly after the addition a
nd not weeks later)
– Summary of new additions for release NEWS
– Notifications for manual bindings to catchup
– Check if new APIs have documentation

17. 17Samsung Open Source Group
Steps Forward: Pre-push CI
●
Two entry points we should cover:
– Developers: Allow dev branches to be run
through Jenkins
– Contributors: Run incoming phab submissions
through Jenkins and mark them as
passed/failed
●
Both could have the same Jenkins jobs at the
backend

18. 18Samsung Open Source Group
Q&A Topics

19. 19Samsung Open Source Group
Q&A Topics
●
Mandatory unit tests for new public APIs?
●
5% coverage increase per release a sane goal?
Who would help with this work?
●
How to motivate the writing of unit tests?
●
Anyone willing to drive the exactness integration
work until the end?
●
Is there interest for build hooks for phab
submissions?

20. 20Samsung Open Source Group
Thank you.

21. 21Samsung Open Source Group
We are hiring.
jobs@osg.samsung.com

22. 22Samsung Open Source Group
Attributions
● Brunoschalch:
https://www.flickr.com/photos/bsgphotos/7594903774/in/ph
otostream/
● Katja Schulz:
https://www.flickr.com/photos/treegrow/14404553548
● Fernando Flores: https://www.flickr.com/photos/ferjflores/86
97860914
● David Brossard: https://www.flickr.com/photos/string_bass_d
ave/8152835443
● Anne Reeves:
https://www.flickr.com/photos/charlock/10677411025
● Andrew: https://www.flickr.com/photos/polandeze/7413117
58