Stefan Schmidt, from Samsung Open Source Group's EFL development team, discusses current Enlightenment for Linux Quality Assurance Statistics at the recent EFL Developers Day in Dusseldorf, Germany.

1. EFL QA Statistics
EFL Developer Day 2014
Samsung Open Source Group 1
Stefan Schmidt
Samsung Open Source Group
s.schmidt@samsung.com

2. Agenda
● Introduction
● Jenkins
● Coverity
● Clang Static Analyzer
● Unit Tests
● Phab Review
● Phab Bugs
● Miscellaneous
● Future
Samsung Open Source Group 2

3. Introduction
● Recording QA stats since 2013-07-15
● 15 months full of numbers might give
some insights
Samsung Open Source Group 3

4. Jenkins
● Almost all our QA tools are triggered from Jenkins
● Jobs are running on every repository change and
nightly
● Failure rates vary a lot
Samsung Open Source Group 4

5. Coverity
● Issue count got down lately due to Raster's Coverity addiction
● Elementary, Emotion generic players and Terminology already at 0
● Enlightenment, EFL and Evas generic loaders are not far away
● We should be able to bring them down to 0 for 1.12 or 1.13
● This will allow us to react only on newly created issues
Samsung Open Source Group 5

6. Clang Static Analyzer
● Nobody really takes a look at these
● Reported issues are harder to understand compared to Coverity
● No way to mark something as false positive in a web UI.
● I expected way more clang issues to vanish when Coverity issues got down
– Maybe that means clang reports still have some value for us
● Problems with understanding our hash_add, list_add function (leak reports)
Samsung Open Source Group 6

7. Unit Tests
● Less than 40% of the code base covered (with 400 tests)
● Biggest problem from a QA perspective
● What should we do to improve this situation?
– Man power to write tests covering the existing code
– Make test cases for new API's mandatory
– Usage of Exactness for UI testing
– Regression tests for fixed bugs?
Samsung Open Source Group 7

8. Phab Review
● Only 60 open from 1527 review requests
● Do not rely on the web UI only for review
– Always have a look at the output of git show before pushing
– Author not setup but phab ID shows instead
– White space changes not shown depending on UI setting (default should be fine now)
● Someone motivated to hook Phab up with jenkins?
– We could run new patches through jenkins and add feedback
Samsung Open Source Group 8

9. Phab Bugs
● 396 open bugs from 1722 reports
● 179 of them are whishlist items
Samsung Open Source Group 9

10. Miscellaneous
● Is the weekly report mail still interesting?
● Change something?
● More statistics?
– Would need a volunteer to write a script fetching
data from the web services
Samsung Open Source Group 10

11. Future
Potential improvements:
– Integrate ABI checker with jenkins to find problems earlier
– Try out more -fsanitize options from gcc and clang
– Volunteer for setting up qemu ARM builds?
– Volunteer for setting up expedite benchmarking on E3?
– Volunteer to finish exactness setup on jenkins?
– More testing on platforms not used by main devs (*BSD, Windows, Mac OSX)
QA work seems not to attract many people in the community.
Companies could help on this to reduce their internal QA
costs.
Samsung Open Source Group 11

12. Thank you.
Samsung Open Source Group 12

13. We are hiring.
Contact me if you want to hear details.
s.schmidt@samsung.com
Samsung Open Source Group 13