The Semantic Web vision emerged with a promise to collect and interlink semantically relevant data from diverse sources in order to to achieve a full potential of the Web. After more than a decade of diligent research, it is the time to start summing up what has been accomplished and how mature Semantic Web research is, so that plans for the future can be charted. One of the key trails of a mature discipline is to have well-designed research methods allowing researchers to establish evidence about the effectiveness of the research ideas. It is equally important to to have knowledge translation methods that allow for transferring the established evidence to decision makers in practice. In this talk, we will first share some experience and challenges in conducting experiments in the area of the Semantic Web. We will next discuss findings of systematic reviews conducted to estimate the level of quality of the existing research results based on the criteria well-known in medical research and recently adopted in empirical software engineering. We will conclude the talk by discussing the importance and potential milestones for the Semantic Web in order to become an evidence-based discipline (similar to medicine or education) capable of producing strong research evidence transferable to practice.

1. https://semtech.athabascau.ca Evidence-based Semantic WebJust a Dream or the Way to Go? http://goo.gl/SJFaf DraganGašević

3. Semantic Web To create a universal medium for the exchange of data. … to smoothly interconnect personal information management, enterprise application integration and the global sharing of commercial, scientific and cultural data. Semantic Web Activity Statementhttp://www.w3.org/2001/sw/Activity

4. What evidence do we have? If something works or not and why?

5. The rest of the talk Evidence-based Semantic Web Evaluating Semantic Web research knowledge Quality assessment of research evidence What can we do better

6. Part IEvidence-based Semantic Web

7. Evidence-based Semantic Web As the integration of best research evidence with practitioner expertise and stakeholder values The goal made up based on

8. Evidence-based BPM

9. Evidence-based BPM

10. Evidence-based BPM Current best evidence from research to integrate withpractical experience and human valuesin the decision making process

11. Evidence-based BPM

12. Systematic reviews

13. Measures matter!

14. Measures matter! But, how much really?! ~1/3 out of the 19 studies presented empirical results Very few of them report empirical validation as critical Sanchez Gonzalez et al., 2010 BPM Journal 16 (1), pp. 114-134

15. Measures matter! But, how much really?! Not found reported research on interoperability, compliance, security, maturity, learnability, analyzability, and testability Sanchez Gonzalez et al., 2010 BPM Journal 16 (1), pp. 114-134

16. Community Engineering Awareness can only help!

17. Part IIEvaluating Semantic Web Research Knowledge A case study- Semantic Web User Interfaces -

18. Systematic Review General overview of the area "Guidelines for performing Systematic Literature Reviews in Software Engineering,” http://goo.gl/2HqQQ

19. Publication Venues

20. Subjects under study

21. Method

22. Experiments with Users

23. Empirical validation!? Any evidence from user experiments?

24. Is evidence just a dream?

25. Part III Quality Assessment of Research Evidence

26. Objective Systematic quality analysis of user experiments in Semantic Web research

27. Method Assess the quality of each paper with a sound instrument

28. Method T. Greenhalgh, How to Read a Paper, second ed., BMJ Publishing Group, London, 2001. B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, J. Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721–734.

29. Quality Assessment Instrument

30. Method Systematically select the papers from the major venues ISWC, ESWC, ASWC, WWW, JWS, and IEEE Int. Sys.

31. Method Several raters to assign the scores to the papers Inter-rater reliability to be computed

32. Results Descriptive stats Mean = 5.61 SD = 1.32 Max = 9.00 Min = 3.00 Reliability Kappa= 0.73 Agree= 0.82

33. Results

34. Results

35. Systematic reviews

36. Systematic reviews

37. Systematic reviews

38. Systematic reviews

39. Results 0.91/ 0.58 0.24/ 0.64 0.30/ 0.17 0.82/ 0.61 0.76/ 0.64 0.03/ 0.00

40. Is there a way to go?

41. Part IV What Can We Do Better

44. Clarity of variables For example, grounding on standards ISO 9126 standard, Software engineering — Product quality

45. Software Quality L. C. Briand, J. W¨ust, S. V. Ikonomovski, and H. Lounis, “Investigating quality factors in object-oriented designs: an industrial case study,” in ICSE ’99: Proceedings of the 21st International Conference on Software Engineering, 1999, pp. 345–354.

46. Does visualization help?

47. 5 Vis-fmp

48. Perceived is not bad! Sometimes the only instrument

49. LOCO-Analyst LOCO-Analyst for learning analytics

50. Q5 – Sampling Sampling is not just about higher numbers But, how to accomplish valid findings

51. About myths 1 – Higher than 75th percentile (> 20) 19 papers 2 – Between 50-75th percentile (13-20) 22 papers 3 – Between 25-50th percentile (8-12) 15 papers 4 – Lower than 25th percentile (<8) 23 papers

52. Estimate statistical power OntoGen Text2Onto

53. Families of Experiments

55. Effect of visualization: Vis-fmpvsfmp

57. http://www.semanticdoc.org/ Effectiveness of Semantic Documents* * Similar done in Vis-fmp and OntoGen vs. Text2Onto

58. Q8 – Analysis rigor Rigorous analysis To test significance of the effect

59. One size does not fit all Depends on research questions

60. Text2Onto vsOntoGen

61. Open ended-questions Content analysis Statistical tests can be also applied

62. Text2Onto vsOntoGen * χ2 (1, N=27) = 3.89, p=0.049 Easy - easy to use NVE - not very easy to use VP - hard to manipulate the visualization LF - lack of feedback NC - user has no control over the process

63. Effects of visualization Changeability tasks (time) H1: (Easy) t (38) = 2.11, p = 0.041* H1: (Complex) t (38) = 3.47, p = 0.001* Understandability tasks (time) H3: (Easy) t (38) = 1.42, p = 0.164 H4: (Complex) t (38) = 2.71, p = 0.009* No significant effect on correctness

64. Effects of Semantic Docs Time to complete tasks

65. LOCO-Analyst Predictors of the perceived utility information about interactions of students social networking students’ comprehension of content collaborative tagging Multiple regression

66. Q9 - Bias Oh, that bias Remember, mean value was 0.02

67. Randomized Control Trails Schulz KF, Altman DG, Moher D; for the CONSORT Group (2010). "CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials". Br Med J340: c332. doi:10.1136/bmj.c332

68. Bias control is difficult Double blind process as a direction

69. Bias Control Blind allocation to treatment groups, material distribution, marking, analysis, & data collection Systematic subject difference skill, gender, and race by blocking, covariate analysis, or cross-over designs Replicated studies by those who have no vested interest in the outcome e.g., Text2Onto vs. OntoGen Barbara A. Kitchenham, Tore Dybå, Magne Jørgensen: Evidence-Based Software Engineering. ICSE 2004: 273-281

70. Q10 - Findings Semantic Web research has no limitations Yeah, sure!?

71. Threats to validity must be reported Conclusion, construct, internal, and external validity

77. Acknowledgements MarekHatala, EbrahimBagheri, AmalZouaq, JelenaJovanovic, Marko Boskovic, FaezehEnsan, Mohsen Asadi, IvanaOgnjanovic, Jeff Rusk, Luis Rocha, SamanehSoltani, GhislainHachey, Vid Prezel, Tony Lenihan, EsanMurugesupillai, Glenn Brand,…

78. Thank you!Questions? https://semtech.athabascau.ca