https://semtech.athabascau.ca<br />Evidence-based Semantic WebJust a Dream or the Way to Go?<br />http://goo.gl/SJFaf<br /...
Semantic Web<br />To create a universal medium for the exchange of data. <br />… to smoothly interconnect personal informa...
What evidence do we have? <br /> If something works or not and why?<br />
The rest of the talk<br />Evidence-based Semantic Web<br />Evaluating Semantic Web research knowledge<br />Quality assessm...
Part IEvidence-based Semantic Web<br />
Evidence-based Semantic Web<br /> As the integration of best research evidence with practitioner expertise and stakeholder...
Evidence-based BPM<br />
Evidence-based BPM<br />
Evidence-based BPM<br />Current best evidence from research<br />to integrate withpractical experience and human valuesin ...
Evidence-based BPM<br />
Systematic reviews<br />
Measures matter!<br />
Measures matter!<br />But, how much really?!<br />~1/3 out of the 19 studies presented empirical results<br />Very few of ...
Measures matter!<br />But, how much really?!<br />Not found reported research on interoperability, compliance, security, m...
Community Engineering<br />Awareness can only help!<br />
Part IIEvaluating Semantic Web Research Knowledge<br />A case study- Semantic Web User Interfaces -<br />
Systematic Review<br />General overview of the area<br />"Guidelines for performing Systematic Literature Reviews in Softw...
Publication Venues<br />
Subjects under study<br />
Method<br />
Experiments with Users<br />
Empirical validation!?<br />Any evidence from user experiments?<br />
Is evidence just a dream? <br />
Part III Quality Assessment of Research Evidence<br />
Objective<br />Systematic quality analysis of user experiments in Semantic Web research<br />
Method<br />Assess the quality of each paper with a sound instrument<br />
Method<br />T. Greenhalgh, How to Read a Paper, second ed., BMJ Publishing Group, London, 2001.<br />B.A. Kitchenham, S.L....
Quality Assessment Instrument<br />
Method<br />Systematically select the papers from <br />the major venues<br />ISWC, ESWC, ASWC, WWW, JWS, and IEEE Int. Sy...
Method<br />Several raters to assign the scores to the papers<br />Inter-rater reliability to be computed<br />
Results<br />Descriptive stats<br />Mean =	5.61<br />SD =	1.32<br />Max =	9.00<br />Min =	3.00<br />Reliability<br />Kappa...
Results<br />
Results<br />
Systematic reviews<br />
Systematic reviews<br />
Systematic reviews<br />
Systematic reviews<br />
Results<br />0.91/<br />0.58<br />0.24/<br />0.64<br />0.30/<br />0.17<br />0.82/<br />0.61<br />0.76/<br />0.64<br />0.03...
Is there a way to go? <br />
Part IV What Can We Do Better<br />
Q4 - Methods<br />Methods not needed!<br />Yeah, right!?<br /><ul><li>Weak research questions
Unclear variables and measures of interest</li></li></ul><li>Questionnaire is not the only instrument!<br />
Clarity of variables<br />For example, grounding on standards<br />ISO 9126 standard, Software engineering — Product quali...
Software Quality<br />L. C. Briand, J. W¨ust, S. V. Ikonomovski, and H. Lounis, “Investigating quality factors in object-o...
Does visualization help? <br />
5<br />Vis-fmp<br />
Perceived is not bad!<br />Sometimes the only instrument<br />
LOCO-Analyst<br />LOCO-Analyst for learning analytics<br />
Q5 – Sampling<br />Sampling is not just about higher numbers <br />But, how to accomplish valid findings<br />
About myths<br />1 – Higher than 75th percentile (> 20)<br />19 papers<br />2 – Between 50-75th percentile (13-20)<br />22...
Estimate statistical power <br />OntoGen<br />Text2Onto<br />
Families of Experiments <br />
Q6 – Control groups<br />Control groups <br />Useful to test significance of some effect<br /><ul><li>Onto Learning: Text2...
Effect of visualization: Vis-fmpvsfmp
LOCO-Analyst – roles of participants</li></li></ul><li>Repeated measures are not a sin<br />If they are done right<br />
http://www.semanticdoc.org/<br />Effectiveness of Semantic Documents*<br />* Similar done in Vis-fmp and OntoGen vs. Text2...
Q8 – Analysis rigor<br />Rigorous analysis <br />To test significance of the effect<br />
One size does not fit all<br />Depends on research questions<br />
Text2Onto vsOntoGen<br />
Open ended-questions<br />Content analysis <br />Statistical tests can be also applied<br />
Text2Onto vsOntoGen<br />* χ2 (1, N=27) = 3.89, p=0.049<br />Easy -	easy to use<br />NVE - 	not very easy to use<br />VP -...
Effects of visualization<br />Changeability tasks (time)<br />H1: (Easy)		 t (38) = 2.11, p = 0.041*<br />H1: (Complex)	 t...
Effects of Semantic Docs<br />Time to complete tasks<br />
LOCO-Analyst<br />Predictors of the perceived utility<br />information about interactions of students<br />social networki...
Q9 - Bias<br />Oh, that bias <br />Remember, mean value was 0.02<br />
Randomized Control Trails <br />Schulz KF, Altman DG, Moher D; for the CONSORT Group (2010). "CONSORT 2010 Statement: upda...
Bias control is difficult<br />Double blind process as a direction<br />
Bias Control<br />Blind<br />allocation to treatment groups, material distribution, marking, analysis, & data collection<b...
Q10 - Findings<br />Semantic Web research has no limitations<br />Yeah, sure!?<br />
Threats to validity must be reported<br />Conclusion, construct, internal, and external validity<br />
Upcoming SlideShare
Loading in …5
×

Evidence-based Semantic Web Just a Dream or the Way to Go?

1,354 views
1,291 views

Published on

The Semantic Web vision emerged with a promise to collect and interlink semantically relevant data from diverse sources in order to to achieve a full potential of the Web. After more than a decade of diligent research, it is the time to start summing up what has been accomplished and how mature Semantic Web research is, so that plans for the future can be charted. One of the key trails of a mature discipline is to have well-designed research methods allowing researchers to establish evidence about the effectiveness of the research ideas. It is equally important to to have knowledge translation methods that allow for transferring the established evidence to decision makers in practice. In this talk, we will first share some experience and challenges in conducting experiments in the area of the Semantic Web. We will next discuss findings of systematic reviews conducted to estimate the level of quality of the existing research results based on the criteria well-known in medical research and recently adopted in empirical software engineering. We will conclude the talk by discussing the importance and potential milestones for the Semantic Web in order to become an evidence-based discipline (similar to medicine or education) capable of producing strong research evidence transferable to practice.

1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
1,354
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Evidence-based Semantic Web Just a Dream or the Way to Go?

  1. 1. https://semtech.athabascau.ca<br />Evidence-based Semantic WebJust a Dream or the Way to Go?<br />http://goo.gl/SJFaf<br />DraganGašević<br />
  2. 2.
  3. 3. Semantic Web<br />To create a universal medium for the exchange of data. <br />… to smoothly interconnect personal information management, enterprise application integration and the global sharing of commercial, scientific and cultural data.<br /> Semantic Web Activity Statementhttp://www.w3.org/2001/sw/Activity<br />
  4. 4. What evidence do we have? <br /> If something works or not and why?<br />
  5. 5. The rest of the talk<br />Evidence-based Semantic Web<br />Evaluating Semantic Web research knowledge<br />Quality assessment of research evidence<br />What can we do better<br />
  6. 6. Part IEvidence-based Semantic Web<br />
  7. 7. Evidence-based Semantic Web<br /> As the integration of best research evidence with practitioner expertise and stakeholder values<br />The goal made up based on <br />
  8. 8. Evidence-based BPM<br />
  9. 9. Evidence-based BPM<br />
  10. 10. Evidence-based BPM<br />Current best evidence from research<br />to integrate withpractical experience and human valuesin the decision making process<br />
  11. 11. Evidence-based BPM<br />
  12. 12. Systematic reviews<br />
  13. 13. Measures matter!<br />
  14. 14. Measures matter!<br />But, how much really?!<br />~1/3 out of the 19 studies presented empirical results<br />Very few of them report empirical validation as critical <br />Sanchez Gonzalez et al., 2010 <br />BPM Journal 16 (1), <br />pp. 114-134<br />
  15. 15. Measures matter!<br />But, how much really?!<br />Not found reported research on interoperability, compliance, security, maturity, learnability, analyzability, and testability<br />Sanchez Gonzalez et al., 2010 <br />BPM Journal 16 (1), <br />pp. 114-134<br />
  16. 16. Community Engineering<br />Awareness can only help!<br />
  17. 17. Part IIEvaluating Semantic Web Research Knowledge<br />A case study- Semantic Web User Interfaces -<br />
  18. 18. Systematic Review<br />General overview of the area<br />"Guidelines for performing Systematic Literature Reviews in Software Engineering,” http://goo.gl/2HqQQ<br />
  19. 19. Publication Venues<br />
  20. 20. Subjects under study<br />
  21. 21. Method<br />
  22. 22. Experiments with Users<br />
  23. 23. Empirical validation!?<br />Any evidence from user experiments?<br />
  24. 24. Is evidence just a dream? <br />
  25. 25. Part III Quality Assessment of Research Evidence<br />
  26. 26. Objective<br />Systematic quality analysis of user experiments in Semantic Web research<br />
  27. 27. Method<br />Assess the quality of each paper with a sound instrument<br />
  28. 28. Method<br />T. Greenhalgh, How to Read a Paper, second ed., BMJ Publishing Group, London, 2001.<br />B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, J. Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721–734.<br />
  29. 29. Quality Assessment Instrument<br />
  30. 30. Method<br />Systematically select the papers from <br />the major venues<br />ISWC, ESWC, ASWC, WWW, JWS, and IEEE Int. Sys.<br />
  31. 31. Method<br />Several raters to assign the scores to the papers<br />Inter-rater reliability to be computed<br />
  32. 32. Results<br />Descriptive stats<br />Mean = 5.61<br />SD = 1.32<br />Max = 9.00<br />Min = 3.00<br />Reliability<br />Kappa= 0.73<br />Agree= 0.82<br />
  33. 33. Results<br />
  34. 34. Results<br />
  35. 35. Systematic reviews<br />
  36. 36. Systematic reviews<br />
  37. 37. Systematic reviews<br />
  38. 38. Systematic reviews<br />
  39. 39. Results<br />0.91/<br />0.58<br />0.24/<br />0.64<br />0.30/<br />0.17<br />0.82/<br />0.61<br />0.76/<br />0.64<br />0.03/<br />0.00<br />
  40. 40. Is there a way to go? <br />
  41. 41. Part IV What Can We Do Better<br />
  42. 42. Q4 - Methods<br />Methods not needed!<br />Yeah, right!?<br /><ul><li>Weak research questions
  43. 43. Unclear variables and measures of interest</li></li></ul><li>Questionnaire is not the only instrument!<br />
  44. 44. Clarity of variables<br />For example, grounding on standards<br />ISO 9126 standard, Software engineering — Product quality<br />
  45. 45. Software Quality<br />L. C. Briand, J. W¨ust, S. V. Ikonomovski, and H. Lounis, “Investigating quality factors in object-oriented designs: an industrial case study,” in ICSE ’99: Proceedings of the 21st International Conference on Software Engineering, 1999, pp. 345–354.<br />
  46. 46. Does visualization help? <br />
  47. 47. 5<br />Vis-fmp<br />
  48. 48. Perceived is not bad!<br />Sometimes the only instrument<br />
  49. 49. LOCO-Analyst<br />LOCO-Analyst for learning analytics<br />
  50. 50. Q5 – Sampling<br />Sampling is not just about higher numbers <br />But, how to accomplish valid findings<br />
  51. 51. About myths<br />1 – Higher than 75th percentile (> 20)<br />19 papers<br />2 – Between 50-75th percentile (13-20)<br />22 papers<br />3 – Between 25-50th percentile (8-12)<br />15 papers<br />4 – Lower than 25th percentile (<8)<br />23 papers<br />
  52. 52. Estimate statistical power <br />OntoGen<br />Text2Onto<br />
  53. 53. Families of Experiments <br />
  54. 54. Q6 – Control groups<br />Control groups <br />Useful to test significance of some effect<br /><ul><li>Onto Learning: Text2Onto vs. OntoGen or IT vsnonIT
  55. 55. Effect of visualization: Vis-fmpvsfmp
  56. 56. LOCO-Analyst – roles of participants</li></li></ul><li>Repeated measures are not a sin<br />If they are done right<br />
  57. 57. http://www.semanticdoc.org/<br />Effectiveness of Semantic Documents*<br />* Similar done in Vis-fmp and OntoGen vs. Text2Onto<br />
  58. 58. Q8 – Analysis rigor<br />Rigorous analysis <br />To test significance of the effect<br />
  59. 59. One size does not fit all<br />Depends on research questions<br />
  60. 60. Text2Onto vsOntoGen<br />
  61. 61. Open ended-questions<br />Content analysis <br />Statistical tests can be also applied<br />
  62. 62. Text2Onto vsOntoGen<br />* χ2 (1, N=27) = 3.89, p=0.049<br />Easy - easy to use<br />NVE - not very easy to use<br />VP - hard to manipulate the visualization<br />LF - lack of feedback<br />NC - user has no control over the process<br />
  63. 63. Effects of visualization<br />Changeability tasks (time)<br />H1: (Easy) t (38) = 2.11, p = 0.041*<br />H1: (Complex) t (38) = 3.47, p = 0.001*<br />Understandability tasks (time)<br />H3: (Easy) t (38) = 1.42, p = 0.164<br />H4: (Complex) t (38) = 2.71, p = 0.009*<br />No significant effect on correctness<br />
  64. 64. Effects of Semantic Docs<br />Time to complete tasks<br />
  65. 65. LOCO-Analyst<br />Predictors of the perceived utility<br />information about interactions of students<br />social networking <br />students’ comprehension of content<br />collaborative tagging<br />Multiple regression<br />
  66. 66. Q9 - Bias<br />Oh, that bias <br />Remember, mean value was 0.02<br />
  67. 67. Randomized Control Trails <br />Schulz KF, Altman DG, Moher D; for the CONSORT Group (2010). "CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials". Br Med J340: c332. doi:10.1136/bmj.c332<br />
  68. 68. Bias control is difficult<br />Double blind process as a direction<br />
  69. 69. Bias Control<br />Blind<br />allocation to treatment groups, material distribution, marking, analysis, & data collection<br />Systematic subject difference<br />skill, gender, and race by blocking, covariate analysis, or cross-over designs<br />Replicated studies<br />by those who have no vested interest in the outcome<br />e.g., Text2Onto vs. OntoGen<br />Barbara A. Kitchenham, Tore Dybå, Magne Jørgensen: Evidence-Based Software Engineering. ICSE 2004: 273-281<br />
  70. 70. Q10 - Findings<br />Semantic Web research has no limitations<br />Yeah, sure!?<br />
  71. 71. Threats to validity must be reported<br />Conclusion, construct, internal, and external validity<br />
  72. 72.
  73. 73.
  74. 74.
  75. 75.
  76. 76.
  77. 77. Acknowledgements<br />MarekHatala, EbrahimBagheri, AmalZouaq, JelenaJovanovic, Marko Boskovic, FaezehEnsan, Mohsen Asadi, IvanaOgnjanovic, Jeff Rusk, Luis Rocha, SamanehSoltani, GhislainHachey, Vid Prezel, Tony Lenihan, EsanMurugesupillai, Glenn Brand,…<br />
  78. 78. Thank you!Questions? <br />https://semtech.athabascau.ca<br />

×