Iterative usability evaluation of DSLs

187 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
187
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • {}
  • Iterative usability evaluation of DSLs

    1. 1. Ankica Barišić, PhD student Supervisors: Vasco Amaral, Miguel Goualo
    2. 2. A language is a means of communication  The user interface is a realization of a language A language is a model that describes the allowed terms and how to compose them into valid sentences 2
    3. 3. General Purpose (programming) Languages (GPLs)  User has to master programming concepts  User has to master domain concepts Domain Specific (modeling) Languages (DSLs)  Meant to close gap between PROBLEM DOMAIN and SOLUTION DOMAIN Reduce the use of computation domain concepts Focus on the domain concepts 3
    4. 4. Increasingly popular  Raise the abstraction level (closer to the domain)  Narrow the design space Several benefits claimed, in well-defined domains  Productivity gains  Better time to market  Avoid error-prone mappings between domain and software development concepts  Leverage the expertise of domain experts 4
    5. 5. ? SE U RE [Mernik, 2005] 5
    6. 6.  a model of the   User Different knowledge sets Characteristics chosen are dependent on application domain  the hardware-software   set of computing, sensing, communication, and interaction resources e.g. operating systems, memory size, network bandwidth, input and output interaction devices  the social and physical  Platform Environment Where the interaction is actually taking place Different languages may have different contexts of use  Their users are likely to have different knowledge sets  A minimum set of ontological concepts is required to use the language 6
    7. 7. The user's view of the Quality of a product Measured in terms of the result of using the product, rather than its properties 7
    8. 8. In general, software language engineers DO NOT evaluate their languages with respect to their impact in the software development process in which the DSLs will be integrated Or, if they do, they are extremely shy about it 8
    9. 9. 9
    10. 10. What to measure? 10
    11. 11. [Barisic, 2011c] 11
    12. 12. • Introduce DSLs’ Usability evaluation during DSLs’ life-cycle • • • • iterations Design an effective experimental evaluation of DSLs that will provide qualitative and quantitative feedback for DSLs developers Produce user-centered design of DSL Foresee the Quality of a DSL while in an iterative evolution step Merge the Software Language development process with the Usability Engineering process 12
    13. 13. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 13
    14. 14. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 14
    15. 15. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 15
    16. 16. Language engineer Domain expert Usability engineer 16
    17. 17. 17
    18. 18. per group 18
    19. 19. Language goals: Support users in creating and manipulating Evaluation: Compare two different DSL solutions within same domain sharing the same goal For the purpose of having means to reason about what is more usable: Abstract syntax definition  Transformation  Interaction  19
    20. 20. Development environment and methodology: Epsilon Model-Driven development Domain: Management and organization Transformation output: MS Excel template Place: Summer school on MDD of DSLs, Faculty of Organizational Science's 20
    21. 21. Domain Experts: subjects Usability Experts: experiment designers Language Developers: lecturers, subjects End Users: subjects Lecturers Experiment designers Subjects 2 2 39 21
    22. 22. Instrument: background questionnaire 22
    23. 23. Our evaluation technique was tested with two individuals (two physics experts) in order to verify it and to test the teaching materials and questionnaires As time constrants and equipment turn out to be adequat there was no need to change prepared materials [Howatt 1995] 23
    24. 24. Instrument: background questionnaire RQ1:Is querying with Pheasant more effective than with C+ +/BEE? RQ2:Is querying with Pheasant more efficient than with C+ +/BEE? RQ3:Are participants querying with Pheasant more confident on their performance than with C++/BEE? Our goal is to:  analyze the performance of Pheasant programmers plug-ins  for the purpose of comparing it with a baseline alternative (C+ +/BEE)  with respect to the efficiency, effectiveness and confidence of defying queries in Pheasant  from the point of view of a researcher trying to assess the Pheasant DSL,  in the context of a case study on selected queries 24
    25. 25. H1null Using Pheasant vs. C++/BEE has no impact on the effectiveness of querying the analysis framework H1alt Using Pheasant vs. C++/BEE has a significant impact on the effectiveness of querying the analysis framework H2null Using Pheasant vs. C++/BEE has no impact on the efficiency of querying the analysis framework H2alt Using Pheasant vs. C++/BEE has a significant impact on the efficiency of querying the analysis framework H3null Using Pheasant vs. C++/BEE has no impact on the confidence of querying the analysis framework H3alt Using Pheasant vs. C++/BEE has a significant impact on the confidence of querying the analysis framework 25
    26. 26.  We focus on presenting six examples, each focusing in some of the features we chose to evaluate  Participants are asked to give themselves a mark for feeling of correctness of their trial  Session take the time needed for each group to understand the examples 26
    27. 27. Every participant has four queries, specified in English, to be rewritten in previously learned language Subjects makes selfassessment of his replay rating his feeling of correctness Example:  Build the decay of a D0 particle to a Kaon Pion 27
    28. 28. Results obtained with Pheasant were clearly better then those with C++/BEE Pheasant allowed non-programmers to correctly define their queries. The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using Pheasant The feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative. 28
    29. 29. The participants were asked to judge the intuitiveness, suitability and effectiveness of the query language. The goal was to evaluate: Overall reactions Query language constructs Affect to query language was rated by: Query language constructs Participants’ comments 29
    30. 30. Foster higher productivity of DSL end users Increased quality in use and quality of experience while using a DSL Replication of experiments Reusing data and recommendations Reduce complexity Traceability of usability recommendations and design changes Reduced cost of evaluation, development and support Increased reliability of usability evaluation results 30
    31. 31. 31
    32. 32. 32
    33. 33. Foster higher productivity of DSL end users Increased quality in use and quality of experience while using a DSL Replication of experiments Reusing data and recommendations Reduce complexity Traceability of usability recommendations and design changes Reduced cost of evaluation, development and support Increased reliability of usability evaluation results 33
    34. 34. 34
    35. 35. Literature  [Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to develop domain-specific languages, 2005, ACM Computing Surveys  [Gabriel2010] Gabriel, P., Goulão, M. & Amaral, V. (2010). Do Software Languages Engineers Evaluate their Languages? in XIII Congreso Iberoamericano en "Software Engineering" (CIbSE'2010)  [Barisic2011a] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of DSLs: Current Evaluation Methods’. Proc. 3rd INForum - Simpósio de Informática (INForum2011), Coimbra, Portugal, September 2011  [Barisic2011b] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of Domain Specific Languages: a Case Study’. Proc. Evaluation and Usability of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011  [Barisic2011c] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘How to reach a usable DSL? Moving toward a Systematic Evaluation’, Electronic Communications of the EASST, 2011  [Barisic2012] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Evaluating the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI Global, 2012) 35

    ×