ER 2019 tutorial - Data Driven RE
•
2 likes•571 views
Tutorial given at ER 2019. It provides a state of the art in data-driven requirements engineering dealing with topics like feedback management, decision-making and others
More Related Content
Similar to ER 2019 tutorial - Data Driven RE
Similar to ER 2019 tutorial - Data Driven RE (20)
More from Xavier Franch
More from Xavier Franch (12)
Recently uploaded
Recently uploaded (20)
ER 2019 tutorial - Data Driven RE
- 1. Data‐Driven Requirements Engineering Xavier Franch Universitat Politècnica de Catalunya (UPC‐BarcelonaTech) Tutorial at ER’19, November 2019 Salvador, Brasil
- 6. 6 ER 2019Xavier Franch How to ensure that the system is delivering the right value to stakeholders? ...business value being one of them “80% of products lose money due to wrongly set product objectives that result in building products that customers do not need” vs. Several reasons for this...
- 11. 11 Research areas • Explicit Feedback • Implicit Feedback • Repository mining • Decision‐making • Processes ER 2019Xavier Franch Combination
- 14. 14 ER 2019Xavier Franch Behind the curtains: NLP & ML • NLP: processing textual information • ML: every context is different
- 15. 15 ER 2019Xavier Franch • From text to lexical/syntactical units Tokenization • Splitting input into parts Stemming / Lemmatization • Lemmatization more accurate Phrasing • Part‐of‐speech tagging “I have a problem when saving the document, please check it” I/PRP have/VBP a/DT problem/NN when/WRB saving/VBG the/DT document/NN ,/, please/VBP check/VB it/PRP Speech‐art: Requestive Preprocessing
- 16. 16 ER 2019Xavier Franch Feedback gathering Several topics • Communication style Push vs. pull • Mode From linguistic to multi‐modal • Channel App stores, forums, social media, … • Advanced features E.g., feedback to feedback
- 17. 17 Feedback analysis • Categorization • Sentiment analysis • Topic modeling
- 18. 18 ER 2019Xavier Franch Categorization Typical: • Bug report • Feature request Sometimes difficult to distinguish • “It’s not a bug, it’s a feature” Also: comments, general ideas, just praise, ...
- 20. 20 ER 2019Xavier Franch Process of assigning a quantitative value to a piece of text expressing an affect or mood Use of dictionaries: But of course, not easy...: “Great, I love this new feature that gives me this wonderful headache” Models using advanced ML techniques Sentiment analysis
- 21. 21 ER 2019Xavier Franch Instability problems (“order effects”) Fails to capture rich topical correlations • Identifying topics that best describes a corpus Usually, latent (emerge during the process) • Each corpus described by a distribution of topics Each topic described by a distribution of words • Most popular algorithm: Latent Dirichlet Allocation Topic modeling
- 26. 26 Implicit feedback • Getting feedback from the user without her involvement • Two main instruments Monitoring infrastructure QoS Log files system usage May be (usually are) inter‐related
- 30. 30 ER 2019Xavier Franch (context, behavior) Drowsy driver Ring alarm Satisfaction: given cr = (ctx, beh) eval(cr) = ctx dom(Oper) evaluable(ctx) ctx beh Objective function: minimize {cri: 0 < i ≤ m: NOT EvalCR(cri)} Contextual requirements
- 32. 32 ER 2019Xavier Franch Usage logs • Information about usage of the system • What can be discovered? Which functionalities are most used Which navigational paths prevail (or not) Which calls results often in error codes … Especially if QoS also available
- 33. 33 COMBINING IMPLICIT AND EXPLICIT FEEDBACK
- 39. 39 ER 2019Xavier Franch Mining repositories for quality
- 41. 41 ER 2019Xavier Franch Raw metric Size metric Normalized metric Attribute + Data source Business value Data availability Raw metric Data source Code complexity SonarQube Function’s cyclomatic complexity Number of functions Average cyclomatic complexity GitHub Nb. commits 1. Definition of metrics
- 42. 42 ER 2019Xavier Franch Norm. metric Attribute Norm. metric Thresholds coming from benchmarking Determination of metrics’ impact onto attributes Use of (linear) utility functions o Utility: strength of preference a decision‐maker has among alternatives regarding a specific quality aspect 2. Evaluation of attributes
- 43. 43 ER 2019Xavier Franch 3. Aggregation of factors Synthesis of individual utilities into global one Weights for contributing factors Attribute Attribute Attribute +w1 w2 w3 ... Weights coming from existing data or expert opinion Code complexity Coupling degree Code modifiability
- 44. 44 ER 2019Xavier Franch Translation of utility values into a QA value Interpretation model transforming from ratio to ordinal Could be done at the level of general quality, a characteristic or a subcharacteristic E.g., analogy with school grade Attribute Subcharac‐ teristic Attribute v1 v2 Code modifiability API stability Modifiability 4. Interpretation
- 46. 46 ER 2019Xavier Franch Keys to decision making • Having at hand all information to make informed decisions Analytics tools • Allowing all stakeholders to have a part in the process Prioritization
- 47. 47 ER 2019Xavier Franch Code analytic tools Good, but not strategic enough
- 52. 52 ER 2019Xavier Franch • Stakeholders can evaluate requirements based on different interest dimensions Most stakeholders might not be appropiate to evaluate all requirements accurately Stakeholders can delegate votes to experts selectively Liquid democracy
- 53. 53 PROCESSES
- 54. 54 Data‐driven RE in context • How to connect data‐driven RE to the software process ER 2019Xavier Franch How can practitioners use this information and integrate it into their processes and tools to decide about what should be done
- 55. 55 ER 2019Xavier Franch The data‐driven cycle
- 56. 56 ER 2019Xavier Franch Example: The Q‐Rapids Approach
- 57. 57 Challenges to adoption • Organizational Tailoring to the company Integration with company WoW Shared vocabulary • Value Informativeness Transparency • Technological Simplify tool installation Efficienty tool configuration ER 2019Xavier Franch
- 58. 58 Lessons learned • Organizational Incremental adoption Monitor progress with strategic indicators Involve experts • Value Transparency as a business value Tailoring to different scopes • Technological Single access point to software quality related data ER 2019Xavier Franch
- 59. 59 Online Controlled Experimentation • Collecting data from users based in two competing versions wrt to some difference D • Difference in metrics due to D (modulo statistical significance) • Intensively used by big players as Google, Microsoft, etc. Not so easy for smaller companies ER 2019Xavier Franch
- 60. 60 Experimentation model growthER 2019Xavier Franch Experimen‐ tation pervasiveness Experimentation platform capability Technical focus Feature team self‐efficiency Experimentation team organization Overall Evaluation Criteria Experimen‐ tation Impact Don’t read me
- 61. 61 Experimentation model growthER 2019Xavier Franch Level Example: Technical Focus Crawl Collection Contextual logging Walk Standardization Set of metrics Run Comprehensiveness Data pipeline. Automated analysis Fly Consolidation Standardized process
- 62. 62 Lessons learned • Data quality fosters trustworthiness • Experimentation platform accelerates experimentation growth • Qualitative insights enrich experiment learnings • Iterating OCEs with small changes yields large ROIs • The impact of experimentation is boundless • Involve different stakeholders (business, dev, customers, …) • Internalize OCE values and principles (fast learning, experimental mindset, …) • Assign resources enough and establish necessary competencies ER 2019Xavier Franch
- 63. 63 CONCLUSIONS
- 65. 65 ER 2019Xavier Franch Data‐driven RE • Offers a great opportunity for delivering more business value to systems’ stakeholders • But… Not a hammer for every nail Data‐driven needs data • Still traditional methods at least to start with • The role of traditional RE in the loop is a matter of debate
- 66. 66 ER 2019Xavier Franch Beyond RE • Data is prevalent in many fields… … and also in conceptual modeling • We had a couple of examples yesterday…
- 70. Data‐Driven Requirements Engineering Xavier Franch Universitat Politècnica de Catalunya (UPC‐BarcelonaTech) Tutorial at ER’19, November 2019 Salvador, Brasil