Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Old wine in new bottles? Is it the case for how we design an ddata collection and analysis strategies?

3 views

Published on

15th International Advanced School on Empirical Software Engineering (IASESE 2018)

Published in: Software
  • Be the first to comment

  • Be the first to like this

Old wine in new bottles? Is it the case for how we design an ddata collection and analysis strategies?

  1. 1. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 1 Is it the case for how we design and implement data collection and analysis strategies? Presented by: Andreas Jedlitschka (Fraunhofer IESE, Kaiserslautern, Germany) Brought to you by: Old win in new bottles? © Fraunhofer IESE Introduction Motivation Conceptualize innovation Evaluate innovation Example – Q-Rapids Conclusions 2
  2. 2. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 2 © Fraunhofer IESE +20 years of experience with measurement, analysis and prediction October 22, 2018 3 1 Characterize 2 Set Goals 3 Choose Process 5 Analyze 6 Package 4 Execute 1 Characterize 2 Set Goals 3 Choose Process 5 Analyze 6 Package 4 Execute © Fraunhofer IESE Motivation Creation of software-based products and services 4
  3. 3. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 3 © Fraunhofer IESE Motivation Conceptualize innovation Evaluate innovation Example – Q-Rapids Conclusions 5 © Fraunhofer IESE Data-driven vs. Business-driven 6 Business-driven Big Data Which data & tools would we need to gain business-relevant knowledge? Data-driven Big Data What business-relevant knowledge can we gain using existing data and tools? Business Case Big Data Big Data Solution Innovation Idea Empirical approach Data-X approach
  4. 4. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 4 © Fraunhofer IESE (Business Solution) Evaluation Cycle 7 IDEASIDEAS PROTOTYPEPROTOTYPEFEEDBACKFEEDBACK DESIGNDESIGN TESTTEST IMPROVEIMPROVE Explore and improve business solution ideas  Ideate and design business solutions  Gather, cluster, prioritize hypotheses  Plan evaluation to test hypotheses  Actions before opinions  Execute tests and collect evidence  Understand and improve solution ideas © Fraunhofer IESE Understand the business (Context) Example: Shoe shop 8 Domain & Organization  Retail of brand shoes in medium price range segment  Traditional family enterprise with over 30 years of experience  Nationwide network of stores  Online shop for 5 years  Monthly newsletter Data Assets & Analytics Expertise  Integrated data warehouse for 3 years (customer, product, and transaction data)  Available data are not used beyond classical reporting Scope of Potential Analysis  Online sales stagnate despite general boom in online business  Online store business unit Shoe Shop Management Board OnlineStores … Sales IT … System Admin DBMS Manager Analysis & BI Marketing Customer service Web service Inventory & logistics Online IT DBMS Manager Analysis & BI Marketing Customer Service Web Service Inventory & Logistics
  5. 5. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 5 © Fraunhofer IESE Customer Segments Key Resources Revenue Streams Cost Structure Key Partners Value Propositions Channels Key Activities Customer Relationships Current Situation Shoe shop: Current business model for online shopping 9 Source: A. Osterwalder, Y. Pigneur, “Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers.” John Wiley and Sons, 2010. Mass market Shoes Web developers BI & Reporting Data warehouse Online shop Sales margin Web dev. costs Website Newsletter, Vouchers Shoe producers Shoe wholesalers Inventory, Shipment Shipment & return cost Sale-in cost © Fraunhofer IESE ! Business Needs Shoe shop: Customers’ current jobs to do for online shopping 10 Find shoe High return costs Order shoe Pay for order Return shoe Fast delivery Poor usability of store Great user experience Online shop Web development Inventory managmt. Sell-in, Sell- out Shipping & returns High inventory costs Many shoe providers Large #customers CustomersShoe Shop Easy payment Large selection Return time & cost Overnight delivery Multiple payment options Large selection Fully automated returns Charge-free returns Source: A. Osterwalder, Y. Pigneur, “Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers.” John Wiley and Sons, 2010.
  6. 6. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 6 © Fraunhofer IESE Customer Segments Key Resources Revenue Streams Cost Structure Key Partners Value Propositions Channels Key Activities Customer Relationships Business Needs Shoe shop: Deficits of current business model 11 Wrong size information High return costs Unsatisfied customers Shoes Sales  Revenue  Data warehouse 60% Returns 1,200 k€ annually Negative InformativePositive Source: A. Osterwalder, Y. Pigneur, “Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers.” John Wiley and Sons, 2010. © Fraunhofer IESE Solution Options Shoe shop: Business innovation and improvement ideas 12 Key Partners Key Activities Customer Segments Value Propositions Cost Structure Revenue Streams Key Resources Customer Relationships Channels Reduced return costs Size recom- mendation Satisfied customers Sales  Revenue  Right size right away Sale & return data Data analytics Source: A. Osterwalder, Y. Pigneur, “Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers.” John Wiley and Sons, 2010.
  7. 7. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 7 © Fraunhofer IESE Aspects of the Solution Idea to Evaluate Test the hypotheses on which the success of the business solution is based 13 Is your business solution idea viable?  Desirable: Do customers really need the solution?  Interest and relevance  Willingness and ability to pay  Priorities and preferences  Can the model survive in a changing environment?  Profitable: Is the underlying cost-revenue structure sound?  What revenues does it expect to generate?  Can the model sustain profits?  Feasible: Is it possible to implement the solution?  Cost, time, technology, regulations, etc.  Can the model scale up? © Fraunhofer IESE Motivation Conceptualize innovation Evaluate innovation Example – Q-Rapids Conclusions 14
  8. 8. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 8 © Fraunhofer IESE How to evaluate it? Experiments Case studies Interviews Observations Survey Document analysis Focus groups … 15 © Fraunhofer IESE Desirability of the Business Model Does the business model address relevant needs? 16 Process EXAMPLE: Shoe ShopPROCESS Prioritize Hypotheses Consider criticality and testing cost Prioritize Hypotheses Consider criticality and testing cost Prepare Test Select test type. Plan test (who, what, when, how). Prepare Test Select test type. Plan test (who, what, when, how). Analyze Results Analyze evidence. Prove hypotheses. Analyze Results Analyze evidence. Prove hypotheses. Execute Test Execute plan. Collect evidence. Execute Test Execute plan. Collect evidence. Adjust Idea & Iterate Improve solution idea. Test new hypotheses. Adjust Idea & Iterate Improve solution idea. Test new hypotheses. Hypothesis: Shoe return is a relevant pain. Customers will use a recommendation system. survey among customers. Test: Survey Plan: Design and implement an online survey among customers. Analyze return frequency, experiences with other recommendation systems. Distribute and collect online survey. Distribute vouchers among participants. Optimize recommendation system idea and associated business model.
  9. 9. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 9 © Fraunhofer IESE AB Testing 17 © Fraunhofer IESE Motivation Conceptualize innovation Evaluate innovation Example – Q-Rapids Conclusions 18
  10. 10. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 10 © Fraunhofer IESE CONTEXT SWITCH 19 © Fraunhofer IESE Customer Segments Key Resources Revenue Streams Cost Structure Key Partners Value Propositions Channels Key Activities Customer Relationships Business Needs NICE INC. Deficits of current business model 20 App-Store High demands Happy customers NICE-APP Motivated Development team High ratings Negative InformativePositive Source: A. Osterwalder, Y. Pigneur, “Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers.” John Wiley and Sons, 2010. Continuous Deployment We sell their data Maintain Develop Maintenance cost Development cost Major Pain: Technical Debt
  11. 11. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 11 © Fraunhofer IESE Build Quality model (1/2) 21 Data gathering  Data analysis Software quality workshops (On‐site) User stories and quality models per use case Consolidated quality modelStudy of quality problems Landscape of data sources Interviews by two researchers Validation by the industry partners Architecture specification © Fraunhofer IESE Creation (and evolution) of a  quality model for actionable  analytics Business‐goal‐oriented  the selection of relevant and  business‐oriented metrics Aggregation of  heterogeneous data sources  into product factors  transparency Automatic interpretation of  raw data (i.e., assessment) Build Quality model (2/2) 22 Product Quality Code Quality Commented files Comments lines,  lines of code,… SonarQube Non‐complex files File cyclomatic complexity, No.  functions,… SonarQube Absence of  duplications Duplicated lines,  lines of code,… SonarQube Testing Status Passed tests Unit tests errors,  Unit tests passed,  … Jenkins, GitLab Software stability Ratio of open/in  progress bugs No. of open bugs,  No. of open  issues,… JIRA, Redmine,  GitLab Blocking Blocking code Fulfillment of  critical/blocker  quality rules Critical issues,  blocking issues,… SonarQube Issues’  specification Issues completely  specified Filled description in a issue, Filled due date,… JIRA, Redmine,  GitLab Test performance Fast tests’ builds Test duration,… Jenkins Details in: S. Martínez‐Fernández, A. Jedlitschka,  L. Guzmán, and A. M. Vollmer, “A Quality Model  for Actionable Analytics in Rapid Software  Development,” in Euromicro SEAA 2018 Major Pain: Effort to build/maintain modelsMajor Pain: Effort to build/maintain models
  12. 12. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 12 © Fraunhofer IESE Identification of the data  produced during: process system usage Heterogeneous data sources: sources carrying  different information storage in  heterogeneous formats  and tools diversity versions,  usage, environments,  security options,…  Identify Data sources for  software analytics 23 Details in: S. Martínez‐Fernández, P.  Jovanovic, X. Franch, and A. Jedlitschka,  “Towards Automated Data Integration in  Software Analytics,” in BIRTE@VLDB, 2018.  © Fraunhofer IESE Q‐Rapids tool overview Data sources Collected runtime, development data Insights, Actions qr‐connectqr‐eval Real‐time analytics for the whole life‐cycle 24
  13. 13. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 13 © Fraunhofer IESE The Q‐Rapids Project ‐ http://q‐rapids.eu 25 Q-Rapids vision: Supporting rapid SW development Quality requirements proposed through data mining and intelligent analysis of data © Fraunhofer IESE Q‐Rapids tool 26 Package: source code, artifacts, instructions for three modules qr‐connect module  data gathering qr‐eval module  implementation of quality model for data analysis Objects in Kibana  raw data visualization for actionable analytics Details in:  L. López, S. Martínez‐Fernández, C. Gómez, M.  Choras, R. Kozik, L. Guzmán, A. M. Vollmer, X. Franch, and A.  Jedlitschka, “Q‐Rapids Tool Prototype: Supporting Decision‐ Makers in Managing Quality in Rapid Software  Development,” in CAiSE Forum 2018
  14. 14. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 14 © Fraunhofer IESE Support for different data sources: nine  connectors Heterogeneous data: providing  valuable information about the  process, system, usage Scalability: able to ingest huge amount of data  per second (e.g., usage) >19 millions data points from use cases Initial infrastructure for Big data  attractive for companies to adopt Customizations: tutorial for modular  “connectors” Companies developing their own  connectors Tool modules: qr‐connect for data gathering 27 © Fraunhofer IESE Tool modules: qr‐eval for data analysis Providing valuable information to be shown  in the dashboard Assessment using the preferences  and judgments of experts and/or  learned data Evaluation results from the use cases: Understandable metrics and factors Relevant for identifying deficiencies 28
  15. 15. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 15 © Fraunhofer IESE Objects in Kibana portable while maintaining defined indexes  generated by qr‐connect It helps decision making. Examples: Prioritizing critical open bugs Solving quality rule violations Improvement after the evaluation of the proof‐of‐ concept in the use cases software organizations want to be able to  base their decisions on the latest set of  available data and the real‐time analytics  derived from them Tool modules: raw data visualizations for actionable analytics 29 © Fraunhofer IESE Q‐Rapids Quality Models ‐ Combining empirical models with data‐based models 30 SI1 F1 F2 M1 M6 M6 M1 M6 M6 Strategic  Indicators Assessed Metrics Data Source Dependencies Raw Data Quality Model (QM) SI1 M1 M6 M6 M1 M6 M6 SI = f(M1, …,  M6) Expert-based QM Data-based QM Context- Specific Model
  16. 16. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 16 © Fraunhofer IESE Q‐Rapids Architecture 31 © Fraunhofer IESE Motivation Conceptualize innovation Evaluate innovation Example – Q-Rapids Conclusions 32
  17. 17. IASESE 2018 @ ESEIW 2018 - Oulu, Finland October, 17. 2018 Copyright: Andreas Jedlitschka Fraunhofer IESE 17 © Fraunhofer IESE Old wine in new bottles?  YES for  Many things in analytics and empiricism, we have be using before  NO for  Systematic, goal-oriented approach for data-driven approaches  Integration of empirical models with data-based models  Open Question: Is there a difference in the level of evidence?  Open Question: What is about credibility of the approach? 33 © Fraunhofer IESE Dr. Andreas Jedlitschka Data Engineering DE Phone:+49 (631) 6800 2260 Fax: +49 (631) 6800 9 2260 Email: andreas.jedlitschka@iese.fraunhofer.de Dr. Silverio Martínez-Fernández Data Engineering DE Phone: +49 (0) 631-6800-2271 Email: Silverio.martinez@iese.fraunhofer.de 34

×