215 Towards a Framework to Support Large Scale Sampling in Software Engineering Surveys

University of Southampton

Context: The low quality and small size of samples in empirical studies in software engineering hamper the interpretation and generalization of their results. Therefore, enlarging sample sizes and improving their quality represent an important research challenge. Goal: We aim to define a conceptual framework, including requirements for establishing adequate sources for sampling subjects in software engineering surveys. Method: We use previous experience on applying systematic sampling strategies combined with contemporary web technologies in previously executed surveys, to organize the conceptual framework. We analyze its application to different sources of sampling. Results: The framework was observed to be feasible after its application to nine different large-scale sources of sampling. Conclusions: The analyzed crowdsourcing tools do not support essential requirements to be considered sources of sampling, while free- lancing tools and professional social networks do.

Towards a Framework to Support Large Scale
Sampling in Software Engineering Surveys
Rafael Maiani de Mello
rmaiani@cos.ufrj.br
Pedro Correa da Silva
pedrorez@poli.ufrj.br
Per Runeson
per.runeson@cs.lth.se
Guilherme Horta Travassos
ght@cos.ufrj.br
ese.cos.ufrj.br - serg.cs.lth.se

2
Motivation
• Small and non-probabilistic samples usually:
• reduce external validity;
• make replication difficult;
• limit possibilities of aggregation, and;
• hamper the evaluation of SE technologies .
• Particularly, SE surveys have their results affected
when inadequate samples are used.

3
Challenge
How to support the improvement of Software
Engineering surveys samples’ quality ?

4
Challenge
How to support the improvement of Software
Engineering surveys samples’ quality ?
Size
Confidence
Heterogeneity
Representativeness
Randomness
Reusability

5
Research Objective
To depict a framework, composed by a set of concepts
and processes, aimed at supporting researchers on
establishing adequate populations and samples for SE
surveys

6
Preliminary Studies
Do systematic strategies for large scale recruitment of
subjects in Software Engineering surveys contribute for
improving the samples' quality?

7
Preliminary Studies
Do systematic strategies for large scale recruitment of
subjects in Software Engineering surveys contribute for
improving the samples' quality?
Sampling and
Recruitment Protocol
Professional Social
Networks

10
Search Unit
• Characterizes how one or more units of observation can be
retrieved from a specific source

11
Source of Sampling
• Consists on a database (automated or not) from
which adequate sampling frames from the target
population can be systematically retrieved and
randomly sampled
• Essential Requirements
• Desirable Requirements
– Accuracy
– Clearness
– Completeness

13
Search Plan
• Describes how search units will be systematically
retrieved from a Source of sampling and
evaluated in order to be included or not into a
sampling frame

14
Sampling Strategy
• Describes the steps that must be followed in
order to sample and access the units of
observation that will be investigated in a study
trial

15
Sources of Sampling
Professional Social Networks
Source
Essential Accuracy Clearness Completeness
1 2 3 4 1 2 3 1 2 1 2 3 4
Linkedin (Groups)             
LinkedIn (Members)     - - - - - - - - -
ResearchGate             
Academia.edu             

16
Sources of Sampling
Crowdsourcing Tools
Source
Essential Accuracy Clearness Completeness
1 2 3 4 1 2 3 1 2 1 2 3 4
MTurk     - - - - - - - - -
MicroWorkers     - - - - - - - - -
ClickWorkers     - - - - - - - - -

17
Sources of Sampling
Freelancing Tools
Source
Essential Accuracy Clearness Completeness
1 2 3 4 1 2 3 1 2 1 2 3 4
eLancers             
ODesk             
FreeLancers             

18
Next Steps
• Investigating relevant attributes for contextualizing
units in SE surveys
• Develop processes for supporting the application of
the proposed framework
• Evaluate the framework

19
Towards a Framework to Support Large Scale
Sampling in Software Engineering Surveys
Rafael Maiani de Mello
rmaiani@cos.ufrj.br
Pedro Correa da Silva
pedrorez@poli.ufrj.br
Per Runeson
per.runeson@cs.lth.se
Guilherme Horta Travassos
ght@cos.ufrj.br
ese.cos.ufrj.br - serg.cs.lth.se

Gene Network Central (GNC) Pro is an online pathway analysis and systems biology software that integrates multiple bioinformatics resources. It allows users to analyze genes of interest within biological networks and pathways, helping expedite research. GNC Pro displays genes as nodes connected by edges representing relationships. Users can interactively explore networks, view tissue-specific expression data, and access supporting information for genes and relationships. The tool draws on curated data from literature text-mining, public databases, and SA Biosciences' own predictive algorithms.

Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...

Tim Menzies

The document discusses requirements traceability and how analysts can work efficiently with automated traceability tools. It presents the results of two experimental studies that evaluated the impact of different analyst behaviors - including with/without feedback and global vs local ordering - on precision, recall, and effort when developing requirements traceability matrices. The most efficient behaviors were local ordering with feedback, as they resulted in the highest precision and recall while examining the fewest candidate links.

AI Chatbot Service Framework based on Backpropagation Network for Predicting ...

資彥解

Thesis Presentation

nirvdrum

The document discusses a study evaluating different user feedback systems for search engines. It describes previous work using implicit and explicit user feedback to improve search result relevance. The study collected both voluntary and mandatory feedback from users in controlled and uncontrolled search scenarios. Decision trees were built from the collected data and showed that voluntary feedback systems achieved higher classification accuracy than mandatory systems, supporting the hypothesis that voluntary feedback is higher quality.

Risk assessment

Spoorthi Sham

This document summarizes a study on risk assessment models and methods for small and medium software development projects. It identifies 12 risk assessment approaches across two categories: models and methods. The models category includes prototypes like RAT and SRAEM that assess risk through matrices and estimations. The methods category consists of techniques like expectation maximization algorithms and source-based SRAM that use facts and probability assessments. The author concludes no single approach is ideal for all small projects due to their rapid timelines and budgets, and proposes a 5-step process to strategically assess risks.

JHU Data Science specialization_certificate

Gianfranco Campana

This document is a certificate from Johns Hopkins University recognizing the completion of their online Data Science Specialization. The specialization consisted of 10 courses covering the data science pipeline from R programming to machine learning to developing data products, as well as a capstone project. It provides students with the concepts and tools needed to think analytically about problems, manage large data, perform statistical analysis, create visualizations, build machine learning models, publish analyses, and develop data products.

get_specialization_certificate

Chao Yang

This document is a certificate from Johns Hopkins University recognizing an individual's completion of their Data Science Specialization. The specialization consisted of 10 courses covering the data science pipeline from R programming to machine learning to developing data products, as well as a capstone project. It provided participants with the concepts and tools needed to work with large data sets and create reproducible and visual analyses using statistical and machine learning techniques.

Introduction to Survey Data Quality

The document is a certificate from Johns Hopkins University recognizing that Russell Robbins successfully completed the Data Science Specialization through Coursera. The specialization consisted of 10 courses covering the concepts and tools needed for data science, including R programming, getting and cleaning data, exploratory data analysis, reproducible research, statistical inference, regression models, practical machine learning, developing data products, and a capstone project. The specialization provided participants with skills in using data science tools, analyzing complex problems, managing large datasets, applying statistical principles, creating visualizations, building and evaluating machine learning models, and publishing reproducible analyses.

It’s Not a Bug, It’s a Feature:Does Misclassification Affect Bug Localization?

Pavneet Singh Kochhar

Information Retrieval and User-centric Recommender System Evaluation

Alan Said

User Personality and the New User Problem in a Context-Aware Point of Interes...

KMI - Knowledge Media Institute, The Open University, UK.

The new user problem is an important and challenging issue that Context-Aware Recommender Systems (CARSs) must deal with, especially in the early stage of their deployment. It occurs when a new user is added to the system and there is not enough information about the user’s preferences in order to compute appropriate recommendations. It is common to address this problem in the recommendation algorithm, by using demographic attributes such as age, gender, and occupation, which are easy to collect and are reasonably good predictors of the user preferences. However, as we show here, user’s personality provides even better information for generating context-aware recommendations for places of interest (POI), and it is still easy to assess with a simple questionnaire. In our study, using a rating data set collected by a mobile app called STS (South Tyrol Suggests), we have found that by considering the user personality the system can better rank the recommendations for the new users.

Recognition and reward system for peer-reviewers

Empirical Evaluation of Active Learning in Recommender Systems

IEEEFINALYEARSTUDENTPROJECTS

The accuracy of collaborative-filtering recommender systems largely depends on three factors: the quality of the rating prediction algorithm, and the quantity and quality of available ratings. While research in the field of recommender systems often concentrates on improving prediction algorithms, even the best algorithms will fail if they are fed poor quality data during training. Active learning aims to remedy this problem by focusing on obtaining better quality data that more aptly reflects a user’s preferences. In attempt to do that, an active learning strategy selects the best items to be presented to the user in order to acquire her ratings and hence improve the output of the RS. In this seminar, I present a set of active learning strategies with different characteristics and the evaluation results with respect to several evaluation measures (i.e., MAE, NDCG, Precision, Coverage, Recommendation Quality, and, Quantity of the acquired ratings and contextual conditions). The traditional evaluation of active learning strategies has two major flaws: (1) Performance has been evaluated for each user independently (ignoring system-wide improvements) (2) Active learning strategies have been evaluated in isolation from unsolicited user ratings (natural acquisition). Addressing these flaws, I present that an elicited rating has effects across the system, so a typical user-centric evaluation which ignores any changes of rating prediction of other users also ignores these cumulative effects, which may be more influential on the performance of the system as a whole (system-centric). Hence, I present a novel offline evaluation methodology and use it to evaluate some novel and state of the art rating elicitation strategies. While the first set of experiments was done offline, the true value of active learning must be evaluated in an online setting. Hence, in the second part of the seminar, I present a novel active learning approach that exploits some additional information of the user (i.e. the user’s personality) to deal with the cold start problem in an up-and-running mobile context-aware RS called STS, that provides users with recommendations for places of interest (POIs). The results of live user studies, have shown that the proposed AL approach significantly increases the quantity of the ratings and contextual conditions acquired from the user as well as the recommendation accuracy.

Ml part2

Leon Gladston

This document discusses several machine learning algorithms including principal component analysis (PCA), naive Bayes classifiers, and artificial neural networks (ANN). It provides steps for PCA including centering data, calculating the covariance matrix, and selecting principal components. It describes multinomial, Bernoulli, and Gaussian naive Bayes classifiers and their pros and cons. Applications of naive Bayes include text classification, spam filtering, and sentiment analysis. The document also discusses the layers of ANN including hidden layers and activation functions, and provides pros and cons as well as applications of ANN.

Project Data Incorporating Qualitative Factors for Improved Software Defect P...

Tim Menzies

The document discusses a project that aimed to improve software defect prediction by incorporating qualitative factors. It provides background on previous statistical and causal models for defect prediction. It then describes the development of a causal Bayesian network model that considers qualitative factors like testing staff experience, documented test cases, and how well-defined the testing process is, in addition to quantitative factors. The model was developed using both qualitative expert judgement and quantitative data from companies. It was validated on a set of projects selected for having reliable defect data and key people available. The results showed good predictions of actual versus predicted defects using only a few of the model inputs. Some caveats are mentioned regarding biases and limitations of the data. In conclusion, the causal model was able to provide

IEEE 2014 JAVA DATA MINING PROJECTS Security evaluation of pattern classifier...

Recommender Systems and the Human Factor

Mark Graus

Mark Graus gave a presentation on recommender systems and the importance of considering the human factor. He discussed how recommender systems use machine learning algorithms like collaborative filtering and content-based filtering to make predictions, but that machine learning alone is not enough. User behavior data from recommender systems is limited and does not capture things like user preferences, privacy concerns, or choice overload. It is important to conduct A/B testing and user experience evaluations to understand how users actually interact with and feel about recommendation systems. The key takeaway is that recommender systems require both strong machine learning and a focus on the human perspective.

The Human Factor in Digital Recommender Systems

SIMAdmin

The document discusses a study on how users perceive and interact with Netflix's recommender systems. 8 interviews were conducted where participants discussed their criteria for evaluating recommendations, experiences with recommender systems, and opinions on Netflix's recommendations. A coding schema was developed to analyze the interviews and looked at factors like viewing preferences, the role of serendipity, and the importance of trust in recommendations. Key findings included that common genres and talents were important finding aids for participants and that their level of trust in a recommendation source impacted its perceived value.

Coursera Data Science 2015

Frank Hasbani

The document is a certificate from Johns Hopkins University congratulating Frank Hasbani for successfully completing the Data Science Specialization online program. The specialization consisted of 10 courses covering the concepts and tools for data science, including R programming, getting and cleaning data, exploratory data analysis, reproducible research, statistical inference, regression models, practical machine learning, developing data products, and a capstone project. The specialization trained participants in using data science tools, analyzing complex problems, managing large datasets, applying statistical principles, creating visualizations, building and evaluating machine learning models, and publishing reproducible analyses.

Coursera Data Science 2015

Fernando Gonzalez Prada

The document is a certificate confirming that Fernando Sebastian Gonzalez Prada successfully completed the Data Science Specialization through Coursera. The specialization consisted of 10 courses covering the data science pipeline and tools, including R programming, getting and cleaning data, exploratory data analysis, reproducible research, statistical inference, regression models, practical machine learning, developing data products, and a capstone project. The specialization trained participants to use data science tools, analyze complex problems, manage large datasets, apply statistical principles, create visualizations, build machine learning models, publish analyses, and develop data products.

Optimizing Protocol Planning, Feasibility, and Site Selection through an Inte...

will buckley

moraes-a2017ictir

Felipe Moraes

This document discusses the components of dynamic search systems and evaluates their impact through simulations. It introduces four hypotheses: (1) early effectiveness is influenced by baseline ranker precision, (2) later effectiveness is influenced by baseline ranker recall, (3) inaccurate aspect modeling hinders effectiveness, and (4) improved coverage estimates can enhance effectiveness, especially for narrow queries. It simulates variations in baseline rankers, aspect modeling accuracy/completeness, and coverage matrices to evaluate the hypotheses. The authors plan to extend their analysis to additional interaction elements and concrete system implementations.

Modern elicitation trends asma & ayesha paper presentation

Asma Sajid

The document discusses modern trends in requirement elicitation techniques. It outlines traditional elicitation methods like conversational, observational, analytical and synthetic. It then analyzes the effect of elicitation on projects, the most commonly used methods, and methods for specific project types. Finally, it proposes guidelines for elicitation and a plan combining different techniques based on the project characteristics and development process.

About the benefits and pitfalls of relying on analytical methods

Pragmatic Cohesion Consulting, LLC

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...

InsightInnovation

Data quality isn’t always the sexiest topic, but it’s critical and one that buyers and suppliers often neglect to have. The ramifications of ignoring it can cost millions of dollars. Some of the industry’s largest buyers and suppliers have found a simple solution though and it’s one that is available to everyone else too. Come here about how the issue of data quality concerns haven’t gone away, and what others are doing to make sure they and their insights are protected.

Data science in 10 steps

QuantUniversity

This document outlines a 10 step framework for developing data science applications. It begins with articulating the business problem and data questions. Next steps include developing a data acquisition and preparation strategy, exploring and formatting the data, defining the goal, and shortlisting techniques. Later steps evaluate constraints, establish evaluation criteria, fine tune algorithms, and plan for deployment and monitoring. The document also provides background on the speaker and organization. They offer data science, quant finance, and machine learning programs and consulting using Python, R, and MATLAB on their online sandbox platform.

Philips john huffman

BigDataExpo

1. The document discusses the perspectives of John Huffman, CTO of Philips Healthcare Informatics, on big data, analytics, and AI in healthcare. 2. It outlines the multi-stage process for advanced analytics, including data ingestion, model training, evaluation, and production. It also discusses challenges in data collection/processing and model deployment. 3. Key points made are that data is more important than algorithms/methods; analytics requires clean, interoperable data; and the stack provides tools but not full solutions - data is the intellectual property.

Sdlc 4

The document discusses the system development life cycle (SDLC), which includes preliminary investigation, requirements analysis, system design, software development, system testing, and implementation and maintenance. It describes the purpose and history of SDLC as emerging in the 1960s to address the "software crisis". It also outlines the main steps and activities in each phase of the SDLC process.

Software Development Life Cycle

The document discusses the system development life cycle (SDLC), which involves 6 main steps: 1) preliminary investigation, 2) requirements analysis, 3) system design, 4) system acquisition and development, 5) system testing, and 6) implementation and maintenance. It describes each step in detail, including gathering user requirements, designing and selecting a software model, testing the system, training users, and evaluating the results. The SDLC aims to efficiently develop high-quality software through a structured process of analysis, design, implementation, and maintenance activities.

What's hot

ds_certificate

Russell Robbins

It’s Not a Bug, It’s a Feature:Does Misclassification Affect Bug Localization?

Pavneet Singh Kochhar

Information Retrieval and User-centric Recommender System Evaluation

Alan Said

User Personality and the New User Problem in a Context-Aware Point of Interes...

KMI - Knowledge Media Institute, The Open University, UK.

Recognition and reward system for peer-reviewers

Empirical Evaluation of Active Learning in Recommender Systems

IEEEFINALYEARSTUDENTPROJECTS

Ml part2

Leon Gladston

Project Data Incorporating Qualitative Factors for Improved Software Defect P...

Tim Menzies

IEEE 2014 JAVA DATA MINING PROJECTS Security evaluation of pattern classifier...

Recommender Systems and the Human Factor

Mark Graus

The Human Factor in Digital Recommender Systems

SIMAdmin

Coursera Data Science 2015

Frank Hasbani

Coursera Data Science 2015

Fernando Gonzalez Prada

What's hot (13)

ds_certificate

It’s Not a Bug, It’s a Feature:Does Misclassification Affect Bug Localization?

Information Retrieval and User-centric Recommender System Evaluation

User Personality and the New User Problem in a Context-Aware Point of Interes...

Recognition and reward system for peer-reviewers

Empirical Evaluation of Active Learning in Recommender Systems

Ml part2

Project Data Incorporating Qualitative Factors for Improved Software Defect P...

IEEE 2014 JAVA DATA MINING PROJECTS Security evaluation of pattern classifier...

Recommender Systems and the Human Factor

The Human Factor in Digital Recommender Systems

Coursera Data Science 2015

Similar to 215 Towards a Framework to Support Large Scale Sampling in Software Engineering Surveys

Optimizing Protocol Planning, Feasibility, and Site Selection through an Inte...

will buckley

moraes-a2017ictir

Felipe Moraes

Modern elicitation trends asma & ayesha paper presentation

Asma Sajid

About the benefits and pitfalls of relying on analytical methods

Pragmatic Cohesion Consulting, LLC

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...

InsightInnovation

Data science in 10 steps

QuantUniversity

Philips john huffman

BigDataExpo

Sdlc 4

Software Development Life Cycle

Varendra University Rajshahi-bangladesh

Infiltrate 2015 - Data Driven Offense

Ram Shankar Siva Kumar

1) The document discusses how machine learning (ML) techniques can be used by red teams to drive offensive cyber operations in a more intelligent and adaptive manner. 2) Specifically, it provides examples of how ML can be used for spear phishing by analyzing past user click data to select the most relevant phishing emails tailored to each individual user's role. 3) The document argues that embedding intelligence into attacks using ML can make attacks more effective while making it more difficult for blue teams to build attackers' tactics, techniques, and procedures (TTPs).

software project management

The document discusses software project management. It defines what a project and project management are, and describes the key characteristics of a software project. It outlines several software development lifecycles and methodologies including waterfall, prototype, spiral, agile, Scrum, extreme programming (XP), and rapid application development (RAD). It also discusses software project roles, risk management, project monitoring, defining a lifecycle model, software team organization structures, communication and coordination practices, and factors to consider when selecting a lifecycle model.

Human factors in software reliability engineering - Research Paper

Muhammad Ahmad Zia

In this paper, Maria Spichkova et al. have presented theri vision of the integration of human factors engineering into the software development process. The aim of this approach is to improve the quality of software and to deal with human errors in a systematic way. Categories and Subject Descriptors D.4.5 [Reliability]: Fault-tolerance; D.2.5 [Software Engineering]: Testing and Debugging General Terms Reliability, Verification

Tommi kramer 2013-06-21-caise-re2-kramer

caise2013vlc

The document describes a method and tool called SODA (Software Outsourcing Decision Aid) that helps small and medium enterprises decide what software development work to outsource and what to do in-house. SODA uses a three phase process: 1) it models software requirements as a graph, 2) it clusters the requirements graph to find groups of related requirements, and 3) it analyzes the structure of the requirements graph and clusters to make outsourcing recommendations based on modularity, coupling, cohesion and other measures. An evaluation of SODA found that it produced high quality clusters and scalable results for different sized projects.

Software Analytics

anapaula_spbr

- The document discusses software analytics and presents a ranked list of 145 questions that software engineers want data scientists to answer. It describes conducting surveys of over 1,500 engineers to develop this list. - The list identifies important topics for researchers, practitioners, and educators to focus on based on the needs of industry. These include understanding software processes and practices, and the impact of code quality. - Developing this catalog of questions can guide the collection and analysis of data, as well as the development of tools and techniques, to better support decision-making in software projects.

Gr 4 system theory and methodologies

The document discusses systems theory and methodologies. It provides definitions of key concepts in systems theory such as open vs closed systems. It also describes different system development methodologies including the waterfall model, prototyping model, and rapid application development (RAD) model. The waterfall model involves sequential stages of requirements, design, implementation, verification and maintenance. Prototyping involves building scaled-down versions of the system to help determine requirements. RAD model involves developing components in parallel and assembling them into a working prototype to get early user feedback.

Gr 4 system theory and methodologies

This document discusses systems theory and methodologies. It provides information on open vs closed systems, system concepts like hierarchy and boundaries, and different system development methodologies including waterfall, prototyping, and rapid application development (RAD). The waterfall model involves sequential stages of requirements, design, implementation, verification, and maintenance. Prototyping builds a scaled down working version of the system to help determine requirements. RAD involves parallel development of components in short time boxes to quickly provide a working prototype for feedback. Information systems in firms are used for operational excellence, productivity, efficiency, agility, developing new products/services, customer intimacy, improved decision making, and competitive advantage.

Measuring the usefulness of Knowledge Organization Systems in Information Ret...

GESIS

This document discusses three case studies that demonstrate the usefulness of knowledge organization systems (KOS) in information retrieval applications. The case studies show that: 1) Controlled vocabulary searches using term mappings between domains improved search effectiveness compared to searches without mappings. 2) Query expansion using recommendations from discipline-specific search term recommendation services led to better search results than using a general recommendation service. 3) Usage of search term recommendations in an interactive IR system was found to positively correlate with successful search outcomes in subsequent search steps.

Fact finding techniques

imthiyasbtm

The document discusses various fact-finding techniques used by systems analysts to identify requirements, including sampling existing documentation, observation, questionnaires, interviews, prototyping, and joint requirements planning. It describes the difference between functional and non-functional requirements, and the importance of properly identifying and managing requirements to avoid cost overruns, delays, user dissatisfaction, and other issues. The fact-finding process involves problem analysis, requirements discovery, documentation, and ongoing management of requirements as needs change over the project lifecycle.

SAD _ Fact Finding Techniques.pptx

SharmilaMore5

Towards a Metamodel for a Requirements Engineering Process of Embedded Systems

Tarcísio Couto

In the embedded systems (ES) area, more than 50% of problems occur at system delivery and are related to misconconceptions in capturing requirements. According to our systematic literature review (SLR), no evidence explicitly depicts how an embedded system must be elicited and specified. However, understanding the embedded systems and their environment is a strenuous activity. Even though current approaches present some contributions, the definition of a systematic requirements engineering process remains a challenging issue. Based on this shortcoming, we developed a metamodel that defines concepts and relationships that must be taken into account the development of an ES. From the metamodel, we define a precise requirements engineering process. This research presents the main results of a SLR, a resource model, and a sketch of a process to guide the requirements development of embedded systems.

Similar to 215 Towards a Framework to Support Large Scale Sampling in Software Engineering Surveys (20)

Optimizing Protocol Planning, Feasibility, and Site Selection through an Inte...

moraes-a2017ictir

Modern elicitation trends asma & ayesha paper presentation

About the benefits and pitfalls of relying on analytical methods

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...

Data science in 10 steps

Philips john huffman

Sdlc 4

Software Development Life Cycle

Infiltrate 2015 - Data Driven Offense

software project management

Human factors in software reliability engineering - Research Paper

Tommi kramer 2013-06-21-caise-re2-kramer

Software Analytics

Gr 4 system theory and methodologies

Measuring the usefulness of Knowledge Organization Systems in Information Ret...

Fact finding techniques

SAD _ Fact Finding Techniques.pptx

Towards a Metamodel for a Requirements Engineering Process of Embedded Systems

More from ESEM 2014

Keynote 2 - The 20% of software engineering practices that contribute to 80% ...

This presentation will challenge the application of metrics and other software engineering practices in commercial companies that do not have to comply with safety/ regulatory standards and thus can chose the SDLC approach that they feel more appropriate and cost effective for the intended purpose. Which are the practices that are really applied to make things happen under the tight constraints of time to market and profitability? A snapshot of the "hands-on" situation from the perspective of a large consulting company engaged with many customers in various markets and domains. Bio: Gualtiero Bazzana is Chairman of ITA-STQB, Head of Marketing WG for ISTQB and Managing Director of Alten Italia. He has been working in the IT domain since 20 years with a long lasting experience in the areas of testing, process improvement, quality. He has authored 50+ papers at international conferences on such subjects

Keynote 1 - Engineering Software Analytics Studies

Performing quantitative software analytics studies can be an immensely rewarding activity for scientists performing empirical research. However, such studies often pose numerous engineering challenges. The researcher must hunt down appropriate data sets, devise bespoke collection and processing tools, and optimise performance to match the size of the collected data. I will discuss principles and strategies that can be used to deal with these problems, and present examples of associated tools and techniques. Some particularly effective strategies associated with data set construction involve recursion, web searching, synthesis, probing, instrumentation, and the nurturing of alliances. On the processing front approaches include the opportunistic scavenging of tool front-ends, the exploratory development of pipelines, as well as the exploitation of tool interoperability, scripting languages, and their rich libraries. The required performance can be obtained through parallelism, stream processing, the judicious use of low-level facilities, and the choice of appropriate samples. I will finish the presentation with an overview of open problems and challenges in software analytics in vertical domains, data analysis, and under-represented stakeholders.

33 - On Knowledge Transfer Skill in Pair Programming

Context: General knowledge transfer is often considered a valuable effect or side-effect of pair programming, but even more important is its role for the success of the pair programming session itself: The partners often need to explain an idea to carry the process forward. Goal: Understand the mechanisms at work when knowledge is transferred during a pair programming session; provide practical advice for constructive behavior. Method: Qualitative data analysis of recordings of actual industrial pair programming sessions. Results: Some pairs are much more efficient in their knowledge transfer than others. These pairs manage to (1) not attempt to explain multiple things at once, (2) not lose sight of a topic, (3) clarify dicult points in stages. Conclusions: Pair programming requires skill beyond software development skill. To be able to identify knowledge needs and then push such knowledge to or pull it from the partner successfully is one aspect of such skill. We characterize a number of its elements.

222 - Design Pattern Decay: The Case for Class Grime

Context: We investigate class grime, a form of design pattern decay, wherein classes of the pattern realization have extraneous attributes or methods, which obfuscate the intended design of a pattern. Goal: To expand the taxonomy of class grime using properties of class cohesion. Using this expanded taxonomy we explore the effect that forms of class grime have on pattern realization understandability. Method: A pilot study utilizing a formal experiment to explore the effects of class grime on design pattern understandability. The experiments used simulated injection of 8 types of class grime into design pattern realizations randomly selected from 16 design pattern types from a set of 6541 realizations from 520 distinct software systems. Results: We found that for each of the 8 identified class grime forms, understandability was negatively affected. Conclusion: This work serves as early communication of research for the validation of the extended taxonomy as well as the method of grime injection used in the experiment.

210 - Software Population Pyramids: The Current and the Future of OSS Develop...

This document presents a study on software population pyramids, which graphically represent the distribution of contributors to open source software projects over time. The study analyzed activity data from 90 open source projects on GitHub. It identified four common shapes of software population pyramids and showed how the pyramids for two projects, homebrew and blueprint-css, evolved over multiple time periods. The software pyramids provide insights into the current status and future sustainability of open source development communities.

169 - Bridging the Gap: SE Technology Transfer into Practice - Study Design a...

Background: Particularly during and after research projects, technology transfer into practice plays an important role for academia to get technologies into use and for industry to improve their development. Objective: Our goal was to gain more and current knowledge about how technology transfer from software engineering (SE) research into industrial practice is accomplished best and how to measure the effectiveness of this transfer. Method: We conducted a study in the context of two German research projects, covering many different organizations from industry and academia. Results: This paper presents the design of the study and the survey performed. After introducing the concept of technology transfer we used and adapted, we present preliminary results. Conclusions: We observed that traditional means such as meetings or workshops are still the most widely used mediums for technology transfer in SE. We also discovered that, even though the duration of transfer depends on the object being transferred, the average duration is three years, which is far less than previously published (~18 years).

196 - Evaluation in Practice: Artifact-based Requirements Engineering and Sc...

Context: In the context of the research and development project ARAMiS, multiple partners from research and industry are collaborating in the development of new methods and technologies in the field of multicore systems. Goal: We designed and executed studies for evaluating the results of the ARAMiS sub-project responsible for requirements engineering: an artifact-based requirements engineering approach, its tooling, and a cross-domain scenario. Method: This evaluation was performed along with the dissemination of the results in the project. The evaluation included two studies aimed at collecting the opinions of the project participants regarding the requirements engineering results from the viewpoints of industry and research. Results: The mainly positive results showed us that the different parts of the requirements engineering approach in this project are being accepted. Conclusions: Nonetheless, especially the results for the scenario revealed some weaknesses, such as the so-called “ARAMiS gap”, i.e., a gap between the high-level requirements engineering artifacts and the detailed engineering artifacts.

42- Using Templates to Elicit Implied Security Requirements from Functional R...

Context: Security requirements for software systems can be challenging to identify and are often overlooked during the requirements engineering process. Existing functional requirements of a system can imply the need for security requirements. Systems having similar security objectives (e.g., confidentiality) often also share security requirements that can be captured in the form of reusable templates and instantiated in the context of a system to specify security requirements. Goal: We seek to improve the security requirements elicitation process by automatically suggesting appropriate security requirement templates implied by existing functional requirements. Method: We conducted a controlled experiment involving 50 graduate students enrolled in a software security course to evaluate the use of automatically-suggested templates in eliciting implied security requirements. Participants were divided into treatment (automatically-suggested templates) and control groups (no templates provided). Results: Participants using our templates identified 42% of all the implied security requirements in the oracle as compared to the control group, which identified only 16% of the implied security requirements. Template usage increased the efficiency of security requirements identified per unit of time. Conclusion: Automatically-suggested templates helped participants (security non-experts) think about security implications for the software system and consider more security requirements than they would have otherwise. We found that participants need more incentive than just a participatory grade when completing the task. Further, we recommend to ensure task completeness, participants either need a step-driven (i.e., wizard) approach or progress indicators to identify remaining work.

166 - ISBSG variables most frequently used for software effort estimation: A ...

Background: The International Software Benchmarking Standards Group (ISBSG) dataset makes it possible to estimate a project’s size, effort, duration, and cost. Aim: The aim was to analyze the ISBSG variables that have been used by researchers for software effort estimation from 2000, when the first papers were published, until the end of 2013. Method: A systematic mapping review was applied to over 167 papers obtained after the filtering process. From these, it was found that 133 papers produce effort estimation and only 107 list the independent variables used in the effort estimation models. Results: Seventy-one out of 118 ISBSG variables have been used at least once. There is a group of 20 variables that appear in more than 50% of the papers and include Functional Size (62%), Development Type (58%), Language Type (53%), and Development Platform (52%) following ISBSG recommendations. Sizing and Size attributes altogether represent the most relevant group along with Project attributes that includes 24 technical features of the project and the development platform. All in all, variables that have more missing values are used less frequently. Conclusions: This work presents a snapshot of the existing usage of ISBSG variables in software development estimation. Moreover, some insights are provided to guide future studies.

112 - The Role of Mentoring and Project Characteristics for Onboarding in Ope...

Context: Onboarding is a process that helps newcomers become integrated members of their organisation. Successful onboarding programs can result in increased performance in conventional organisations, but there is little guidance on how to onboard new developers in Open Source Software (OSS) projects. Goal: In this study, we examine how mentoring and project characteristics influence the effectiveness and efficiency of the onboarding process. We study a collaboration program involving a total of nine Open Source Software projects and more than 120 students from dfferent universities around the world as part of Facebook’s Education Modernization Program. Method: We use quantitative measurements of source code repositories, issue tracking systems, and discussion fora to examine how newcomers become contributing members of their OSS projects. Results: We found that developers receiving deliberate onboarding support through mentoring were more active at an earlier stage than developers entering projects through conventional means. Also, we found that project size and lifetime influenced onboarding. Conclusion: Empirical decision support can contribute to a more effective onboarding process in OSS projects. Mentor support in critical stages can accelerate the process, but project maturity is also a significant factor that increases the effect of onboarding.

224 - Factors Impacting Rapid Releases: An Industrial Case Study

Context: Software release teams try to reduce the time needed for the transit of features or bug fixes from the development environment to the production, crossing all the quality gates. However, little is known about the factors that influence the time-to-production and how they might be controlled in order to speed up the release cycles. Goal:This paper examines step by step the release process of an industrial software organization aiming to identify factors that have a significant impact on the lead time and outcomes of the software releases. Method:Over 14 months of release data have been analyzed (246 releases from the isolated source code branches to the production environment). Results:We discuss three dimensions under which a series of factors could be addressed: technical, organizational, and interactional. We present our findings in terms of implications for release process improvements. Conclusions: Our analyzes reveal that testing is the most time consuming activities (86%) along with the need for more congruence among teams, especially in the context of parallel development.

214 - Sampling Improvement in Software Engineering Surveys

Context: Small and non-probabilistic samples represent relevant issues when discussing the external validity of empirical studies in Software Engineering. Goal: To investigate alternatives to improve the quality of samples (size, heterogeneity and level of confidence). Method: To replicate a survey on characteristics of agility in software processes by applying a systematic recruitment strategy over a professional social network. Results: It resulted in a sampling frame composed by 19 groups stratified according two perspectives: sharing of groups' members and main software engineering skills reported by the subjects. In total, 7,745 subjects were randomly recruited, resulting in 291 contributions. Conclusions: This sample was significantly larger, more heterogeneous and presents some strata with higher confidence levels than previous trials samples.

201 - Using Qualitative Metasummary to Synthesize Empirical Findings in Liter...

This document describes a study that used qualitative metasummary to synthesize findings from multiple empirical studies on software engineering teams. It discusses the metasummary method, which involves extracting findings, grouping them, abstracting them, and calculating frequency and intensity effect sizes. The researchers applied this method to studies on software engineering team performance. They found it produced a synthesis highly connected to the original findings but had limitations in comparability and integrating mixed data. Overall, qualitative metasummary was found to be useful for literature reviews in software engineering but could be improved.

130 - Motivated software engineers are engaged and focused, while satisfied o...

Context – Motivation and job satisfaction are not the same thing, and although business organisation research recognised this a long time ago, in Software Engineering research, we have not. As a result, thirty years of research on motivation in software engineering has produced knowledge on what makes software engineers generally happier, but not about how to increase their motivation. Goal – In this article, we aim to identify visible signs of a software engineer who is motivated to work. Method – We describe a field study in which 62 practitioners in Brazil reported their view of "motivation" in the context of their practical work. Data was collected by means of audio-recorded semi-structured interviews, and a thematic analysis was applied to identify the most relevant descriptors of motivation. Results – Our data reveal that (1) motivated Software Engineers are engaged, focused, and collaborative; and (2) the term "motivation" is used as an umbrella term to cover several distinct organizational behaviours that are not necessarily related to the individual ́s desire to work. Conclusions – Without a clear picture of the difference between these two concepts, work-based motivation programs may not be designed effectively to address either turnover or performance issues. Overall, this work indicates the need for a more effective conceptual system to investigate and encourage both job satisfaction and work motivation in software engineering research and practice.

178 - A replicated study on duplicate detection: Using Apache Lucene to searc...

Context: Duplicate detection is a fundamental part of issue management. Systems able to predict whether a new defect report will be closed as a duplicate, may decrease costs by limiting rework and collecting related pieces of information. Goal: Our work explores using Apache Lucene for large- scale duplicate detection based on textual content. Also, we evaluate the previous claim that results are improved if the title is weighted as more important than the description. Method: We conduct a conceptual replication of a well-cited study conducted at Sony Ericsson, using Lucene for searching in the public Android defect repository. In line with the original study, we explore how varying the weight- ing of the title and the description affects the accuracy. Results: We show that Lucene obtains the best results when the defect report title is weighted three times higher than the description, a bigger difference than has been previously acknowledged. Conclusions: Our work shows the potential of using Lucene as a scalable solution for duplicate detection.

124 - Impact of Developer Reputation on Code Review Outcomes in OSS Projects:...

This document studies the impact of developer reputation on code review outcomes in open source software projects. It analyzes code review data from 8 open source projects to test hypotheses about whether core developers, who have higher reputation, have shorter feedback and review intervals, higher acceptance rates, and fewer required patchsets than peripheral developers. The results provide empirical evidence that core developers do receive preferential treatment in code reviews, supporting all the hypotheses. This suggests reputation influences code review behaviors and outcomes for developers in open source projects.

18 - Impact of Process Conformance on the Effects of Test-driven Development

Context: One limitation of the empirical studies about test- driven development (TDD) is knowing whether the developers followed the advocated test-code-refactor cycle. Re- search dealt with the issue of process conformance only in terms of internal validity, while investigating the role of other confounding variables that might explain the contro- versial e↵ects of TDD. None of the research included process conformance as a fundamental part of the analysis. Goal: We aim to examine the impact of process conformance on the claimed effects of TDD on external quality, developers’ productivity and test quality. Method: We used data collected during a previous study to create regression models in which the level of process conformance was used to predict external quality, productivity, and tests thoroughness. Result: Based on our analysis of the available data (n = 22), we observe that neither quality (p value = 0.21), productivity (p value = 0.80), number of tests (p value = 0.39) nor coverage (pvalue = 0.09) was correlated with the level of TDD process conformance. Conclusion: While based on a small sample, we raise concerns about how TDD is interpreted. We also question whether the cost of strictly following TDD will pay-o↵ in terms of external quality, productivity, and tests thoroughness.

65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...

Context: Real-time speech translation technology is today available but still lacks a complete understanding of how such technology may affect communication in global software projects. Goal: To investigate the adoption of combining speech recognition and machine translation in order to overcome language barriers among stakeholders who are remotely negotiating software requirements. Method: We performed an empirical simulation-based study including: Google Web Speech API and Google Translate service, two groups of four subjects, speaking Italian and Brazilian Portuguese, and a test set of 60 technical and non-technical utterances. Results: Our findings revealed that, overall: (i) a satisfactory accuracy in terms of speech recognition was achieved, although significantly affected by speaker and utterance differences; (ii) adequate translations tend to follow accurate transcripts, meaning that speech recognition is the most critical part for speech translation technology. Conclusions: Results provide a positive albeit initial evidence towards the possibility to use speech translation technologies to help globally distributed team members to communicate in their native languages.

52 - The Impact of Test Ownership and Team Structure on the Reliability and E...

Context: Software testing is a crucial step in most software development processes. Testing software is a key component to manage and assess the risk of shipping quality products to customers. But testing is also an expensive process and changes to the system need to be tested thoroughly which may take time. Thus, the quality of a software product depends on the quality of its underlying testing process and on the effectiveness and reliability of individual test cases. Goal: In this paper, we investigate the impact of the organizational structure of test owners on the reliability and effectiveness of the corresponding test cases. Prior empirical research on organizational structure has focused only on developer activity. We expand the scope of empirical knowledge by assessing the impact of organizational structure on testing activities. Method: We performed an empirical study on the Windows build verification test suites (BVT) and relate effectiveness and reliability measures of each test run to the complexity and size of the organizational sub-structure that enclose all owners of test cases executed. Results: Our results show, that organizational structure impacts both test effectiveness and test execution reliability. We are also able to predict effectiveness and reliability with fairly high precision and recall values. Conclusion: We suggest to review test suites with respect to their organizational composition. As indicated by the results of this study, this would increase the effectiveness and reliability, development speed and developer satisfaction.

167 - Productivity for proof engineering