To consolidate a body of knowledge built upon evidence, experimental results have to be extensively verified. Experiments need replication at other times and under other conditions before they can produce an established piece of knowledge. Several replications need to be run to strengthen the evidence.
Most SE experiments have not been replicated. If an experiment is not replicated, there is no way to distinguish whether results were produced by chance (the observed event occurred accidentally), results are artifactual (the event occurred because of the experimental configuration but does not exist in reality) or results conform to a pattern existing in reality.
The immaturity of experimental SE knowledge has been an obstacle to replication. Context differences usually oblige SE experimenters to adapt experiments for replication. As key experimental conditions are yet unknown, slight changes in replications have led to differences in the results that prevent verification.
There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment?
The aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. We need to learn which types of replications are feasible in SE as well as the acceptable changes for each type and the level of verification provided.
What use are experiments for industry?
1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions
2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development
What use are experiments for industry?
1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions
2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development
Parameter Optimization of Shot Peening Process of PMG AL2024 Alloy CoverIOSRJMCE
Shot peening leads to local plastic deformations in the near-surface regions, which result in the development of compressive residual stress and the improvement of surface hardness in the aerospace structural components. These properties can be enhanced by careful selection of the peening parameters. PMG Cover of AL2024 Aluminum Alloy is widely used in the generator manufacturing cover due to its high specific static strength. In this study a Taguchi Grey Relational Analysis is presented to optimize the surface properties of residual stress, micro hardness. The effects of four peening parameters (Shot Diameter, Shot Velocity, Impact Angle, Nozzle Distance) on micro hardness and residual stress are investigated Design of Experiment work is carried out by MINITAB 14 software tools of Taguchi Grey relational method, for getting excellent shot peening process parameter combination by MAT LAB R2009 software tools of advanced Optimization method as Genetic Algorithm, Simulated Annealing. Compare of the above reading for the investigation.
INVITED EDITORIAL Lets Do It Again A Call for Replications TatianaMajor22
INVITED EDITORIAL: Let's Do It Again: A Call for Replications
in Psi Chi J o u rn a l o f Psychological Research
J o h n E. E d lu n d
R o c h e s te r In s titu te o fT e c h n o lo g y
S
c ie n c e is said to b e su ffe rin g fro m a crisis
o f r e p li c a b i li t y ( I o a n n i d is , 2 0 0 5 ). T h is
crisis o c c u rs w h e n s c ie n tific s tu d ie s fail
to b e s u p p o r te d by s u b s e q u e n t re s e a r c h . T h e
challenges posed by th e replication crisis address
the fundam ental n atu re o f science an d the p ublic’s
u n d e r s t a n d i n g o f it. N u m e r o u s c o n t r i b u t i n g
reasons for th e rep licatio n crisis have b e e n n o ted
in c lu d in g d a ta fa lsific a tio n (S te e n , 2 0 1 1 ), th e
pressures o f ten u re an d p ro m o tio n (Varian, 1998),
questionable research practices (Simmons, Nelson,
& Sim onsohn, 2011), th e tendency o f jo u rn a ls to
w ant to publish particularly novel papers (Steen,
2011), and the p reference for publishing significant
results (de W inter & H appee, 2013). T hese factors
all in c re a se th e o d d s o f in a c c u ra te in fo rm a tio n
b e in g p u b lish e d , w hich in tu rn is in c o rp o r a te d
in to tex ts, as h a p p e n e d w ith th e d e ta ils o f th e
original investigation in th e Kitty G enovese case,
which led to th e fam ous bystander apathy studies
(Griggs, 2015). T he p rim ary goal o f this editorial is
to briefly discuss th e factors th at have co n trib u ted
to th e re p lic atio n crisis, tech n iq u e s em ployed by
various jo u rn a ls in th e field to deal with the crisis,
and how Psi Chi Journal o f Psychological Research (PCf)
is responding.
Build U p to th e R eplication Crisis
Perhaps th e biggest indication o f the psychological
re p lic atio n crisis was a series o f p ap e rs th a t were
com pletely fabricated by several differen t authors
(L e v elt C o m m itte e , 2 0 1 2 ). In th e s e cases, th e
au th o rs in question were discovered to have com
pletely fabricated th e ir d ata based o n a n u m b e r o f
factors ra n g in g from th e inability o f coauthors to
get access to data to a statistical analysis o f raw data
from the papers suggesting th at the d ata was faked
(S im onsohn, 2013).
Some have looked at the data fabrication crisis
as a serie s o f u n r e la te d a n d iso la te d in c id e n ts,
perhaps driven by personal flaws o r am bition. O th
ers, however, have looked at systemic features in aca
dem ia as a potential influence on this p h e n o m e n a
(Nosek, Spies, & Motyl, 2012). For instance, it has
long been noted that ten u re an d prom otion in aca
dem ia is driven largely by the n u m b er an d quality of
publications (Varian, 1998). Early career research
ers in the field (graduate students, post-docs, an d
assistant faculty) are pressured to publish early and
often, an d this can lead ...
There’s More Than One Way to Conduct a Replication StudyBey.docxrhetttrevannion
There’s More Than One Way to Conduct a Replication Study:
Beyond Statistical Significance
Samantha F. Anderson and Scott E. Maxwell
University of Notre Dame
As the field of psychology struggles to trust published findings, replication research has begun to become
more of a priority to both scientists and journals. With this increasing emphasis placed on reproducibility,
it is essential that replication studies be capable of advancing the field. However, we argue that many
researchers have been only narrowly interpreting the meaning of replication, with studies being designed
with a simple statistically significant or nonsignificant results framework in mind. Although this
interpretation may be desirable in some cases, we develop a variety of additional “replication goals” that
researchers could consider when planning studies. Even if researchers are aware of these goals, we show
that they are rarely used in practice—as results are typically analyzed in a manner only appropriate to a
simple significance test. We discuss each goal conceptually, explain appropriate analysis procedures, and
provide 1 or more examples to illustrate these analyses in practice. We hope that these various goals will
allow researchers to develop a more nuanced understanding of replication that can be flexible enough to
answer the various questions that researchers might seek to understand.
Keywords: replication, data analysis, confidence interval, effect size, equivalence test
Replication, a once largely ignored premise, has recently be-
come a defining precept for the future of psychology. Reproduc-
ibility has been referred to as the “cornerstone” (Simons, 2014, p.
76) and “Supreme Court” (Collins, 1985, p. 19) of science, and as
“the best and possibly the only believable evidence for the reli-
ability of an effect” (Simons, 2014, p. 76). In fact, “findings that
do not replicate are worse than fairy tales” (Wagenmakers, Wet-
zels, Borsboom, van der Maas, & Kievit, 2012, p. 633).
The idea of replication is not new. Even prior to Sir Ronald
Fisher and the advent of modern experimental design (circa 1935),
the field of agriculture used replication to assess accuracy and
reliability (Yates, 1964). In fact, Fisher himself emphasized the
importance of replication, believing that experimental findings are
only established if “a properly designed experiment rarely fails to
give . . . significance” (Fisher, 1926, p. 504). In 1969, Tukey noted
that “confirmation comes from repetition” and that ignoring the
need for replication would “lend[s] itself to failure and more
probably destruction” (Tukey, 1969, p. 84). However, replications
were rarely conducted due to lack of incentive and rarely published
due to lack of novelty (Nosek & Lakens, 2014). This lack of
incentive gradually started to change when concerns about “the
reliability of research findings in the field” began to emerge
(Pashler & Wagenmakers, 2012, p. 528). The field has been amid
“a crisis of confidence” (P.
Malec, T. & Newman, M. (2013). Research methods Building a kn.docxcroysierkathey
Malec, T. & Newman, M. (2013). Research methods: Building a knowledge base. San Diego, CA: Bridgepoint Education, Inc. ISBN-13: 9781621785743, ISBN-10: 1621785742.
Chapter 5: Experimental Designs – Determining Cause-and-Effect Relationships
hapter 5
Experimental Designs—Determining Cause-and-EffectRelationships
Cosmo Condina/Stone/Getty Images
Chapter Contents
· Experiment Terminology
· Key Features of Experiments
· Experimental Validity
· Experimental Designs
· Analyzing Experiments
· Wrap-Up: Avoiding Error
· Critiquing a Quantitative Study
· Mixed Methods Research Designs
One of the oldest debates within psychology concerns the relative contributions that biology and the environment make in shaping ourthoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Dopeople take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? Thereare, in fact, several ways to address these types of questions. In fact, a consortium of researchers at the University of Minnesota has spent thepast 2 decades comparing pairs of identical and fraternal twins to tease apart the contributions of genes and environment. You can read moreat the research group’s website, Minnesota Center for Twin and Family Research, http://mctfr.psych.umn.edu/.
Creatas Images/Thinkstock
Researchers at the University ofMinnesota work with twins in order tostudy the impact of genetics versusupbringing on personality traits.
An alternative to using twin pairs to separate genetic and environmental influence is through the use of experimental designs, which have the primary goal of explaining the causes of behavior. Recall fromChapter 2 (Section 2.1, Overview of Research Designs) that experiments can speak to cause and effectbecause the experimenter has control over the environment and is able to manipulate variables. Oneparticularly ingenious example comes from the laboratory of Michael Meaney, a professor of psychiatryand neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, &Meaney, 1999). Meaney’s research revealed that the parenting ability of female rats can be reliablyclassified based on how attentive they are to their rat pups, as well as how much time they spendgrooming the pups. The question tackled in this study was whether these behaviors were learned fromthe rats’ own mothers or transmitted genetically. To answer this question experimentally, Meaney andcolleagues had to think very carefully about the comparisons they wanted to make. It would have beeninsufficient to simply compare the offspring of good and bad mothers—this approach could notdistinguish between genetic and environmental pathways.
Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they wereborn. This resulted in four combinations of rats: (1) thos ...
Parameter Optimization of Shot Peening Process of PMG AL2024 Alloy CoverIOSRJMCE
Shot peening leads to local plastic deformations in the near-surface regions, which result in the development of compressive residual stress and the improvement of surface hardness in the aerospace structural components. These properties can be enhanced by careful selection of the peening parameters. PMG Cover of AL2024 Aluminum Alloy is widely used in the generator manufacturing cover due to its high specific static strength. In this study a Taguchi Grey Relational Analysis is presented to optimize the surface properties of residual stress, micro hardness. The effects of four peening parameters (Shot Diameter, Shot Velocity, Impact Angle, Nozzle Distance) on micro hardness and residual stress are investigated Design of Experiment work is carried out by MINITAB 14 software tools of Taguchi Grey relational method, for getting excellent shot peening process parameter combination by MAT LAB R2009 software tools of advanced Optimization method as Genetic Algorithm, Simulated Annealing. Compare of the above reading for the investigation.
INVITED EDITORIAL Lets Do It Again A Call for Replications TatianaMajor22
INVITED EDITORIAL: Let's Do It Again: A Call for Replications
in Psi Chi J o u rn a l o f Psychological Research
J o h n E. E d lu n d
R o c h e s te r In s titu te o fT e c h n o lo g y
S
c ie n c e is said to b e su ffe rin g fro m a crisis
o f r e p li c a b i li t y ( I o a n n i d is , 2 0 0 5 ). T h is
crisis o c c u rs w h e n s c ie n tific s tu d ie s fail
to b e s u p p o r te d by s u b s e q u e n t re s e a r c h . T h e
challenges posed by th e replication crisis address
the fundam ental n atu re o f science an d the p ublic’s
u n d e r s t a n d i n g o f it. N u m e r o u s c o n t r i b u t i n g
reasons for th e rep licatio n crisis have b e e n n o ted
in c lu d in g d a ta fa lsific a tio n (S te e n , 2 0 1 1 ), th e
pressures o f ten u re an d p ro m o tio n (Varian, 1998),
questionable research practices (Simmons, Nelson,
& Sim onsohn, 2011), th e tendency o f jo u rn a ls to
w ant to publish particularly novel papers (Steen,
2011), and the p reference for publishing significant
results (de W inter & H appee, 2013). T hese factors
all in c re a se th e o d d s o f in a c c u ra te in fo rm a tio n
b e in g p u b lish e d , w hich in tu rn is in c o rp o r a te d
in to tex ts, as h a p p e n e d w ith th e d e ta ils o f th e
original investigation in th e Kitty G enovese case,
which led to th e fam ous bystander apathy studies
(Griggs, 2015). T he p rim ary goal o f this editorial is
to briefly discuss th e factors th at have co n trib u ted
to th e re p lic atio n crisis, tech n iq u e s em ployed by
various jo u rn a ls in th e field to deal with the crisis,
and how Psi Chi Journal o f Psychological Research (PCf)
is responding.
Build U p to th e R eplication Crisis
Perhaps th e biggest indication o f the psychological
re p lic atio n crisis was a series o f p ap e rs th a t were
com pletely fabricated by several differen t authors
(L e v elt C o m m itte e , 2 0 1 2 ). In th e s e cases, th e
au th o rs in question were discovered to have com
pletely fabricated th e ir d ata based o n a n u m b e r o f
factors ra n g in g from th e inability o f coauthors to
get access to data to a statistical analysis o f raw data
from the papers suggesting th at the d ata was faked
(S im onsohn, 2013).
Some have looked at the data fabrication crisis
as a serie s o f u n r e la te d a n d iso la te d in c id e n ts,
perhaps driven by personal flaws o r am bition. O th
ers, however, have looked at systemic features in aca
dem ia as a potential influence on this p h e n o m e n a
(Nosek, Spies, & Motyl, 2012). For instance, it has
long been noted that ten u re an d prom otion in aca
dem ia is driven largely by the n u m b er an d quality of
publications (Varian, 1998). Early career research
ers in the field (graduate students, post-docs, an d
assistant faculty) are pressured to publish early and
often, an d this can lead ...
There’s More Than One Way to Conduct a Replication StudyBey.docxrhetttrevannion
There’s More Than One Way to Conduct a Replication Study:
Beyond Statistical Significance
Samantha F. Anderson and Scott E. Maxwell
University of Notre Dame
As the field of psychology struggles to trust published findings, replication research has begun to become
more of a priority to both scientists and journals. With this increasing emphasis placed on reproducibility,
it is essential that replication studies be capable of advancing the field. However, we argue that many
researchers have been only narrowly interpreting the meaning of replication, with studies being designed
with a simple statistically significant or nonsignificant results framework in mind. Although this
interpretation may be desirable in some cases, we develop a variety of additional “replication goals” that
researchers could consider when planning studies. Even if researchers are aware of these goals, we show
that they are rarely used in practice—as results are typically analyzed in a manner only appropriate to a
simple significance test. We discuss each goal conceptually, explain appropriate analysis procedures, and
provide 1 or more examples to illustrate these analyses in practice. We hope that these various goals will
allow researchers to develop a more nuanced understanding of replication that can be flexible enough to
answer the various questions that researchers might seek to understand.
Keywords: replication, data analysis, confidence interval, effect size, equivalence test
Replication, a once largely ignored premise, has recently be-
come a defining precept for the future of psychology. Reproduc-
ibility has been referred to as the “cornerstone” (Simons, 2014, p.
76) and “Supreme Court” (Collins, 1985, p. 19) of science, and as
“the best and possibly the only believable evidence for the reli-
ability of an effect” (Simons, 2014, p. 76). In fact, “findings that
do not replicate are worse than fairy tales” (Wagenmakers, Wet-
zels, Borsboom, van der Maas, & Kievit, 2012, p. 633).
The idea of replication is not new. Even prior to Sir Ronald
Fisher and the advent of modern experimental design (circa 1935),
the field of agriculture used replication to assess accuracy and
reliability (Yates, 1964). In fact, Fisher himself emphasized the
importance of replication, believing that experimental findings are
only established if “a properly designed experiment rarely fails to
give . . . significance” (Fisher, 1926, p. 504). In 1969, Tukey noted
that “confirmation comes from repetition” and that ignoring the
need for replication would “lend[s] itself to failure and more
probably destruction” (Tukey, 1969, p. 84). However, replications
were rarely conducted due to lack of incentive and rarely published
due to lack of novelty (Nosek & Lakens, 2014). This lack of
incentive gradually started to change when concerns about “the
reliability of research findings in the field” began to emerge
(Pashler & Wagenmakers, 2012, p. 528). The field has been amid
“a crisis of confidence” (P.
Malec, T. & Newman, M. (2013). Research methods Building a kn.docxcroysierkathey
Malec, T. & Newman, M. (2013). Research methods: Building a knowledge base. San Diego, CA: Bridgepoint Education, Inc. ISBN-13: 9781621785743, ISBN-10: 1621785742.
Chapter 5: Experimental Designs – Determining Cause-and-Effect Relationships
hapter 5
Experimental Designs—Determining Cause-and-EffectRelationships
Cosmo Condina/Stone/Getty Images
Chapter Contents
· Experiment Terminology
· Key Features of Experiments
· Experimental Validity
· Experimental Designs
· Analyzing Experiments
· Wrap-Up: Avoiding Error
· Critiquing a Quantitative Study
· Mixed Methods Research Designs
One of the oldest debates within psychology concerns the relative contributions that biology and the environment make in shaping ourthoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Dopeople take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? Thereare, in fact, several ways to address these types of questions. In fact, a consortium of researchers at the University of Minnesota has spent thepast 2 decades comparing pairs of identical and fraternal twins to tease apart the contributions of genes and environment. You can read moreat the research group’s website, Minnesota Center for Twin and Family Research, http://mctfr.psych.umn.edu/.
Creatas Images/Thinkstock
Researchers at the University ofMinnesota work with twins in order tostudy the impact of genetics versusupbringing on personality traits.
An alternative to using twin pairs to separate genetic and environmental influence is through the use of experimental designs, which have the primary goal of explaining the causes of behavior. Recall fromChapter 2 (Section 2.1, Overview of Research Designs) that experiments can speak to cause and effectbecause the experimenter has control over the environment and is able to manipulate variables. Oneparticularly ingenious example comes from the laboratory of Michael Meaney, a professor of psychiatryand neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, &Meaney, 1999). Meaney’s research revealed that the parenting ability of female rats can be reliablyclassified based on how attentive they are to their rat pups, as well as how much time they spendgrooming the pups. The question tackled in this study was whether these behaviors were learned fromthe rats’ own mothers or transmitted genetically. To answer this question experimentally, Meaney andcolleagues had to think very carefully about the comparisons they wanted to make. It would have beeninsufficient to simply compare the offspring of good and bad mothers—this approach could notdistinguish between genetic and environmental pathways.
Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they wereborn. This resulted in four combinations of rats: (1) thos ...
Tester contribution to Testing Effectiveness. An Empirical ResearchNatalia Juristo
Software verification and validation might be improved by gaining knowledge of the variables influencing the effectiveness of testing techniques. Unless automated, testing is a human-intensive activity. It is not unreasonable to suppose then that the person applying a technique is a variable that affects effectiveness. We have conducted an empirical study on the extent to which tester and technique contribute to the effectiveness of two testing techniques (equivalence partitioning and branch testing). We seed three programs with several faults. Next we examine the theoretical effectiveness of the technique (measured as how likely a subject applying the technique strictly as prescribed is to generate a test case that exercises the fault).
To measure the observed effectiveness, we then conduct an observational study where the techniques are applied by master’s students. The difference between theoretical and observed effectiveness suggests the tester contribution. To determine the nature of this contribution, we conduct a qualitative empirical study in which we ask the students to explain why they do or do not detect each seeded fault. We have found that the human component can reduce or increase testing technique effectiveness. We have also found that testers do not contribute equally to the different techniques.
Software Usability Implications in Requirements and DesignNatalia Juristo
There are so many software products and systems with immature usability that it is for sure that most people have enough frustrating experiences to acknowledge the low level of use that usability strategies, models and methods have in software construction.
However, usability is not at all an extra but a basic for a software system: people productivity and comfort is directly related to the usability of the software they use (in their work or at home) and several quality attribute classifications agree on the importance of considering usability as a quality attribute the seminar will discuss and debunk three myths that stand in the way of the proper incorporation of usability features into software systems. These myths are:
• usability problems can be fixed in the later development stages.
• usability has implications only for the non-functional requirements.
• the general statement of a usability feature (“The system must incorporate the undo feature”) is a sufficient specification.
A pattern-oriented solution that support developers in incorporating usability features into their requirements and designs is presented
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Accelerate your Kubernetes clusters with Varnish Caching
Towards Understanding SE Experiments Replication (ESEM'13 Keynote)
1. Towards Understanding
the Replication of
SE Experiments
Natalia Juristo
Universidad Politecnica de Madrid (Spain)
&
University of Oulu (Finland)
ESEM Conference Baltimore (USA) October 11th, 2013
2. Scope & Terminology
This talk focuses on the replication of
experiments
I will refer to the study whose results we
want to check as the baseline experiment
3. Replication Intuitive Definition
Deliberate repetition of research
procedures in a second investigation for
the purpose of determining if earlier
results can be reproduced
4. Content
Replication & the experimental paradigm
State of replication in ESE: practice & theory
Shedding a bit of light
Purposes for replicating
Replication functions
Some answers
Replication limits
Baseline and replication minimum degree of similarity
Admissible changes
Reproduction of results
Threats to reuse materials
Summary
6. Searching for Regularities
Science does not settle for anecdotes. A
scientific law or theory describes a regular
occurrence in the world
Regularities existing in reality are identified by
reproducing the same event in different
replications
The result of one experiment is an isolated
event
7. One Result, Three Meanings
Without reproduction of results it is impossible
to distinguish whether they
occurred by chance
are artifactual
the event occurs only in the experiment, not in reality
really correspond to a regularity
9. Not Enough Replications
Most SE experiments have not yet been
replicated
Two reviews provide empirical data to
support this point
Let us look at their results
10. Experiments in Leading Journals
& Conferences (1993-2002)
5,453 articles published from 1993 to 2002
in major SE journals and conference
proceedings
113 experiments
20 (17.7%) described as replications
Sjøberg et al. “A survey of
controlled experiments in SE” TSE, 2005
11. All Publications
96 papers reporting replications
133 replications of 72 baseline studies
Any type of empirical study
Quasi-Experiments 35 49%
Controlled Experiments 21 29%
Case Study 15 21%
Survey 1 1%
da Silva et al. “Replication of Empirical Studies in
Software Engineering Research: A Systematic
Mapping Study” EMSE 2013
13. First Paper Published on a
Replication [Da Silva et al. 2012]
In SE, the first article that explicitly reported
a replication of an empirical study was
published in 1994
Daly, Brooks, Miller, Roper & Wood
Verification of Results in Sw Maintenance Through
External Replication
Intl Conf on Software Maintenance
14. EMSE Special Issue on
Replication
The large number of submissions was
admittedly more than we expected
We received a total of 16 submissions
Encouraging the publication of replications
will foster researchers to replicate more
studies
16. First Theoretical Publication on
Replication
In 1999, a paper discussed a framework to
organize sets of related experiments
(families) and the generation of knowledge
from such sets
Basili, Shull & Lanubile. Building knowledge through
families of experiments. TSE
Is a family exactly a set of replications?
…experiments can be viewed as part of common
families of studies, rather than being isolated
events…
17. More Activity in the Last 10 Years
Shull, Basili, Carver, Maldonado, Travassos, Mendonça & Fabbri Replicating
software engineering experiments: Addressing the tacit knowledge problem.
ISESE 2002
Vegas, Juristo, Moreno, Solari & Letelier Analysis of the influence of
communication between researchers on experiment replication. ISESE 2006
Brooks, Roper, Wood, Daly & Miller Replication’s role in software engineering.
Guide to Advanced Empirical SE. Springer 2008
Juristo & Vegas Using differences among replications of software engineering
experiments to gain knowledge. ESEM 2009 [Juristo & Vegas The Role of Non-
Exact Replications in SE EMSE Journal 2011]
Krein & Knutson A Case for replication: Synthesizing research methodologies in
SE. RESER 2010
Gómez, Juristo & Vegas Replications types in experimental disciplines. ESEM
2010
Juristo, Vegas, Solari, Abrahao & Ramos A Process for Managing Interaction
between Experimenters to Get Useful Similar Replications IST 2013
18. State of the Theory
There is no agreement yet on terminology,
typology, purposes, operation and other
replication issues
There is not even agreement on what a
replication is!!
Different authors consider different types of
changes to the baseline experiment as
admissible
19. Example of Divergent Views
Some researchers advise the use of different protocols
and materials to preserve independence and prevent
error propagation in replications by using the same
configuration
Kitchenham
The role of replications in ESE - a word of warning EMSE 2008
Other researchers recommend the reuse of materials to
assure that replications are similar enough for results to
be comparable
Shull, Carver, Vegas & Juristo
The role of replications in ESE EMSE 2008
21. The Two Roles of Replication
Validation
Learning
22. Learning Relevant Conditions
As more replications of Thompson and
McConnell’s baseline experiment were run
different conditions influencing the results of this
experiment were identified
After several hundred experiments had been run
experimenters managed to identify around 70
conditions influencing the behavior of this type of
invertebrate
23. Which are the Important
Variables?
“…In fact, the principle of Transversely
Excited Atmospheric (TEA) lasers,
scientists did not know that the inductance
of the top was important”
A physicist quoted in
Changing Order: Replication and Induction in Scientific Practice
Harry Collins 1992
24. First Learn, Then Validate
“In the early stages, failure to get the expected
results is not falsification but a step in the
discovery of some interfering factor.
For immature experimental knowledge, the first
step is … to find out which experimental
conditions should be controlled”
Validity and the Research Process
Brinberg and McGrath 1985
25. SE Problems with Identical
Replications
SE has tried to repeat experiments identically,
but no exact replications have yet been
achieved
The complexity of the software development
setting prevents the many experimental
conditions from being reproduced identically
Yet this is a regular rather than an exceptional
situation
26. In the Beginning Most is Unkown
“Most aspects are unknown when we start
to study a phenomenon experimentally.
Even the tiniest change in a replication
can lead to inexplicable differences in the
results”
Validity and the Research Process
Brinberg and McGrath 1985
27. Start with Similar Replications
“The less that is known about an area the more
power a very similar experiment has ... This is
because, in the absence of a well worked out set
of crucial variables, any change in the
experiment configuration, however trivial in
appearance, may well entail invisible but
significant changes in conditions”
Changing Order:
Replication and Induction in Scientific Practice
Harry Collins 1992
28. Learning & Validation Process
1. Start with identical replications
At the beginning of experimental research, equality, even if
targeted, will not happen
There will be either invisible but significant changes in conditions or
induced changes due to context adaptation or both
Failure to get the expected results should not be construed as
falsification, but as a step towards the discovery of some new
factor
2. Later on, both knowledge discovery and testing
can be more systematic
Changes in the configuration will be made purposely to learn
more variables and rule out artifactual results
29. Learning is Even More Important
Replication is needed not merely to
validate one’s findings, but more
importantly, to establish the increasing
range of radically different conditions
under which the findings hold, and the
predictable exceptions
The design of replicated studies. American Statistician
Lindsay and Ehrenberg 1993
32. Verification Functions
Control experimental errors
Control protocol independence
Understand operationalization limits
Understand population limits
33. Control Experimental Errors
Verify that the results of the baseline experiment
are not a chance product of an error
All elements of the experiment must resemble
the baseline experiment as closely as possible
Collateral benefit
Provide an understanding of the natural (random)
variation of the observed results
critical for being able to decide whether or not results hold in
dissimilar replications
34. Control Protocol Independence
Verify that the results of the baseline experiment
are not artifactual
An artifactual result is due to the experimental
configuration and cannot be guaranteed to exist in
reality
The experimental protocol needs to be changed
for this purpose
If an experiment is replicated several times using the
same materials, the observed results may occur due
to the materials
The same applies for all protocol elements
35. Understanding Operationalization
Limits
Learn how sensitive results are to different
operationalizations
Treatment operationalizations
treatment application procedures, treatment
instructions, resources, treatment
transmission …
Effect operationalizations
Metrics, measurement procedures …
36. Understand Population Limits
Learn the extent to which results hold for
other subject types or other types of
experimental objects
Learn to which specific population the
experimental sample belongs and what
the characteristics of such population are
37. Changes & Replication Functions
Experimental
Configuration
Control
Experimental
Error
Control Protocol
Independence
Understand
Operationalization
Limits
Understand
Population Limits
Operationalization = = ≠ =
Population = = = ≠
Protocol = ≠ = =
Function of Replication
LEGEND: = the element is equal to, or as similar as possible to, the baseline experiment
≠ the element varies with respect to the baseline experiment
40. Establishing Limits
Amount of Changes
Run
Same data
different
models or
statistical
methods
Identical
same site &
researchers
+
Protocol
changes
Operationa-
lization
changes
Population
changes
Just the
hypothesis
kept
RE-ANALYSIS REPETITION REPLICATION REPRODUCTION
Not Run
41. Question 2
What level of similarity should an
experiment have to be
considered a replication rather
than a new experiment?
42. Unchanged elements of a
replication
A replication must share the hypothesis
with the baseline experiment
Same response variable
although not same metric
Same treatments
although not same operationalization
at least two treatments in common
43. Partials Replications
Exp. A (Baseline OR Replication)
Exp. B (Replication OR Baseline)
Exp. D
Replication C
RV3
RV2
RV1
T5
T4
T3
T2
T6
T1
45. Levels of Verification
Similarity between the baseline
experiment and a replication serve
different verification purposes depending
on the changes made
Replicating an experiment as closely as possible
Verifies results are not accidental
Varying the experimental protocol
Verifies results are not artifactual
Varying the population properties
Verifies types of populations for which the results hold
Varying the operationalization
Verifies range of operationalizations for which the results hold
46. Can I Change Everything?
It is better not to change everything at the
same time
We can understand the source of differences
in results better if only one change is made at
a time
But a replication with a lot of changes is
not rendered useless or doomed to failure
We will just need to wait until other
replications are run
50. Question 4
What is the level of similarity that
results must have to be
considered as reproduced?
51. Understanding the Natural
Variation
Identical replications are useful for
understanding the range of variability of
results
This provides an estimate for other
experimenters to use as a baseline when
they replicate the experiment
53. Accomodating Opposing Views
The possible threat of errors being propagated
by experimenters exchanging materials does not
mean discarding replications sharing materials
Replications with identical materials and protocols
(and possibly the same errors) are a necessary step
for verifying that an exact replication run by others
reproduces the same results
Other replications that alter the design and other
protocol details should be performed in order to
assure that the results are not induced by the protocol
54. Accomodating Opposing Views
Replication functions accommodate
opposing views within a broader
framework
Contrary stances are really tantamount to
different types of replication conducted for
different purposes
Different ways of running replications are
useful for gradually advancing towards
verified experimental results
56. Main Ideas
Replication in ESE
Replication plays an essential role in the experimental paradigm
Replication is not a regular practice in ESE today
More methodological research on the adoption and tailoring of
replication in ESE is still necessary
Clarifying conceptions
Replication is necessary not merely to validate findings, but,
more importantly, to discover the range of conditions under
which the findings hold
Replications provide different knowledge depending on the
changes to the baseline experiment
Knowledge gained from a replication needs to relate changes
and findings
57. Towards Understanding
the Replication of
SE Experiments
Natalia Juristo
Universidad Politecnica de Madrid (Spain)
&
University of Oulu (Finland)
ESEM Conference Baltimore (USA) October 11th, 2013