The document discusses issues with reproducibility and replication in biomedical research. It notes decades of failed translational stroke research due to biases, low statistical power, and other issues. It argues that both exploration and confirmation are needed, but they are often conflated. Exploration aims to generate new theories while confirmation aims to demonstrate treatment effects. The document provides suggestions to reduce biases, increase power, practice open science through preregistration and data sharing, and consider effect sizes over sole reliance on p-values. Failure to replicate may be due to cuting-edge research pushing boundaries rather than a false positive original result. Both replication and non-replication can provide valuable scientific insights when properly interpreted.
An analysis of the Quantitative and Qualitative approaches and collection of data in terms of transport facility design was conducted for the "Transport Facility Design" module during semester 3 of the bachelors (Hons) degree program in Transport and Logistics Management at University of Moratuwa.
How to combine results from randomised clinical trials on the additive scale with real world data to provide predictions on the clinically relevant scale for individual patients
An analysis of the Quantitative and Qualitative approaches and collection of data in terms of transport facility design was conducted for the "Transport Facility Design" module during semester 3 of the bachelors (Hons) degree program in Transport and Logistics Management at University of Moratuwa.
How to combine results from randomised clinical trials on the additive scale with real world data to provide predictions on the clinically relevant scale for individual patients
Reproducibility, argument and data in translational medicineTim Clark
Failures in reproducibility and robustness of scientific findings are explored from statistical, historical, and argumentation theory perspectives. The impact of false positives in the literature is connected to failures in T1 and T2 biomedical translation, and is shown to have a significant impact on the costs of therapeutic development and availability of needed treatments to the public. Technological and social approaches to resolve these issues are presented. "Reproducibility" initiatives are critiqued as unsustainable and non-authoritative; improved requirements and methods for scientific communication of findings including data, methods and material are supported as the best approaches for improved reproducibility.
Dichotomania and other challenges for the collaborating biostatisticianLaure Wynants
Conference presentation at ISCB 41 in the session
"Biostatistical inference in practice: moving beyond false
dichotomies"
A comment in Nature, signed by over 800 researchers, called for the scientific community to “retire statistical significance”. The responses included a call to halt the use of the term „statistically significant”, and changes in journal’s author guidelines. The leading discourse among statisticians is that inadequate statistical training of clinical researchers and publishing practices are to blame for the misuse of statistical testing. In this presentation, we search our collective conscience by reviewing ethical guidelines for statisticians in light of the p-value crisis, examine what this implies for us when conducting analyses in collaborative work and teaching, and whether the ATOM (accept uncertainty; be thoughtful, open and modest) principles can guide us.
Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelin...John Hoey
Much published health sciences literature is misleading and biased
Efforts to correct this include use of reporting guidelines- criteria for doing science and reporting the results properly
Also discussion of conflicts of interest - how to report them.
What's the Science in Data Science? - Skipper SeaboldPyData
The gold standard for validating any scientific assumption is to run an experiment. Data science isn’t any different. Unfortunately, it’s not always possible to design the perfect experiment. In this talk, we’ll take a realistic look at measurement using tools from the social sciences to conduct quasi-experiments with observational data.
D. Mayo: Replication Research Under an Error Statistical Philosophy jemille6
D. Mayo (Virginia Tech) slides from her talk June 3 at the "Preconference Workshop on Replication in the Sciences" at the 2015 Society for Philosophy and Psychology meeting.
This lecture looks at:
- An explanation of each of the steps in the research process flowchart
- Types of data
- Generating and testing theories
- Measurement error
- Validity
- Reliability
A comment in Nature, signed by over 800 researchers, called for a rise up against statistical significance. This was followed by a special issue of The American Statistician aimed at halting the use of the term "statistically significant", and new guidelines for statistical reporting in the New England Journal of Medicine. These slides discuss the broader context of the "p-value crisis" and alternatives for communicating the conclusions after statistical analyses.
Target audience: Medical researchers; Scientists involved in conducting or interpreting analyses and communicating the results of scientific research, as well as readers of scientific publications.
Learning objectives:
To understand the context of the reproducibility crisis in medical research.
To learn about problems with p-values and alternatives to report findings.
To understand how (not) to interpret significant and insignificant findings.
To learn how to communicate research findings in a modest, thoughtful, and transparent way.
Clinical trials are the gold standard of evidence-based medicine. Properly designed clinical trials can lead to chance findings and potentially lead to erroneous conclusions.
Importantly, clinical trials can also be badly designed on purpose to increase the risk of false or chance findings leading to support misleading claims. Such techniques are frequently used by bad researchers and charlatans to substantiate their claims with biased clinical trials. It is therefore important to be weary of the limitations of clinical trials and understand how causal inference should be approach. In that presentation, I discuss the situations under which the risk of erroneous conclusions from clinical trials is increased and I discuss ways to identify and prevent bad clinical research.
The views expressed and presented in that presentation are my own views and may not represent the views of the National Institute for Health and Care Excellence.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.Sérgio Sacani
The return of a sample of near-surface atmosphere from Mars would facilitate answers to several first-order science questions surrounding the formation and evolution of the planet. One of the important aspects of terrestrial planet formation in general is the role that primary atmospheres played in influencing the chemistry and structure of the planets and their antecedents. Studies of the martian atmosphere can be used to investigate the role of a primary atmosphere in its history. Atmosphere samples would also inform our understanding of the near-surface chemistry of the planet, and ultimately the prospects for life. High-precision isotopic analyses of constituent gases are needed to address these questions, requiring that the analyses are made on returned samples rather than in situ.
Slide 1: Title Slide
Extrachromosomal Inheritance
Slide 2: Introduction to Extrachromosomal Inheritance
Definition: Extrachromosomal inheritance refers to the transmission of genetic material that is not found within the nucleus.
Key Components: Involves genes located in mitochondria, chloroplasts, and plasmids.
Slide 3: Mitochondrial Inheritance
Mitochondria: Organelles responsible for energy production.
Mitochondrial DNA (mtDNA): Circular DNA molecule found in mitochondria.
Inheritance Pattern: Maternally inherited, meaning it is passed from mothers to all their offspring.
Diseases: Examples include Leber’s hereditary optic neuropathy (LHON) and mitochondrial myopathy.
Slide 4: Chloroplast Inheritance
Chloroplasts: Organelles responsible for photosynthesis in plants.
Chloroplast DNA (cpDNA): Circular DNA molecule found in chloroplasts.
Inheritance Pattern: Often maternally inherited in most plants, but can vary in some species.
Examples: Variegation in plants, where leaf color patterns are determined by chloroplast DNA.
Slide 5: Plasmid Inheritance
Plasmids: Small, circular DNA molecules found in bacteria and some eukaryotes.
Features: Can carry antibiotic resistance genes and can be transferred between cells through processes like conjugation.
Significance: Important in biotechnology for gene cloning and genetic engineering.
Slide 6: Mechanisms of Extrachromosomal Inheritance
Non-Mendelian Patterns: Do not follow Mendel’s laws of inheritance.
Cytoplasmic Segregation: During cell division, organelles like mitochondria and chloroplasts are randomly distributed to daughter cells.
Heteroplasmy: Presence of more than one type of organellar genome within a cell, leading to variation in expression.
Slide 7: Examples of Extrachromosomal Inheritance
Four O’clock Plant (Mirabilis jalapa): Shows variegated leaves due to different cpDNA in leaf cells.
Petite Mutants in Yeast: Result from mutations in mitochondrial DNA affecting respiration.
Slide 8: Importance of Extrachromosomal Inheritance
Evolution: Provides insight into the evolution of eukaryotic cells.
Medicine: Understanding mitochondrial inheritance helps in diagnosing and treating mitochondrial diseases.
Agriculture: Chloroplast inheritance can be used in plant breeding and genetic modification.
Slide 9: Recent Research and Advances
Gene Editing: Techniques like CRISPR-Cas9 are being used to edit mitochondrial and chloroplast DNA.
Therapies: Development of mitochondrial replacement therapy (MRT) for preventing mitochondrial diseases.
Slide 10: Conclusion
Summary: Extrachromosomal inheritance involves the transmission of genetic material outside the nucleus and plays a crucial role in genetics, medicine, and biotechnology.
Future Directions: Continued research and technological advancements hold promise for new treatments and applications.
Slide 11: Questions and Discussion
Invite Audience: Open the floor for any questions or further discussion on the topic.
More Related Content
Similar to Excursions into the garden of the forking paths
Reproducibility, argument and data in translational medicineTim Clark
Failures in reproducibility and robustness of scientific findings are explored from statistical, historical, and argumentation theory perspectives. The impact of false positives in the literature is connected to failures in T1 and T2 biomedical translation, and is shown to have a significant impact on the costs of therapeutic development and availability of needed treatments to the public. Technological and social approaches to resolve these issues are presented. "Reproducibility" initiatives are critiqued as unsustainable and non-authoritative; improved requirements and methods for scientific communication of findings including data, methods and material are supported as the best approaches for improved reproducibility.
Dichotomania and other challenges for the collaborating biostatisticianLaure Wynants
Conference presentation at ISCB 41 in the session
"Biostatistical inference in practice: moving beyond false
dichotomies"
A comment in Nature, signed by over 800 researchers, called for the scientific community to “retire statistical significance”. The responses included a call to halt the use of the term „statistically significant”, and changes in journal’s author guidelines. The leading discourse among statisticians is that inadequate statistical training of clinical researchers and publishing practices are to blame for the misuse of statistical testing. In this presentation, we search our collective conscience by reviewing ethical guidelines for statisticians in light of the p-value crisis, examine what this implies for us when conducting analyses in collaborative work and teaching, and whether the ATOM (accept uncertainty; be thoughtful, open and modest) principles can guide us.
Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelin...John Hoey
Much published health sciences literature is misleading and biased
Efforts to correct this include use of reporting guidelines- criteria for doing science and reporting the results properly
Also discussion of conflicts of interest - how to report them.
What's the Science in Data Science? - Skipper SeaboldPyData
The gold standard for validating any scientific assumption is to run an experiment. Data science isn’t any different. Unfortunately, it’s not always possible to design the perfect experiment. In this talk, we’ll take a realistic look at measurement using tools from the social sciences to conduct quasi-experiments with observational data.
D. Mayo: Replication Research Under an Error Statistical Philosophy jemille6
D. Mayo (Virginia Tech) slides from her talk June 3 at the "Preconference Workshop on Replication in the Sciences" at the 2015 Society for Philosophy and Psychology meeting.
This lecture looks at:
- An explanation of each of the steps in the research process flowchart
- Types of data
- Generating and testing theories
- Measurement error
- Validity
- Reliability
A comment in Nature, signed by over 800 researchers, called for a rise up against statistical significance. This was followed by a special issue of The American Statistician aimed at halting the use of the term "statistically significant", and new guidelines for statistical reporting in the New England Journal of Medicine. These slides discuss the broader context of the "p-value crisis" and alternatives for communicating the conclusions after statistical analyses.
Target audience: Medical researchers; Scientists involved in conducting or interpreting analyses and communicating the results of scientific research, as well as readers of scientific publications.
Learning objectives:
To understand the context of the reproducibility crisis in medical research.
To learn about problems with p-values and alternatives to report findings.
To understand how (not) to interpret significant and insignificant findings.
To learn how to communicate research findings in a modest, thoughtful, and transparent way.
Clinical trials are the gold standard of evidence-based medicine. Properly designed clinical trials can lead to chance findings and potentially lead to erroneous conclusions.
Importantly, clinical trials can also be badly designed on purpose to increase the risk of false or chance findings leading to support misleading claims. Such techniques are frequently used by bad researchers and charlatans to substantiate their claims with biased clinical trials. It is therefore important to be weary of the limitations of clinical trials and understand how causal inference should be approach. In that presentation, I discuss the situations under which the risk of erroneous conclusions from clinical trials is increased and I discuss ways to identify and prevent bad clinical research.
The views expressed and presented in that presentation are my own views and may not represent the views of the National Institute for Health and Care Excellence.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.Sérgio Sacani
The return of a sample of near-surface atmosphere from Mars would facilitate answers to several first-order science questions surrounding the formation and evolution of the planet. One of the important aspects of terrestrial planet formation in general is the role that primary atmospheres played in influencing the chemistry and structure of the planets and their antecedents. Studies of the martian atmosphere can be used to investigate the role of a primary atmosphere in its history. Atmosphere samples would also inform our understanding of the near-surface chemistry of the planet, and ultimately the prospects for life. High-precision isotopic analyses of constituent gases are needed to address these questions, requiring that the analyses are made on returned samples rather than in situ.
Slide 1: Title Slide
Extrachromosomal Inheritance
Slide 2: Introduction to Extrachromosomal Inheritance
Definition: Extrachromosomal inheritance refers to the transmission of genetic material that is not found within the nucleus.
Key Components: Involves genes located in mitochondria, chloroplasts, and plasmids.
Slide 3: Mitochondrial Inheritance
Mitochondria: Organelles responsible for energy production.
Mitochondrial DNA (mtDNA): Circular DNA molecule found in mitochondria.
Inheritance Pattern: Maternally inherited, meaning it is passed from mothers to all their offspring.
Diseases: Examples include Leber’s hereditary optic neuropathy (LHON) and mitochondrial myopathy.
Slide 4: Chloroplast Inheritance
Chloroplasts: Organelles responsible for photosynthesis in plants.
Chloroplast DNA (cpDNA): Circular DNA molecule found in chloroplasts.
Inheritance Pattern: Often maternally inherited in most plants, but can vary in some species.
Examples: Variegation in plants, where leaf color patterns are determined by chloroplast DNA.
Slide 5: Plasmid Inheritance
Plasmids: Small, circular DNA molecules found in bacteria and some eukaryotes.
Features: Can carry antibiotic resistance genes and can be transferred between cells through processes like conjugation.
Significance: Important in biotechnology for gene cloning and genetic engineering.
Slide 6: Mechanisms of Extrachromosomal Inheritance
Non-Mendelian Patterns: Do not follow Mendel’s laws of inheritance.
Cytoplasmic Segregation: During cell division, organelles like mitochondria and chloroplasts are randomly distributed to daughter cells.
Heteroplasmy: Presence of more than one type of organellar genome within a cell, leading to variation in expression.
Slide 7: Examples of Extrachromosomal Inheritance
Four O’clock Plant (Mirabilis jalapa): Shows variegated leaves due to different cpDNA in leaf cells.
Petite Mutants in Yeast: Result from mutations in mitochondrial DNA affecting respiration.
Slide 8: Importance of Extrachromosomal Inheritance
Evolution: Provides insight into the evolution of eukaryotic cells.
Medicine: Understanding mitochondrial inheritance helps in diagnosing and treating mitochondrial diseases.
Agriculture: Chloroplast inheritance can be used in plant breeding and genetic modification.
Slide 9: Recent Research and Advances
Gene Editing: Techniques like CRISPR-Cas9 are being used to edit mitochondrial and chloroplast DNA.
Therapies: Development of mitochondrial replacement therapy (MRT) for preventing mitochondrial diseases.
Slide 10: Conclusion
Summary: Extrachromosomal inheritance involves the transmission of genetic material outside the nucleus and plays a crucial role in genetics, medicine, and biotechnology.
Future Directions: Continued research and technological advancements hold promise for new treatments and applications.
Slide 11: Questions and Discussion
Invite Audience: Open the floor for any questions or further discussion on the topic.
Earliest Galaxies in the JADES Origins Field: Luminosity Function and Cosmic ...Sérgio Sacani
We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest
imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters
spanning 0.4−0.9µm) and novel JWST images with 14 filters spanning 0.8−5µm, including 7 mediumband filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data
at > 2.3µm to construct an ultradeep image, reaching as deep as ≈ 31.4 AB mag in the stack and
30.3-31.0 AB mag (5σ, r = 0.1” circular aperture) in individual filters. We measure photometric
redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts
z = 11.5 − 15. These objects show compact half-light radii of R1/2 ∼ 50 − 200pc, stellar masses of
M⋆ ∼ 107−108M⊙, and star-formation rates of SFR ∼ 0.1−1 M⊙ yr−1
. Our search finds no candidates
at 15 < z < 20, placing upper limits at these redshifts. We develop a forward modeling approach to
infer the properties of the evolving luminosity function without binning in redshift or luminosity that
marginalizes over the photometric redshift uncertainty of our candidate galaxies and incorporates the
impact of non-detections. We find a z = 12 luminosity function in good agreement with prior results,
and that the luminosity function normalization and UV luminosity density decline by a factor of ∼ 2.5
from z = 12 to z = 14. We discuss the possible implications of our results in the context of theoretical
models for evolution of the dark matter halo mass function.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
Richard's aventures in two entangled wonderlandsRichard Gill
Since the loophole-free Bell experiments of 2020 and the Nobel prizes in physics of 2022, critics of Bell's work have retreated to the fortress of super-determinism. Now, super-determinism is a derogatory word - it just means "determinism". Palmer, Hance and Hossenfelder argue that quantum mechanics and determinism are not incompatible, using a sophisticated mathematical construction based on a subtle thinning of allowed states and measurements in quantum mechanics, such that what is left appears to make Bell's argument fail, without altering the empirical predictions of quantum mechanics. I think however that it is a smoke screen, and the slogan "lost in math" comes to my mind. I will discuss some other recent disproofs of Bell's theorem using the language of causality based on causal graphs. Causal thinking is also central to law and justice. I will mention surprising connections to my work on serial killer nurse cases, in particular the Dutch case of Lucia de Berk and the current UK case of Lucy Letby.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
1. EXCURSIONS INTO THE GARDEN OF THE
FORKING PATHS
P-VALUE FETISHISATION, REPLICATION CRISIS, AND THE
TENSION BETWEEN INNOVATION AND CONFIRMATION
http://bit.ly/helmholtzdirnagl
2. Personal motivation:
Decades of futile translational stroke research
• Millions of animals killed
• Hundreds (thousands?) of neutral or
negative clinical trials
• Thousands of researchers and
clinicians globally
• Many billions spent on preclinical
research ?
3. Take home I:
The garden of the forking paths
http://bit.ly/2q2gtXqhttp://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
http://bit.ly/2JzblTR
4. Take home II
No scientific progress without reproducibility failures
To boldly go where no man…
Exploration at low base rate
Innovation
‚Paradigm shift‘
Incompetence
Bad designs
Tacit knowledge (bad reporting)
Low validity (bias)
Misconduct
The Good The Bad
Essential non-reproducibility
(Kuhn)
Detrimental non-reproducibility
(Popper)
5. Taken home III
Confirmation – weeding out the false positives of exploration
Jonathan
Kimmelman
PLoS Biol. (2014) 12:e1001863.
7. Modfied after Gary Larson
Bias: Subjective reality informed by ones preferences
8. Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research:
A Focus for Improvement. PLoS Biol 13: e1002273.
Low prevalence of methods to prevent bias
9. Alzheimer's disease models
models
Blinded conduct of
experiment
Blinded assessment
of outcome
Blinded assessment of
outcome
Stroke models (NXY-095)
Blinded assessment of behavioural outcome
No Yes
Improvementinbehaviouraloutcome
(StandardisedEffectSize)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Blinded assessment of behavioural outcome
No Yes
Improvementinbehaviouraloutcome
(StandardisedEffectSize) 0.0
0.2
0.4
0.6
0.8
1.0
1.2
Blinded assessment of behav
No
Improvementinbehaviouraloutcome
(StandardisedEffectSize)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Reductionininfarctsize
Reductionininfarctsize
> 30 studies > 500 animals
Bias inflates effect sizes
20. In exploratory investigation, researchers
should aim at generating robust
pathophysiological theories of disease.
Currently we often see a mixup of both modes. This prevents us
from tailoring our study designs accordingly.
In confirmatory investigation, researchers
should aim at demonstrating strong and
reproducible treatment effects in relevant
animal models.
Exploration vs Confirmation
21. Exploratory Confirmatory
Hypothesis (+) +++
Establish pathophysiology +++ (+)
Sequence and details of experiments established
at onset
(+) +++
Primary endpoint - ++
Sample size calculation (+) +++
Blinding +++ +++
Randomization +++ +++
External validity (aging, comorbidities, etc.) - ++
In/Exclusion criteria ++ +++
Test statistics + +++
Preregistration (-) +++
Sensitivity (Type II error) Find what might work ++ +
Specificity (Type I error) Weed out false positives + +++
Stroke 2016; 47:2148-2153
24. ‚ .. non-reproducible single occurrences
are of no significance to science …‘
The Logic of Scientific Discovery (1934)
Sir Karl Popper
(1902-1994)
‘We do not take even our own
observations quite seriously, or accept
them as scientific observations, until we
have repeated and tested them. Only by
such repetitions can we convince ourselves
that we are not dealing with a mere
isolated ‘coincidence’, but with events
which, on account of their regularity and
reproducibility, are in principle inter-
subjectively testable.’
25. The lexicon of reproducibility
Methods reproducibility: Same data, same tools, same
results? Adds no additional evidence!
Results reproducibility (aka „replication“): Technically
competent repetition, i.e. a new study. Could be strict:
identical conditions: or conceptual: altered conditions (does
causal claim extend to previously unsampled settings?)
Inferential reproducibility: Same conclusions from study
replication or re-analysis? Not all scientists come to the
same conclusions from same results, or may make different
analytic choices. What is concluded or recommended from
a study is often the only thing that matters!
Adapted from Goodman et al. Sci Transl Med. 2016;8:341ps12.
26. What do we mean by 'reproducible'?
Significance and P values: Evaluating replication effect against null
hypothesis of no effect
Evaluating replication effect against original effect size: Is the
original effect size within the 95% CI of the effect size estimate
from the replication. Alternatively: Comparing original and
replication effect sizes
Meta-analysis combining original and replication effects:
Combining original and replication effect sizes for cumulative
evidence
Subjective assessment of “Did it replicate?”
From the Open Science Collaboration, Psychology Replication, Science. 2015 ;349(6251):aac4716
28. The emptiness of failed replication (?)
Mitchell J (2014) On the evidentiary evidence of failed replication
http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_science_2014.pdf
29. The emptiness of failed replication
Does a failure to replicate mean that the original
result was a false positive? Or was the failed
replication a false negative?
Does successful replication mean that the original
result was correct? Or are both results false positives?
30. Hidden moderators - Contextual
sensitivity – Tacit knowledge
‚We analyzed 100 replication attempts in psychology and found that the
extent to which the research topic was likely to be contextually sensitive
(varying in time, culture, or location) was associated with replication
success. This relationship remained a significant predictor of replication
success even after adjusting for characteristics of the original and
replication studies that previously had been associated with replication
success (e.g., effect size, statistical power).‘
Proc Natl Acad Sci. 2016;113:6454-9.
32. p = 0.049 (p< α = 0.05)
Assume that the experimental result is correct, i.e.
measured difference equals (unknown) treatment effect.
Repeat experiment under identical conditions (i.e. 'strict
replication').
What is the probability to reproduce the significant
findings?
50 %!
How likely is strict replication ?
33. Replication failure as an indicator of
cutting edge research?
Dirnagl (2017) How likely are your hypotheses, really?
https://dirnagl.com/2017/04/13/how-original-are-your-scientific-hypotheses-really/
34.
35. The garden of the forking paths
http://bit.ly/2q2gtXqhttp://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
http://bit.ly/2JzblTR
37. Resolving the tension:
Discovery & Replication
Suggested reading:
Wagenmakers EJ, Dutilh G, Sarafoglou A.
Perspect Psychol Sci. 2018 Jul;13(4):418-427
Chang and Eng Bunker circa 1865. Foto Hulton/Getty
38. No scientific progress without
nonreproducibility
To boldly go where no man…
Exploration at low base rate
Innovation
‚Paradigm shift‘
Incompetence
Bad designs
Tacit knowledge (bad reporting)
Low validity (bias)
Misconduct
The Good The Bad
Essential non-reproducibility
(Kuhn)
Detrimental non-reproducibility
(Popper)
39. Reduce Bias!
Use blinding, randomization,in/exclusion criteria.
Report results according to guidelines (e.g. ARRIVE).
Increase Power!
Check your power. Achieve at least 80%.
Do apriori sample size calculations.
Probably you need to increase n‘s.
Replicate.
Use statistics sensibly!
P-values do not provide evidence regarding a model or hypothesis.
Test statistics are overrated (and overused) in exploration.
Think biological significance, think effect size.
Replicate.
Practice Open Science
Preregister.
Publish NULL results.
Make the original data available.
Don’t get lost in the garden of the forking paths
Lots of potential biases are lurking which may impact on our experimental results. From a very recent paper study from Malcolm's group we know that randomization, blinding, and conflict of interest statements are still not as prevalent as we might hope, in fact in preclinical stroke research below 40 %. I should mention that stroke research is not doing any worse than other fields in neuroscience, for example in MS research.
9
This is particularly true when group sizes are small. This is in fact the case in experimental stroke research....
let me explain
Would this have any practical consequence? How different are these designs?
Kant was perhaps the first to realize that the objectivity of scientific statements is closely connected with the construction of theories — with the use of hypotheses and universal statements. Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested — in principle — by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’, but with events which, on account of their regularity and reproducibility, are in principle inter-subjectively testable.
Every experimental physicist knows those surprising and inexplicable apparent ‘effects’ which in his laboratory can perhaps even be reproduced for some time, but which finally disappear without trace. Of course, no physicist would say in such a case that he had made a scientific discovery (though he might try to rearrange his experiments so as to make the effect reproducible). Indeed the scientifically significant physical effect may be defined as that which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed. No serious physicist would offer for publication, as a scientific discovery, any such ‘occult effect,’ as I propose to call it — one for whose reproduction he could give no instructions. The ‘discovery’ would be only too soon rejected as chimerical, simply because attempts to test it would lead to negative results. (It follows that any controversy over the question whether events which are in principle unrepeatable and unique ever do occur cannot be decided by science: it would be a metaphysical controversy.)
– Karl Popper (1959/2002), The Logic of Scientific Discovery, pp. 23-24.
• Recent hand-wringing over failed replications in social psychology is largely
pointless, because unsuccessful experiments have no meaningful scientific
value.
• Because experiments can be undermined by a vast number of practical mistakes,
the likeliest explanation for any failed replication will always be that the replicator
bungled something along the way. Unless direct replications are conducted
by flawless experimenters, nothing interesting can be learned from them.
• Three standard rejoinders to this critique are considered and rejected. Despite
claims to the contrary, failed replications do not provide meaningful
information if they closely follow original methodology; they do not necessarily
identify effects that may be too small or flimsy to be worth studying; and they
cannot contribute to a cumulative understanding of scientific phenomena.
• Replication efforts appear to reflect strong prior expectations that published
findings are not reliable, and as such, do not constitute scientific output.
• The field of social psychology can be improved, but not by the publication of
negative findings. Experimenters should be encouraged to restrict their
“degrees of freedom,” for example, by specifying designs in advance.
• Whether they mean to or not, authors and editors of failed replications are
publicly impugning the scientific integrity of their colleagues. Targets of failed
replications are justifiably upset, particularly given the inadequate basis for
replicators’ extraordinary claims.
power irrelevant, as experiment reproduced under identical conditions
ARTICLE
Switch to Standard View
mTORC1 Inactivation Promotes Colitis-Induced Colorectal Cancer but Protects from APC Loss-Dependent Tumorigenesis
Marta Brandt
, Tatiana P. Grazioso
, Mohamad-Ali Fawal
, Krishna S. Tummala
, Raul Torres-Ruiz
, Sandra Rodriguez-Perales
, Cristian Perna
, Nabil Djouder4,Correspondence information about the author Nabil DjouderEmail the author Nabil Djouder
4Lead Contact
PlumX Metrics
DOI: http://dx.doi.org/10.1016/j.cmet.2017.11.006 |
Article Info
The Amazing American Story of the Original Siamese Twins
Few newcomers to the U.S. have crossed more daunting barriers than Chang and Eng Bunker