The document discusses several case studies involving predicting cancer patient response to chemotherapy using gene expression signatures. It summarizes that researchers claimed they could predict drug response using microarray data from cell lines, but upon external analysis several issues were found, such as gene lists not matching and poor performance on test data. The document examines multiple studies that claimed to validate the approach but found inconsistencies in results and undocumented changes to data and methods. It emphasizes the importance of reproducibility in high-throughput studies and encourages readers to examine the raw data themselves.
IntelliManage provides a unique service that helps shorten the duration of projects for companies managing a wide range of technology, engineering and organizational projects.
IntelliManage מספקת שירות ייחודי של קיצור משך הזמן של פרויקטים, עבור חברות המנהלות פרוייקטים טכנולוגיים, הנדסיים וארגוניים בקשת תחומים רחבה.
DEZBATERILE GRUPULUI DE REFLECȚIE PRIVIND DEMOCRATIA REALĂEmanuel Pope
A apărut al doilea caiet din ANALELE GRUPULUI DE REFLECȚIE PRIVIND DEMOCRAȚIA REALĂ care poate fi citit si descărcat accesând situl grupului din portalul internet www.cartesiarte.ro
Caietul nr. 10 al Grupului de reflecție privind problemele democrației reale Emanuel Pope
Vă informăm că in spațiul de dezbateri „PLURALITAS” de pe prima pagină a portalului ingternet www.cartesiarte.ro a apărut Caietul nr. 10 al Grupului de reflecție privind problemele democrației reale pe care vi-l prezentăm în atașament
A presentation about war and the severe damages , how un human can war be .
Be the change you want to see in the world.
Mahatma Ghandi
-Made by :Shahd Hamouri
IntelliManage provides a unique service that helps shorten the duration of projects for companies managing a wide range of technology, engineering and organizational projects.
IntelliManage מספקת שירות ייחודי של קיצור משך הזמן של פרויקטים, עבור חברות המנהלות פרוייקטים טכנולוגיים, הנדסיים וארגוניים בקשת תחומים רחבה.
DEZBATERILE GRUPULUI DE REFLECȚIE PRIVIND DEMOCRATIA REALĂEmanuel Pope
A apărut al doilea caiet din ANALELE GRUPULUI DE REFLECȚIE PRIVIND DEMOCRAȚIA REALĂ care poate fi citit si descărcat accesând situl grupului din portalul internet www.cartesiarte.ro
Caietul nr. 10 al Grupului de reflecție privind problemele democrației reale Emanuel Pope
Vă informăm că in spațiul de dezbateri „PLURALITAS” de pe prima pagină a portalului ingternet www.cartesiarte.ro a apărut Caietul nr. 10 al Grupului de reflecție privind problemele democrației reale pe care vi-l prezentăm în atașament
A presentation about war and the severe damages , how un human can war be .
Be the change you want to see in the world.
Mahatma Ghandi
-Made by :Shahd Hamouri
The presentation I gave in the plenary session at the ICCS (Presentation A3). It has been slightly modified for publishing. Please contact me if you have any questions!
Kevin R. Fox, M.D., Director, Rena Rowan Breast Center, Perelman Center for Advanced Medicine at Penn Medicine - Newest Approach to Breast Cancer
Presented at New Frontiers in the Management of Solid and Liquid Tumors hosted by the John Theurer Cancer Center at Hackensack University Medical Center. jtcancercenter.org/CME
Presentation of my M.Sc. graduation work, titled "Determination of canal pool characteristics with experimental modeling". It describes the use of System Identification techniques to identify (irrigation) canal pool characteristics needed for automatic controller design from tests on the real canal rather than on model representations. There is a special focus on potential resonance behavior of canal pools and the identification of these characteristics. Experiments on irrigation canal pools in Arizona, USA are described.
Talk at the University of Tokyo on history of Retraction Watch, our database, and current trends. Includes titles in Japanese, courtesy of Iekuni Ichikawa.
My June 14, 2017 talk at the Friends of the National Library of Medicine conference, "Consequential Clinical Research Accelerating Continuous Improvement"
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Baggerly presentation from CSE
1. The Importance of Reproducibility in
High-Throughput Studies: Case
Studies in Forensic Bioinformatics
Keith A. Baggerly
Bioinformatics and Computational Biology
UT M. D. Anderson Cancer Center
kabagg@mdanderson.org
CSE, May 3, 2011
2. G ENOMIC S IGNATURES 1
Why is Reproducibility Important in H-T-S?
Our intuition about what “makes sense” is very poor in high
dimensions. To use “genomic signatures” as biomarkers, we
need to know they’ve been assembled correctly.
Without documentation, we may need to employ forensic
bioinformatics to infer what was done to obtain the results.
Let’s examine some case studies involving an important
clinical problem: can we predict how a given patient will
respond to available chemotherapeutics?
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
3. G ENOMIC S IGNATURES 2
Using the NCI60 to Predict Sensitivity
Potti et al (2006), Nature Medicine, 12:1294-1300.
The main conclusion is that we can use microarray data from
cell lines (the NCI60) to define drug response “signatures”,
which can be used to predict whether patients will respond.
They provide examples using 7 commonly used agents.
This got people at MDA very excited.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
4. G ENOMIC S IGNATURES 3
Fit Training Data
We want the test data to split like this...
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
5. G ENOMIC S IGNATURES 4
Fit Testing Data
But it doesn’t. Did we do something wrong?
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
6. G ENOMIC S IGNATURES 5
5-FU Heatmaps
Nat Med Paper
7. G ENOMIC S IGNATURES 5
5-FU Heatmaps
Nat Med Paper Our t-tests
8. G ENOMIC S IGNATURES 5
5-FU Heatmaps
Nat Med Paper Our t-tests Reported Genes
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
9. G ENOMIC S IGNATURES 6
Their List and Ours
> temp <- cbind(
sort(rownames(pottiUpdated)[fuRows]),
sort(rownames(pottiUpdated)[
fuTQNorm@p.values <= fuCut]);
> colnames(temp) <- c("Theirs", "Ours");
> temp
Theirs Ours
...
[3,] "1881_at" "1882_g_at"
[4,] "31321_at" "31322_at"
[5,] "31725_s_at" "31726_at"
[6,] "32307_r_at" "32308_r_at"
...
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
10. G ENOMIC S IGNATURES 7
Offset P-Values: Other Drugs
Topotecan+1 Etoposide+1 Adriamycin+1
1.0
1.0
1.0
0.8
0.8
0.8
q
0.6
0.6
0.6
P Value
P Value
P Value
q
0.4 q
0.4
0.4
0.2
0.2
0.2
q
0.0
0.0
0.0
q
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqq
0 50 100 150 0 10 20 30 40 50 0 20 40 60 80
Index Index Index
Paclitaxel+1 Docetaxel+1 Cytoxan+1
1.0
1.0
1.0
q qq
q
q q q
q q
q q
0.8
0.8
0.8
q
q
q
q
q
0.6
0.6
0.6
P Value
P Value
q
P Value
q
qq q
qq
q q
q
0.4
0.4
0.4
q q
qq
q q
qqqq
q
q
0.2
0.2
0.2
qq
q
q
qq
q
q
q q
q qq q
q
0.0
0.0
0.0
qqqqqqqqqqqqqq
qqqqqqqqqqqqqq qqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqq qqq
qqq
0 5 15 25 35 0 10 20 30 40 50 0 5 15 25 35
Index Index Index
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
11. G ENOMIC S IGNATURES 8
Using Their Software
Their software requires two input files:
1. a quantification matrix, genes by samples, with a header
giving classifications (0 = Resistant, 1 = Sensitive, 2 = Test)
2. a list of probeset ids in the same order as the quantification
matrix. This list must not have a header row.
What do we get?
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
12. G ENOMIC S IGNATURES 9
Heatmaps Match Exactly for Most Drugs!
From the paper:
From the software:
13. G ENOMIC S IGNATURES 9
Heatmaps Match Exactly for Most Drugs!
From the paper:
From the software:
We match heatmaps but not gene lists? We’ll come back to
this, because their software also gives predictions.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
14. G ENOMIC S IGNATURES 10
Predicting Docetaxel (Chang 03)
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
15. G ENOMIC S IGNATURES 11
Predicting Adriamycin (Holleman 04)
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
16. G ENOMIC S IGNATURES 12
There Were Other Genes...
The 50-gene list for docetaxel has 19 “outliers”.
The initial paper on the test data (Chang et al) gave a list of
92 genes that separated responders from nonresponders.
Entries 7-20 in Chang et al’s list comprise 14/19 outliers.
The others: ERCC1, ERCC4, ERBB2, BCL2L11, TUBA3.
These are the genes named to explain the biology.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
17. G ENOMIC S IGNATURES 13
A Repro Theme: Don’t Take My Word For It!
Read the paper! Coombes, Wang & Baggerly, Nat Med, Nov
6, 2007, 13:1276-7, author reply 1277-8.
Try it yourselves! All of the raw data, documentation*, and
code* is available from our web site (*and from Nat Med):
http://bioinformatics.mdanderson.org/
Supplements/ReproRsch-Chemo.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
18. G ENOMIC S IGNATURES 14
Potti/Nevins Reply (Nat Med 13:1277-8)
Labels for Adria are correct – details on their web page.
They’ve gotten the approach to work again. (Twice!)
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
19. G ENOMIC S IGNATURES 15
Adriamycin 0.9999+ Correlations (Reply)
20. G ENOMIC S IGNATURES 15
Adriamycin 0.9999+ Correlations (Reply)
Redone Aug 08, “using ... 95 unique samples” (also wrong)
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
21. G ENOMIC S IGNATURES 16
Validation 1: Hsu et al
J Clin Oncol, Oct 1, 2007, 25:4350-7.
Same approach, using Cisplatin and Pemetrexed.
For cisplatin, U133A arrays were used for training. ERCC1,
ERCC4 and DNA repair genes are identified as “important”.
With some work, we matched the heatmaps. (Gene lists?)
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
22. G ENOMIC S IGNATURES 17
The 4 We Can’t Match (Reply)
203719 at, ERCC1,
210158 at, ERCC4,
228131 at, ERCC1, and
231971 at, FANCM (DNA Repair).
The last two probesets are special.
23. G ENOMIC S IGNATURES 17
The 4 We Can’t Match (Reply)
203719 at, ERCC1,
210158 at, ERCC4,
228131 at, ERCC1, and
231971 at, FANCM (DNA Repair).
The last two probesets are special.
These probesets aren’t on the U133A arrays that were used.
They’re on the U133B.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
24. G ENOMIC S IGNATURES 18
Validation 2: Bonnefoi et al
Lancet Oncology, Dec 2007, 8:1071-8. (early access Nov 14)
Similar approach, using signatures for Fluorouracil, Epirubicin
Cyclophosphamide, and Taxotere to predict response to
combination therapies: FEC and TET.
Potentially improves ER- response from 44% to 70%.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
25. G ENOMIC S IGNATURES 19
We Might Expect Some Differences...
High Sample Correlations Array Run Dates
after Centering by Gene
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
26. G ENOMIC S IGNATURES 20
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TET
and FEC. Let P() indicate prob sensitive. The rules used are
as follows.
27. G ENOMIC S IGNATURES 20
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TET
and FEC. Let P() indicate prob sensitive. The rules used are
as follows.
P (T F AC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
28. G ENOMIC S IGNATURES 20
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TET
and FEC. Let P() indicate prob sensitive. The rules used are
as follows.
P (T F AC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
29. G ENOMIC S IGNATURES 20
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TET
and FEC. Let P() indicate prob sensitive. The rules used are
as follows.
P (T F AC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
5 1
P (F EC) = [P (F ) + P (E) + P (C)] − .
8 4
30. G ENOMIC S IGNATURES 20
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TET
and FEC. Let P() indicate prob sensitive. The rules used are
as follows.
P (T F AC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
5 1
P (F EC) = [P (F ) + P (E) + P (C)] − .
8 4
Each rule is different.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
31. G ENOMIC S IGNATURES 21
Predictions for Individual Drugs? (Reply)
Does cytoxan make sense?
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
32. G ENOMIC S IGNATURES 22
Temozolomide Heatmaps
Augustine et al., 2009, Clin
Can Res, 15:502-10, Fig 4A.
Temozolomide, NCI-60.
33. G ENOMIC S IGNATURES 22
Temozolomide Heatmaps
Augustine et al., 2009, Clin Hsu et al., 2007, J Clin
Can Res, 15:502-10, Fig 4A. Oncol, 25:4350-7, Fig 1A.
Temozolomide, NCI-60. Cisplatin, Gyorffy cell lines.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
34. G ENOMIC S IGNATURES 23
Some Timeline Here...
Nat Med Nov 06*, Nov 07*, Aug 08. JCO Feb 07*, Oct 07*.
Lancet Oncology Dec 07*. PLoS One Apr 08. CCR Jan 09*.
(* errors reported)
35. G ENOMIC S IGNATURES 23
Some Timeline Here...
Nat Med Nov 06*, Nov 07*, Aug 08. JCO Feb 07*, Oct 07*.
Lancet Oncology Dec 07*. PLoS One Apr 08. CCR Jan 09*.
(* errors reported)
May/June 2009: we learn clinical trials had begun.
2007: pemetrexed vs cisplatin, pem vs vinorelbine.
2008: docetaxel vs doxorubicin, topotecan vs dox (Moffitt).
36. G ENOMIC S IGNATURES 23
Some Timeline Here...
Nat Med Nov 06*, Nov 07*, Aug 08. JCO Feb 07*, Oct 07*.
Lancet Oncology Dec 07*. PLoS One Apr 08. CCR Jan 09*.
(* errors reported)
May/June 2009: we learn clinical trials had begun.
2007: pemetrexed vs cisplatin, pem vs vinorelbine.
2008: docetaxel vs doxorubicin, topotecan vs dox (Moffitt).
Sep 1. Paper submitted to Annals of Applied Statistics.
Sep 14. Paper online at Annals of Applied Statistics.
Sep-Oct: Story covered by The Cancer Letter, Duke starts
internal investigation, suspends trials.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
37. G ENOMIC S IGNATURES 24
Jan 29, 2010
Their investigation’s results “strengthen ... confidence in this
evolving approach to personalized cancer treatment.”
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
38. G ENOMIC S IGNATURES 25
Why We’re Unhappy...
“While the reviewers approved of our sharing the report with
the NCI, we consider it a confidential document” (Duke). A
future paper will explain the methods.
39. G ENOMIC S IGNATURES 25
Why We’re Unhappy...
“While the reviewers approved of our sharing the report with
the NCI, we consider it a confidential document” (Duke). A
future paper will explain the methods.
There was also a major new development that the restart
announcement didn’t mention.
In mid-Nov (mid-investigation), the Duke team posted new
data for cisplatin and pemetrexed (in trials since ’07).
These included quantifications for 59 ovarian cancer test
samples (from GSE3149) used for predictor validation.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
40. G ENOMIC S IGNATURES 26
We Tried Matching The Samples
43 samples are mislabeled; 16 don’t match at all.
The first 16 don’t match because the genes are mislabeled.
We reported this to Duke and to the NCI in mid-November.
All data was stripped from the websites within the week.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
41. G ENOMIC S IGNATURES 27
More Timeline
“While the reviewers approved of our sharing the report with
the NCI...”
42. G ENOMIC S IGNATURES 27
More Timeline
“While the reviewers approved of our sharing the report with
the NCI...”
April, 2010. Review report sought from the NCI under the
Freedom of Information Act (FOIA).
May, 2010. Redacted report supplied; gaps noted.
43. G ENOMIC S IGNATURES 27
More Timeline
“While the reviewers approved of our sharing the report with
the NCI...”
April, 2010. Review report sought from the NCI under the
Freedom of Information Act (FOIA).
May, 2010. Redacted report supplied; gaps noted.
May, 2010. NCI and CALGB pull lung metagene signature
from an ongoing phase III trial.
Duke trials continue.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
44. G ENOMIC S IGNATURES 28
July 16, 2010
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
45. G ENOMIC S IGNATURES 29
Subsequent Events
July 19/20: letter to Varmus; Duke resuspends trials
July 30: Varmus & Duke request IOM Involvement
Oct 22/29: call to retract JCO paper
Nov 9: Duke announces trials terminated
Nov 19: call to retract Nat Med paper, Potti resigns
There have been at least three developments since.
These involve the NCI, the FDA, and Duke.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
46. G ENOMIC S IGNATURES 30
Dec 20, 2010: The IOM Meets, the NCI Speaks
At the first meeting of the IOM review panel, Lisa McShane
outlined some NCI interactions with the Duke group.
An MP3 is available from the Cancer Letter.
Our questions prompted the NCI to ask questions of its own.
The NCI released about 550 pages of documents giving the
details.
We posted our annotation of these documents and an overall
timeline on Jan 14th.
We’re going to look at one case they examined.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
47. G ENOMIC S IGNATURES 31
The Lung Metagene Score (LMS)
One of “the most signficant advances on the front lines of
cancer.” – Ozols et al., JCO, 25:146-62, 2007, ASCO survey
of 2006.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
48. G ENOMIC S IGNATURES 32
The Background of CALGB 30506
When this trial was proposed to the NCI, they had questions
about the amount of blinding, so they asked for one more test.
49. G ENOMIC S IGNATURES 32
The Background of CALGB 30506
When this trial was proposed to the NCI, they had questions
about the amount of blinding, so they asked for one more test.
The LMS failed.
It almost achieved significance going the wrong way.
50. G ENOMIC S IGNATURES 32
The Background of CALGB 30506
When this trial was proposed to the NCI, they had questions
about the amount of blinding, so they asked for one more test.
The LMS failed.
It almost achieved significance going the wrong way.
After post-hoc adjustments (adjusting for batches), the Duke
group attained some success (with IB, not IA).
At CALGB’s urging, this went forward, but the NCI only
allowed the LMS to be used for stratification, not allocation.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
51. G ENOMIC S IGNATURES 33
After our Paper
The NCI asked to see the data and code associated with the
post-hoc adjustment.
Using the code and data provided, the NCI couldn’t
reproduce the success of the post-hoc adjustment.
More, they noticed another problem.
They ran the algorithm and got predictions.
52. G ENOMIC S IGNATURES 33
After our Paper
The NCI asked to see the data and code associated with the
post-hoc adjustment.
Using the code and data provided, the NCI couldn’t
reproduce the success of the post-hoc adjustment.
More, they noticed another problem.
They ran the algorithm and got predictions.
They ran it again.
53. G ENOMIC S IGNATURES 33
After our Paper
The NCI asked to see the data and code associated with the
post-hoc adjustment.
Using the code and data provided, the NCI couldn’t
reproduce the success of the post-hoc adjustment.
More, they noticed another problem.
They ran the algorithm and got predictions.
They ran it again.
The predictions changed.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
54. G ENOMIC S IGNATURES 34
The Changes Weren’t Subtle
Some scores changed from 5% to 95%.
From one run to the next, high/low classifications changed
about 25% of the time.
55. G ENOMIC S IGNATURES 34
The Changes Weren’t Subtle
Some scores changed from 5% to 95%.
From one run to the next, high/low classifications changed
about 25% of the time.
The NCI was unhappy. They had understood the rule to be
“locked down”, so that predictions wouldn’t change.
56. G ENOMIC S IGNATURES 34
The Changes Weren’t Subtle
Some scores changed from 5% to 95%.
From one run to the next, high/low classifications changed
about 25% of the time.
The NCI was unhappy. They had understood the rule to be
“locked down”, so that predictions wouldn’t change.
The NCI publicly yanked the LMS from CALGB 30506 in May
2010.
57. G ENOMIC S IGNATURES 34
The Changes Weren’t Subtle
Some scores changed from 5% to 95%.
From one run to the next, high/low classifications changed
about 25% of the time.
The NCI was unhappy. They had understood the rule to be
“locked down”, so that predictions wouldn’t change.
The NCI publicly yanked the LMS from CALGB 30506 in May
2010.
The NEJM paper was retracted March 2, 2011.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
58. G ENOMIC S IGNATURES 35
The FDA Visits
On Jan 28, 2011, the Cancer Letter reported that an FDA
Audit team was visiting Duke to examine how the trials had
been run.
A key question was whether Investigational Device
Exemptions (IDEs) were obtained before the signatures were
used to guide therapy. They weren’t.
59. G ENOMIC S IGNATURES 35
The FDA Visits
On Jan 28, 2011, the Cancer Letter reported that an FDA
Audit team was visiting Duke to examine how the trials had
been run.
A key question was whether Investigational Device
Exemptions (IDEs) were obtained before the signatures were
used to guide therapy. They weren’t.
Some aspects of this issue were anticipated by the IOM
committee (Baggerly and Coombes, Clinical Chemistry,
57(5):688-90).
Per the FDA, genomic signatures are medical devices.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
60. G ENOMIC S IGNATURES 36
Duke’s Initial Investigation
One thing that had puzzled us since the trial restarts was how
the external reviewers missed the new errors we reported.
61. G ENOMIC S IGNATURES 36
Duke’s Initial Investigation
One thing that had puzzled us since the trial restarts was how
the external reviewers missed the new errors we reported.
Jan 11, 2011: Nature talks to Duke.
The Duke deans overseeing the investigation, in consultation
with the acting head of Duke’s IRB, decided not to forward
our report to the reviewers.
This was done to avoid “biasing the review”.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
62. G ENOMIC S IGNATURES 37
The IOM Meets Again: Mar 30-31
Science: Ned Calogne, Rich Simon, Chuck Perou, Sumitha
Mandrekar
Institutional Responsibilty: Peter Pronovost (Hopkins)
Case Studies: Laura Van’t Veer, Steve Shak, Joe Nevins
Responsible Parties: journal editors (Cathy DeAngelis,
JAMA, Veronique Kiermer, Nature, Katrina Kelner, Science),
authors and PIs (Stott Zeger), institutions (Albert Reece, U
Md, Harold Paz, Penn State, Scott Zeger, Hopkins)
Forensics: Keith Baggerly
http://bioinformatics.mdanderson.org/
Supplements/ReproRsch-All/Modified/IOM/
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
63. G ENOMIC S IGNATURES 38
Some Cautions/Observations
We’ve seen problems like these before.
The most common mistakes are simple.
Confounding in the Experimental Design
Mixing up the sample labels
Mixing up the gene labels
Mixing up the group labels
(Most mixups involve simple switches or offsets)
This simplicity is often hidden.
Incomplete documentation
Unfortunately, we suspect
The most simple mistakes are common.
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
64. G ENOMIC S IGNATURES 39
How Should We Pursue Reproducibility?
AAAS, Feb 19: All slides and an MP3 of the audio are
available from
http://www.stanford.edu/˜vcs/AAAS2011/
ENAR, Mar 23: Panel on ethics in biostatistics. Handouts are
available from our Google Group.
What’s Missing? Nature, Mar 23
AACR, Apr 5: Difficulties in moving biomarkers to the clinic.
CSE, May 3
NCI, Jun 23-4 What should the NCI be looking for in grants
that it funds?
Check back...
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
65. G ENOMIC S IGNATURES 40
What Should the Norm Be?
For our group? Since 2007, we have prepared reports in
Sweave.
For papers? (Baggerly, Nature, Sep 22, 2010)
Things we look for:
1. Data (often mentioned, given MIAME)
2. Provenance
3. Code
4. Descriptions of Nonscriptable Steps
5. Descriptions of Planned Design, if Used.
For clinical trials?
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
66. G ENOMIC S IGNATURES 41
Some Acknowledgements
Kevin Coombes
Shannon Neeley, Jing Wang
David Ransohoff, Gordon Mills
Jane Fridlyand, Lajos Pusztai, Zoltan Szallasi
MDACC Ovarian SPORE, Lung SPORE, Breast SPORE
Now in the Annals of Applied Statistics! Baggerly and
Coombes (2009), 3(4):1309-34.
http://bioinformatics.mdanderson.org/
Supplements/ReproRsch-All
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes
67. G ENOMIC S IGNATURES 42
Index
Title
Cell Line Story
Trying it Ourselves
Matching Features
Using Software/Making Predictions
Outliers
The Reply
Adriamycin Followup
Hsu et al (Cisplatin)
Timeline, Trials, Cancer Letter
Trial Restart and Objections
Final Lessons
c Copyright 2007-2011, Keith A. Baggerly and Kevin R. Coombes