Selective Correlations-
NotVoodoo
Malach Lab 8.5.2014	



Jonathan Rosenblatt- WIS
Acknowledgments
• Part of PhD dissertation under guidance of
Prof.Yoav Benjamini	

• Prof. Russ Poldrack	

• Ms. Neomi Sin...
Outline
• Tom et al., 

“The Neural Basis of Loss Aversion in
Decision-Making Under Risk.”

Science 315 (January 26, 2007)...
Tom et al. (2007)
Reported Correlations
EnterVul
• Vul, Edward, Christine Harris, Piotr
Winkielman, and Harold Pashler. 

“Puzzlingly High Correlations in fMRI
St...
And The People Rejoice
• Diener, Ed.“Editor’s Introduction toVul et Al. (2009) and Comments.” Perspectives on Psychologica...
The Usual Suspects
• Multiplicity control	

• Small samples => underpowered	

• Reporting standards
n.subjects: 12 n.subjects: 29 n.subjects: 47 n.subjects: 64 n.subjects: 82 n.subjects: 100
●
●
● ● ● ● ●
●
● ● ● ● ● ● ●
●...
Cureton, Edward (1950)
• When a validity coefficient is computed
from the same data used in making an item
analysis, this c...
Selective Estimation
• A.k.a.“circular inference”,“double dipping”,
“voodoo correlations”,...	

• Estimation with quality ...
Ingredients of Estimation
• Point/interval?	

• Error criterion?	

• Algorithms with uniform error bounds?
False Coverage statement Rate
• An error criterion for selective interval
estimation.	

• R= selected parameters	

• V= fa...
FCR of Nominal CIs
• Parameters selected: R=3 (big dots)	

• Parameters not covered:V=2 (grey bars)	

• FCP=V/R= 0.66	

• ...
FCR Adjusted CIs
• Benjamini,Yekutieli (2005)	

• Motivation: conservative nominal CIs.	

• In practice:
Conditional CIs
• Weinstein, Fithian, Benjamini (2013)	

• Motivation: invert acceptance region of
conditional distributio...
(non)Uniqueness of Acceptance Region
• Conditional Shortest Length (CSR):

Short interval, but indifferent to sign
ambigui...
Properties
Remarks
• “Simple”: varying the value of a selected i’th
estimator in its selectable range, does not
change R.	

• FCR adj...
Applying FCR Adjusted CIs
“Confidence Calibration Plot”
Applying CQC CIs
“Confidence Calibration Plot”
Conditional or FCR Adjusted?
“If the functional contrast is
demonstrably independent of the
effects to be estimated for the
selected data, then the sam...
●
●●●●●●●●
● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00
FullAndSplit
Split...
CI Agreement with Half Split
CI Agreement with Half Split
• In less than 0.1% of voxels, splitting will
provide strong sign determination and
CQC will ...
Summary
• Selective estimation in social-neuroscience:
Acknowledged but untreated. 	

• FCR controlling CIs as a general r...
Open Problems
• Conditioning	

• Inference on aggregates	

• Other Error measures
Thank you
<jonathan.rosenblatt@weizmann.ac.il>
Upcoming SlideShare
Loading in …5
×

Selective Correlations- Not Voodoo

265 views

Published on

On a solution to selective estimation using post selection confidence intervals.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
265
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Selective Correlations- Not Voodoo

  1. 1. Selective Correlations- NotVoodoo Malach Lab 8.5.2014 
 Jonathan Rosenblatt- WIS
  2. 2. Acknowledgments • Part of PhD dissertation under guidance of Prof.Yoav Benjamini • Prof. Russ Poldrack • Ms. Neomi Singer, Mr. Omri Perez, Prof. Talma Hendler
  3. 3. Outline • Tom et al., 
 “The Neural Basis of Loss Aversion in Decision-Making Under Risk.”
 Science 315 (January 26, 2007) • Selection Bias- Problem & Remedy. • Revisiting Tom et al. • Discussion.
  4. 4. Tom et al. (2007)
  5. 5. Reported Correlations
  6. 6. EnterVul • Vul, Edward, Christine Harris, Piotr Winkielman, and Harold Pashler. 
 “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 274–290.
  7. 7. And The People Rejoice • Diener, Ed.“Editor’s Introduction toVul et Al. (2009) and Comments.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 272–273. • Fiedler, Klaus.“Voodoo Correlations Are Everywhere—Not Only in Neuroscience.” Perspectives on Psychological Science 6, no. 2 (March 1, 2011) • Jabbi et al.“Response to ‘Voodoo Correlations in Social Neuroscience’ byVul et Al.–summary Information for the Press.” Accessed July 30, 2013. • Kriegeskorte et al.“EverythingYou Never Wanted to Know about Circular Analysis, but Were Afraid to Ask.” Journal of Cerebral Blood Flow & Metabolism 30, no. 9 (September 2010): 1551–1557. • Lazar, Nicole A.“Discussion of ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition’ byVul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 308–309. • Lieberman, Matthew D., Elliot T. Berkman, and Tor D.Wager.“Correlations in Social Neuroscience Aren’tVoodoo: Commentary onVul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 299–307. doi:10.1111/j.1745-6924.2009.01128.x. • Lindquist, Martin A., and Andrew Gelman.“Correlations and Multiple Comparisons in Functional Imaging:A Statistical Perspective (Commentary onVul et Al., 2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 310–313. doi:10.1111/j.1745-6924.2009.01130.x. • Nichols,Thomas E., and Jean-Baptist Poline.“Commentary onVul et Al.’s (2009) ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.’” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 291–293. doi:10.1111/j.1745-6924.2009.01126.x. • Poldrack, Russell A., and Jeanette A. Mumford.“Independence in ROI Analysis:Where Is theVoodoo?” Social Cognitive and Affective Neuroscience 4, no. 2 (June 1, 2009): 208–213. doi:10.1093/scan/nsp011. • Vul, Edward, Christine Harris, Piotr Winkielman, and Harold Pashler.“Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 274–290. doi:10.1111/j.1745-6924.2009.01125.x. • “Reply to Comments on ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.’” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 319–324. doi:10.1111/j.1745-6924.2009.01132.x. • Vul, Edward, and Nancy Kanwisher.“Begging the Question:The Non-Independence Error in fMRI Data Analysis.” Foundational Issues for Human Brain Mapping (2010): 71–91. • Yarkoni,Tal.“Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary onVul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 294–298. doi:10.1111/j.1745-6924.2009.01127.x.
  8. 8. The Usual Suspects • Multiplicity control • Small samples => underpowered • Reporting standards
  9. 9. n.subjects: 12 n.subjects: 29 n.subjects: 47 n.subjects: 64 n.subjects: 82 n.subjects: 100 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4 0.6 0.8 1.0 0.4 0.6 0.8 1.0 0.4 0.6 0.8 1.0 resels:5000resels:10000resels:3e+05 0.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.00 True Correlation MeanofSelectedCorrelations Selection Bias
  10. 10. Cureton, Edward (1950) • When a validity coefficient is computed from the same data used in making an item analysis, this coefficient cannot be interpreted uncritically.And, contrary to many statements in the literature, it cannot be interpreted “with caution” either.There is one clear interpretation for all such validity coefficients.This interpretation is– “Baloney”
  11. 11. Selective Estimation • A.k.a.“circular inference”,“double dipping”, “voodoo correlations”,... • Estimation with quality guarantees following a parameter selection stage. • Relation to selective testing.
  12. 12. Ingredients of Estimation • Point/interval? • Error criterion? • Algorithms with uniform error bounds?
  13. 13. False Coverage statement Rate • An error criterion for selective interval estimation. • R= selected parameters • V= false coverage statements • FCP=V/R and 0 if none are selected. • FCR=E(FCP)
  14. 14. FCR of Nominal CIs • Parameters selected: R=3 (big dots) • Parameters not covered:V=2 (grey bars) • FCP=V/R= 0.66 • Desired FCR= 1-confidence level= 0.1 • Nominal CIs do not control the FCR. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 0 ● ● ●● ● ●
  15. 15. FCR Adjusted CIs • Benjamini,Yekutieli (2005) • Motivation: conservative nominal CIs. • In practice:
  16. 16. Conditional CIs • Weinstein, Fithian, Benjamini (2013) • Motivation: invert acceptance region of conditional distribution. • In practice: selectiveCI R package.
  17. 17. (non)Uniqueness of Acceptance Region • Conditional Shortest Length (CSR):
 Short interval, but indifferent to sign ambiguity. • Conditional Modified Pratt (CMP): 
 Best sign determination, while no larger than r times the CSR. • Conditional Quasi Conventional (CQC):
 Shortest interval with penalty for sign flip
  18. 18. Properties
  19. 19. Remarks • “Simple”: varying the value of a selected i’th estimator in its selectable range, does not change R. • FCR adjusted CIs are B-H selection duals. 
 Duality does not hold in general. • Some selection rules are very hard to condition on.
  20. 20. Applying FCR Adjusted CIs “Confidence Calibration Plot”
  21. 21. Applying CQC CIs “Confidence Calibration Plot”
  22. 22. Conditional or FCR Adjusted?
  23. 23. “If the functional contrast is demonstrably independent of the effects to be estimated for the selected data, then the same data may be used for effect estimation. Otherwise, independent data are required to render the effect estimate independent.” Kriegeskorte (2013)
  24. 24. ● ●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 FullAndSplit SplitAndFull2 Split.count ● ● ● ● ● 0 10000 20000 30000 40000 Spatial Agreement of a Half Split
  25. 25. CI Agreement with Half Split
  26. 26. CI Agreement with Half Split • In less than 0.1% of voxels, splitting will provide strong sign determination and CQC will not. • In more than 50% of possible splits, 1/3 of jointly selected voxels will have opposing strong sign determination. • Conclusion: CQC has higher probability of catching the right effect sign.
  27. 27. Summary • Selective estimation in social-neuroscience: Acknowledged but untreated. • FCR controlling CIs as a general remedy. • Better than splitting small samples.
  28. 28. Open Problems • Conditioning • Inference on aggregates • Other Error measures
  29. 29. Thank you <jonathan.rosenblatt@weizmann.ac.il>

×