Team Shared Cognitive Constructs:   A Meta-Analysis Exploring the Effects of Shared     Cognitive Measures on Team Perform...
Meta-Analysis / Shared Cognitive ConstructsMeta-Analysis                                     Shared Cognitive•	 “Statistic...
Meta-Analysis / Effect SizesMeta-Analyses analyze the Effect Size from different studies that  represent the same or simil...
Shared Cognitive ConstructsConstruct Abbr. DescriptionShared Mental     SMM   Team members overlapping representation of k...
Research Questions1)	 Which construct produces the best overall effect on perfor-mance?2)	 How do the measures for the six...
Research Questions -cont.-A quality measure for each research article in this analysis wasconducted. Each article was code...
Data Collection MethodsERIC-EBSCOhost Bibliographic databaseSearch Time Period: January 1990 - April 2012.Criteria: ‘in Ab...
Data Collection Methods -cont.-TMS:	  Criteria: ‘Transactive Memory’	  Initial:	 8 articles, 4 with quantitative data	  Fi...
Summary of Articles AnalyzedID#       Researchers        Year   Quality Predictors                Outcome                 ...
Summary of Articles Analyzed -cont.-ID #      Researchers       Year   Quality Predictors                     Outcome     ...
Summary of Articles Analyzed -cont.-ID #     Researchers         Year   Quality Predictors                  Outcome       ...
Only One!Effect Sizes need to be independent from oneanother.“If a study presents more than one effect size for a construc...
Standardize Reported ESFisher’s Z (for correlation):                                Variance of Z:Standard Error of Z:    ...
Fi                Share	  Mental	  Models	  (SMM)	  -­‐	  01                                                              ...
Estimated VariancesFixed Effects Model               Random Effects Model        ; within study variance            ; Vwit...
Test of Heterogeneity             To determine whether the variance calculated was more                than what would be ...
Test of Heterogeneity -cont.-For all Effect Sizes in Study:Q (17) = 177.53χ 2 (17)   = 27.59Q (17) > χ (17): reject null h...
Test of Constructs       Calculate the random effects per shared             cognition construct group:         SMM      T...
Test of Constructs -cont.-Calculate Difference btwn                  Example: IS -vs- TMMTwo Constructs:Diff      *= M * −...
Test of Constructs -cont.-                        Random-­‐effects	  model	  (separate	  estimates	  of	  T 2),	  Calculat...
Quality Ranking ComparisonQuality Measure:“Questions to Ask Yourself When Evaluating a Report of a Quanti-tative Study” (G...
Quality Ranking Comparison -cont.-Interrater Reliability:Each article evaluated by researcherOne half of the articles were...
Quality Ranking Comparison -cont.-    Comparison ‘Low and Medium Quality’        -vs- ‘High Quality’ Ranking:Diff         ...
ConclusionIS was shown to be the better predictor of per-formance:                            Construct   r* (corrected r)...
Conclusion -cont.- Kontoghiorghes et al. (2005) recommended the followingto transform into innovative and adaptive entitie...
Conclusion -cont.-Kontoghiorghes et al. (2005) recommended:“focusing first on... open communications, team-work, resource ...
QUESTIONS ?                       THANK YOUThe University of North TexasJohn R. Turnerjrthpt@gmail.comwww.unt.lt.edu      ...
References:Beretvas, N. S. (2010). Meta-Analysis. In Hancock, G. R., & Mueller, R. O, (Eds.), The Reviewr’s Guide to Quant...
Meta analysis-sloan
Upcoming SlideShare
Loading in...5
×

Meta analysis-sloan

246

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
246
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Meta analysis-sloan

  1. 1. Team Shared Cognitive Constructs: A Meta-Analysis Exploring the Effects of Shared Cognitive Measures on Team Performance The University of North Texas Department of Learning Technologies Denton, Texas John R. Turner - Presenter Qi Chen, Ph.D. Shelby Danksjrthpt@gmail.comwww.lt.unt.edu 1
  2. 2. Meta-Analysis / Shared Cognitive ConstructsMeta-Analysis Shared Cognitive• “Statistical synthesis of results from a Constructs series of studies” (Borenstein et al., 2009, Prefix). • The distributed &/or overlapping of knowledge structures and belief struc-• “focuses on the aggregation and com- tures (Mohammed & Dumville, 2001). parison of the findings of different stud- ies” (Lipsey & Wilson, 2001, p. 2). • The shared information and knowledge among team / group members.• An analysis of anlalyses. • Knowing who knows what and who has• A quantitative literature review. what skills. 2
  3. 3. Meta-Analysis / Effect SizesMeta-Analyses analyze the Effect Size from different studies that represent the same or similar constructs and their outcome. Standardized Mean Difference / Gain, d Correlation, r Unstandardized Mean Difference / Gain, D 3
  4. 4. Shared Cognitive ConstructsConstruct Abbr. DescriptionShared Mental SMM Team members overlapping representation of knowl-Models edge (tasks, equipment, working relationships, situa- tions, etc...) (Bossche et al., 2011)Team Mental TMM The “organized understanding of relevant knowledgeModels that is shared by team members” (Mohammed & Dumville, 2001, p. 89)Information IS The “transfer of tacit and explicit knowledge from indi-Sharing viduals within the organization to the collective” (Bontis et al., 2011, p. 240)Transactive TMS Where team members encode, store, and retrieve rel-Memory Systems evant information together (Liang et al., 1995)Cognitive CC Team members determine best response for the aggre-Consensus gate - majority rules.Group Learning GL Where students encourage and facilitate one another’s goal achievements (Onwuegbuzie et al., 2009) 4
  5. 5. Research Questions1) Which construct produces the best overall effect on perfor-mance?2) How do the measures for the six shared cognition constructscompare to one another in relation to performance? 5
  6. 6. Research Questions -cont.-A quality measure for each research article in this analysis wasconducted. Each article was coded categorically as being either‘low quality’, ‘medium quality’, or ‘high quality’. Meta-Analyses should be conducted using quality articles so thereis less of a chance that the effect sizes are found unreliable (Beretvas,2010).3) What differences are there in the effect sizes reported fromthose ranked as low quality articles compared to those ranked ashigh quality articles? 6
  7. 7. Data Collection MethodsERIC-EBSCOhost Bibliographic databaseSearch Time Period: January 1990 - April 2012.Criteria: ‘in Abstract’ & ‘English’SMM & TMM: Criteria: ‘Team Mental Models’ Initial: 38 articles 25 relevant after Abstracts reviewed 4 with quantitative data Final: 2 articles relating to SMM 2 articles relating to TMM (SMM & TMM were batched together in database)IS: Criteria: ‘Information Sharing’ Initial: 832 articles reduced via Abstract review: exclusion: K-12 education, classroom, & international education articles inclusion: organizational, higher education, & training Second: 53 articles, 5 with quatitative data (1 non-relevant) Final: 4 articles relating to IS 7
  8. 8. Data Collection Methods -cont.-TMS: Criteria: ‘Transactive Memory’ Initial: 8 articles, 4 with quantitative data Final: 4 articles relating to TMSGL: Criteria: ‘Group Learning’ Initial: 4,572 articles, reduced to include ‘Academic Journals’ only Second: 2,556 articles, reduced by changing ‘in Abstract’ to ‘in Title’ Third: 577 articles reduced via Abstract review: exclusion: K-12 education, classroom, & international education articles inclusion: organizational, higher education Fourth: 38 articles, 9 with quatitative data (6 non-relevant) Final: 3 articles relating to GLCC: Criteria: ‘Cognitive AND Consensus’ Initial: 67 articles Final: 3 articles relating to CC. 8
  9. 9. Summary of Articles AnalyzedID# Researchers Year Quality Predictors Outcome Type of Reported ES / Ranking Measure Avg. rSMM1003101 Bossche et al. 2011 High SMM - Concept Perceived Team Perf. P r = .16 to .51 SMM - Statement Team Perf. - Actual A r = .397 Team Perf. - Goodwill A1014101 Johnson & Lee 2008 Medium SMM Team-Related Team Perf. Knowledge A r = .27 to .49 Skill A r = .366 Attitude A Dynamicity A Environment ATMM1007102 Burtscher et al. 2011 High TMM - Similarity Team Perf. A r = -.08 to .12 TMM - Accuracy A r = .021131102 Lim & Klein 2006 High Taskwork MM Similarity Team Perf. A r = .21 to .42 Teamwork MM Similarity A r = .29 Taskwork MM Accuracy A Teamwork MM Accuracy A 9
  10. 10. Summary of Articles Analyzed -cont.-ID # Researchers Year Quality Predictors Outcome Type of Reported ES / Ranking Measure Avg. rIS1045103 Garg 2010 Low Information Sharing (Com- Perceived Inc. Cust. Satisfaction P r = .22 to .45 posite) r = .322 Perceived Inc. Effectiveness P Perceived Overall Perf. P Perception of Inc. Productivity P1030103 Bontis et al. 2011 Medium Internal Information Sharing Efficiency P r = .54 to .67 Customer Focus P r = 6051068103 Kontoghiorghes 2005 High Open Comm. & IS Rapid Change Adaptation P r = .36 to .52 et al. r = .4391046103 Weldy & Gillis 2010 High Embedded Systems Financial Perf. P r = .55 to .63 Knowledge Perf. P r = .59TMS1087104 Liang et al. 1995 High Group vs Individual Trained Team Assembly Errors A r = .387 r = .3871083104 Michinov et al. 2009 High Specialization Perf. A r = -.09 to .42 Coordination Perf. Improvement A r = .187 Credibility1082104 Pearsall et al. 2009 High Transactive Memory Team Perf. A r = -.53 to .5 Psychological Withdrawal A r = -0.01 Problem-Solving Coping A Avoidant Coping A1080104 Gino et al. 2010 High Transactive Memory Team Creativity Level A r = .3 to .7 r = .51084104 Michinov et al. 2009 Medium Transactive Memory Group Perf. A r = .37 r = .37 10
  11. 11. Summary of Articles Analyzed -cont.-ID # Researchers Year Quality Predictors Outcome Type of Reported ES / Ranking Measure Avg. rCC1136105 Kirkman et al. 2001 High Consensus Gain over Ag- Productivity TL-P r = .22 to .45 gregate r = .308 Customer Service TL-P Team Org. Citizenship Behaviors TL-P1137105 Collins & Smith 2006 High Knowledge Exchange / Com- % Sales Growth A r = .49 to .54 bination r = .515 Revenue (new product & srvcs) AGL1098106 Pazos et al. 2010 Medium Group Interaction Style Self Efficacy A r = .22 r = .221102106 Onwuegbuzie et al. 2009 Medium Cooperation Article Critique Scores A r = -.22 r = -.221114106 Williams et al. 2006 High Teamwork Orientation Overall Student Learning P r = .22 to .45 Student Team-Source Learning A r = .335 11
  12. 12. Only One!Effect Sizes need to be independent from oneanother.“If a study presents more than one effect size for a construct... they should not beincluded in the same analysis as if they were independent data points” (Lipsey & Wilson,2001, p. 113).Lipsey and Wilson (2001) recommend the following when more than one effect size ispresented in a study:1) Average the effect size so that one effect size represents the study.2) Use one effect size from the study, omit the others.This meta-analysis averaged the effect sizes. 12
  13. 13. Standardize Reported ESFisher’s Z (for correlation): Variance of Z:Standard Error of Z: 13
  14. 14. Fi Share  Mental  Models  (SMM)  -­‐  01 sh Team  Mental  Models  (TMM)  -­‐  02 er SE Information  Sharing  (IS)  -­‐  03 VZ ’s Transactive  Memory  Systems  (TMS)  -­‐  04 Z Cognitive  Congruence  -­‐  05 Z Group  Learning  -­‐  06 Outcome   n  per  PASW-­‐ID ID# Construct Outcome Correlation Mean Outcome  SD N n-­‐teams teams Avg.  r fishers  z  (Yi) Variance  of  Z SEz r z  =  .5[ln(1+r)/(1-­‐r)] Vz  =  1  /  (n-­‐3) SEz  =  SQRT(Vz)1003101 3A SMM-­‐conc Perceived  Team  Perf. 0.28 5.99 0.64 81 27 3 0.3967 0.420 0.042 0.204 3B SMM-­‐conc Actual  Team  Perf.:  Equity 0.51 10128539.60 20343459.60 3C SMM-­‐conc Actual  Team  Perf.:  Goodwill 0.5 9477871.80 6965830.30 3D SMM-­‐stat Perceived  Team  Perf. 0.16 5.99 0.64 81 27 3 3E SMM-­‐stat Actual  Team  Perf.:  Equity 0.43 10128539.60 20343459.60 3F SMM-­‐stat Actual  Team  Perf.:  Goodwill 0.5 9477871.80 6965830.30 SMM-­‐conc 6.00 2.41 SMM-­‐stat 10.18 10.51 ns n tio io la rre MM) e lat Co (S C orr M) d ed y (SM o rte tudy g ep 1 S era Stud R m Av 1 fro for 14
  15. 15. Estimated VariancesFixed Effects Model Random Effects Model ; within study variance ; Vwithin + Vbetween ; weight assigned to ; weight assigned to each study each study ; weighted Mean ; weighted Mean (Borenstein et al., 2009) 15
  16. 16. Test of Heterogeneity To determine whether the variance calculated was more than what would be expected from random error.Table 3Random Effects for all Effect Sizes Study ID Y VW VB VT W* W*Y 1003101 .42 0.042 0.0551 0.097 10.302 4.327 1014101 .38 0.5 0.0551 0.555 1.802 0.692 1007102 .02 0.036 0.0551 0.091 10.980 0.220 ... ... ... ... ... ... ... 1098106 .23 0.006 0.0551 0.061 16.374 3.731 1102106 -.22 0.043 0.0551 0.098 10.196 -2.281 1114106 .35 0.038 0.0551 0.093 10.744 3.744Total 1.801 241.16 86.51 16
  17. 17. Test of Heterogeneity -cont.-For all Effect Sizes in Study:Q (17) = 177.53χ 2 (17) = 27.59Q (17) > χ (17): reject null hypothesis that all the shared cognition construct studies 2share a common effect size.This sample is a sample of heterogeneity in which the variance is more than what isexpected from error.Overall:Weighted Mean: M* = .359 (VM* = .0041)Interval Estimate for M*: 95% CI (.233, .485)Weighted Corrected Correlation: r* = .344Interval Estimate for r*: 95% CI (.228, .450) 17
  18. 18. Test of Constructs Calculate the random effects per shared cognition construct group: SMM TMM IS TMS CC GLM* 0.417 0.196 0.568 0.308 0.449 0.145VM* 0.0387 0.0177 0.0069 0.0111 0.0157 0.0196SEM* 0.197 0.133 0.083 0.105 0.126 0.14LLM* 0.032 -0.065 0.405 0.101 0.204 -0.128ULM* 0.803 0.456 0.73 0.514 0.696 0.421ZM* 2.119 1.472 6.853 2.919 3.585 1.047p 0.017 0.071 <.001 <.001 <.001 0.148Q 0.0024 1.509 32.691 14.817 3.319 4.812r* 0.394 0.193 0.513 0.298 0.422 0.146LLr* 0.032 -0.065 0.384 0.101 0.201 -0.127ULr* 0.666 0.427 0.623 0.474 0.602 0.398 18
  19. 19. Test of Constructs -cont.-Calculate Difference btwn Example: IS -vs- TMMTwo Constructs:Diff *= M * − M * Diff * = (0.5678) − (0.1958) = 0.372 IS TMMZ Test for Significance: SE Diff * = (.0068) + (.0177) = 0.157 Diff *Z Diff * = SEDiff * Z = 0.372 = 2.372 Diff * 0.157SE Z Diff * = VM * + VM * p = 0.01769 (p < .05) IS TMMEstimate p-value (Stat Tables)or =(1-(NORMSDIST(ABS(Z))))*2(Borenstein et al., 2009) 19
  20. 20. Test of Constructs -cont.- Random-­‐effects  model  (separate  estimates  of  T 2),  Calculated  Z-­‐values SMM TMM IS TMS CC GLSMM -­‐TMM -­‐0.932 -­‐IS 0.705 **  2.372 -­‐TMS -­‐0.4901 0.6593 *  -­‐1.939 -­‐CC 0.14 1.389 -­‐0.7842 0.8674 -­‐GL -­‐1.12 -­‐0.254 **  -­‐2.588 -­‐0.9192 -­‐0.5886 -­‐*  Sign  at  p  =  .10**  Sign  at  p  =  .05IS  >  TMM CV  at  .05  =  1.96IS  >  GL CV  at  .10  =  1.645IS  >  TMS 20
  21. 21. Quality Ranking ComparisonQuality Measure:“Questions to Ask Yourself When Evaluating a Report of a Quanti-tative Study” (Gall, Gall, & Borg, 2010, pp. 537-540)Total of 18 Questions:3-point scale 0 to 2 (0 = N0, 1 = Somewhat, 2 = Yes)Scores Coded:‘Low Quality Ranking’ (< 18)‘Medium Quality Ranking’ (between 18 and 27)‘High Quality Ranking’ (>27) 21
  22. 22. Quality Ranking Comparison -cont.-Interrater Reliability:Each article evaluated by researcherOne half of the articles were evaluated by second researcherChronbach’s Alpha = .800Classification:‘Low Quality Ranking’ - 1‘Medium Quality Ranking’ - 5‘High Quality Rankig’ - 12Re-Classification:‘Low and Medium Quality Ranking’ - 6‘High Quality Ranking’ - 12 22
  23. 23. Quality Ranking Comparison -cont.- Comparison ‘Low and Medium Quality’ -vs- ‘High Quality’ Ranking:Diff * High−Low = (0.3795)− (0.3162) = 0.0633SE Diff * = (0.00342) + (0.02416) = 0.166 0.0633 No Sign. DifferenceZ Diff * = 0.166 = 0.3812 btwn ‘Low and Medium Quality’ and ‘High Qual-p = 0.7037 ity’ Ranked Articles. 23
  24. 24. ConclusionIS was shown to be the better predictor of per-formance: Construct r* (corrected r) Highest IS 0.513 CC 0.4218 SMM 0.394 TMS 0.2984 TMM 0.1933 Lowest GL 0.1456• IS Highest ES of all Shared Cognition Constructs• IS statistically significant compared to GL and TMM• IS marginally significant compared to TMS 24
  25. 25. Conclusion -cont.- Kontoghiorghes et al. (2005) recommended the followingto transform into innovative and adaptive entities in today’s highly complex environment:• Provide employees / students with: - time - facts Relating to Task - information - tools• Allow employees / students the freedom to: - try new ideas - to be risk takers Double-Loop Learning - to challenge the norms Creative Thinking - to be creative 25
  26. 26. Conclusion -cont.-Kontoghiorghes et al. (2005) recommended:“focusing first on... open communications, team-work, resource availability, and risk taking, andthen on building learning network and continuous learning culture” (p. 206). 26
  27. 27. QUESTIONS ? THANK YOUThe University of North TexasJohn R. Turnerjrthpt@gmail.comwww.unt.lt.edu 27
  28. 28. References:Beretvas, N. S. (2010). Meta-Analysis. In Hancock, G. R., & Mueller, R. O, (Eds.), The Reviewr’s Guide to Quantitative Methodsin the Social Sciences (pp. 255-263). New York, NY: Routledge.Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. West Sussex, UK: JohnWiley & Sons.Gall, M. D., Gall, J. P., & Borg, W. R. (2010). Applying Educational Research: How to Read, Do, and Use Research to SolveProblems of Practice (6th ed.). Boston, MA: Pearson.Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis (Vol. 49). Thousand Oaks, CA: SAGE.Mohammed, S., & Dumville, B. C. (2001). Team mental models in a team knowledge framework: expanding theory and measure-ment across disciplinary boundaries. Journal of Organizational Behavior, 22(2), 89-106. Retrieved from www.onlinelibrary.wiley.com/journal/10.1002/(ISSN)1099-1379 28
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×