 Self report (trials and tribulations) Rise of objective measures  (1981 ~ 10 articles, 2011 ~ 800 articles) Special is...
   Bypass report bias   Raw data vs. black box   Include other measures    (heart rate, Galvanic    Skin Response)   S...
PROS                         CONS   Some research-grade       Many not (well)    monitors are validated     validated  ...
 Different measures are related to or  explain different phenomena, and… Give you types of different information
   Sample:    › Combined baseline data from 3 related studies    › N=105 (complete data)    › 74% Latino, 75% female, mea...
   Accelerometer (Actigraph GT1M)     › Wear for 7 days (Inclusion criteria: 4 days 10 hours)                       Moder...
3 or more of the following 5 features:         Feature                                            CriteriaElevated Triglyc...
MetS          Non-MetS                                                                    p-value                         ...
600                                                                                                 *                     ...
Within Time Dimension for MVPA:               Subjective                                   Objective                (3DPAR...
Adjusted OR for MetS aAccelerometer (min)b MVPA                   0.49 (95% CI: 0.25-0.95) Sedentary Behavior    2.56 (95%...
Adjusted OR for MetS a      Adjusted OR for MetS a                                                    (keep both MVPA & SB...
15
Unadjusted correlation      Accelerometer                                  MVPA              SBTriglycerides              ...
Unadjusted correlation                 Partial correlation        Accelerometer                                         MV...
Accelerometer                           3DPARPhysical           • Triglycerides↓Activity↑          • Waist circumference↓ ...
 Assimilation (United States Orientation), Separation (Home Country Orientation), Integration (Both Countries Orientati...
Mean (SD)Age                                           12.1 ± 3.1BMI %tile                                     87.0 ± 20.0...
MVPA                    SED             3DPAR-MVPA                 3DPAR-SED          β          p          β           p ...
Not Marginalized               MarginalizedSleeping                       SleepingSitting in class               Sitting i...
 Valid? Appropriate for targeted population? Train (either participant or staff) to  administer? HUMAN FACTOR2: even a...
 Both *GOOD* objective and *GOOD*  subjective measures of activity contribute to  understanding activity and health Powe...
 NCIUSC Center for Transdisciplinary Research on Energetics and Cancer (U54 CA 116848)
 dmetz@usc.edu www.metzlab.net
2.2 - Good Self-Report Measures of Physical Activity vs. Sensors: Let’s not throw the baby out with the bathwater
2.2 - Good Self-Report Measures of Physical Activity vs. Sensors: Let’s not throw the baby out with the bathwater
Upcoming SlideShare
Loading in …5
×

2.2 - Good Self-Report Measures of Physical Activity vs. Sensors: Let’s not throw the baby out with the bathwater

782 views
688 views

Published on

Tuesday, October 23, 2012
Technical Session #2: Energy Expenditure

Donna Spruijt-Metz (University of Southern California, US), Britni Belcher (National Cancer Institute, US), Ya-Wen Hsu (Chia Nan University of Pharmacy & Science, US)

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
782
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • MSSE 1: The conference, “Objective Measurement of Physical Activity: Closing the Gaps in the Science of Accelerometry,” took place at the University of North Carolina on December 13–15, 2004. ‘‘Objective Measurement of Physical Activity: Best Practices and Future Direction,’’ held on July 20–21, 2009, was to update best practice recommendations for the use of wearable monitors to assess physical activity.
  • The data presented here today is the combined data from 3 related studiesthat share common measures. Total sample size contains 201 participants. Mean age is 13.7, with age ranged from 8 to 18. Distributions of gender and ethnicity were listed in this table. It shows that there are more Latino youths (63.2%) and more girls (72.6%) in our study sample.As part of our study protocol , participants were admitted for an overnight visit (in the afternoon) to the GCRC. In the afternoon, they underwent a brief physical examination and completed body composition test and anthropometry measures. The morning following an overnight fast, Fasting glucose and plasma lipids were collected after an overnight fast===============================================Body Composition: total body fat & lean tissue by air plethysmography (BodPodTM)FSIVGTT: frequently sampled intravenous glucose tolerance tests blood pressure by triplicate using the right arm after resting for 5 minutes waist circumference measured at the umblicus and recorded to the nearest 0.1 cm================SANO: Strength and nutrition outcomes for Latino adolescent, 16-week intervention study to improve metabolic risk factors including insulin resistant, high body fat..visceral fatIntervention: 16 week study with 3 groups: ControlNutrition EducationNutrition Education + Strength TrainingSTAND: Strength training and nutrition development for African American youth – sister study of SANO-LATRANSITIONS: Insulin Resistance and Declining Physical Activity Levels in African American and Latina girls, it is a 4-yr longitudinal observational studyBody composition: total body fat & lean tissue by air plethysmography (BodPodTM )
  • =====================Nonwear was defined by an interval of at least 60 consecutive minutes of zero activity intensity counts, with allowance for 1–2 min of counts between 0 and 100.Count= output from accelerometers; is an estimate of acceleration intensityCut points= MET- based activity level thresholds used to classify counts into activity levelsAccelerometer is designed to filtered out the low-intensity levels so it could have better sensibility in capturing high indensitty?Why include these 3 DB in 3DPAR? They are conceived as unhealthy recreational activity (screen time) associated with the advanced in technology which has been found to related to obesity. Also, research found that snacking may occur while spending time on these SB. EX: HW and sitting in class are sedentary, too, but we don’t want to cause misinterpretation of the data that studying for your health..internalize may increase EE, tooFreedson age-adjusted cut points1Measures acceleration (m/s2 ) over timeProvides output in “counts” over a user-defined time interval (epoch)Counts mapped to energy expenditure (METs)*Cut-points used to define MVPAMinutes summed above thresholds Wear over the right hip on an elasticized beltNonwear time was defined by an interval of at least 60 consecutive minutes of 0 activity intensity counts, with allowance for 1-2 mon of counts between 0 and 100.Recorded every 15 esc epoc, and we collapse them into 60 sec epoc, to be consistent with NHANES.Indirect calorimetry used to estimate the energy expenditure (kcal) or intensity level (MVPA)Sometimes unclear which cut-points to use in which populationsUse of different cut-points can result in difference of 2 hours in time spent MVPA1Different criterion methodsLab-based (treadmill)Field-based (running on track, spontaneous playing)Individual cut-points or counts per minute may be better assessment of activity1Guinhouya, Obesity, 2006
  • There is no standardized pediatric definition.Some popular pediatric definitions of MetS that have been proposed based on ATP III. I used the combined pediatric definitions (based on ATP3) based on Cruz et al and Cook et al. Participants with 3 or more of the 5 features will be classified as having MetS. We used Cruz’s definition of the MetS, except for for hyperglycemia, we used fasting glucose for hyperglycemia based on Cook’s definition, because we do not have the OGTT for all 3 studies. In addition to dichotomizing the MetS, we also use individual MS features for correlation analysis. =====================Differences between impaired fasting glucose and impaired glucose tolerance?Cruz et al and Cook et al definitions based on ATP3 definitions. For impaired fasting glucose, Cook originally used >=110, Shaibi et al updated to >= 100 based on ADA Participants were dichotomized into those with and without MetS, which was defined as having at least 3 of the following.
  • As for ethnicity, there is a cell that is <5, Fisher's exact test.The chi-square is not really valid when you have empty cells (or cells with expected values less than 5). In such cases, you can request Fisher's exact test (which is valid under such circumstances) with the exact option as shown below.For bmi percentile, fat mass, lean mass, use Wilcoxon rank sum testHere this table shows the clinical data of people with MetS and compared to those without. Significant findings are circled in red. The percentage of MetS was higher in males. As we expected, youth with MetS have higher BMI percentiles, fat mass, and lean mass. (search if all in expected directions)Although the ethnic difference did not reach significance, p-values shows that it is marginal significant and that the prevalence of MetS is higher in Latino than in AA youth. This is consistent with NHANES data.For participants with MetS, the mean BMI percentile is above 95Th BMI percentile..which is defined as obese For participants without MetS, the mean o BMI5 percentile is above 85Th BMI percentile..which is defined as overweight
  • This figure shows the differences in MVPA and sedentary behavior by MetS. On the left hand side are the MVPA and SB assessed by accelerometers, and on the right are those assessed by 3DPAR. Significant between-group differences are marked by the *. We see that MVPA is significantly lower and SB is significantly higher in those with MetS than without. For MVPA:For each half-hour block, 3DPAR measures time spent at the dominant intensity level for the dominant activity while Accelerometers detect the actual time of activity. 3DPAR may include time spent warming up/resting and is more likely to provide higher estimates than Accelerometers.In addition, we see that the error bars are larger for 3DPAR, this could be explained by the fact that estimates by 3DPAR is subjective, so it is influenced by respondent’s self-perception of fitness capacity or concern on social desirability (self-reported). Accelerometers , on the other hand, are more like robots, although there are differences between each machine, their differences are usually relatively small when compared to humans.In addition to the associations activity levels and MetS, another clear differences is due to different types of the measurement. It appears that 3DPAR estimates a much longer time spent in MVPA than accelerometry does. For Sb, on the other hand , accelerometry estimates a much longer time spent in MVPA than 3DPAR does. It also shows that… those with and without MetS spend similar amounts of time being sedentary based on the data by accelerometry. For SB in 3DPAR, the definition of SB is user-defined. It allows us to: 1) select those sedentary activities which has been found to related to obesity and adverse health outcomes. Accordingly, it may partially explain why we see a greater differences in SB between youth with and without MetS 2) identify whether certain type of sedentary activity is related to health risk, once the association is established, such sedentary behavior could be further targeted for behavioral change.For SB in Accel: it generally capture all activities under a certain cut-offs of counts. No information is provided on type of activates. In our study, we see similar time in SB, but we don’t know what they are.========The youngest age group recorded twice as much MVPA than the older age groups. The 6-11 age group: 87 minutes 12-15 age group: 32 minutes16-19 age group: 26 minutes
  • 3DPAR measures time spent in dominant intensity level for the dominant activity for each half-hour block while Accelerometers detect actual spent in specific intensity levels of physical activity. 3DPAR may include time spent warming up/resting and are more likely to provide a higher estimates than Accelerometers.In addition, we see that the error bars are larger for 3DPAR, this could be explained by the fact that estimates by 3DPAR is subjective, so it is influenced by respondent’s perception of fitness levels, social desirability, and memory. As 3DPAR is self-reported, more variations are introduced by humans. On the other hand, Accelerometers are determined by continuous wearable monitoring (take human reporting errors out of the equation). They are more like robots, although there are differences between each machine, their differences are usually relatively small when compared to humans.For SB in 3DPAR, the definition of SB is user-defined. It allows us to: 1) select those sedentary activities which has been found to related to obesity and adverse health outcomes. Accordingly, it may partially explain why we see a greater differences in SB between youth with and without MetS 2) identify whether certain type of sedentary activity is related to health risk, once the association is established, such sedentary behavior could be further targeted for behavioral change.For SB in Accel: it generally capture all activities under a certain cut-offs of counts. No information is provided on type of activates. In our study, we see similar time in SB, but we don’t know what they are.===========================Higher estimated by 3DPAR may be due to the fact that 3DPAR measures time spent in dominant intensity for the dominant activity for each half-hour block. By summing these blocks, it is likely that we may over estimate the actual duration. Also, because it is self-perceived intensity,
  • This slide shows partial correlations between PA/SB and MetS. After controlling for age, gender, ethnicity, tanner stage, fat mass, lean mass, it looks more clearly that accel and 3DPAR data had significant findings on different individual features of MetS. There is on overlap like previous slide on blood pressure.Here, accel showed negative correlations bt PA and triglycerides and waist circumference. While PAR showed negative correlations between PA and blood pressure.As for SB, still, only 3DPAR data showed a positive correlation bt SB and blood pressure.======Variables were not normally distributed so statistical tests were run with log-transformed data.Parameters were adjusted for age, gender, ethnicity, Tanner stage, fat mass, and lean tissue mass.
  • Spearman correlations??This slide shows partial correlations between PA/SB and MetS. After controlling for age, gender, ethnicity, tanner stage, fat mass, lean mass, it looks more clearly that accel and 3DPAR data had significant findings on different individual features of MetS. There is on overlap like previous slide on blood pressure.Here, accel showed negative correlations bt PA and triglycerides and waist circumference. While PAR showed negative correlations between PA and blood pressure.As for SB, still, only 3DPAR data showed a positive correlation bt SB and blood pressure.======Variables were not normally distributed so statistical tests were run with log-transformed data.Parameters were adjusted for age, gender, ethnicity, Tanner stage, fat mass, and lean tissue mass.
  • Why no sig findings based on 3DPAR for MVPA?Our results indicated that PA was negatively associated with MetS as measured by accelerometry and 3DPAR, while SB was positively associated with MetS as measured by 3DPAR. Accelerometer data showed that: youth with MetS have lower levels of MPA and MVPA than those withoutgreater participation in MPA (OR=0.47) and MVPA (OR=0.48) were related to lower risk of MetS3DPAR data showed that: youth with MetS have higher levels of SB than those without greater participation in SB was related to greater risk of MetS (OR=2.43)When comparing the results by measurement method in this study (objectively and subjectively measured activity levels), although accelerometry measures of PA/SB objectively, it may not always capture sedentary behavior, which could have an impact on MeteS. The accelerometer, because it assesses physical activity objectively, avoids the recall bias inherent in self-report measures and hence appears to detect differences in activity levels better among youth with and without MetS. 1However, the accuracy of the accelerometry is dependent on activity type.107Accelerometry is not able to capture activity intensity well during movements with static hip position (i.e. biking, strength training) or for water sports (i.e. swimming).107 Additionally, because accelerometry does not accurately measure movement of the upper body performed during low levels of activities, 218 it may not accurately differentiate sedentary from light intensity activities. A unique advantage of self-report measures, and in particular the 3DPAR, is their ability to provide rich contextual data on the types sedentary activities, allowing researchers to focus on the influence of activities that are considered recreational sedentary behavior such as small screen recreation. Self-report of activity levels allows research on sedentary activities directly related to advances in technology that promote prolonged inactive life styles and have been found to have adverse effects on metabolic health. Despite its advantages, there are limitations to self-reported measures of behavior. One relevant limitation is that overweight youth may underreport their time in sedentary behavior due to social desirability.11 However, in this study this type of bias would only make the association between sedentary behavior and MetS stronger than observed, and indicating that our results add support for utilizing a carefully validated self-report measure (3DPAR) to assess sedentary behavior.
  • 2.2 - Good Self-Report Measures of Physical Activity vs. Sensors: Let’s not throw the baby out with the bathwater

    1. 1.  Self report (trials and tribulations) Rise of objective measures (1981 ~ 10 articles, 2011 ~ 800 articles) Special issue MSSE 2005 Troiano + 2007 NHANES pub  objective PA in large samples = feasible & informative Self-report falling out of favor (Sirard + 2001, Spruijt-Metz + 2009) Special issue MSSE 2012: Intille+ recognizes value of self-report.
    2. 2.  Bypass report bias Raw data vs. black box Include other measures (heart rate, Galvanic Skin Response) Syncs to apps & has cool user interfaces Gamefied Will give you participant data Will give you data in real time
    3. 3. PROS CONS Some research-grade  Many not (well) monitors are validated validated Paired with other  Might be expensive or sensors (like GPS) = user-unfriendly valuable synced levels  Miss swimming, biking, of data weight lifting, other Ubiquitous activities If wirelessly enabled,  No contextual can give you real time information (what are data you doing, who are If comfy, low you with?) participant burden
    4. 4.  Different measures are related to or explain different phenomena, and… Give you types of different information
    5. 5.  Sample: › Combined baseline data from 3 related studies › N=105 (complete data) › 74% Latino, 75% female, mean age=13±3 yrs (range: 8-18) Protocols: Overnight admission to GCRC › Body composition: air plethysmography (BodPodTM ) › Blood pressure: in triplicate › Waist circumference: measured at the umbilicus › Fasting glucose, plasma lipids: collected after an overnight fast
    6. 6.  Accelerometer (Actigraph GT1M) › Wear for 7 days (Inclusion criteria: 4 days 10 hours) Moderate-to-Vigorous Sedentary Behavior 2 Physical Activity (MVPA) 1 Threshold ≥4 MET <100 counts/min 3-day Physical Activity Recall (3DPAR) › 7.00 am-12.00 am/day › Identify main activities in half-hour intervals and rate the relative intensity level for each activity MVPA 1 Sedentary Behavior 3 Threshold ≥4 MET playing video game, talking on phone, watching TV/movie METs= Metabolic equivalent tasks, or the ratio of working to resting metabolic rate 1 Freedson et al 1998 2 Matthews et al 2008 3 Pate et al 2003 8
    7. 7. 3 or more of the following 5 features: Feature CriteriaElevated Triglycerides ≥ 90th % for Age and Sex, and Ethnicity (NHANES Ⅲ)1Low HDL-cholesterol ≤ 10th % for Age and Sex, and Ethnicity (NHANES Ⅲ) 1Abdominal adiposity ≥ 90th % for Age Gender, and Ethnicity (NHANES Ⅲ) 1Elevated Blood Pressure ≥90th % for Age, Sex, and Height (NHBPEP)* 1 Impaired Fasting Glucose ≥100 mg/dl (ADA guideline)Hyperglycemia 2 * NHBPEP: National High Blood Pressure Education Program 1 Cruz et al, 2004 2 Cook et al , 2003 9
    8. 8. MetS Non-MetS p-value (N=17/16%) (N=88/84%)Gender Female 9 (11%) 70 (89%) 0.02 * Male 8 (31%) 18 (69%)Ethnicity Latino 15 (19%) 63 (81%) 0.22 African American 2 (7%) 25 (93%)BMI Percentile 98.5 (±1.6) 89.0 (±18.1) <0.001 ‡Fat mass (kg) 41.8 (±19.1) 25.2 (±15.8) 0.001 **Lean mass (kg) 56.0 (±14.0) 46.0 (±16.3) 0.007 **p <.05, ** p <.01, ‡ p <.001x2 tests were used for categorical variables, independent t tests/ Wilcoxonrank sum test for continuous variables 10
    9. 9. 600 * 500Activity levles in minute 462.2 461.8 400 MetS 300 Non-MetS 291.8 * p<0.01 200 210.1 * 147.2 100 102.9 26.5 31.6 0 MVPA Sedentary Behavior MVPA Sedentary Behavior Accelerometer 3DPAR 11
    10. 10. Within Time Dimension for MVPA: Subjective Objective (3DPAR) (Accelerometry) Dominant activity for each half-hour block  Detectable time spent in specific intensity levels of physical activity May explain the relatively higher estimates of MVPA obtained by 3DPAR Influenced by respondent’s perceived of  Determined by continuous wearable fitness levels, social desirability, memory monitoring (take human reporting errors out of the equation) May explain the relatively larger variations in MVPA obtained by 3DPARWithin Time Dimension for Sedentary Behavior: Subjective Objective (3DPAR) (Accelerometry) Sedentary activities selected based on  Captures all detected activities empirical evidence under a certain cut-off May explain greater differences between MetS groups 12
    11. 11. Adjusted OR for MetS aAccelerometer (min)b MVPA 0.49 (95% CI: 0.25-0.95) Sedentary Behavior 2.56 (95% CI: 0.06-119.63)3DPAR (min)b MVPA 0.86 (95% CI: 0.58-1.26) Sedentary Behavior 4.44 (95% CI: 1.41-14.02) a. Adjusted for age, gender, ethnicity, Tanner stage, fat mass, lean mass b. Log-transformed values were used 13
    12. 12. Adjusted OR for MetS a Adjusted OR for MetS a (keep both MVPA & SB in )Accelerometer (min)b MVPA 0.49 (95% CI: 0.25-0.95) 0.49 (95% CI: 0.25-0.98) Sedentary Behavior 2.56 (95% CI: 0.06-119.63) 1.01 (95% CI: 0.02-61.70)3DPAR (min)b MVPA 0.86 (95% CI: 0.58-1.26) 0.99 (95% CI: 0.64-1.56) Sedentary Behavior 4.44 (95% CI: 1.41-14.02) 4.44 (95% CI: 1.33-14.79) a. Adjusted for age, gender, ethnicity, Tanner stage, fat mass, lean mass b. Log-transformed values were used 14
    13. 13. 15
    14. 14. Unadjusted correlation Accelerometer MVPA SBTriglycerides -0.21* ̶HDL-cholesterol ̶ ̶Waist circumference -0.42 ‡ 0.32 **Fasting glucose ̶ ̶Systolic Blood Pressure -0.33 ** 0.21 *Diastolic Blood Pressure -0.27** ̶ Unadjusted correlation 3DPAR MVPA SBTriglycerides ̶ ̶HDL-cholesterol ̶ -0.22 *Waist circumference ̶ ̶Fasting glucose ̶ ̶Systolic Blood Pressure ̶ 0.31**Diastolic Blood Pressure ̶ ̶SB=sedentary behavior - : not significant * p <0.05, ** p <.01, ‡ p <.001 16
    15. 15. Unadjusted correlation Partial correlation Accelerometer MVPA SB MVPA SBTriglycerides -0.21* ̶ ̶ ̶HDL-cholesterol ̶ ̶ ̶ ̶Waist circumference -0.42 ‡ 0.32 ** ̶ ̶Fasting glucose ̶ ̶ -0.21* ̶Systolic Blood Pressure -0.33 ** 0.21 * -0.25* ̶Diastolic Blood Pressure -0.27** ̶ ̶ ̶ Unadjusted correlation Partial correlation 3DPAR MVPA SB MVPA SBTriglycerides ̶ ̶ ̶ ̶HDL-cholesterol ̶ -0.22 * ̶ -0.21*Waist circumference ̶ ̶ ̶ ̶Fasting glucose ̶ ̶ ̶ ̶Systolic Blood Pressure ̶ 0.31** ̶ 0.26 **Diastolic Blood Pressure ̶ ̶ ̶ ̶Adjusted for age, gender, ethnicity, tanner stage, fat mass, lean mass * p <0.05, ** p <.01, ‡ p <.001 17
    16. 16. Accelerometer 3DPARPhysical • Triglycerides↓Activity↑ • Waist circumference↓ • Fasting glucose↓ • Systolic blood pressure↓ • Diastolic blood pressure↓ • Odds of MetS↓Sedentary • Waist circumference ↑ • HDL-cholesterol↓Behavior↑ • Systolic blood pressure ↑ • Systolic blood pressure↑ • Odds of MetS ↑Only significant findings were reported, those with underline were adjusted for covariates  Findings regarding MVPA and MetS were based on the accelerometer data  The adjusted associations between sedentary behavior and MetS were based on the data by 3DPAR Hsu et al, MSSE 2011
    17. 17.  Assimilation (United States Orientation), Separation (Home Country Orientation), Integration (Both Countries Orientation), Marginalization (Neither Country Orientation)1. Physical activity measured by accelerometry and 3-Day PAR 1Unger et al 2002
    18. 18. Mean (SD)Age 12.1 ± 3.1BMI %tile 87.0 ± 20.0% body fat 32.5 ± 11.2Mean mins. MVPA per day (accelerometer) 31.4 ± 21.1Mean mins. MVPA per day (self-report) 96.9 ± 74.2Mean mins. Sedentary per day (accelerometer) 467.0 ± 96.0Mean mins Sedentary per day (self-report) 187.7 ± 107.7
    19. 19. MVPA SED 3DPAR-MVPA 3DPAR-SED β p β p β p β pAGE -3.3 <0.0001 11.0 0.02 -7.2 0.03 -3.9 0.50% fat -0.4 0.05 0.9 0.49 -0.1 0.90 0.2 0.91Margin-ization 12.6 <0.0001 -31.4 0.10 -12.8 0.08 29.1 0.02
    20. 20. Not Marginalized MarginalizedSleeping SleepingSitting in class Sitting in classWatching TV or movie Watching TV or movieHanging around Eating a mealEating a meal HomeworkRiding in a car/bus Hanging aroundPlaying video games/ surfingInternet while sitting Lunch/free time/study hallDoing house chores Riding in a car/busTravel by walking Doing house choresHomework Working (e.g., part-time job)
    21. 21.  Valid? Appropriate for targeted population? Train (either participant or staff) to administer? HUMAN FACTOR2: even a great self- report measure can’t protect you from poor data collection.
    22. 22.  Both *GOOD* objective and *GOOD* subjective measures of activity contribute to understanding activity and health Powerful methodologies will combine objective measures, a mix of sensors, and prompted real-time self-report
    23. 23.  NCIUSC Center for Transdisciplinary Research on Energetics and Cancer (U54 CA 116848)
    24. 24.  dmetz@usc.edu www.metzlab.net

    ×