Landis Lewis, Z. et al.: Measuring Efficiency of Use in a Web-based EMR Developed for Malawi: Lessons Learned From Performing a Preliminary CogTool Analysis This slideshow, presented at Medicine 2.0’08 , Sept 4/5 th , 2008, in Toronto, was uploaded on behalf of the presenter by the Medicine 2.0 team Do not miss the next Medicine 2.0 congress on 17/18th Sept 2009 ( www.medicine20congress.com ) Order Audio Recordings (mp3) of Medicine 2.0’08 presentations at http://www.medicine20congress.com/mp3.php

This slideshow, presented at Medicine 2.0’08 , Sept 4/5 th , 2008, in Toronto, was uploaded on behalf of the presenter by the Medicine 2.0 team

Do not miss the next Medicine 2.0 congress on 17/18th Sept 2009 ( www.medicine20congress.com )

Order Audio Recordings (mp3) of Medicine 2.0’08 presentations at http://www.medicine20congress.com/mp3.php

Measuring Efficiency of Use in a Web-based EMR Developed for Malawi: Lessons Learned From Performing a Preliminary CogTool Analysis Zach Landis Lewis, MLIS Gerald Douglas, MSIS Valerie Monaco, PhD, MHCI University of Pittsburgh Department of Biomedical Informatics

Background: Malawi 12.4 million 119,282 km 2 Pennsylvania 13.6 million Population (2007 est.) 118,484 km 2 Area Malawi 43.5 77.8 Life expectancy at birth in years 900,000 (14.2%) 18,000 (0.15%) Number of people living with HIV/AIDS

Background: Baobab Anti-Retroviral Therapy system (BART)

Objective Our research objective is to determine how efficiently novice users complete tasks using the touchscreen interface of the EMR.

Our research objective is to determine how efficiently novice users complete tasks using the touchscreen interface of the EMR.

Methods Predict skilled task performance Select tasks Use CogTool software application to generate prediction 2. Measure novice task performance Collect timestamp data from user interface events (e.g. pressing a button ) Repeat each task three times 3. Compare prediction with results of novice performance

Predict skilled task performance

Select tasks

Use CogTool software application to generate prediction

2. Measure novice task performance

Collect timestamp data from user interface events (e.g. pressing a button )

Repeat each task three times

3. Compare prediction with results of novice performance

Methods: CogTool validated and used in the field of Human Computer Interaction over 100 papers validating or using human performance modeling for evaluation or design of interfaces Q: What is predictive human performance modeling? A: A method for predicting how long a skilled user will take to complete a task Examples of real-world applications: - Web pages and browsers - Telephone operator workstations - Space operations database system - Television control system - Intelligent tutoring system - IRS office automation system - Police in vehicle systems - Firefox tab feature

validated and used in the field of Human Computer Interaction

over 100 papers validating or using human performance modeling for evaluation or design of interfaces

Examples of real-world applications:

- Web pages and browsers

- Telephone operator workstations

- Space operations database system

- Television control system

- Intelligent tutoring system

- IRS office automation system

- Police in vehicle systems

- Firefox tab feature

Methods: CogTool

Methods: CogTool

Methods: CogTool The five clusters of colored bars represent all the button presses required to perform this task, separated by thinking time. “ 5” “ 8” “ .” “ 3” “ Next”

Methods: CogTool This is the final hand movement operator for pressing the button labeled “5”. This pane shows a close-up view of a sequence of cognitive resources being used. Here we see the activities for pressing the button labeled “5”

Methods: CogTool This is a “trace” of production rules fired by the ACT-R production rule system during the task performance The highlighted production rules correspond with cognitive activities occurring while a user is pressing the button labeled “5”.

Results: CogTool Selected 31 routinely performed tasks in BART Used CogTool to predict skilled task performance Predicted performance times in seconds for each task

Selected 31 routinely performed tasks in BART

Used CogTool to predict skilled task performance

Predicted performance times in seconds for each task

Results: Novice Performance Rate of errors: Errors are any deviation from the optimal sequence of steps required to complete a task 77% (286) of task performances were error-free and were compared with CogTool predictions 4 of the 31 tasks were performed without error by all subjects on all repetitions

Rate of errors:

Errors are any deviation from the optimal sequence of steps required to complete a task

77% (286) of task performances were error-free and were compared with CogTool predictions

4 of the 31 tasks were performed without error by all subjects on all repetitions

Results: Comparison of CogTool Prediction with Novice Performance

Discussion 1. CogTool allowed us to rapidly generate predictions of skilled performance 2. Novice subjects demonstrated a low error rate 3. Novices performed faster than CogTool predictions on average : Tasks were modeled independently, but users interleaved some tasks CogTool's assumptions for inserting "Think" events may not be applicable for wizard format interfaces

1. CogTool allowed us to rapidly generate predictions of skilled performance

2. Novice subjects demonstrated a low error rate

3. Novices performed faster than CogTool predictions on average :

Tasks were modeled independently, but users interleaved some tasks

CogTool's assumptions for inserting "Think" events may not be applicable for wizard format interfaces

Discussion, continued 4. Unexpected findings: Pittsburgh subjects occasionally used more than one hand to manipulate the interface – (but we haven’t observed that in Malawi… yet) Communication time varied greatly between tasks, sometimes resulting in prolonged dialog rather than a single question and answer

4. Unexpected findings:

Pittsburgh subjects occasionally used more than one hand to manipulate the interface – (but we haven’t observed that in Malawi… yet)

Communication time varied greatly between tasks, sometimes resulting in prolonged dialog rather than a single question and answer

Future Work 1. Update the CogTool model to reflect current, more sophisticated understanding of tasks and user actions - We are working with the CogTool team to be able to adjust the models and CogTool itself to fit the assumptions to our tasks and users 2. Characterize the use of the system in a real-world setting Collect anonymized user interface event data in Malawi from a representative group of users Measure system use by novices and skilled users

1. Update the CogTool model to reflect current, more sophisticated understanding of tasks and user actions - We are working with the CogTool team to be able to adjust the models and CogTool itself to fit the assumptions to our tasks and users

2. Characterize the use of the system in a real-world setting

Collect anonymized user interface event data in Malawi from a representative group of users

Measure system use by novices and skilled users

Acknowledgements The National Institutes of Health and the National Library of Medicine, USA - Grant # 5T15LM007059-22 for funding this research Bonnie John, PhD The CogTool Project - http://www.cs.cmu.edu/~bej/cogtool/ Greg Cooper, MD, PhD Mike McKay Joe Rauch, DDS Yolanda DiBucci Margaret Henry

The National Institutes of Health and the National Library of Medicine, USA

- Grant # 5T15LM007059-22 for funding this research

Bonnie John, PhD

The CogTool Project - http://www.cs.cmu.edu/~bej/cogtool/

Greg Cooper, MD, PhD

Mike McKay

Joe Rauch, DDS

Yolanda DiBucci

Margaret Henry