Gary Marrer presented at the 25th Annual Distance Teaching and Learning Conference on assessing the effectiveness and efficiency of mobile teaching and learning. He discussed various methodologies for assessing mobile learning, including traditional assessment techniques as well as newer measures like tracking student retention and skills development. Marrer also reviewed examples of empirical research on mobile learning and frameworks for evaluating mobile learning at the micro, meso, and macro levels. He emphasized that assessing mobile learning effectiveness is still a work in progress due to the newness of the technology.

1. 25th Annual Distance Teaching and Learning ConferenceUniversity of Wisconsin –Madison ________________________ Overview of Current Research on Effectiveness and Efficiency of Mobile Teaching and Learning Gary Marrer Glendale Community College Maricopa Community College District Glendale Arizona

2. Presentation Goals Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 My Methodology… Mobile learning (effectiveness and efficiency) = assessment Survey existing research on mobile learning assessment. How should we assess mobile learning?

3. My Methodology (perspective & bias) Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 I am teacher not researcher so I my view is from the classroom A preference for “hands on” Community college faculty 20 years corporate I/T Experience. An occupational / technical perspective I teach technology courses and have written programs for smart phones with Java, Net and Android technology I have developed software tools for mobile learning content delivery I have a passionfor technology and learning Effectiveness and Evaluation of mobile learning is still a work in progress

4. How does mobile learning fit it? It is a Blended World. Hybrid Example Hybrid Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

5. Effectiveness and Efficiency = Assessment Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 what does it mean to assess the effectiveness of a pedagogy or technology? “For many instructors, assessment just means grading, but it actually entails much more than this. When designed thoughtfully and implemented carefully, assessments can provide important evidence about what and how students are learning and whether course goals have been met.”1 1Strategies for Assessing Learning Effectiveness : by Constance A. Steinkuehler & Sharon J. Derry University of Wisconsin–Madison Copyright© NJIT & Constance Steinkuehler 2001 . http://www.alnresearch.org/HTML/AssessmentTutorial/

6. Mobile Learning Effectiveness and Efficiency Combination of all or any of these techniques Pre-Post Assessment Survey’s Test/Quiz Feedback Observation Combination of all or any of these techniques plus … Evidence of student retention Evidence of skills for a life time of learning Validation of personalized and student centered approaches Enhanced Data Collection – Digital Tracking Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Traditional Assessment Techniques Potential New Assessment Measures

7. Seeds of assessment discussion and dissent …. Does the use of mobile devices improve learning or is it just taking a popular technology and fitting into coursework? (the skeptics...) Is technology effective in learning? Is it more cost effective than other alternatives? (the administrators…) Are traditional assessment mechanisms for mobile learning valid? Does mobile learning need newer assessment mechanism? (the users…) Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

8. Survey Criteria I researched the answer to this question: As a reluctant teacher or a teacher who needs to provide a convincing argument to others who are less confident, what research on mobile learning effectiveness and efficiency can I review to help me out? Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

9. Research Sample1 - Empirical Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Pragmatic/Empirical – Comparison along similar variables / Data Driven A literature review of M-learning effectiveness by Charles Rich, http://komodo.utsystem.edu/med/crich/PMCH_6381/PMCH_6381_Assignment_3.pdf Mobile learning in higher education: an empirical assessment of a new educational tool http://www.scribd.com/doc/4494956/MOBILE-LEARNING-IN-HIGHER-EDUCATION-AN-EMPIRICAL-ASSESSMENT-OF-A-NEW-EDUCATIONAL-TOOL MLearning: pedagogical, technical and organizational hypes and realities, Pul B. Muyinda, Department of Distance Education, Institute of Adult and Continuing Education, Makerere University, Kampala, Uganda, and Department of Information Systems, Faculty of Computing and IT, Makerere University, Kampala, Uganda http://www.emeraldinsight.com/Insight/viewPDF.jsp?Filename=html/Output/Published/EmeraldFullTextArticle/Pdf/1650240202.pdf

10. Research Sample 2 – Institutional Commitment Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Mobile learning work at Abilene Christian University College with “focused” commitment to Mobile Learning Over 1000 iPhone/iTouch devices for Students, Faculty and Staff. Customized ACU content ACU's iPhone initiative: a year later by Brett Terpstra http://www.tuaw.com/2009/07/24/acus-iphone-initiative-a-year-later/

11. Research Sample 3 – An Expanded View Mobile Learning Assessment Criteria (Traxler, 2007) Rigorous, meaning roughly that conclusions must be trustworthy and transferable Efficient, in terms of cost, effort, time, or some other resource Ethical, specifically in relation to the nuances of evolving forms of provision, in terms of standards from • Legal to • Normative Proportionate, that is, not more ponderous, onerous, or time-consuming than the learning experience or the delivery and implementation of the learning itself (bearing in mind earlier remarks about the learners’ experiences of mobile learning) Appropriate to the specific learning technologies, to the learners, and to the ethos of the learning – ideally built in, not bolted on Consistent with the teaching and learning philosophy and conceptions of teaching and learning of all the participants Authentic, in accessing what learners (and perhaps teachers and other stakeholders) really mean, really feel, and sensitive to the learners’ personalities within those media Aligned to the chosen medium and technology of learning Consistent across: • different groups or cohorts of learners in order to provide generality • time, that is, the evaluation is reliably repeatable • whatever varied devices and technologies are used Traxler, J. 2007. Defining, Discussing and Evaluating Mobile Learning: The moving finger writes and having writ . . . .. The International Review of Research in Open and Distance Learning [Online] 8:2. Available: http://www.irrodl.org/index.php/irrodl/article/view/346/875 Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

12. Research Sample 4 – A Three Level Framework (systematic) Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Micro level: the micro level examines the individual activities of the technology users and assesses usability and utility. In the case of Myartspace the activities included collecting objects through exhibit codes, making notes, contacting people who have collected a particular item, recording audio, and taking pictures. Mesolevel: the meso level examines the learning experience as a whole, to identify learning breakthroughs and breakdowns; it also examines how well the learning experience integrates with other related learning 300 experiences. In the case of Myartspace, evaluation at this level involved exploring whether there was a successful connection between learning in the museum and in the classroom as well as identifying critical incidents that reveal new patterns and forms of learning or where learning activity is impeded. Macro level: the macro level examines the longer term impact of the new technology on established educational and learning practice. For Myartspace this related to the organisation of school museum visits. The evaluation at this level looked, for example, at the appropriation of the new technology by teachers, the emergence of new museum practices in supporting school visits, and how they related to the original project visions. Vavoula, G & M. Sharples. (2008). Challenges in Evaluating Mobile Learning. Proceedings of the mLearn Conference Ironbridge Gorge, Shropshire, UK. http://www.scit.wlv.ac.uk/brendan/mLearn2008.pdf YouTube Presentation at: http://ow.ly/hM6f

13. mLearning Resources …continued Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Great sites to start your mobile learning investigation mLearnopedia - http://mlearnopedia.com/ Educause - www.Educause.com (search keyword mLearning or mobile learning) Abilene Christian University Mobile Resources Page http://www.acu.edu/technology/mobilelearning/mLearning_Resources/index.html Great mobile learning articles Are You Ready for Mobile Learning? By Joseph Rene Corbeil and Maria Elena Valdes-Corbeil , http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/AreYouReadyforMobileLearning/157455 REPORT 11: FUTURELAB SERIES - Laura Naismith, Peter Lonsdale, Giasemi Vavoula, Mike Sharples University of Birmingham, http://www.futurelab.org.uk/resources/documents/lit_reviews/Mobile_Review.pdf Defining, Discussing and Evaluating Mobile Learning: the moving finger writes and having writ . . . . John Traxler University of Wolverhampton, UK http://www.irrodl.org/index.php/irrodl/article/view/346/875

14. Links to My mLearning Resources Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 My Links for my Mobile Sabbatical and Panel Discussion http://docs.google.com/View?id=dhc6qkdw_60776vmfccq

15. From Research: Things to think about… Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Maybe more questions than answers – Why? Homogeneity in mobile learning difficult or non-existent Questions of using assessment for continuous improvement versus determining a “winner” Newness of technology makes standards problematic. Research validity problems identified… Hawthorne effects Populations not similar Bias by design

16. As teachers we ask … Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Is traditional measures of assessment enough to gauge effectiveness and efficiency of mobile learning? What should our definition be for mobile learning assessment? What is the impacts of blended learning on assessment? Can we compare apples to apples? What are impacts of assessment when technologies that personalized instruction are used? Can it be measured when each students experience could be different? What is the best way to develop a plan for mobile assessment? How do we continue the discussion on mobile learning assessment (i.e. Blog’s, wiki’s, etc.)?

17. Thank You… Questions Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 Gary Marrer http://www.google.com/profiles/gmarrer My email gary.marrer@gcmail.maricopa.edu Presentation http://www.slideshare.net/gmarrer/uo-w-25th-annual-distance-learning-conference My web sites: mobiledot.ning.com(a mobile learning community) www.laptoppress.com (technology and learning projects) My Blog on mobile teaching and learning: www.garymarrer.com/wordpress

18. Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

19. Systematic Approach…… Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

21. System Stats

22. Usage Data

23. Statistical Analysis

24. Trend Analysis

25. Pre/Post Testing

26. Survey

27. ObservationGary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009

28. Mobile Learning EXperiment(MLEX) Gary Marrer – Annual Conference on Distance Teaching and Learning - Summer 2009 A MS-Windows Application that will assist the web author in designing an xHTML page viewable on PC or Mobile Device. http://mobiledot.ning.com http://www.laptoppress.com