SlideShare a Scribd company logo
Strategies for Evaluating
Visitors’ Experiences
Using Technologies
Jackie Armstrong, Landau Education Fellow, The Museum of Modern Art
Karen Plemons, Assistant Museum Educator for Program Evaluation, The
Metropolitan Museum of Art
Part 1
Evaluating Visitors’
Experiences Using
Technologies
Mobile
Apps
Mobile
Apps
User Testing
an App
Activity with
Children
User Testing
an App
Activity with
Children
Interactives
in Museums
Interactives
in Museums
Interactives
in Museums
Interactives
in Museums
Interactives
in Museums
Mobile
Devices in
the Galleries
QR Codes
QR Codes
QR Codes
Activity
Scanning QR Codes
Part 2
Using Technologies to
Evaluate Visitors’
Experiences
Online
Surveys
Online
Surveys
Using iPads
Photography
Google
Analytics
Google
Analytics
Google
Analytics
Any Questions?
Thank you!

More Related Content

Recently uploaded

Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
Anna Sz.
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
Jisc
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptx
RaedMohamed3
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
kaushalkr1407
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
Vivekanand Anglo Vedic Academy
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
Mohd Adib Abd Muin, Senior Lecturer at Universiti Utara Malaysia
 
Acetabularia Information For Class 9 .docx
Acetabularia Information For Class 9  .docxAcetabularia Information For Class 9  .docx
Acetabularia Information For Class 9 .docx
vaibhavrinwa19
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
MIRIAMSALINAS13
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345
beazzy04
 
678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf
CarlosHernanMontoyab2
 
"Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe..."Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe...
SACHIN R KONDAGURI
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
TechSoup
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Atul Kumar Singh
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
DhatriParmar
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
Special education needs
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
Tamralipta Mahavidyalaya
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
TechSoup
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
Jean Carlos Nunes Paixão
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
Jisc
 

Recently uploaded (20)

Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptx
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
 
Acetabularia Information For Class 9 .docx
Acetabularia Information For Class 9  .docxAcetabularia Information For Class 9  .docx
Acetabularia Information For Class 9 .docx
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345
 
678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf
 
"Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe..."Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe...
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
 

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
Marius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
Rajiv Jayarajah, MAppComm, ACC
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
Christy Abraham Joy
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
Vit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
MindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

Nycmer strategies for evaluating visitors' experiences using technologies (website version) may 2, 2013

Editor's Notes

  1. - Karen introduces herself, then Jackie introduces herself -Jackie: - We’ll going to be talking with you today about strategies for evaluating visitors’ experiences using technologies -just to give you an idea about how this presentation is laid out, in the first section we’ll be talking about strategies for evaluating visitors’ experiences that are primarily analog or more traditional, while in the 2 nd part we’ll be sharing our experiences of using technologies to evaluate visitor experiences. -we are also going to have all of you test out some technologies, as well as leave time for some discussion at the end -one thing to keep in mind as we share our examples is that there is no perfect evaluation -often as evaluators we need to get the best data we can out of the time, resources and constraints of a particular situation -also, although we will mostly be focusing on digital technologies, it’s important not to overlook analog technologies or to assume digital is better than analog -in fact, analog technologies often inform the creation and use of digital technologies -we hope our experiences will help all of you create evaluation strategies that work for you in whatever situations you are facing -and now I’m going to turn this presentation over to Karen to talk about mobile apps
  2. Overview of project -Mobile application designed to accompany Light: Installations by Bruce Munro at Longwood Gardens was the first large-scale mobile application (Light App) developed by Longwood staff -App featured videos of installation, audio from the artist talking about each piece, and an overview of the exhibition from the director, photographs of the artworks, a map of where the pieces were located, information regarding the materials used in each piece, kids features with kid centered questions for guided looking, a place to share photos and/ or comments to the museums website and general information about the gardens including dining options and hours -Overall a lot of information to navigate
  3. Evaluation Strategy -Visitor surveys in the gardens, and member surveys sent online -Docent Focus groups (docents had been trained to use ipads with the apps loaded to show aspects included in the app with visitors) Findings -Visitors were most frequented the map that helped visitors in navigating the gardens and finding the pieces related to the show -When asked what they would like in future apps visitors rated a map of the garden to be able to get around as their first choice, and plant names/ information, and a history of the gardens and the area as their second choice Strengths of the App -Worked well at night when the exhibit was at it’s best.  If there had been extensive signage it would have had to be well lit taking away from the installations -Worked well for providing variety of media including text, images and videos Shortcomings of the App -There was not enough wifi to use the app throughout the gardens, leading to frustration and visitors giving up on using it -Most visitors surveyed did not know about the app, or that they could check out in iTouch to use the app- which could have been remedied through having the ticket sales team let visitors know about the apps, and setting up multiple stations to distribute itouches- particularly near where docents were stationed with iPads which was a 10-15 minute walk from entrance.   -If visitors were trying to use the app on a sunny day it was difficult to see the screen and the videos and they would have to find shade Shortcomings of the evaluation strategy -Because the evaluation began at the end of the exhibit, changes to the app, or distribution of ipads, or location of wifi hotspots could not be changed.  If the evaluation had been integrated from the start of the project the gardens would have known more about what visitors were interested in an app, and the logistical challenges -Because evaluation was an afterthought, they did not turn on the google analytics to the site until the exhibition was halfway over.  Meaning that we had a limited amount of data to look at in terms of the actual use of the app and site developed for the exhibit.  
  4. Overview of Project -one evaluation that was somewhat tricky and required a lot of patience was testing out a new activity for the MoMA Art Lab App -The creators of the new activity, inspired by Sol LeWitt’s work “Lines, Not Straight, Not Touching, 4 Colors” were unsure about whether the instructions were clear and also wanted to know if children would think the activity was fun -for this particular evaluation we needed to get the data back to the designers within a pretty short timeframe, as well we needed to recruit children 7 years and older to test this out but it was pre March break so there wasn’t many children around in that age range -the patience part of this evaluation came into play because I spent a lot of time waiting for children who could participate in this user testing -there is actually often a lot of waiting around in evaluation in terms of collecting the data you need so it’s important to work that into your plan -for example if you are in this position of waiting to collect data, that time can be used to review data already collected, expand on notes, photograph the space you are in, basically anything that will add to your study
  5. Evaluation Strategy and Findings -For this particular study, ideally we would have mocked up interactive wireframes onto a device but there wasn’t time or resources for that so we paper tested and tried to make that experiences as engaging as possible -generally, children aren’t as receptive to paper testing as adults and it’s more difficult for them to envision how a paper format of an activity would translate into a digital one -what was helpful was recruiting from and conducting the testing in the MoMA Art Lab space as I think that environment helped set a playful tone and put children into a more receptive and at ease mood for user testing -Also, rather than using think-aloud protocol as we typically do for user testing with adults (where participants voice everything they do and make comments and ask questions as they arise), I chatted with children as they worked on the activity which put them at ease and encouraged feedback about what they thought of the activity -Paying attention to expressions and body language was also key as sometimes children are more apt to says things to please than adults are -I also had children choose a face on a mood scale to find out how they’d characterize their feelings/enjoyment of the activity -something else that I found helpful was fleshing out my notes immediately after each testing finished so as not to distract or make the children feel uncomfortable while they were doing the activity -I also found that really approaching it more as another activity in the Lab space rather than as user testing helped to get more honest feedback, and also kept parents at bay since sometimes they want to get overly involved in an activity -early on in the testing, a couple parents really wanted their child to get the activity right which sort of defeats the purpose of trying to figure out how children are really responding to the activity -another strategy which is useful, but requires more setup, is having children work in pairs to play test -children will more naturally communicate their feelings about an activity with one another while engaged in play together and this simulates a kind of think-aloud protocol that researchers rely on when user testing with adults Findings: -The drawings children created in testing out this activity also provided information about how clearly they understood the instructions and how they interpreted them, for example did they use only one color or select four as the rules indicated -Children were really interested in having options to choose from (number of drawings to create, color choices) and being able to share what they made -there was also interest in seeing other examples of Sol LeWitt’s work -interestingly, some children tried to copy the image rather than interpreting the rules their own way -in terms of testing, children also liked being able to contribute their ideas and hearing that their feedback would impact the App activity that was being created
  6. -Observations and interviews with families regarding their experience in a technology rich hands on exhibit on animal intelligence
  7. MoMA Media Lounge -one evaluation I’m in the midst of right now is for MoMA’s Media Lounge -for this study I’m using observations (including timing) and interviews, thinking about the location of this interactive and its purpose -the Media Lounge is meant to give visitors an opportunity to study /research videos in MoMA’s collection and our main question is how are people using the space and why are they using it in those ways -located in a busy hallway that can get quite noisy there seems to be an obvious disconnect between its location and purpose, as a place of research -there are 7 kiosk –like structure in the Media Lounge and while some people do stop to watch and listen to videos (generally in the 3 kiosks closest to the Café and the Education Information desk), others use the kiosks to relax, use mobile devices, re-arrange their tote bags and even breast feed
  8. -one thing clear from evaluation so far is that visitors who do use the Media Lounge are very frustrated by technological problems, such as videos cutting out, images freezing and audio not working -the observations and interviews are also revealing visitor response to the comfort (or lack of comfort) in the space, for example how long people are willing to sit on a low wooden stool -you’ll notice from the image that even though there are screens up around the monitors and iPads, there is a lot of glare that comes in from the windows along that hallway -the name Media Lounge also doesn’t readily tell people what the spaces is all about -hopefully findings from this evaluation will help ensure that improvements made to the Media Lounge will enable it to better connect to its purpose and improve the visitor experience it currently offers -as well, seeing how people are using this space also gives us insight into the needs of visitors in general, for example looking for more places to rest/sit, charge their mobile devices , make calls on their cell phones etc.
  9. -Another evaluation I’m currently in the midst of is for MoMA’s LaBrouste exhibition -there are a few different reasons why evaluation is being done for this exhibition but one is because the curator wanted to know more about why people were not using the interactive shown in this slide as much as he thought they would -Right now, I’m doing tracking and timing in the exhibition to help answer this question and will likely conduct interviews as well -Just to explain briefly about tracking in case anyone is unfamiliar with this approach, you basically take a floor plan of the exhibition and create a map from that, labeling objects, text, seating and anything else in the exhibition -once you have a map created you can get started -what I do is stand at the start of the exhibition, trying to be as unobtrusive as possible and the first person I see about to enter is the person I will track -I immediately record their start time as well as noting things like gender, age, who they are with etc. -from that moment on, I make note of every place they stop, what they read, what they take photos of, as well as using lines and arrows to record movement through the exhibition -I follow that person throughout the exhibition and note the time they leave -after I’ve tracked about 50 individuals, all of those times calculated to get an average of time spent -for the LaBrouste exhibition, I’m also noting how long visitors spend in each of its 3 sections to see if there is drop off at any point -in terms of this particular interactive, there seem to be a few things affecting how little visitors are using it -the placement of the interactive, at the end of exhibition, may be one factor affecting how infrequently visitors stop to use this -the exhibition is also information heavy, lots and lots of text panels, tombstone labels, extended labels and other didactic panels, and labeled architectural plans -it’s very possible that by the time they get to the interactive they have exhibition fatigue -I’ve also noticed some visitors will stop to look at the interactive but seem unsure of how to interact with it - although it’s a touch screen, how to go about searching the information is not particularly intuitive -it’s also possible that the organization and content on this interactive, translated directly from French to English, may contribute to people’s uncertainty about how to explore this -interestingly, there are several small computer screens showing images of architectural drawings and buildings that also do not attract a lot of attention while the projected images of buildings and landscapes do cause many to pause, and a fairly large percentage of people stop to use the 3D glasses to watch a clip of Hugo (actually opposite the all where this interactive is) -Again, I think that location and function play a big role in this, as well as the time people are willing to spend in one exhibition -While I’m sure some visitors would love to go deep into the material presented in this exhibition, a large interactive, which you explore by standing up, might not be the most inviting way to do this
  10. MoMA Art Lab App (installed in Cafe 2) -Another interactive that I’ve spent time observing how people are using it, is the iPad and large screen installed in Café 2 with the MoMA Art Lab app -from observations we’ve been able to find out who’s using it, how long they are using it, how they are using it, any issues people run into while using it, and how spectators are responding to when someone else is using the App -two of the findings that I think are most interesting is how engaged people can get while using this App (spending a considerable amount of time, taking photos of what they create), despite the sometimes crowded and noisy conditions of its location, as well as how involved spectators get (e.g. shouting out suggestions of what to draw)
  11. Mobile Devices in Galleries -so as we are all aware, mobile devices are so prevalent in today’s society that people seem to take them everywhere they go -while some museums may try to fight against these tendencies, (e.g. prohibiting the use of mobile devices in galleries), in many ways it’s better to understand these behaviors and see how they can be used to enhance the visitors’ experience -in thinking about the way visitors use audio devices at MoMA and the app currently available, last spring we also looked at what people were doing with the mobile devices they brought into the galleries -Over a number of days and in all parts of the museum, visitors observed using their mobile devices were interrupted briefly to answer a few questions -we found out that 92% of visitors did not use their own device to listen to MoMA Audio even though about half realized they could do this -20% of visitors looked up information about MoMA or art in general on their devices -70% took photos with their devices and 50% of those who took photos said they shared some immediately with friends/family -visitors were very much interested in showing others where they were and what they were doing -38% made comments on Twitter, Facebook or other social media during their visit -There was also interest in being able to customize their experience at MoMA using their devices as well as having access to information not found on labels or text panels -several visitors also willingly showed what they had posted on Twitter/Facebook, photos they took and in a few instances, games they were playing to relax in between exploring galleries -What’s been great about collecting this kind of data is how much it has been able to inform our thinking about certain initiatives, such as participatory experiences in the galleries, in-gallery games design, the creation of a new App, the kinds of content visitors are interested in, the importance of having areas to recharge, as well as offering more customized and personal experiences to our visitors
  12. -Almost as pervasive as mobile devices in society, QR Codes (Quick Response Codes) seem to be everywhere including in museums -I sometimes think there’s this idea that using QR codes will automatically make something more interactive or an assumption that people want to use their mobile devices to access more information -But without an appropriate context and well thought out strategies for engagement, using QR codes can actually take away from an experience -one exhibition at MoMA where QR codes worked well was in the exhibition “Talk to Me: Design and Communication between People and Objects” -I’m not going to talk about this exhibition since I wasn’t at MoMA at that time but I did want to mention that the reason it was so successful was because the QR codes were fully integrated into the exhibition design and thought process of the exhibition -in the Cindy Sherman exhibition at MoMA this past summer, QR codes were more of an afterthought and didn’t really fit the situation -since we knew from informal observations and analytics that visitors were not scanning the QR codes much we conducted a study to find out more -there were 10 QR codes in Cindy Sherman on tombstone labels in the exhibition, and if visitors scanned them they could access videos featuring artists, curators and critics discussing their favorite works by the artist -for this study, each of the 10 QR code locations in the exhibition was observed for 20 minutes to not only determine whether visitors were scanning the codes but to find out about their experiences doing so -visitors observed scanning (or attempting to scan) were approached and asked questions about their motivations, expectations and experiences scanning QR codes in this exhibition -as well, visitors who looked at the labels with the QR codes but did not scan were also interviewed to find out if they were aware of the codes and if they were interested in scanning them -the QR codes in Cindy Sherman were rarely scanned by visitors -visitors found out about QR codes by seeing them on the object labels -Many visitors demonstrated confusion about how to scan. Some assumed that their MoMA Audio guide could scan the codes and didn’t realize they needed a smart phone and app to scan them -For visitors who had a smart phone but not a QR reader, it took several minutes to search for and download the appropriate App. Part of this had to do with our wireless signal not being consistent throughout the Museum at that time -No visitors were aware of what content they might access by scanning the QR codes, making many hesitant to take the time to do so (and use their data plan and battery life) -When visitors accessed the videos of individuals discussing their favorite photography by Cindy Sherman, many expressed some disappointment and said they were interested in knowing the story behind the photograph or hearing directly from the artist, not seeing talking heads -as well, when visitors scanned the QR codes they did not go the exact video related to the photo they were viewing but instead had to scroll down a list to find the right one
  13. The size and placement of the QR codes also posed some challenges: Since most visitors stood back from the large scale photographs, many did not notice the QR codes Getting close to the labels to scan made many security guards nervous about visitor proximity to the works of art The distribution of QR codes was uneven: there were 4 concentrated in the first gallery (setting up an expectations for a higher density of QR codes), but the remaining 6 were spread fairly consistently throughout the remaining galleries Is it the right exhibition? -Sometimes QR codes just don’t work with a particular exhibition -Cindy Sherman was the kind of exhibition were visitors stepped back to take in the photographs and enjoyed discussing with one another what they thought was going on in the photograph. The QR codes did not further engagement with the content Who has access? -Not everyone has a smart phone, and those who do may not have a QR code reader installed. Even if someone has both, many visitors are simply unwilling to use their device in that way Troubleshooting -Some of the confusion and frustration felt by visitors can be eased by putting up a few clear signs about how to scan QR codes. Let visitors know what they need to scan and what app they need to download. Making sure the WiFi is strong enough to support app downloads and streaming video is crucial before offering this content Content: If I scan, what will I get? -Visitors want to have an idea of what they will get if they scan -including a short tagline on the label briefly describing the medium and content (e.g. “Watch a video of Glenn Lowry sharing his thoughts about Untitled #263”) gives visitors a chance to decide whether they want to make the effort to scan or not. Including the runtime of the video is also helpful
  14. Follow-up -Follow up observations were conducted in Cindy Sherman to see if adding directions and a description of content to the object labels would encourage more visitors to scan QR codes -in addition, each QR code was updated to link to the individual video about that photograph, rather than a web page featuring all 10 videos -in 80% of the locations, visitor use of the QR codes was higher after the taglines were added -additionally, in several instances it was clear that taglines helped visitors to make more informed decisions about whether to scan or not
  15. Scanning our QR code: What worked? What didn’t Examples of QR codes that were successfully used in a museum context? Examples where the use of QR codes was unsuccessful in a museum context?
  16. - online surveys are frequently used in evaluation and with the availability of companies such as Survey Monkey anyone has access to these tools -the best part is that if the survey questions are carefully thought out, this tool can really yield some invaluable data -one of my favorite ways to approach an online survey is to collect e-mails from visitors at the museum, particularly those taking part in some of our more participatory in-gallery programming There are a number of advantages to this strategy: -one is that the nature of the programming does not always allow for in-depth interviewing so collecting e-mails is a more nimble way to go about gathering feedback -with the increasingly greater amount of participatory programming taking place in our galleries, it also makes collecting data more feasible since anyone can help to collect the e-mails -Visitors are often more reflective when completing online surveys since they’ve had time away from the experience and do not feel put on the spot. Online surveys let visitors decide when they want to provide feedback which makes them more likely to provide thorough responses -as well, if you are interested in how memorable an experience was, delaying the send off is a good approach to take
  17. -embedding online surveys into sites like MoMA Learning and the MoMA Talks Tumblr offers a way to not only continue to collect data over time but also is a way to look at the feedback in light of changes made to the sites, allowing for continual improvement
  18. HOB Study -Used ipads to conduct surveys with visitors to the farmers market to get a sense of their thoughts on the museums and institutions in Brooklyn, and social issues related to the institutions - Benefits to using the iPads : Could filter questions based on their responses in real time Did not have to do the data entry Had a much higher response rate Cons: Expensive In the sun there was too much of a glare to be able to see
  19. -Photography can be a really effective and straightforward way of evaluating visitors’ experiences and the nice thing is that cameras are generally readily available -when I’m conducting observations, I always take photographs in addition to taking notes to fully document what’s going on -photographs are a way to communicate ideas, experiences and other key information to people who may not be familiar with a particular situation or did not do any observations -photographs also help trigger thoughts when working on a report and trying to piece together data -like qualitative data, photographs can even be coded -sometimes the best way to convey critical info. is through a good photo and great visitor quote
  20. Google Analytics (for Apps, websites etc.) -What information can we learn -What information can’t we learn