Is it working


Published on

Panel at AAM2013 regarding evaluating the effectiveness of mobile apps in museums. This slide deck represents the opening presentation, introducing the speakers, my company's research into real space social engagement, and the criteria for evaluating success that our panel identified.

Other presentations in this session:

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Thank you for joining us this morning. This session poses the question we all want to answer about our mobile apps. Is it working? But in order to answer this question we must first answer a much more complex one:
  • How do we define success? Before going into that, I’d like to introduce our session panelists.
  • I am Matthew Fisher, president of Night Kitchen Interactive. Our panelists are LoicTallon, of Pocket Proof, Nancy Proctor, Head of Mobile Strategy & Initiatives at the Smithsonian, and Matthew Petrie, of Fusion Research and Analytics. Each of the three of our panelists brings a unique and compelling perspective to the questions of how and perhaps more importantly what do we measure in determining the success of museum mobile apps? Before we hear from our panelists, allow me to briefly provide my perspective on this question and why it led me to not only organize this session,
  • but to co-author a paper for the museums and the web conference last month entitled Rousing the Mobile Herd: Apps that Encourage Real Space Engagement.
  • I became interested in this topic last summer when, together with my team at Night Kitchen, we downloaded over 75 of the top museum apps in the app store in an effort to familiarize ourselves with the current state of the industry. My primary interest was in understanding how these apps fulfill what I see as a key strength that museums hold in the continuum of informal learning environments.
  • Museums and their peers are unique in that continuum, in that they inhabit the rare intersection of real space and social space. They provide direct access to material culture and natural phenomena on the one hand AND they support a wide range of social behaviors on the other. In short, they are what I call “real social”.
  • To tease out this concept of “real social”, we mapped social behaviors into four quadrants with anti-social on the left, social on the right, the virtual below and the real above. Those activities in the lower left quadrant, that are both anti-social and virtual, take the least advantage of the museum space. Imagine, for example, a tween sitting on a bench playing angry birds. Those activities in the upper right quadrant, that are both social and real, take the greatest advantage of the museum space. Imagine a family on a walk through the nature center, sharing thoughts and reflections as all five senses come alive with the experience.
  • But don’t take my word for it. As you probably know, there is a wellspring of research identifying social engagement as a key ingredient in effective museum experiences, as our paper explores in greater depth. I’ll simply reference two key points, one being that: “conversation is a primary mechanism of knowledge construction and distributed meaning-making”, and the other, thatgroups observe each other to learn, to understand exhibit interactions and model behavior.
  • To promote“real social” engagement, we considered the types of behaviors to be encouraged in the museum space, such as talking about the exhibit, playing a game in the space, engaging in facilitated discussions, participating in multi-person interactives, and photographing or observing others. Yet when we analyzed the apps that were available last summer, we found that the vast majority of them provided features that did not promote real social engagement. Instead, by and large the apps provided features that were anti-social and virtual, such as audio and media tours, single-person games and research, and features that fell into the social and virtual quadrant, such as social sharing and commenting.
  • In our paper we were not questioning the benefits that many of these apps provide. We would say that the majority of them effectively meet the goals of engaging visitors in the space, providing visitors with context and insights, and allowing visitors to share and engage with others online. But on a cautionary note, we would say that the more that museums define success in terms of app usage and social media metrics, the more we will design apps that encourage virtual, and often anti-social, behaviors, rather than “real social” behaviors.
  • In our paper we simply suggest that IF you agree that “real social” engagement is a key strength for museums, and if this sort of visitor engagement is central to your museum’s mission, then its important to design mobile apps with real social goals in mind, and measure the effectiveness of your apps in meeting those goals. There are several examples in our paper and available on slideshare that I will not go into here.
  • So, back to the question at hand: How do we define success?
  • There are many ways to do this, so many, in fact, that it is difficult to know where to begin. Our panel gathered a range of metrics and I have attempted to place them into categories, including usage analytics, visitor participation, critical assessment, museum goals, and the bottom line.
  • There are usage analytics, such as downloads, visitor tracking, and pre, during and post visit usage comparisons and comparisons of online and onsite visitors.
  • You can look at visitor participation, including in-app commenting and social sharing, contributed content when solicited, or even across the entire engagement pyramid.
  • There are also critical assessments, including press and social media reviews, the quality and quantity of reviews and ratings in the app stores, visitor surveys, and tracking against other apps.
  • There are also many museum goals to track against, including the desire to innovate and provide unique functionality, integrate with museum offerings and systems, reach new audiences and improve audience diversity, and meeting both onsite and beyond the walls requirements.
  • And finally there are some bottom line considerations, such as what is the cost per user, is it sustainable, and does mobile usage translate to ticket sales and contributions?
  • Is it working

    1. 1. Is it Working?Evaluating the Effectiveness of Mobile Apps5/21/13?
    2. 2. How do we define success?
    3. 3. Introductions• Matthew Fisher, Night Kitchen Interactive (chair)• Loic Tallon, Pocket-Proof• Nancy Proctor, Smithsonian Institution• Matthew Petrie, Fusion Analytics
    4. 4. Rousing the Mobile HerdApps that Encourage Real Space
    5. 5. American Folk Art Presents: "Infinite Variety"American Museum of Natural History:Cosmic DiscoveriesAmerican Museum of Natural History:DinosaursAmerican Museum of Natural History:ExplorerAquarium of the PacificAsian Civilizations Museum: TerracottaWarriorsBalboa Park: Spotlight MobileBean Life Science Museum at BrighamYoung University: Identify MeBean Life Science Museum at BrighamYoung University: Dichot KeyBritish Museum: Book of the DeadBrooklyn Museum of ArtCanadian Museum of CivilizationCanadian War MuseumCentre PompidouChina HeartCity of Calgary Downtown Public Art CircuitCoastal Maine Botanical GardensCrystal Bridges Museum of American Art:CBMuseumDenver Art Museum: DAM_SCOUTDenver Museum of Nature & ScienceGraphic Design MuseumGuggenheim BilbaoGuggenheim: CattelanHammer MuseumHigh Museum: Art ClixJewish Museum: Radical CameraLACMAMemphis Brooks Museum of ArtMilwaukee Art MuseumMinneapolis Institute of ArtsMinneapolis Institute of Arts: iAfricaMissouri History Museum: Historic St. LouisMoMAMusee des Beaux Arts de Lyons: MBA LiteMusee du LouvreMuseum HuntMuseum of Modern Art Vienna: MumokMuseum of London: StreetmuseumMuseum Victoria: Field Guide to FuanaMuseum of Science: Then and NowNational GalleryNICH: eMuseumNLS: Explore the John Murray ArchiveNMS: Concorde AppNational Naval Aviation MuseumNational September 11 Memorial & MuseumNezu MuseumNew York City Museum Guide (Star and Light)Norton Simon MuseumPace Gallery: 50 Years at PacePowerhouse Museum: LovelacePowerhouse Museum: PHM WalksPrado Museum: Audio GuideQueensland Art Gallery of Modern ArtRijksmuseum (Amsterdam)Royal Ontario Museum GuideSI NASM: PixPopSI: Access American StoriesSI: MeanderthalSI: Stories from Main StreetSI MobileSI NMAI: American Indian MagazineSolomon R. Guggenheim MuseumStorm King Art CenterStrawberry Banke MuseumSuntory Museum of ArtTate Modern: How It IsTate Modern: Magic Tate BallTate Modern: Magritte Your WorldTate Modern: MuybridgizerTate Modern: Race Against TimeTeylers Museum: Gadgets and GamesThe Albuquerque Museum of Art andHistory: ABQ Museum PainterThe Field Museum: SpecimaniaThe Metropolitan Museum of Art: MetGuitarsThe Metropolitan Museum of Art: Murder atthe MetThe State Hermitage MuseumToledo Museum of ArtUniversity of Melbourne: FormativeHistories Walk75+ museum apps in iTunes store@mefisher
    6. 6. Things we doin real spacewith other people
    7. 7. Social engagement supports moreeffective museum experiencesconversation is a primary mechanism ofknowledge construction and meaning-making.Lienhardt & Crowley, 1998groups observe each other to learn, to understandexhibit interactions and model behavior.Falk & Storksdieck, 2005@mefisher
    8. 8. Talking about the exhibitGames in the spaceFacilitated discussionsTouch-tables & multi-person interactivesPhotographing/observingothers
    9. 9. Meeting some goals…• Engaging visitors with objects• Providing context and insights• Social sharing• Online engagement@mefisher
    10. 10. …while not meeting othersRousing the Mobile HerdApps that Encourage Real Space
    11. 11. How do we define success?
    12. 12. How do we define success?• Usage analytics• Visitor Participation• Critical Assessment• Serving the Mission• Bottom Line
    13. 13. Usage analytics• Downloads• Visitor tracking• Pre-, during and Post-visit usage• Mobile visitors compared to online and onsitevisitors (quantity, demographics)
    14. 14. Visitor Participation• Commenting• Social Sharing• Contributed content & crowdsourcing• Engagement pyramid:– Curating– Producing– Commenting– Sharing– Watching
    15. 15. Critical Assessment• Press reviews• Social media reviews• Reviews/ratings in app stores• Visitor surveys• Compared to other apps
    16. 16. Serving the Mission• Innovation & experimentation(functionality unique to the platform)• Integration with museum offerings(interpretative and educational)• Integration with museum systems(collections, website, social media)• New audience reach(reaching target or underserved demographics)• Improving audience diversity(cultural and socio-economic, multi-lingual, accessibility)• Onsite and beyond-the-walls
    17. 17. Bottom Line• Cost per user (compared to other offerings)• Sustainability (ease/cost of maintenance)• Translation to ticket sales or contributions