Understanding online audiences: How to evaluate your website and why World wide wonder: Museums on the web V&A Sackler Centre 10 June 2009 Martin Bazley Online experience consultant Martin Bazley & Associates

Martin Bazley Science Museum, London, Internet Projects (7yrs) E-Learning Officer, MLA South East (3yrs) Chair of E-Learning Group for Museums Consultancy, websites, training, user testing, evaluation … Martin Bazley & Associates Notes in pack for afterwards

Science Museum, London, Internet Projects (7yrs)

E-Learning Officer, MLA South East (3yrs)

Chair of E-Learning Group for Museums

Consultancy, websites, training, user testing, evaluation … Martin Bazley & Associates Notes in pack for afterwards

www.martinbazley.com

Why do we need to research our online audiences?

Why do we need to research our online audiences?

Because otherwise users don’t ‘get’ what we are offering: a real–world example (10 Jun09: 43% of non-users don’t want web at home - even for free) Re value/status of Wikipedia – motivation , then information – anyone involved in any serious research will discover errors very quickly – try and get links into museum content via Wikipedia and other channels

Because otherwise users don’t ‘get’ what we are offering:

a real–world example (10 Jun09: 43% of non-users don’t want web at home - even for free)

Re value/status of Wikipedia – motivation , then information – anyone involved in any serious research will discover errors very quickly – try and get links into museum content via Wikipedia and other channels

Even a slight difference in viewpoints… … can cause real problems for users

A conflict between visual affordance … … and written instructions visual affordance almost always wins

Another example

Another example

Hmm… the button is really small… And it’s not green… You can’t push it in… Just push the big green button by the gate

Huge green button

How can we get a sense of who our online visitors are and what they do with our online content? How do we gather data to help us improve what we do? How do we measure success from the user's point of view, and against our own objectives and constraints? For example, how justify investment in social networks etc?

How can we get a sense of who our online visitors are and what they do with our online content?

How do we gather data to help us improve what we do?

How do we measure success from the user's point of view, and against our own objectives and constraints?

For example, how justify investment in social networks etc?

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Define audience research goal Analyse data Collect data Use results to guide changes Plan methodology

Reasons for doing audience research: Evaluation Did your project/product/service do what you wanted it to do? Provide information for stakeholders Gauge audience satisfaction

Did your project/product/service do what you wanted it to do?

Provide information for stakeholders

Gauge audience satisfaction

Reasons for doing audience research: Promotion Improve your offer for your target audiences Increase usage Widen access

Improve your offer for your target audiences

Increase usage

Widen access

Reasons for doing audience research: Planning Inform development of a new product/service Inform business planning Prove interest in a related activity (e.g. exhibit)

Inform development of a new product/service

Inform business planning

Prove interest in a related activity (e.g. exhibit)

Tools available Qualitative – focus groups, “free text” questions in surveys, interviews Quantitative – web statistics, “multiple choice” questions in surveys, visitor tracking Observational – user testing, ethnographic

Qualitative – focus groups, “free text” questions in surveys, interviews

Quantitative – web statistics, “multiple choice” questions in surveys, visitor tracking

Observational – user testing, ethnographic

When to evaluate or test and why Before funding approval – project planning Post-funding - project development Post-project – summative evaluation

Before funding approval – project planning

Post-funding - project development

Post-project – summative evaluation

Testing is an iterative process Testing isn’t something you do once Make something => test it => refine it => test it again

Testing isn’t something you do once

Make something

=> test it

=> refine it

=> test it again

Before funding – project planning *Evaluation of other websites Who for? What for? How use it? etc awareness raising: issues, opportunities contributes to market research possible elements, graphic feel etc *Concept testing check idea makes sense with audience reshape project based on user feedback Focus group Research

*Evaluation of other websites

Who for? What for? How use it? etc

awareness raising: issues, opportunities

contributes to market research

possible elements, graphic feel etc

*Concept testing

check idea makes sense with audience

reshape project based on user feedback

Post-funding - project development *Concept testing refine project outcomes based on feedback from intended users Refine website structure does it work for users? *Evaluate initial look and feel graphics,navigation etc Focus group Focus group One-to-one tasks

*Concept testing

refine project outcomes based on feedback from intended users

Refine website structure

does it work for users?

*Evaluate initial look and feel

graphics,navigation etc

Post-funding - project development 2 *Full evaluation of a draft working version usability AND content: do activities work, how engaging is it, what else could be offered, etc Observation of actual use of website by intended users , using it for intended purpose , in intended context – workplace, classroom, library, home, etc

*Full evaluation of a draft working version

usability AND content: do activities work, how engaging is it, what else could be offered, etc

Video clip Moving Here key ideas not lesson plans

Video clip Moving Here key ideas not lesson plans

Post-funding - project development 3 Acceptance testing of ‘finished’ website last minute check, minor corrections only often offered by web developers Summative evaluation report for funders, etc learn lessons at project level for next time

Acceptance testing of ‘finished’ website

last minute check, minor corrections only

often offered by web developers

Summative evaluation

report for funders, etc

learn lessons at project level for next time

Website evaluation and testing Need to think ahead a bit: what are you trying to find out? how do you intend to test it? why? what will do you do as a result ? The Why? should drive this process

Need to think ahead a bit:

what are you trying to find out?

how do you intend to test it?

why? what will do you do as a result ?

The Why? should drive this process

User test early Testing one user early on in the project… … is better than testing 50 near the end

Testing one user early on in the project…

… is better than testing 50 near the end

Two usability testing techniques “ Get it” testing - do they understand the purpose, how it works, etc Key task testing ask the user to do something, watch how well they do Ideally, do a bit of each, in that order

“ Get it” testing

- do they understand the purpose, how it works, etc

Key task testing

ask the user to do something, watch how well they do

Ideally, do a bit of each, in that order

User testing – who should do it? The worst person to conduct (or interpret) user testing of your own site is… you! Beware of hearing what you want to hear… Useful to have an external viewpoint First 5mins in a genuine setting tells you 80% of what’s wrong with the site

The worst person to conduct (or interpret) user testing of your own site is…

you!

Beware of hearing what you want to hear…

Useful to have an external viewpoint

First 5mins in a genuine setting tells you 80% of what’s wrong with the site

Extracts from London Museums Hub Online audience research

Data used for different things: Reporting and diagnostics Diagnostics – aimed at improvement of resources, identifying new areas for development, etc Reporting – to DCMS and other funders, internal etc

Diagnostics – aimed at improvement of resources, identifying new areas for development, etc

Reporting – to DCMS and other funders, internal etc

Google Analytics = Most popular web-based web stats tool. But also Piwik, Mint, Clicky etc Also AWstats, HitStats, etc

Google Analytics =

Most popular web-based web stats tool.

But also Piwik, Mint, Clicky etc

Also AWstats, HitStats, etc

Stats solutions Exclusion of bots / internal traffic most people aware of need to do so but there is also pressure to provide higher figures…so maybe leave them in?

Exclusion of bots / internal traffic

most people aware of need to do so

but there is also pressure to provide higher figures…so maybe leave them in?

Issues Browser-based statistics applications (like GA) cannot be installed on the whole of a site because a) no access to the code of the site, or b) there are just too many legacy pages to change

Browser-based statistics applications (like GA) cannot be installed on the whole of a site because

a) no access to the code of the site, or

b) there are just too many legacy pages to change

Issues Barriers to effective usage of web stats Subsections of websites or microsites have different statistics packages => separate analysis for each part Perhaps outside supplier controls access to log files or stats package and are unwilling to add Google Analytics etc

Barriers to effective usage of web stats

Subsections of websites or microsites have different statistics packages => separate analysis for each part

Perhaps outside supplier controls access to log files or stats package and are unwilling to add Google Analytics etc

Issues Off-website activity not reflected in server stats More of users’ interaction with museum web content is happening away from the museum websites themselves, via blogging, Flickr, YouTube, wikis, etc. Activity via RSS feeds such as blogs or podcasts can be measured…

Off-website activity not reflected in server stats

More of users’ interaction with museum web content is happening away from the museum websites themselves, via blogging, Flickr, YouTube, wikis, etc.

Activity via RSS feeds such as blogs or podcasts can be measured…

Issues As museums engage further with web 2.0 services their online identities become fuzzier => May soon not be feasible for museums to measure impact in absolute numbers

As museums engage further with web 2.0 services their online identities become fuzzier

=> May soon not be feasible for museums to measure impact in absolute numbers

Issues Ways to ameliorate: Inbound RSS feeds from those services could be embedded into the museum sites. Strategies to gather statistics from external services can be developed. But a line needs to be drawn to avoid wasting too much effort, following the law of diminishing returns. There is also the copyright-related question of to what extent such content is yours given that it is generated ‘out there’ rather than on your site. Therefore, the precise strategy to adopt should be determined by the ways in which the information will be used. This is one reason for adopting a more universal approach to data collection and interpretation, as outlined below.

Ways to ameliorate:

Inbound RSS feeds from those services could be embedded into the museum sites. Strategies to gather statistics from external services can be developed.

But a line needs to be drawn to avoid wasting too much effort, following the law of diminishing returns. There is also the copyright-related question of to what extent such content is yours given that it is generated ‘out there’ rather than on your site.

Therefore, the precise strategy to adopt should be determined by the ways in which the information will be used. This is one reason for adopting a more universal approach to data collection and interpretation, as outlined below.

Concerns Benchmarks: people want to know what is “good” and what is not when it comes to web statistics >P re Google benchmarks People don’t always know what to do with the results

Benchmarks: people want to know what is “good” and what is not when it comes to web statistics >P re Google benchmarks

People don’t always know what to do with the results

Geffrye Museum web stats linked with events etc

Key points What we did Online questionnaire Visitor survey Focus groups What we found out Some key findings Recommended next steps

What we did

Online questionnaire

Visitor survey

Focus groups

What we found out

Some key findings

Recommended next steps

Online questionnaire SurveyMonkey – questions developed with museums then refined following trialling Links to questionnaire added to museum websites Run for several months, 000s responses collected

SurveyMonkey – questions developed with museums then refined following trialling

Links to questionnaire added to museum websites

Run for several months, 000s responses collected

EM Online collections research by MHM

Visitor survey gathered data for comparison with OQs recruitment for focus groups

gathered data for comparison with OQs

recruitment for focus groups

Focus groups 18-24 / families / adults 25+

18-24 / families / adults 25+

What we found out Link between websites and physical visits Majority of web use is for planning visits Only about half use the website before visiting the museum Some websites failing to create appropriate impressions of the physical museums, so failing to attract undecided visitors >N Confusion re difference between stored and displayed collections

Link between websites and physical visits

Majority of web use is for planning visits

Only about half use the website before visiting the museum

Some websites failing to create appropriate impressions of the physical museums, so failing to attract undecided visitors >N

Confusion re difference between stored and displayed collections

What we found out Very little use of websites other than for planning the visit, except: Teachers – looking for teaching materials and ideas Families – looking for online games, homework etc Comment re online collection browsing: researching/searching, following, browsing ( MHM)

Very little use of websites other than for planning the visit, except:

Teachers – looking for teaching materials and ideas

Families – looking for online games, homework etc

Comment re online collection browsing: researching/searching, following, browsing ( MHM)

What we found out Very little interest in ‘user generated content’, but… … this may be because most do not yet understand the possibilities, partly because trends moving so quickly. More research required, based on specific proposals

Very little interest in ‘user generated content’, but…

… this may be because most do not yet understand the possibilities, partly because trends moving so quickly. More research required, based on specific proposals

What we found out Demographic data Motivations for using website Expectations of website etc

Demographic data

Motivations for using website

Expectations of website

etc

Online users

Most commonly requested web content relates to visiting The top 4 overall are: information on what the galleries are like information on the museum’s objects details of events and exhibitions information about collections in store what most online users want is more information about the museum visit .

The top 4 overall are:

information on what the galleries are like

information on the museum’s objects

details of events and exhibitions

information about collections in store

what most online users want is more information about the museum visit .

Value = Reach x Quality

Value = Reach x Quality

Metrics Net Promoter may not be appropriate as Quality metric for cultural sector websites Could a single question (or handful of questions) be agreed as standard, and used to assess ‘satisfaction’ via surveys? How could the results be used? Reporting? Internal investment planning?

Net Promoter may not be appropriate as Quality metric for cultural sector websites

Could a single question (or handful of questions) be agreed as standard, and used to assess ‘satisfaction’ via surveys?

How could the results be used?

Reporting?

Internal investment planning?

Defining market share ‘Reach’ may be very small for niche topics / audiences – does this matter? Could use % of total audience instead? This would mean (a) estimating total audience (often part of project plan / funding application anyway) (b) using unique visitors stats or other means to estimate reach

‘Reach’ may be very small for niche topics / audiences – does this matter?

Could use % of total audience instead?

This would mean

(a) estimating total audience (often part of project plan / funding application anyway)

(b) using unique visitors stats or other means to estimate reach

Metrics for reporting Currently ‘visits’ = default metric for reporting Not v meaningful, but does this matter, given that there is no single metric What should we be reporting on? Audience penetration Audience satisfaction Audience engagement Breadth of coverage …?

Currently ‘visits’ = default metric for reporting

Not v meaningful, but does this matter, given that there is no single metric

What should we be reporting on?

Audience penetration

Audience satisfaction

Audience engagement

Breadth of coverage …?

Metrics for development Use a variety of metrics in combination, based on nature of project issues, future objectives and current sector priorities… No magic bullets on horizon – justifying investment still a ‘creative’ decision Need balance between flexibility between projects, and shareability of data, for successful analysis and skills development ‘ Top tips’ in notes in pack ‘Understanding online audiences’

Use a variety of metrics in combination, based on nature of project issues, future objectives and current sector priorities…

No magic bullets on horizon – justifying investment still a ‘creative’ decision

Need balance between flexibility between projects, and shareability of data, for successful analysis and skills development

‘ Top tips’ in notes in pack ‘Understanding online audiences’