Complete Web Monitoring
A report from the
frontlines of the web

#CWM

I betook
myself to
linking...

...tis some
visitor
entreating
entrance at
my chamber
door

What kind of site are you?

About your company

What’s your job?
Webops (it’s up, and it’s fast)
User experience (it’s easy to use)
Community management and monitoring
Market research (what people think and why)
Support
Other

What we’ll cover
Analytics, interaction, UX, Voice of the Customer
EUEM, synthetic tests, RUM
Online communities, internal communities
Competitive analysis
Integrating data sources

Which pretty much means
We’re going to waste some of your time.

What’s complete web
monitoring?

Smart companies make
mistakes faster

New idea
Execution
Feedback

New idea
Execution
Feedback
Success?

New idea
Execution
Feedback
Success?
Money
left?

New idea
Execution
Feedback
Success?
Money
left?
End

New idea
Execution
Feedback
End
Success?
Money
left?
End

New idea
Execution
Feedback
End
Success?
Money
left?
Risk
End

New idea
Execution
Feedback
End
Success?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
End
Success?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
End
Success?
End
Success?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
End
Success?
End
Success?
End
Success?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
End
Success?
End
Success?
End
Success?
Money
left?
Money
left?
End

New idea
New idea
Execution
Execution
Feedback
Feedback
End
Success?
End
Success?
End
Success?
End
Success?
Money
left?
Money
left?
End
End

Mistakes are good
Most startups don’t win at what they set out to do
Amazon was just a bookstore
eBay sold Pez
F5 made hurricane modeling software
Youtube was video hot-or-not
Mistake speed is critical

Who you are drives what
you watch

Media sites

Transactional sites

Collaborative sites

SaaS sites

Users do what we wanted
Enrolment: They sign up
Purchases: They buy stuff
Invitations: They tell their friends
Stickiness: They stay for longer
Loyalty: They come back
Contribution: They add content

What could we watch?
What we’d like to know Tool set
How much did visitors benefit my business? Internal analytics
Where is my traffic coming from? External analytics
What’s working best (and worst?) Usability testing
How good’s my relationship with my market? Customer surveys, community
How healthy is my infrastructure? Performance monitoring
How am I doing against my competitors? Search, external testing
Where are my risks? Search, alerting
What are people saying about me? Search, community monitoring
How is my content being used elsewhere? Search, external analytics

How much did visitors
benefit my business?
Internal analytics
Conversion and Billing and account use
abandonment
Click-throughs
Offline activity
User-generated content
Subscriptions

Where’s my traffic coming
from?
External analytics
Referring websites
Inbound links from social networks
Visitor motivation

What’s working best (and
worst)?
Usability testing, A/B testing
Site effectiveness Trouble ticketing and
escalation
Upselling effectiveness
Content popularity
Ad and campaign
effectiveness Usability
Findability and search User productivity
effectiveness
Community ranking and
rewards

How good is my relationship
with my market?
Customer surveys, community monitoring
Loyalty
Enrollment
Reach and rewards

How healthy is my
infrastructure?
Performance monitoring
Availability and Impact of performance
performance on outcomes
SLA compliance
Content delivery
Capacity and flash traffic

How am I doing against my
competitors?
Performance monitoring
Site popularity and ranking
How are people finding my competitors?
Relative site performance
Competitor activity

Where are my risks?
Search, alerting
Trolling and spamming
Copyright and legal liability
Fraud, privacy, and account sharing

What are people saying
about me?
Search, community monitoring
Site reputation
Trends
Social network activity

How is my content being
used elsewhere?
Search, external analytics
API access and usage
Mashups, stolen content, and illegal syndication
Integration with legacy systems

The tools at our disposal...

Collection
Server (logfiles)

Server collection
Browser Data center
Server
Analytics
receiver

Server collection
Browser Data center
Server
Analytics
receiver

Server collection
Browser Data center
Server
Analytics
receiver

Server collection
Browser Data center
Server
IT
Analytics
receiver

Collection
Server (logfiles)
Inline (sniffing)

Sniffer collection
Browser Data center
Server
Passive
capture

Sniffer collection
Browser Data center
Server
Passive
capture

Sniffer collection
Browser Data center
Server
Passive
capture

Collection
Server (logfiles)
Inline (sniffing)
Browser (JS)

Analytics
receiver
Marketing
Browser Data center
Server
Browser collection

Analytics
receiver
Marketing
Browser Data center
Server
Browser collection

Analytics
receiver
Marketing
Browser Data center
Client-side
Server
interpreter
Browser collection

Analytics
receiver
Marketing
Browser Data center
Client-side
Server
interpreter
Browser collection

Analytics
receiver
Marketing
Browser Data center
Client-side
Server
interpreter
Browser collection

Collection
Server (logfiles)
Inline (sniffing)
Browser (JS)
Interception

Interception (survey)
collection
Browser Data center
Server
Survey
site

Interception (survey)
collection
Browser Data center
Server
Survey
site

Interception (survey)
collection
Browser Data center
Client-side
Server
interpreter
Survey
site

Interception (survey)
collection
Browser Data center
Client-side
Server
interpreter
Random
selection
Survey
site

Interception (survey)
collection
Browser Data center
Client-side
Server
interpreter
Random
selection
Survey
site

Interception (survey)
collection
Browser Data center
Client-side
Server
interpreter
Random
selection
Survey
site

Collection
Server (logfiles)
Inline (sniffing)
Browser (JS)
Interception

Collection Search & crawlers
Server (logfiles) Search engines
Inline (sniffing) Platform search
Browser (JS)
Interception

Collection Search & crawlers Testing
Server (logfiles) Search engines Test scripts
Inline (sniffing) Platform search
Browser (JS)
Interception

Synthetic
tester
Browser Data center
Server
Synthetic testing

Synthetic
tester
Browser Data center
Server
Synthetic testing

Synthetic
tester
Browser Data center
Server
Synthetic testing

Synthetic … Synthetic
Synthetic
tester tester tester
Browser Data center
Server
Synthetic testing

Analyzing data properly
Always compare
Segment everything
Don’t settle for averages

A complete web monitoring
maturity model
Level 1: Technical details
Level 2: Minding your own house
Level 3: Engaging the Internet
Level 4: Building relationships
Level 5: Web business strategy

Web analytics
What did they do?

Arts & crafts time!

≥≥≥

≥≥≥

Web Analytics is the Mother Ship

A brief history of analytics

Major Issues:
WWW needed to become mainstream
Analytics needed to stop relying on HITS
We needed the ability to segment

blah blah blah ...
A visitor arrives at your website, possibly after following a link that referred
them. They land on a web page, and either bounce (leave immediately) or
request additional pages.
In time, they may complete an transaction that’s good for your business,
converting them from a mere buyer into something more—a customer, a
user, a member, or a contributor—depending on the kind of site you’re
running. On the other hand, they may abandon that transaction and
ultimately exit the website.
That visitor has many external attributes—such as the browser they’re
using, or where they’re surfing from—that let you group them into
segments. They may also see different offers or pages during their visit,
which are the basis for further segmentation.
The goal of analytics, then, is to maximize conversions by optimizing your
website, often by experimenting with different content, layout, and
campaigns, and analyzing the results of those experiments on various
internal and external segments.

Questions?

let’s get practical

The Three Stages of
Analytics
Finding the website
Using the website
Leaving the website

How did they get there

Direct Traffic

Type-in Traffic

Bookmarking

Desktop client

Javascript redirection

Browser inconsistency

Bots, spiders and probes

Organic search

Paid search

Referrals

Linkbacks

Places & Tasks

Landing page:
View one story

Landing page:
View one story
Task: Log in
Enter credentials
Verify
Recovery

Landing page:
View one story
Task: Log in
Enter credentials
Verify
Recovery
Task:
Forward a story
Enter recipients
Enter message
Send

Landing page:
Task: View one story
Create account
Task: Log in
Pick name
Enter credentials
Check if free
Verify
Set Password
Recovery
CAPTCHA
Send mail
Get confirm
Task:
Forward a story
Enter recipients
Enter message
Send

Landing page:
Task: View one story
Create account
Task: Log in
Pick name
Enter credentials
Check if free
Verify
Set Password
Recovery
CAPTCHA
Send mail
Get confirm
Task:
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

Landing page:
Task: View one story
Create account
Task: Log in
Place: View stories
Pick name
Enter credentials
Check if free
Vote up Next 25
Verify
Set Password
Vote down Last 25
Recovery
CAPTCHA
Send mail
Get confirm
Task:
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

Landing page:
Task: View one story
Create account
Task: Log in
Place: View stories
Pick name
Enter credentials
Check if free
Vote up Next 25
Verify
Set Password
Vote down Last 25
Recovery
CAPTCHA
Send mail
Place: Read
Get confirm
poster comments
Vote up Next 25
Task:
Vote down Last 25
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

Landing page:
Task: View one story
Create account
Task: Log in
Place: View stories
Pick name
Enter credentials
Check if free
Vote up Next 25
Verify
Set Password
Vote down Last 25
Recovery
CAPTCHA
Send mail
Place: Read
Get confirm
poster comments
Vote up Next 25
Task:
Vote down Last 25
Forward a story
Task: Submit Enter recipients
a new story Place: My Enter message
account Send
Enter URL
Change My
Describe
address comments
Deduplicate
Change PW See karma
Post it

Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: Read
poster comments
Task:
Forward a story
Task: Submit
a new story Place: My
account

Landing page:
Create acct.
Create acct. View one story
Form uptime Task: Log in
Place: View stories
# started
Bad form
Place: Read
# CAPTCHA poster comments
Mail uptime Task:
Forward a story
Mail bounced
Task: Submit
a new story Place: My
Confirm & return account
Return 3x

Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: View stories
Stories/visit # up/down
Place: Read
poster comments
Time/story Top stories
Task:
Forward a story
Refresh time Views/page
Task: Submit
a new story Place: My
account

Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: Read
poster comments
Task:
Forward a story
Task: Submit
a new story Place: My
account

Places
Efficiency matters
How quickly, how many,
productivity
Learning curve OK
Leave when they’re bored
Collect “aha” feedback
A/B test content for
pages/session, exits

Tasks
Effectiveness matters
Completion, abandonment
Intuitiveness rules
Leave when they change their
mind or it breaks
Collect “motivation” feedback
A/B test layouts for conversion

What did they do?

Parting is such sweet sorrow

Undesirable outcomes

Abandonment & bounce rate

Attrition

Desirable outcomes

Implementing analytics

Defining your site’s goals
Setting up data capture
Setting up filtering
Identifying segments
Tagging page content
Verifying taht everything is working

Identifying segments you
want to analyze

Determine your goals,
challenge your assumptions

Know what normal is

Make a prediction

Test against the original

Run the tests for long
enough

Make sure both A and B
had a fair shake

Web Interaction Analytics
Why did they do it?

http://www.flickr.com/photos/americanlady/3118301118
consume
http://
give data
navigate

Four kinds of issues

Visitors don’t see what you
wanted them to

Visitors don’t interact as you
intended

Visitors don’t input edata
correctly or completely

We have no idea what’s
happening on the site

But your mileage will vary

Usability and affordance

Why you should care about
mouse interactions

segmenting clicks is also
important

All of these aggregate views help you to understand what’s most popular, which links don’t lead to desired outcomes, and places where visitors are clicking mistakenly. In other words, they confirm—or contest—the de
made when you first built the website.

when people abandon forms
(oh noes!)

How many visitors arrived at the form and were
shown it correctly?
How many visitors started to fill out the form?
How far down the form did they get?
How long did they linger at each field?
What was the abandonment for each field?
How many of them submitted the form?
How many of them submitted the form but
missed required fields?

Stalking efficiently: replay

Are my designs working for real visitors?
Why aren’t conversions as good as they should
be?
Why is this visitor having issues?
Why is this problem happening?
What are the steps needed to test this part of
the application?
How do I show them it’s not my fault?

... so .. is my design
working?

A

how can get some of that?

first .. ask yourself a few
questions

paths that drive my business

places that drive users to
ask questions (support)

Places that drive productivity

Issues and concerns

What you wont see with WIA

Voice of the Customer
Why did they do it?

they aren’t doin’ what you
think...

the travel industry’s dilemma

they really aren’t doin’ what
you think...

so what’s this “VOC” thing?

get new ideas

evaluate things you can’t
find out elsewhere

see if the cool shiny
improvements you just

collect demographics data

“customer insights do not
come from quantitative
market research. You cannot
generate insight out of
numbers. Numbers help you
to validate insights. A

What’s in it for you?: insight
and clues

What’s in it for you?:
subjective scoring

What’s in it for you?:
demographics

What’s in it for you?:
surfographics

What’s in it for you?:
collection of visitor data

What VOC isn’t (or shouldn’t
be)

not a substitute for other
forms of collection

not representative of your
user base

not an alternative to a
community

not a substitute for
enrollment

Kickin’ it off

Planning the study

Before/after comparison

Demographics comparison

A/B comparison of two
choices

Comparing the impact of
website latency on

Comparing experience
based on actions

What kind of questions to
ask

Evaluation

Recall

General feedback and
exploration

Mindset

Designing the study’s
navigation

Why surveys fail

never ever ever ever ever
ever ever ever ever ever ever

VOC in action

but the downsides are ...

How many people should I
ask?

Encouraging participation

overlays

P

If things aren’t going right ...

high recruitment and bounce
rates

low recruitment response
rates

poor interception response
rates

poor start rates

poor completion rates