eMetrics SF 2011 - Integrating Analytics and Testing at Dell

Integrating Analytics with
Testing at Dell
Joel Wright & Charles Hua
Dell Online Analytics & Testing Team
March 15th, 2011

Typical Path…
Not Enough People
Not Enough Process
Efficient & Productive
People + Process + Technology

2 Online Analytics, Testing & Targeting

Dell Online Today
Online:
725K B2B Dell.com + Premier + Global Portal + B2B + 34 supported
transactions per year Channel Portal + eSupport + Community languages

Googled 1M 170M visits
times per day to community properties

58K
channel 1B
partners visits per year to
online Dell.com

190K
Premier 13B
Pages server calls
annually

An online order
is placed
every
2 seconds


The Vision

Online Analytics, Testing & Targeting

Online Analytics,
Dell.com
Test & Targeting

To be the To become an
technology innovation engine
Best in Class
e-commerce leader
by offering the most
through insightful
analytics & the
relevant experience best-in-class
and solutions to testing capability to
drive high customer enable constant and
loyalty rapid changes

Our journey started more than a decade ago…

Early Adoption Standardization Maturity

Regional Global analytics
Implementation Ops Reviews
implementation
Regional KPI Standard
KPI’s – global dashboard

In-house testing Global Industry leading Defined Test Life Cycle
capability testing tool testing capability process Management Tool

1999-2007 2008-2009 2010-2011
In-house Advanced Behavioral
VOC solution
clickstream tool segmentation Targeting

Regional teams Global team by BU Global team across
BU’s

Limited expertise Work closely with Grow in-house
and talent bench vendors & expertise
consultants

Low Maturing Technology, People, and Process

Strong back-office
Good people
support

Our Strengths

Committed to
Great technology
testing


Challenge #1: People
• Too much work, Not enough people

• Vague R&R’s

• Lack of process

• No clear accountability


People
Tripled size of the team

50% of team members are
new to Dell…

On-boarding process and
If I were running a company today, I would have one training
priority above all others: to acquire as many of the
best people as I could.
Jim Collins


Team

Global Analytics Global Testing

PMO

eDell Programs Reporting


Process, Roles & pipeline
Responsibilities roadmap

Brainstorm
Recipes
Institutionalize Test
learning Plan

implement Design

Communicate Code

Analytics Manager Lead

Analyze QA Test Manager Lead

Sign off required

Documentation required
Monitor Launch


Challenge#2: Planning

• Balance Strategic and Tactical
• Maximize Value of Testing
• Align Regional Activities


Analytics Manager Lead

Testing Manager Lead

Planning vs. Execution
WK11 WK13 WK2 WK4 WK6 WK8 WK11

• Strategic • Strategic
Areas (NQ) Areas (NQ)
• List of test • List of test
ideas (NQ) ideas (NQ)
• Annualized • Annualized
Analytics
Manager

upside (NQ) upside (NQ)

M1 testing M2 testing M3 testing M1 testing
priorities– priorities– priorities– priorities–
wireframes / wireframes / wireframes / wireframes /
feasibility feasibility feasibility feasibility

Idea Generation
Lock M1 Lock M2 Lock M3
schedule; schedule; schedule;
First test in First test in First test in
M1 ready to
Manager

M2 ready to M3 ready to
Testing

launch launch launch

Test Execution

Weekly Optimization Meeting

Focus on Strategic Areas
ODG

Dell.com homepage

3 Navigation (Masthead + ANAV)

Segment Homepage

6 Category pages (LT, DT)

5 Deals pages

4 Product templates
(Franchise + PD + 3x pages)

Retargeting

2 Config + Builder

Shopping Cart
1 +
Checkout


Global Team Local Execution

Analytics Manager

Testing Manager


Challenge #3: Operations
• How to evaluate results of tests?
• How to manage efficiency?
• How to integrate cross-functional teams?


Evaluation of Results
Before calling test a winner:

• Determine the OEC (overall evaluation criteria)

• Minimum of 100 conversions per recipe

• Result has to be >90% confident with 7 day consistent trend

• Remove extreme orders and outlier data points

• Approval from senior analyst on the results

After, we monetize:

• Determine financial KPI to use

• Estimate value using past traffic data (Outlook)

• Track the actual value post-implementation


Regular Operations Review

Developer
• Financial Outlook
• Testing Activities
• Team Performance
• Alignment

Test Analytics
Manager
Management Manager

Strategic
Programs


Test Life Cycle Management Tool

Data resides in multiple Excel Files Single repository from test idea to execution

Benefits:
• Less maintenance overhead • Better data integrity
• Live information • Archive of tests run
• Shared access & personalized views • Exception reporting
• One ID for all tests • Cross-team integration


Reporting Examples

OAT Overview Mass Update

Gantt Chart Test Detail

Challenge #4: Learning
• How do we avoid repeating mistakes?
• How do we perpetuate what works?
• How to bring new team members up to speed?
• How do we share learnings across organization?


Learning System


Learning System (WIP)

(Process) (Tool) (Process)

Test Results New Test Plans
Analysis Learnings Inform Strategy
Research Historical Perspective

Institutionalize


Results

FY11 Exceeded financial goal

FY10 FY11

FY10
Doubled test volume

# of Tests Rev Upside

Because they lack accountability, they fail to
achieve credibility, and they have no Increased credibility
authenticity.
Jim Collins


Results

WAA Finalist
Best B2B Segment [Practitioner of the Year]
Best B2B Ecommerce
Best B2C Category Page

Which Test Won Awards

Most WAA Certified
Web Analysts


Thank You


eMetrics SF 2011 - Integrating Analytics and Testing at Dell

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (14)

Similar to eMetrics SF 2011 - Integrating Analytics and Testing at Dell

Similar to eMetrics SF 2011 - Integrating Analytics and Testing at Dell (20)

eMetrics SF 2011 - Integrating Analytics and Testing at Dell

Editor's Notes