Wrangling Messy Data - A True Story

Public
Jason Cao/SAP Digital Experience Marketing – Las Vegas
Elizabeth Imm/SAP Market Introduction Services – Berlin
EA104 – Wrangling Messy Data
A True Story

© 2014 SAP SE or an SAP affiliate company. All rights reserved. 2Public
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any other agreement
with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to
develop or release any functionality mentioned in this presentation. This presentation and SAP's
strategy and possible future developments are subject to change and may be changed by SAP at any
time for any reason without notice. This document is provided without a warranty of any kind, either
express or implied, including but not limited to, the implied warranties of merchantability, fitness for a
particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this
document, except if such damages were caused by SAP intentionally or grossly negligent.

Agenda
The Setting – The SAP Community Network (SCN)
The Players – Understanding Roles in Data Analysis
The Plot – Introducing Gamification on SCN
The goals for gamifying SCN
The Conflict – Messy data & other data problems
The Resolution – Approaching the problem
3 SAP Lumira Use-Cases with SCN Gamification Data
The End – “And they lived happily ever after”…or do they?
What’s next for SCN’s Gamification team?

About SAP Community Network (SCN)
A High-Tech, Professional Community
A 11-year old, mature community: Open to all
Need to encourage quality contributions
A place to grow reputation: Contributions showcase expertise
Serious game

What is Gamification: Definition
Gamification is the use of game-
thinking and game mechanics in a
non-game context in order to
engage users and solve problems.
Gamification is used in
applications and processes to
improve user engagement, ROI,
data quality, timeliness, and
learning.
Source: Wikipedia

Why Introduce Advanced Game Mechanics?
Boost
Participation
Build
Reputation
Inject Fun
Drive
Behaviors
Address
Challenges
with Old
System

The SCN Experience
Goal: Boost Participation
Registration:
Missions and badges for
first-time login and
onboarding
Contributions:
Missions and badges
for first contributions
and more

The SCN Experience
Goal: Drive Behaviors
Ethics:
Requirements in
missions to read the
rules of engagement and
how to search
Feedback:
Pay It Forward
mission to reward
positive engagement

The SCN Experience
Goal: Build Reputation
Quality:
Missions that require not
only content creation but
receiving good feedback
on the content
Quality:
Prerequisite of a certain
level before you can
earn some missions

The SCN Experience
Goal: Inject Fun
Hidden missions,
unexpected
recognition
SCN 10-year
anniversary
Data Geek Challenge

Messy Data & Other Data
Problems

Mission Completion (Badges Awarded)

Member Activity Comparison (Total Action Count)

Increase in Active Users (Those who logged an action)
0
5000
10000
15000
20000
25000
30000
35000
40000
APR-28-2013 MAY-5-2013 MAY-12-2013 MAY-19-2013 MAY-26-2013
Active Users
(Logged an Action)
Active Users
WK 1 19,445
WK 2 21,728
WK 3 22,623
WK 4 36,084
143%

The Plot Thickens…
Duplicate data
Senseless data
Disparate data
Corrupted data

People of Many Hats
DECISION MAKERS ANALYSTS DESIGNERS
Leverage Knowledge Activate Data Build Data
Assets

Whiteboarding for our Proof of Concept

Result of Initial Data Cleansing
Total
100M
Rows
3M Clean Rows
Sensible data…

Data
Acquisition Prepare Room (and Object Picker)
Visualize/
Compose
Room
Predict
Room
Share
Room
Seven Common Data Analysis Tasks
Data analyst workflows often consists of seven high-level activity groups:
• Iterative activities, not linear
• Not all equal weight/time
Find
Wrangle
Profile
Enhance
Visualize
Predict
Share

Seven High-level Data Analysis Tasks
Find – Examples:
• Data Connectivity / Opening Files
• Knowing who to ask
Data Analysis

Wrangle – Examples:
• Merge datasets
• Clean data
Data Analysis

Profile:
• Comparing data (e.g. spelling)
• Leading spaces or zero's
• Structured and unstructured data
Data Analysis

Enhance:
• Add/Remove columns
• Hierarchies and calculations
• Geography and time
Data Analysis

Visualize:
• Creating charts and visualizations
• (Note: a table is considered a visualization as well)
Data Analysis

Data Analysis

Share:
• Cloud
• Server
• Email (visualization and dataset)
Data Analysis

Sharing with Lumira Cloud and Lumira Server
Desktop
IT
SAP Lumira
Server
MobileWeb

3 Data Workflows for SAP Lumira Server
Lumira Datasets
(real-time or static)
Lumira Stories
SAP
Lumira
SAP HANA
Studio

Use Case 1
How are missions influencing community adoption?

Playing Around With Visualizations
Too many segments to be useful! Interesting visualization, but irrelevant!

Data Preparation Basics
Replace Filter
Rank

Insight to Help Us Focus on What Matters
Email reminders
add limited value
to onboarding.
Stop mass emails,
& focus on closing
onboarding gap.
Better use of
resources to focus
on activating
members.
Mass email
campaign

Use Case 2
What’s the best day to launch missions on SCN?

Group Replace

Increasing Mission Design Maturity
Design + planning
can influence
mission success.
Revise launch
practice to include
timing and
promotion.
More mature
launch plan, with
greater community
awareness.

Use Case 3
Can we reduce bad behavior?
~ quoted from SCN discussion thread

Observations | 9 Months After Launch
Current Challenges Opportunities
Increased point cheating and plagiarism
(perceived as disproportionate to activity
increase).
Dissatisfaction of loyal, well-established
members and newer members alike.
High operational effort needed to address these
issues.
Strengthen quality requirements in badged
mission.
Further de-emphasize quantity and points in
favor of increased quality and meaningful
engagement.
Think of a transforming approach in 2 steps.

Group Hierarchy and Filter

Turning Data Into Insight

Feedback from Moderators
“Wow! These features really did wonders.
Most of the people we reported from SD
space are gone, that itself indicates the
impact.” (Jyoti Prakash)
“I especially think the removal of points
from Likes has eliminated about 2/3 of the
point games.” (Michael Appleby)
78%
of Moderators who responded
to our poll report that changes
had a positive impact

Creating Awareness for Community Challenges
We can influence
specific member
behaviors.
Create missions
that harness
gaming energy to
desired behaviors.
Greater awareness
of community
topics. Good
behaviors.

Conclusion
Focus on your goal
- cut out the rest.
The more we
know, the more
questions we have.
Start with the end
in mind.

Get into
the
Game!
while (true)

SAP d-code Virtual Hands-on Workshops and SAP d-code Online
Continue your SAP d-code education after the event!
SAP d-code Online
Access replays of keynotes, Demo Jam, SAP d-code
live interviews, select lecture sessions, and more!
Hands-on replays
http://sapdcode.com/online
SAP d-code Virtual Hands-on Workshops
Access hands-on workshops post-event
Starting January 2015
Complementary with your SAP d-code registration
http://sapdcodehandson.sap.com

Further Information
SAP Education and Certification Opportunities
www.sap.com/education
Watch SAP d-code Online
www.sapcode.com/online
SAP Public Web
scn.sap.com/community/lumira
www.saplumira.com
www.sap.com/LearnBI

Feedback
Please complete your session evaluation for
<session EA104>.
Jason Cao {jason.cao@sap.com}
Follow me on Twitter {@JayChaos}
Thanks for attending this SAP TechEd && d-code session.

© 2014 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an
SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE
(or an SAP affiliate company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark
information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its
affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or
SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing
herein should be construed as constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or
release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future
developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for
any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-
looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place
undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

Wrangling Messy Data - A True Story

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Wrangling Messy Data - A True Story

Similar to Wrangling Messy Data - A True Story (20)

Recently uploaded

Recently uploaded (20)

Wrangling Messy Data - A True Story