The adoption of any new technology can be disruptive to one degree or another. If one relies on anecdotal information collecting in the ether, Hadoop and Big Data appear to tip the scale in the direction of “significant” for both impact and complexity. To understand what is really happening, Sand Hill Group surveyed companies working with Hadoop to get a snapshot of the status of their implementation, how Hadoop is being applied and the quality of their experience.

1. Do You Hadoop?
A Survey of Big Data Practitioners
February 12, 2014
Bradley Graham

2. Big Data Research from Sand Hill Group
• Mindset over Data Set: A Big Data Prescription for
Setting the Market Pace
– Presents powerful learnings of some of the most
successful implementers of enterprise Big Data
– Provides prescriptive executive-level guidance for
adopting and using Big Data
– Use as a planning guide or benchmarking tool
– Purchase at http://bit.ly/SH_BD_14_S
• Do You Hadoop? A Survey of Big Data Practitioners
–
–
–
–
–
2
Clarifies Big Data (Hadoop-based) initiative status
Identifies pain points and barriers to adoption
Illuminates usage changes over the next 12-18 months
Use as a benchmarking tool
Download at http://bit.ly/SH_H2014

3. Broad Cross-Sectional View of the User Base
Company Size
Industries
(Number of Employees)
Large
47.4%
Small
33.3%
Other
13.3%
Telecommunications
2.2%
Technology
32.6%
Medium
19.3%
Participant Roles
Consumer Services
11.9%
Education
5.9%
Financial
Services
8.1%
Government
1.5%
Healthcare
2.2%
Industrials
22.2%
Other
8.9%
Academic
3.7%
Technology/Analytics
professional
50.4%
Consultant
14.1%
Business
sponsor/user
23.0%
• Startups and large established companies are leading the charge
• Technology industry use is strongly correlated to startup companies
• Companies serving large and/or diverse customer groups are a natural fit for Big Data
(e.g., retail, media and entertainment, and financial services)
• Business sponsor and user participation renders a more complete picture of Big
Data’s value and impact
3

4. Still Early Days
Hadoop Initiatives Status
(All Companies)
Percent of Total Population
50%
44.4%
40%
30%
20%
16.3%
11.1%
10%
8.1%
9.6%
10.4%
Piloting first
solution
First solution
deployed
Supporting
multiple analytics
0%
Exploring and
educating
Conducting POC Developing first
solution
• Solid progress is being made
• Majority of the companies have identified a business problem to address
• Use of multiple analytics suggest compelling value has been realized
4

5. Beware of Small and Agile Competitors
Hadoop Initiative Status
(by Company Size)
Percent of Category
100%
80%
60%
35.0%
36.4%
63.6%
45.0%
27.3%
18.2%
18.2%
Conducting POC
14.3%
13.3%
36.4%
26.7%
23.1%
35.7%
7.7%
0%
Exploring and
educating
50.0%
69.2%
20.0%
40%
20%
60.0%
Developing first
solution
Small
Medium
Piloting first solution
First solution
deployed
Supporting multiple
analytics
Large
• The median phase by company size is:
– Small: Exploring and educating
– Medium: Conducting POC
– Large: Conducting POC
• Data-centric startups enabled small companies to surpass medium-size companies in
the advanced stages
5

6. Satisfaction is Driving Continued Investment
• Higher overall satisfaction among
business sponsors and users relates to:
Hadoop/Big Data Initiative Satisfaction
(By Role)
100%
16.2%
9.7%
48.5%
61.3%
80%
60%
35.3%
29.0%
0%
Technology/Analytics
professional
Less Than
6
• Technology professionals’ higher than
expected satisfaction is likely attributed
to:
– Success with the technology
– Producing results that satisfied the business
stakeholders
40%
20%
– Profound insights
– More effective actions
Meets
Business sponsor/user
Better Than
• Challenges associated with Big Data are
nevertheless impacting satisfaction
– 3x more were less than satisfied (35.6%) vs.
more than satisfied (11.1%)

7. Mastering the Basics and Moving on to Advanced Applications
Most Commonly Reported #1 Uses of Hadoop
(Current vs. Future)
#1 Current Uses
(as of October 2013)
#1 Future Uses
(in 12 – 18 months)
Change
From
Current
Data Preparation (25.2%)
Advanced Analytics (24.4%)
—
Business Intelligence (17.8%)
Data Preparation (17.8%)
-7.4%
Basic Analytics (17.0%)
Business Intelligence (14.1%)
Archive More Data (14.1%)
-3.7%
—
• Leading current uses are:
– Foundational
– Support or augment the existing
solution portfolio (e.g., DW/BI and
small data analytics)
– Support Big Data experimentation
Top Uses of Hadoop
(Current vs. Future)
Top Current Uses
(as of October 2013)
Basic Analytics (58.5%)
Advanced Analytics (61.5%)
—
Business Intelligence (48.1%)
Business Intelligence (45.9%)
-2.2%
Data Preparation (45.9%)
7
Top Future Uses
(in 12 – 18 months)
Change
From
Current
Data Preparation (40.7%)
-5.2%
—
• Leading uses in 12-18 months
emphasize:
– New data types (streaming, geographic,
syndicated, etc. data)
– Advanced analytics (e.g., risk,
propensity and optimizations)

8. The Data Does Indeed Tell the Story
Data Types in the Hadoop Environment
Percent of Total Population
80%
60.7%
60%
52.6%
45.9%
40%
28.9%
22.2%
19.3%
16.3%
20%
14.1%
0%
Operational
Log
Online
Geographic
Partner
3rd party
Files
(Documents
and Media)
• Declining storage costs encourage a store everything approach
• Most prevalent data types parallel the focus of current usage
• Less frequently hosted data types hint at future Big Data applications
8
Streaming

9. Big Data and Hadoop are Complicated
Challenges Associated with Hadoop/Big Data
Most Commonly Reported
#1 Hadoop-related Challenges
Top Hadoop-related Challenges
Knowledge and experience (46.7%)
Knowledge and experience (65.2%)
Skills availability (20.7%)
Skills availability (52.6%)
Development effort (6.7%)
Development effort (40.7%)
• Resources remain the dominant issue for the foreseeable future
– Internal skills and experience gap
– Limited ability to repurpose existing resources
– Competition for “journey talent” (i.e., those successfully navigating the process at least once)
• Other frustrations are the technology challenges and level of effort related to:
– Implementing, maintaining and provisioning the environment
– Designing, building and maintaining solutions
• Performance, interoperability and other current second tier issues may prove to be
larger than expected issues down the road if left unaddressed
9

10. Success in Numbers
• Navigating the Hadoop/Big Data complexities requires a trusted
partner ecosystem
– It's far too complex at this point to go it alone
• Augment and, through a collaborative working model, edify internal
resources
• Gain access to value-added products and services that simplify:
– Infrastructure implementation and provisioning
– Solution development and use
– Data access and management
Effective partnering can address critical and second tier issues
while reducing time to value
10

11. Thank you!
Bradley Graham
Executive Director, Carpe Datum Rx
bgraham@Amberoon.com
CarpeDatumRx.com
11