CIO Considerations for Big Data: Obtaining real business value from Big Data

An Enterprise Strategy Program paper

CIO considerations for Big Data
Obtaining real business value from Big Data
Abstract
This paper describes the Microsoft vision for Big Data, discusses key industry trends, and evaluates how to
obtain real business value from investments in Big Data. To highlight the potential that Big Data holds for
organizations of all sizes, this paper presents common business scenarios that include customer analytics,
risk management, science and research, and IT business innovation.
Author
Achim Granzen, Architect, Microsoft Services
Publication date
November 2012
Version
1.0
We welcome your feedback on this paper. Please send your comments to the Microsoft Services Enterprise
Architecture IP team at ipfeedback@microsoft.com


Acknowledgments
The author would like to thank the following people who contributed to, reviewed, and helped improve this
paper.
Contributors
Susan Conway, Ulrich Homann, Mike Wise

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date
of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on
the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this
document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic,
mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft
Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in
this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not
give you any license to these patents, trademarks, copyrights, or other intellectual property.
The descriptions of other companies’ products in this document, if any, are provided only as a convenience to you. Any such references
should not be considered an endorsement or support by Microsoft. Microsoft cannot guarantee their accuracy, and the products may
change over time. Also, the descriptions are intended as brief highlights to aid understanding, rather than as thorough coverage. For
authoritative descriptions of these products, please consult their respective manufacturers.
© 2012 Microsoft Corporation. All rights reserved. Any use or distribution of these materials without express authorization of Microsoft
Corp. is strictly prohibited.
Microsoft and Windows are either registered trademarks of Microsoft Corporation in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Microsoft Proprietary and Confidential Information

Page ii


Table of contents
1

Executive summary ..................................................................................................................................... 1

2

Microsoft vision - Democratize Big Data ................................................................................................. 2

3

Business motivation .................................................................................................................................... 3
3.1
3.2

Big Data opportunities ................................................................................................................................................... 3
The evolving Big Data platform .................................................................................................................................. 4

4

Key trends ..................................................................................................................................................... 5

5

Enabling business value out of Big Data investments............................................................................ 6
5.1
5.2
5.3

6

Key scenarios.............................................................................................................................................. 10
6.1
6.2
6.3
6.4

7

Customer analytics ......................................................................................................................................................... 11
Risk and performance management ........................................................................................................................ 11
Science and research .................................................................................................................................................... 12
IT business innovation .................................................................................................................................................. 12

Call to action: Getting started with a Big Data strategy ...................................................................... 13
7.1

8

Sensemaking – Gaining insight from Big Data ...................................................................................................... 6
Insight decisioning .......................................................................................................................................................... 7
Innovating your business with Big Data .................................................................................................................. 9

Our approach to Big Data and Business Intelligence ........................................................................................ 13

References and resources ......................................................................................................................... 14


Page iii


1

Executive summary

Enterprises can now generate, capture, and buy large volumes of data from a variety of sources and then use
analytical methods to gain business insight, support business processes, improve competitive advantage,
and generate profit. The convergence of data availability and analytical innovation is commonly referred to
as Big Data, and many organizations are now starting to use it to produce insights that are relevant to
tactical and strategic business issues.
However, more often than not the use of Big Data starts with a lab approach in IT departments that is driven
by the eagerness of IT staff to examine and understand the latest cutting edge technology. To obtain real
business value out of investments in Big Data, a strategic approach is required to join technical possibilities
with business goals. The Microsoft Enterprise Strategy Program makes resources and offerings available that
can help you get started.


Page 1


2 Microsoft vision - Democratize Big Data
Microsoft envisions all business users having the ability to gain actionable insights from virtually any data,
including insights that were previously hidden in unstructured data.
Microsoft is expanding the vision of Business Intelligence to provide business insight as a service layer for
applications to increase the richness and variety of the Big Data experience. This layer can serve as a new
platform to provide insight into structured and unstructured data of any volume by creating unified and
intuitive approaches to discovering, gathering, storing, indexing, exploring, analyzing, and performing selfservice visualization of Big Data.

Figure 1. Aspects of Big Data
The Microsoft vision of democratizing Big Data aims to provide your organization with the following three
key capabilities:
Manage data of any type or size
▪
▪

Flexible data management layer that supports all data types—structured, semi-structured, and
unstructured data at rest or in motion
Data management and analysis function that can be performed on-premises, in the cloud, or using a
hybrid approach

Enrich your data with the world’s data
▪
▪

Enrichment layer for discovering, transforming, sharing, and governing data
Deeper insights that combine an organization’s data with data and services from external sources

Gain insight from any data
▪
▪

Compelling suite of tools to help users gain insight from analytics
Enable insight decision making for everyone from data scientists to casual users


Page 2


3 Business motivation

Figure 2. Overview of the Big Data Customer Analytics Framework

3.1 Big Data opportunities
Big Data delivers game-changing benefits using a new approach to data acquisition, management, and
visualization for the emerging Big Data platforms.
Organizations have always produced significant amounts of unstructured data from sources such as medical
images, blogs, radio-frequency identification (RFID) tags, and locality sensors. Historically, organizations
threw away most of the data they could collect to avoid what were once considered excessive costs of
managing such a data deluge.
Spurred by plummeting storage and computation costs coupled with a new understanding of the inherent
value of previously discarded data, organizations are demanding new types of business insight from every
bit of data they can access using cost-effective and scalable methods. Examples of new insights include:
▪
▪
▪
▪
▪

Understanding user behavior and online interactions
Identifying trends and popular topics in social media sentiment analytics
Optimizing and targeting advertising campaigns
Discovering medical epidemiological trends (such as identifying the next flu outbreak)
Identifying financial fraud within public sector transactions

Such insights are critical in providing competitive advantages to organizations as well as improving tactical
decisions and controlling costs. To achieve such insights, organizations must invest to create a platform that
accommodates the requirements of Big Data.


Page 3


3.2 The evolving Big Data platform
The initiatives launched in the IT labs of Web 2.0 companies are now evolving into platforms with potential
for broad adoption in all industries. The design, operation, and use of a state of the art Big Data platform has
become easier, and the user circle has widened from data scientists in IT labs to business analysts,
information workers, and, increasingly, information consumers. Eventually, nearly everyone in your
organization can analyze and make more informed decisions with the right tools, as described in the
following example.
Example: Using social media to conduct brand research and promotion
Organizations can conduct brand research and promotion using social media to understand and gain
insight into the opinions of customers and the market (public) about the organization and its products and
services. This research can produce immediate, mid-term, and long-term impact on the top line, such as
market research, brand building, brand protection, product development, and customer service.
Insight obtained from brand research can be used at many levels:
▪
▪
▪
▪

Executive level. Determine new markets, customer segments, products, and services.
Marketing director/CMO level. Develop an organizational brand and overall marketing message.
Marketing campaign managers level. Optimize segments, channels, and campaign for highest returns.
Call center agents. Provide better service to customers by better understanding their motivations and
opinions.

Although its initial uses are mostly tactical, Big Data has the potential to drive real business innovation. After
integrating Big Data analytics into operations to improve day-to-day decision making, your organization will
be poised to start innovating with Big Data (see the “Innovating your business with Big Data” section later in
this paper).


Page 4


4 Key trends

Figure 3. The volume and impact of data is rapidly increasing
A set of broad industry trends are putting pressure on traditional data management and Business
Intelligence platforms and tools. These trends are as follows:
Increasing data volumes
The annual growth of worldwide information volume is roughly 50 percent and continues to rise. This
explosion of new data is driven by a full range of traditional and nontraditional sources that include sensors,
devices, and tools that monitor and catalog content on the Internet, such as bots and crawlers. According to
an IDC study,1 the amount of digital information created and replicated is forecasted to hit 1.8 zettabytes
(1.8 trillion gigabytes) in 2011—and is predicted to grow by a factor of 44 during the 2009-2020 forecast
period.
Increasing complexity of data and analysis
The real growth in data comes from unstructured data in a wide variety of documents, streaming data, and
click-stream data. The success of search engine providers and e-retailers who have unlocked the value of
unstructured data has debunked the presumption that 80 percent of such data has no value. The business
requirement to store, analyze, and mine a combination of structured and unstructured data is becoming the
new norm.
Changing economics and emerging technologies
Cloud computing and commodity hardware have dramatically reduced the acquisition cost of
computational and storage capacity and is fundamentally changing the economics of data processing. To
create platforms for tackling massive data processing tasks, architects are complementing commodity
hardware with new, distributed parallel processing frameworks (such as Hadoop and MapReduce) and a rich
ecosystem of tools.
These trends provide a variety of opportunities for organizations to obtain business insight that supports
effective decision making and innovation in business processes, products, and services.

1

IDC Digital Universe Study, sponsored by EMC, June 2010.


Page 5


5 Enabling business value out of Big Data
investments

5.1 Sensemaking – Gaining insight from Big Data
To obtain pertinent insight, traditional data platforms require you to pre-identify and structure data of
interest, and to define and apply a data model. However, with the increasing volume and types of data,
people cannot reliably anticipate which data will be valuable or decide which data can be discarded without
risking the loss of insight.
With the explosion of potentially meaningful data—some structured, some unstructured (such as signals,
streaming, social, interaction, and transactions)—new types of analyses are needed. The cycle that holds the
greatest promise for gaining knowledge from Big Data is based on the concept of sensemaking developed
by Pirolli and Card in 2005 within the intelligence analysis community.

Figure 4. Obtaining value from all data sources


Page 6


Figure 5. The sensemaking cycle is useful for identifying valuable data and questions.
Sensemaking uses continuous feedback and defines interdependent relationships to create a context that
provides information about data and supports analysis of the data. Big Data expands this cycle to the
unmodeled domains of unstructured content.2
One of the most significant aspects of sensemaking is how it transforms traditional analytical processes:
▪
▪

Traditionally, data comes to you. The data is then structured through an information management
process (such as extract, transform, and load) after which enterprise stakeholders can analyze it.
Now, knowledge and insight come to you. You define your business objectives and then allow the
sensemaking process to map them to data in real time, drawing out knowledge that may form the basis
for immediate action, such as modifying product prices in real time.

Sensemaking is emerging within Big Data environments as a powerful method for culling promising
information from torrents of data, as well as helping identify questions that business decision makers should
ask to meet their business objectives.

5.2 Insight decisioning
The connections between an organization, its data, and its processes defines the technologies that support
traditional structured data as well as unstructured data and advanced analytics within an insight decisioning
platform. The following figure illustrates the conceptual model for an insight decisioning platform.

2

Klein, et al, “A Data/Frame Theory of Sensemaking.” June, 2008


Page 7


Figure 6. Conceptual model of an insight decisioning platform
An insight decisioning platform consists of four different types of components: consumption, insight,
integration, and analysis.
The nature of the insight decisioning process calls for Big Data consumption components from cloud
environments as well as internal business data. The insight platform may use a hybrid environment or a pure
cloud environment that processes and transfers refined Big Data results into an on-premises analysis
environment. Resources that are available in the cloud enable organizations of all sizes to enrich their data
sources, often using a pay-as-you-go model that allows them to apply analytics to selected sets of data,
without incurring significant cost.
Insight components provide the following features: Hot stream data (alert information such as customer
credit issues); cold stream data (latent information, such as unstructured data from search engines that track
banking service inquiries); and multiple decision points about a sale or upsell. Alert features, such as those
provided by StreamInsight have the ability to process time series data in near real-time.
Integration components combine results from the Big Data interrogation with results of similar
interrogation of structured business data. The data analyst facilitates the process by defining the mappings
of the unstructured and structured findings to create an integrated result for presentation and general
consumption.
Analysis components provide a business analyst with results to review, such as a customer’s ability to make
loan payments, or upsell opportunities.


Page 8


5.3 Innovating your business with Big Data
A typical approach for innovating your business with Big Data might consist of the following three
components.
Big Data foundation
A foundational infrastructure is necessary to use, manage, and maintain Big Data. The infrastructure can be
integrated into existing Business Intelligence and operational systems, and may be deployed on-premises, in
the cloud, or as a combination of the two. The infrastructure should also include a sandboxed environment
for employees to acquire new skills and experiment with data.
Key benefits include:
▪
▪
▪

Enable subsequent Big Data activities
Enable development of innovative, customer-specific approaches and solutions
Enable ad hoc analysis and discovery for simple, tactical use cases

Big Data operational analytics
Big Data enriches predictive analytics to improve day-to-day operations and tactical business decision
making. Big Data operational analytics use near real-time data (hot stream) and non-real-time data (cold
stream) to help ensure agile and well-founded business action.
▪
▪

Improve operations in all targeted business areas, building on richer, more accurate and detailed
business insight (for example, achieving high-yield campaigns using customer social media behavior)
Become more agile in business operations and tactical decision making

Big Data innovation
Big Data drives business innovation. Using closed-loop marketing techniques, Big Data can enhance
branding and drive sales. Big Data can also reveal opportunities for market-driven products and services,
and identify new business areas and markets for an organization to pursue.
▪
▪
▪

Widen the idea pipeline
Shorten the reaction time to changing business and sales conditions
Increase the agility of decision making processes and strategic decision making


Page 9


6 Key scenarios

There are four main business scenarios in which Big Data is starting to be used as an extension to the
traditional Business Intelligence and analytical systems:

Figure 7. Big Data scenarios


Page 10


There are additional business scenarios for Big Data, particularly in online search, advertising and managing
social networks. However, this section discusses business scenarios beyond web analysis scenarios.
For overview purposes, this section presents cross-industry scenarios, although some scenarios might be
more common in certain industries. For example, customer analytics is especially important for retail and
telecommunications companies, whereas financial services organizations focused more on risk and
performance management. Also, science and research is a hot scenario for the energy industry to support
the search for natural resources. And pharmaceutical companies might consider all four scenarios of equal
priority.

6.1 Customer analytics
Going beyond classic customer analytics that use internal, structured
data, Big Data customer analytics can create a multifaceted view of a
single customer, with actionable business insights from such diverse
sources as point-of-sale transactions, loyalty data, online/web activity,
lifestyle information, market research, demographic data, marketing
channel responses, direct communications, and social media.
Classic customer analytics tools (such as segmentation, churn, and
cross-/up-sell) are enriched and complemented by analysis of text, sentiment, marketing, and advertising to
enable closed-loop processes for optimizing marketing techniques and portfolios, predicting retail behavior,
improving customer service and satisfaction, conducting brand research, protecting brand value, and
developing new products and services.
Sample use cases involving customer analytics include:
▪
▪
▪

Multichannel campaign management and predictive customer analytics
Social media brand promotion and sentiment analytics
New product and services development

6.2 Risk and performance management
Risk and corporate performance management involves primarily
financial and operational risk/portfolios with a primary goal of obtaining
an optimal state that yields the highest benefits (financial, safety, and so
on). A key application is trading and investment portfolio management,
specifically in high-frequency trading and short-term investment, and in
financial risk management for mid and long-term (insurance).

Sample use cases:
▪
▪
▪

Energy trading and risk management (ETRM)
Improved actuarial analysis
Computerized high-frequency trading and portfolio optimization


Page 11


6.3 Science and research
Science and research involves both the public sector and private sector and uses
varied data source such as meteorological data, fundamental research such as
subatomic research, astronomy, medical/pharmaceutical/genetics research, and
natural resources exploration and exploitation. In a wider sense it also deals with
areas such as real-time monitoring in manufacturing and production.

Sample use cases of science and research include:
▪
▪
▪

Natural resources exploration and prospecting
Pharmaceutical clinical trials
Meteorology and the natural disaster research

6.4 IT business innovation
IT organizations across all industries are constantly being evaluated
against the services and benefits they provide to their enterprises. Simply
“keeping the lights on” has become an outdated metric for many IT
organizations: they are now under pressure to innovate and participate
as a business enabler.
Big Data and cloud technologies provide unique opportunities for IT
organizations to examine and redefine their current services, establishing new creative ways for helping
employees do their business better while enhancing operational efficiency.
Sample use cases of IT business innovation include:
▪
▪

Develop new innovative IT services to enable internal/external business clients
Streamline IT operations and modernize technology platforms to exploit today’s opportunities


Page 12


7

Call to action: Getting started with a
Big Data strategy

Microsoft helps customers assess their approaches to Big Data as part of the Enterprise Strategy Program
(ESP). This program focuses on helping enterprises realize value from their IT investments and consists of
four core elements:
▪
▪
▪
▪

Enterprise architects. Dedicated to the customer and charged with accelerating customers toward their
business goals.
Microsoft network. Subject matter experts from across all areas of Microsoft, including product groups,
research and development, internal IT resources, and Microsoft Research.
Value Realization Framework (VRF). A framework and methodology designed to deliver on the value
proposition of the program.
Library. A collection of exclusive intellectual property including comprehensive guidance, reference
architectures, implementation information, and worked examples from Microsoft engagements.

7.1 Our approach to Big Data and Business Intelligence
Our approach focuses on how you can programmatically extend your traditional Business Intelligence
infrastructure to leverage Big Data techniques, aligning the right technologies to help enable and obtain
measurable benefits from your Big Data strategy.
The Big Data Workshop is specifically designed to help you get started with a strategic, business valuefocused approach to Big Data. On a high level, the workshop provides the following:
▪
▪
▪
▪
▪

Rapidly models the Big Data environment in your enterprise
Identifies gaps in your current Big Data/Business Intelligence infrastructure
Finds opportunities to generate value using Big Data techniques and tools
Proposes initiatives to address those opportunities
Augments the initiatives with metrics that measure adoption and realization of opportunity value

For more information about the Microsoft Enterprise Strategy Program, contact your Microsoft account
representative or visit www.microsoft.com/GoESP.


Page 13


8 References and resources
Sources referenced in this white paper include:
▪
▪

IDC Digital Universe Study, sponsored by EMC, June 2010.
Klein, et al, “A Data/Frame Theory of Sensemaking.” June, 2008

Additional resources from Microsoft include:
▪
▪
▪

Conway, Susan. “Obtaining Insight from Big Data.” Microsoft, 2011
Conway, Susan. “Making Decisions with Insight: A process and platform for generating business insight.”
Microsoft, 2012.
Wise, Mike. “CIO considerations for Business Intelligence: Obtaining competitive insight.” Microsoft, 2012.


Page 14

CIO Considerations for Big Data: Obtaining real business value from Big Data

Recommended

Recommended

More Related Content

More from Microsoft

More from Microsoft (20)

Recently uploaded

Recently uploaded (20)

CIO Considerations for Big Data: Obtaining real business value from Big Data