Management Science
Why Has “Data Warehouse” Become a
Dirty Word?
Bill Dagan
In done to fix some of these situations, and high-
lights what to do if you are embarking on a new
the information technology (IT) industry as
development effort.
it applies to natural gas and electric utilities, it
seems that there are more failed attempts in
DATA WAREHOUSE’S PLACE IN A
building and using a data warehouse than there
BUSINESS INTELLIGENCE SYSTEM
are success stories. In fact, the truly successful
First, let’s put a data warehouse in prospec-
implementations—those that create real value
tive within the big picture. In my last article,1 I
for the organization—are indeed rare. Why are
stated that a data warehouse was an important
so many equating the “DW” acronym to “Dirty
foundation to a comprehensive business intelli-
Word”? Why is there such a negative connota-
gence (BI) system.
tion to a data warehouse when there should be
In itself, a data warehouse is not a BI system
the opposite?
and the true value of implementing a data ware-
house is often not realized until other compo-
nents of your BI system are put in place. How-
Why are so many equating the “DW” acronym to
ever, the foundation is one of the most
“Dirty Word”?
important components of a BI system. While a
data warehouse is not the only methodology
that can be used for a BI foundation, it is the
A data warehouse should bring a smile to
most common and probably the most straight-
your face, especially if you experienced life prior
forward to implement and maintain.
to the existence of a data warehouse. You re-
A brief review of where a data warehouse fits
member—when your organization was bridging
into your entire BI platform will help portray
data from various disparate sources into numer-
some of the failure points discussed later. Ex-
ous spreadsheet reports that could be inconsis-
hibit 1 shows that a data warehouse extracts and
tent or unreliable?
loads data from various sources into a data ware-
You are still doing this? I am sorry to report
house system. The data warehouse system is typ-
that your situation appears to be the norm
ically composed of a staging area; subject-area
rather than the exception, at least in our indus-
data marts; and possibly online analytical pro-
try. This article explains some of the reasons
cessing (OLAP) cubes, which are three-dimen-
that many data warehouse implementations
sional views of data that are typically pre-
have failed, provides advice on what can be
processed so that the users can rapidly roll up
and drill down into the data. The data within is
provided to the user via direct access.
Bill Dagan is president of Energy Data Solu-
WHY ALL THE FAILURES?
tions (endasol.com). He can be reached at
wjdagan@endasol.com.
There are many reasons why data warehouse
projects fail, leaving operating management
18 NATURAL GAS & ELECTRICITY JULY 2007
© 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas

empty and still searching for a better way to sat-
isfy their analytical hunger. Exhibit 1. Data Warehouse System
Lack of Clear Requirements
Any project must start with the end in mind.
What is it that you are building, what will it ac-
complish, and how will it be used? This is par-
ticularly important for a data warehouse. This
sounds fundamental, but many data warehouse
efforts have failed for this very reason: lack of a
clear vision and detailed requirements necessary
to achieve this vision.
You need to determine what data will be in-
cluded. Some IT organizations may feel that
they need to consolidate all the data across the
enterprise so that the business community could
do additional analysis and reporting not
presently possible. While this is a noble goal,
consolidating all data may not be needed.
You need to ask what data must be consolidated
into a data warehouse and why.
You need to ask what data must be consoli-
dated into a data warehouse and why. If this
simple question cannot be answered for any
data element, then this particular data should
not be brought into the data warehouse. You
also need to know how moving data into a data
warehouse will help improve your present re-
porting and analysis. If it is not clear how hav-
ing the data in a data warehouse will help you,
then you are not likely to migrate from your
present process (however cumbersome) to
adopt the new system.
Lack of Business Sponsorship and Data
They should provide subject matter experts
Stewardship
or data stewards that “own” specific subject areas
Another reason that data warehouses fail is
in the data warehouse. These individuals should
that the operating managers have not really
be ultimately responsible for the quality of data
sponsored the effort. Sure, they may have actu-
within their subject area and have a base under-
ally funded the effort out of their budgets, but
standing of the data model. Their management
this is not the same as sponsorship. The operat-
must give them the time and authority to act in
ing managers really need to be part of the effort
this capacity. IT professionals should work with
on a regular basis. They need to help develop
these data stewards to provide the processes to
the requirements, and certainly need to endorse
meet their quality standards and the mechanisms
them. Operating managers also need to be a part
that will allow them to determine when data may
of every major decision affecting the end prod-
be missing or is otherwise questionable.
uct and be involved in acceptance testing.
JULY 2007 NATURAL GAS & ELECTRICITY 19
DOI 10.1002/gas / © 2007 Wiley Periodicals, Inc.

Tackling Too Much at Once Scope Creep
A data warehouse project has the connota- Scope creep or, in some cases, scope ava-
tion of a very long and costly effort. Many lanches are also key culprits in unsuccessful at-
data warehouse projects tried to tackle too tempts in building a data warehouse. This often
much at once. The business community stems from unclear requirements in the first
wanted it all now and/or the IT organization place. However, even with the well-defined re-
was set on delivering an entire enterprise sys- quirements, additional items will surface as the
tem from the start. Many valuable months project progresses. It is not feasible to know
have been devoted to designing the entire en- everything up front.
terprise data model, only to find that it had to How you handle these unexpected situa-
be significantly refined during the develop- tions determines the success of the project. It is
ment process anyway. imperative that you have a well-defined change
Once the enterprise data model was devel- control process in place that the business com-
oped, some have tried the “big bang” approach munity and IT have both endorsed and strictly
by implementing many or all subject areas in adhere to throughout the project. This is not
parallel. This approach has seldom provided real to say that scope additions cannot or should
value in a timely manner. The people in the not occur, but they should only happen when
business got tired of waiting for tangible results, both parties understand and accept the impact
and management grew tired of the money drain on the budget and schedule for the particular
and sometimes pulled the plug in midstream. addition. Without such controls in place, the
Even if the effort continued to some resem- resulting project is often well over budget. As a
blance of completion, the result was a mediocre result, many are not given the opportunity to
system because the business community lost in- reach completion.
terest early in the project.
The alternative is to start small with clear, Inability for Business Users to Use the
tangible deliverables in short periods of time— Data Warehouse
say, three to four months. An effective approach Now IT has built a data warehouse, but op-
is to start with a specific subject area or set and erating management still cannot use it to the
build a data mart. Subsequent phases can either extent needed. There could be a number of rea-
add to that data mart or create new subject-area sons for this. There may not be a clear road map
data marts. for what data is in the data warehouse, how it
relates to other data, and how to access this
data. Industry terminology for such guides is
business metadata.
…start small with clear, tangible deliverables in
It could be in the form of help menus, dia-
short periods of time—say, three to four months.
grams, and/or data dictionaries. Alternatively,
perhaps the business is not using the data ware-
A data mart can be considered a smaller ver- house because it is too difficult to retrieve the
sion of a data warehouse that usually focuses on data they want. This situation may result from a
a specific subject area or serves a particular busi- data model that does not promote reporting and
ness segment. In fact, a complete data ware- user queries. Some remedies to correct this situ-
house is often the result of linking the individ- ation are discussed below.
ual subject-area data marts to form a data Finally, the business may not use the data
warehouse. Doing such a phased approach pro- warehouse because they are not confident of the
vides very visible benefits. Each business seg- quality or completeness of the data. Perhaps
ment gets tangible results in a short amount of some of the early results contained missing or
time. The value of these early results can be as- questionable data. In this case, it may be diffi-
sessed by management, and course corrections cult to convince business users to come back
can be made prior to embarking on the subse- when these issues have been fixed. All of the
quent phases. This will improve the overall qual- above cases could be avoided by having data
ity of the end product. stewards from the business participate through-
20 NATURAL GAS & ELECTRICITY JULY 2007
© 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas

out the development process and take owner- but the structure (data model) is not easily nav-
ship of their assigned subject areas. igated by the business, then you are in fair shape.
Most of the intense effort is complete and prob-
ably sound. However, operating management is
Failure to Maintain the Data Warehouse
A data warehouse is not a static system. In alienated from the very data and analysis that it
most cases, it is never really complete because sought in the first place.
you should continue to refine it to further im- Do not fret, all is not lost. The data model
prove the effectiveness of analysis and reports. may not be arranged to facilitate user naviga-
Also, because the data warehouse retrieves data tion, but the data is there. You can do a number
from various source applications, as those appli- of things to promote use and satisfy the busi-
cations are updated, the data warehouse may ness clients without restructuring the entire
need to be modified accordingly. If properly de- data warehouse.
signed, a data warehouse will also provide noti- If there are specific queries (data sets) that
fication of potential data issues. users frequently seek, then consider develop-
One such notification could involve missing ing views to meet these needs. A view is often
data. If you expect to get data for 24 generating assembled from a combination (join) of data
units and only 22 units were loaded in last from several database tables. This saves the
night’s process, then an alert should be issued. business user the hassle of traversing (joining)
In other cases, new data in the source may be several database tables. In fact, even with a
discovered and it is not evident how to relate sound data warehouse data model, you may
this to other data in the data warehouse. For ex- find that providing views to replace typical
ample, if you have provisions to process data for queries will promote use of the data ware-
24 generating units and data associated with a house.
25th unit suddenly appears, then someone You can also build focused data marts and/or
needs to scrutinize this to ensure that it is valid OLAP cubes pertinent to the specific business
data and, if so, establish the link to other data el- segment (e.g., marketing) from the data within
ements within the data warehouse. the data warehouse.
All of these are fundamental design practices
for a data warehouse. However, if you fail to What Can Be Done If Quality Is Your
properly monitor and act on these notifications, Issue
then the quality of the data warehouse rapidly If your data warehouse is fraught with miss-
deteriorates and people will stop using it. You ing or questionable data, then you have a real
need to maintain a knowledgeable IT staff and problem. You need to assess the situation to
business data stewards to continuously monitor pinpoint the failure points. Do these issues
and refine your data warehouse. The size of this pervade throughout your system or do they
staff depends on the size of your data warehouse. pertain to pulling data from only a handful of
data sources?
CAN YOUR SITUATION BE FIXED?
Assuming that you have at least progressed to
putting some type of data warehouse in place, If your data warehouse is fraught with missing or
can any of the ailments discussed above be cor- questionable data, then you have a real problem.
rected? In most cases, yes, but how much more
you need to spend depends on what’s wrong
with your existing data warehouse and who you Short of going back to square one, you may
engage to help remedy the situation. Below are a be able to selectively correct the data issues. You
few suggestions. should do this in a controlled, phased approach
with clear deliverables with each phase. This
should be a joint effort between the business
Accessing Data Without Rebuilding the
and IT.
Data Warehouse
If you have succeeded in transporting data At this point, you should also consider en-
from numerous sources, the data is complete, gaging outside expertise, because a new set of
JULY 2007 NATURAL GAS & ELECTRICITY 21
DOI 10.1002/gas / © 2007 Wiley Periodicals, Inc.

independent eyes may be in a better position to house and not another set of disjointed data
pinpoint fundamental issues. However, beware silos. Each phase should be about three to
of consultants who immediately insist that you four months.
start all over without doing a full assessment of • Expertise: Chances are that if you are em-
your situation or without a strong case to back barking on your first effort or starting over
this up. Seek a real consulting partner who has from a failed effort, then you do not have
your best interest in mind. Then again, the re- the appropriate expertise in house to develop
sulting evaluation may still determine that the a proper system. Engage outside experts that
existing data warehouse is not likely repairable have developed data warehouses in our in-
and it is best to start all over. dustry. Ensure that these consultants have
references for implementing successful proj-
ects at companies similar to yours. Some
WHAT TO DO IF STARTING A NEW DATA
companies may also be able to provide
WAREHOUSE PROJECT
After reading the above, you may be even “starter models” or “packaged solutions” that
more nervous about embarking on a data ware- may fulfill many of your requirements. Look
house effort. I hope that this is not the case, be- for these, but understand what you are get-
cause a well-developed and maintained data ting and not getting from using these. When
warehouse can be invaluable. There are many you engage outside expertise, you also need
companies within our industry that have an im- to have your own staff, both IT and busi-
pressive data warehouse in place and could not ness, intimately involved throughout the de-
imagine life without this capability. velopment process.
• Business involvement: Ensure that you have
business sponsorship and data stewardship
throughout the project. If operating man-
There are many companies within our industry
agement is not involved and not commit-
that have an impressive data warehouse in place
ted to this process, then do not build it. It’s
and could not imagine life without this capability.
that simple.
• Manage change: Adopt a comprehensive
Unfortunately, there are many more that change control process from the start. This is
have experienced a very painful implementation a necessity. It also requires that you have
effort that has either failed completely or has re- clearly defined requirements for each phase
sulted in a mediocre system that does not pro- of the project and that these requirements are
duce the benefits originally sought. In explain- understood by the business, IT, and your
ing some of the reasons for these failures, I did consulting partner. Then when situations
not mean to discourage you from having a data arise that require changes to these require-
warehouse that will bring real value to your or- ments, all parties can assess the impact on the
ganization, but I hope that I have opened your budget and schedule before and if acting on
eyes to some important considerations necessary the new item.
for a successful implementation.
What should you do when (not if ) you em- By adopting a few sensible practices, you
bark on an initial or renewed effort? Do the should start seeing value in months, not years.
opposite of what does not work. Do at least You can continue to build on this so that in the
the following: end you have a system that not only meets your
basic analytical and reporting needs, but also
• Phases: Break the project into small, measur- provides the needed foundation to support
able phases with clear objectives and tangible more timely decisions and to promote swifter
deliverables in each phase. This methodology action on changes in market conditions.
could be accomplished by building subject-
area data marts and ensuring that the data NOTE
marts can be properly linked to each other so 1. (2007, May). Business intelligence simply stated. Natural
that the end result is an enterprise data ware- Gas & Electricity, pp. 24–27.
22 NATURAL GAS & ELECTRICITY JULY 2007
© 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas