Data Warehouse Dirty Word

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    1 Favorite

    Data Warehouse Dirty Word - Presentation Transcript

    1. Management Science Why Has “Data Warehouse” Become a Dirty Word? Bill Dagan In done to fix some of these situations, and high- lights what to do if you are embarking on a new the information technology (IT) industry as development effort. it applies to natural gas and electric utilities, it seems that there are more failed attempts in DATA WAREHOUSE’S PLACE IN A building and using a data warehouse than there BUSINESS INTELLIGENCE SYSTEM are success stories. In fact, the truly successful First, let’s put a data warehouse in prospec- implementations—those that create real value tive within the big picture. In my last article,1 I for the organization—are indeed rare. Why are stated that a data warehouse was an important so many equating the “DW” acronym to “Dirty foundation to a comprehensive business intelli- Word”? Why is there such a negative connota- gence (BI) system. tion to a data warehouse when there should be In itself, a data warehouse is not a BI system the opposite? and the true value of implementing a data ware- house is often not realized until other compo- nents of your BI system are put in place. How- Why are so many equating the “DW” acronym to ever, the foundation is one of the most “Dirty Word”? important components of a BI system. While a data warehouse is not the only methodology that can be used for a BI foundation, it is the A data warehouse should bring a smile to most common and probably the most straight- your face, especially if you experienced life prior forward to implement and maintain. to the existence of a data warehouse. You re- A brief review of where a data warehouse fits member—when your organization was bridging into your entire BI platform will help portray data from various disparate sources into numer- some of the failure points discussed later. Ex- ous spreadsheet reports that could be inconsis- hibit 1 shows that a data warehouse extracts and tent or unreliable? loads data from various sources into a data ware- You are still doing this? I am sorry to report house system. The data warehouse system is typ- that your situation appears to be the norm ically composed of a staging area; subject-area rather than the exception, at least in our indus- data marts; and possibly online analytical pro- try. This article explains some of the reasons cessing (OLAP) cubes, which are three-dimen- that many data warehouse implementations sional views of data that are typically pre- have failed, provides advice on what can be processed so that the users can rapidly roll up and drill down into the data. The data within is provided to the user via direct access. Bill Dagan is president of Energy Data Solu- WHY ALL THE FAILURES? tions (endasol.com). He can be reached at wjdagan@endasol.com. There are many reasons why data warehouse projects fail, leaving operating management 18 NATURAL GAS & ELECTRICITY JULY 2007 © 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas
    2. empty and still searching for a better way to sat- isfy their analytical hunger. Exhibit 1. Data Warehouse System Lack of Clear Requirements Any project must start with the end in mind. What is it that you are building, what will it ac- complish, and how will it be used? This is par- ticularly important for a data warehouse. This sounds fundamental, but many data warehouse efforts have failed for this very reason: lack of a clear vision and detailed requirements necessary to achieve this vision. You need to determine what data will be in- cluded. Some IT organizations may feel that they need to consolidate all the data across the enterprise so that the business community could do additional analysis and reporting not presently possible. While this is a noble goal, consolidating all data may not be needed. You need to ask what data must be consolidated into a data warehouse and why. You need to ask what data must be consoli- dated into a data warehouse and why. If this simple question cannot be answered for any data element, then this particular data should not be brought into the data warehouse. You also need to know how moving data into a data warehouse will help improve your present re- porting and analysis. If it is not clear how hav- ing the data in a data warehouse will help you, then you are not likely to migrate from your present process (however cumbersome) to adopt the new system. Lack of Business Sponsorship and Data They should provide subject matter experts Stewardship or data stewards that “own” specific subject areas Another reason that data warehouses fail is in the data warehouse. These individuals should that the operating managers have not really be ultimately responsible for the quality of data sponsored the effort. Sure, they may have actu- within their subject area and have a base under- ally funded the effort out of their budgets, but standing of the data model. Their management this is not the same as sponsorship. The operat- must give them the time and authority to act in ing managers really need to be part of the effort this capacity. IT professionals should work with on a regular basis. They need to help develop these data stewards to provide the processes to the requirements, and certainly need to endorse meet their quality standards and the mechanisms them. Operating managers also need to be a part that will allow them to determine when data may of every major decision affecting the end prod- be missing or is otherwise questionable. uct and be involved in acceptance testing. JULY 2007 NATURAL GAS & ELECTRICITY 19 DOI 10.1002/gas / © 2007 Wiley Periodicals, Inc.
    3. Tackling Too Much at Once Scope Creep A data warehouse project has the connota- Scope creep or, in some cases, scope ava- tion of a very long and costly effort. Many lanches are also key culprits in unsuccessful at- data warehouse projects tried to tackle too tempts in building a data warehouse. This often much at once. The business community stems from unclear requirements in the first wanted it all now and/or the IT organization place. However, even with the well-defined re- was set on delivering an entire enterprise sys- quirements, additional items will surface as the tem from the start. Many valuable months project progresses. It is not feasible to know have been devoted to designing the entire en- everything up front. terprise data model, only to find that it had to How you handle these unexpected situa- be significantly refined during the develop- tions determines the success of the project. It is ment process anyway. imperative that you have a well-defined change Once the enterprise data model was devel- control process in place that the business com- oped, some have tried the “big bang” approach munity and IT have both endorsed and strictly by implementing many or all subject areas in adhere to throughout the project. This is not parallel. This approach has seldom provided real to say that scope additions cannot or should value in a timely manner. The people in the not occur, but they should only happen when business got tired of waiting for tangible results, both parties understand and accept the impact and management grew tired of the money drain on the budget and schedule for the particular and sometimes pulled the plug in midstream. addition. Without such controls in place, the Even if the effort continued to some resem- resulting project is often well over budget. As a blance of completion, the result was a mediocre result, many are not given the opportunity to system because the business community lost in- reach completion. terest early in the project. The alternative is to start small with clear, Inability for Business Users to Use the tangible deliverables in short periods of time— Data Warehouse say, three to four months. An effective approach Now IT has built a data warehouse, but op- is to start with a specific subject area or set and erating management still cannot use it to the build a data mart. Subsequent phases can either extent needed. There could be a number of rea- add to that data mart or create new subject-area sons for this. There may not be a clear road map data marts. for what data is in the data warehouse, how it relates to other data, and how to access this data. Industry terminology for such guides is business metadata. …start small with clear, tangible deliverables in It could be in the form of help menus, dia- short periods of time—say, three to four months. grams, and/or data dictionaries. Alternatively, perhaps the business is not using the data ware- A data mart can be considered a smaller ver- house because it is too difficult to retrieve the sion of a data warehouse that usually focuses on data they want. This situation may result from a a specific subject area or serves a particular busi- data model that does not promote reporting and ness segment. In fact, a complete data ware- user queries. Some remedies to correct this situ- house is often the result of linking the individ- ation are discussed below. ual subject-area data marts to form a data Finally, the business may not use the data warehouse. Doing such a phased approach pro- warehouse because they are not confident of the vides very visible benefits. Each business seg- quality or completeness of the data. Perhaps ment gets tangible results in a short amount of some of the early results contained missing or time. The value of these early results can be as- questionable data. In this case, it may be diffi- sessed by management, and course corrections cult to convince business users to come back can be made prior to embarking on the subse- when these issues have been fixed. All of the quent phases. This will improve the overall qual- above cases could be avoided by having data ity of the end product. stewards from the business participate through- 20 NATURAL GAS & ELECTRICITY JULY 2007 © 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas
    4. out the development process and take owner- but the structure (data model) is not easily nav- ship of their assigned subject areas. igated by the business, then you are in fair shape. Most of the intense effort is complete and prob- ably sound. However, operating management is Failure to Maintain the Data Warehouse A data warehouse is not a static system. In alienated from the very data and analysis that it most cases, it is never really complete because sought in the first place. you should continue to refine it to further im- Do not fret, all is not lost. The data model prove the effectiveness of analysis and reports. may not be arranged to facilitate user naviga- Also, because the data warehouse retrieves data tion, but the data is there. You can do a number from various source applications, as those appli- of things to promote use and satisfy the busi- cations are updated, the data warehouse may ness clients without restructuring the entire need to be modified accordingly. If properly de- data warehouse. signed, a data warehouse will also provide noti- If there are specific queries (data sets) that fication of potential data issues. users frequently seek, then consider develop- One such notification could involve missing ing views to meet these needs. A view is often data. If you expect to get data for 24 generating assembled from a combination (join) of data units and only 22 units were loaded in last from several database tables. This saves the night’s process, then an alert should be issued. business user the hassle of traversing (joining) In other cases, new data in the source may be several database tables. In fact, even with a discovered and it is not evident how to relate sound data warehouse data model, you may this to other data in the data warehouse. For ex- find that providing views to replace typical ample, if you have provisions to process data for queries will promote use of the data ware- 24 generating units and data associated with a house. 25th unit suddenly appears, then someone You can also build focused data marts and/or needs to scrutinize this to ensure that it is valid OLAP cubes pertinent to the specific business data and, if so, establish the link to other data el- segment (e.g., marketing) from the data within ements within the data warehouse. the data warehouse. All of these are fundamental design practices for a data warehouse. However, if you fail to What Can Be Done If Quality Is Your properly monitor and act on these notifications, Issue then the quality of the data warehouse rapidly If your data warehouse is fraught with miss- deteriorates and people will stop using it. You ing or questionable data, then you have a real need to maintain a knowledgeable IT staff and problem. You need to assess the situation to business data stewards to continuously monitor pinpoint the failure points. Do these issues and refine your data warehouse. The size of this pervade throughout your system or do they staff depends on the size of your data warehouse. pertain to pulling data from only a handful of data sources? CAN YOUR SITUATION BE FIXED? Assuming that you have at least progressed to putting some type of data warehouse in place, If your data warehouse is fraught with missing or can any of the ailments discussed above be cor- questionable data, then you have a real problem. rected? In most cases, yes, but how much more you need to spend depends on what’s wrong with your existing data warehouse and who you Short of going back to square one, you may engage to help remedy the situation. Below are a be able to selectively correct the data issues. You few suggestions. should do this in a controlled, phased approach with clear deliverables with each phase. This should be a joint effort between the business Accessing Data Without Rebuilding the and IT. Data Warehouse If you have succeeded in transporting data At this point, you should also consider en- from numerous sources, the data is complete, gaging outside expertise, because a new set of JULY 2007 NATURAL GAS & ELECTRICITY 21 DOI 10.1002/gas / © 2007 Wiley Periodicals, Inc.
    5. independent eyes may be in a better position to house and not another set of disjointed data pinpoint fundamental issues. However, beware silos. Each phase should be about three to of consultants who immediately insist that you four months. start all over without doing a full assessment of • Expertise: Chances are that if you are em- your situation or without a strong case to back barking on your first effort or starting over this up. Seek a real consulting partner who has from a failed effort, then you do not have your best interest in mind. Then again, the re- the appropriate expertise in house to develop sulting evaluation may still determine that the a proper system. Engage outside experts that existing data warehouse is not likely repairable have developed data warehouses in our in- and it is best to start all over. dustry. Ensure that these consultants have references for implementing successful proj- ects at companies similar to yours. Some WHAT TO DO IF STARTING A NEW DATA companies may also be able to provide WAREHOUSE PROJECT After reading the above, you may be even “starter models” or “packaged solutions” that more nervous about embarking on a data ware- may fulfill many of your requirements. Look house effort. I hope that this is not the case, be- for these, but understand what you are get- cause a well-developed and maintained data ting and not getting from using these. When warehouse can be invaluable. There are many you engage outside expertise, you also need companies within our industry that have an im- to have your own staff, both IT and busi- pressive data warehouse in place and could not ness, intimately involved throughout the de- imagine life without this capability. velopment process. • Business involvement: Ensure that you have business sponsorship and data stewardship throughout the project. If operating man- There are many companies within our industry agement is not involved and not commit- that have an impressive data warehouse in place ted to this process, then do not build it. It’s and could not imagine life without this capability. that simple. • Manage change: Adopt a comprehensive Unfortunately, there are many more that change control process from the start. This is have experienced a very painful implementation a necessity. It also requires that you have effort that has either failed completely or has re- clearly defined requirements for each phase sulted in a mediocre system that does not pro- of the project and that these requirements are duce the benefits originally sought. In explain- understood by the business, IT, and your ing some of the reasons for these failures, I did consulting partner. Then when situations not mean to discourage you from having a data arise that require changes to these require- warehouse that will bring real value to your or- ments, all parties can assess the impact on the ganization, but I hope that I have opened your budget and schedule before and if acting on eyes to some important considerations necessary the new item. for a successful implementation. What should you do when (not if ) you em- By adopting a few sensible practices, you bark on an initial or renewed effort? Do the should start seeing value in months, not years. opposite of what does not work. Do at least You can continue to build on this so that in the the following: end you have a system that not only meets your basic analytical and reporting needs, but also • Phases: Break the project into small, measur- provides the needed foundation to support able phases with clear objectives and tangible more timely decisions and to promote swifter deliverables in each phase. This methodology action on changes in market conditions. could be accomplished by building subject- area data marts and ensuring that the data NOTE marts can be properly linked to each other so 1. (2007, May). Business intelligence simply stated. Natural that the end result is an enterprise data ware- Gas & Electricity, pp. 24–27. 22 NATURAL GAS & ELECTRICITY JULY 2007 © 2007 Wiley Periodicals, Inc. / DOI 10.1002/gas
    SlideShare Zeitgeist 2009

    + guest08f07guest08f07 Nominate

    custom

    645 views, 1 favs, 0 embeds more stats

    Bill Dagan, Natural Gas & Electricity, July 2007. T more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 645
      • 645 on SlideShare
      • 0 from embeds
    • Comments 0
    • Favorites 1
    • Downloads 42
    Most viewed embeds

    more

    All embeds

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories