• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
White Paper -  Data Warehouse Governance
 

White Paper - Data Warehouse Governance

on

  • 4,081 views

An organisation that is embarking on a data warehousing project is undertaking a long-term development and maintenance programme of a computer system. This system will be critical to the organisation ...

An organisation that is embarking on a data warehousing project is undertaking a long-term development and maintenance programme of a computer system. This system will be critical to the organisation and cost a significant amount of money, therefore control of the system is vital. Governance defines the model the organisation will use to ensure optimal use and re- use of the data warehouse and enforcement of corporate policies (e.g. business design, technical design and application security) and ultimately derive value for money.
This paper has identified five sources of change to the system and the aspects of the system that these sources of change will influence in order to assist the organisation to develop standards and structures to support the development and maintenance of the solution. These standards and structures must then evolve, as the programme develops to meet its changing needs.
“Documentation is not understanding, process is not discipline, formality is not skill”1
The best governance must only be an aid to the development and not an end in itself. Data Warehouses are successful because of good understanding, discipline and the skill of those involved. On the other hand systems built to a template without understanding, discipline and skill will inevitably deliver a system that fails to meet the users’ needs and sooner rather than later will be left on the shelf, or maintained at a very high cost but with little real use.

Statistics

Views

Total Views
4,081
Views on SlideShare
4,081
Embed Views
0

Actions

Likes
1
Downloads
135
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    White Paper -  Data Warehouse Governance White Paper - Data Warehouse Governance Document Transcript

    • Data Management & Warehousing WHITE PAPER Data Warehouse Governance DAVID M WALKER Version: 1.1 Date: 06/04/2007 Data Management & Warehousing 138 Finchampstead Road, Wokingham, Berkshire, RG41 2NU, United Kingdom http://www.datamgmt.com
    • White Paper - Data Warehouse GovernanceTable of ContentsTable of Contents ...................................................................................................................... 2Synopsis .................................................................................................................................... 3Intended Audience .................................................................................................................... 3About Data Management & Warehousing ................................................................................. 3Introduction................................................................................................................................ 4What is governance?................................................................................................................. 5What affects the data warehouse? ............................................................................................ 6What is affected within the data warehouse? ............................................................................ 7How often will change occur?.................................................................................................... 9What organisational structures are needed?........................................................................... 10 Executive Sponsor(s) .......................................................................................................... 10 Steering Committee ............................................................................................................ 11 Programme Management ................................................................................................... 11 Implementation Teams........................................................................................................ 12 Exploitation Teams.............................................................................................................. 12 User Forums ....................................................................................................................... 12 Certification Committee....................................................................................................... 13Project Development Methodologies....................................................................................... 14 Waterfall .............................................................................................................................. 14 Iterative ............................................................................................................................... 14 Agile .................................................................................................................................... 15 Cowboy Coding................................................................................................................... 16Best practices .......................................................................................................................... 17 Momentum .......................................................................................................................... 17 Monitor and Measure .......................................................................................................... 17 Prioritise .............................................................................................................................. 17 Light touch........................................................................................................................... 17 Communication ................................................................................................................... 17 Standards............................................................................................................................ 18 Track processes.................................................................................................................. 18 Understand cost and value ................................................................................................. 18 Continuous Learning and Improvement .............................................................................. 18 Pilot & Prototype ................................................................................................................. 18 Authority to act .................................................................................................................... 18 Plan for the long term.......................................................................................................... 19Governance and Success ....................................................................................................... 19Summary ................................................................................................................................. 20Appendices.............................................................................................................................. 21 Appendix 1 - Programme or Project?.................................................................................. 21 Appendix 2 – The "Declaration of Interdependence" .......................................................... 22 Appendix 3 - Team Values and Principles .......................................................................... 23References .............................................................................................................................. 24 Web resources .................................................................................................................... 24Acknowledgements ................................................................................................................. 24Copyright ................................................................................................................................. 24 © 2007 Data Management & Warehousing Page 2
    • White Paper - Data Warehouse GovernanceSynopsisAn organisation that is embarking on a data warehousing project is undertaking a long-termdevelopment and maintenance programme of a computer system. This system will be criticalto the organisation and cost a significant amount of money, therefore control of the system isvital. Governance defines the model the organisation will use to ensure optimal use and re-use of the data warehouse and enforcement of corporate policies (e.g. business design,technical design and application security) and ultimately derive value for money.This paper has identified five sources of change to the system and the aspects of the systemthat these sources of change will influence in order to assist the organisation to developstandards and structures to support the development and maintenance of the solution. Thesestandards and structures must then evolve, as the programme develops to meet its changingneeds. 1 “Documentation is not understanding, process is not discipline, formality is not skill”The best governance must only be an aid to the development and not an end in itself. DataWarehouses are successful because of good understanding, discipline and the skill of thoseinvolved. On the other hand systems built to a template without understanding, discipline andskill will inevitably deliver a system that fails to meet the users’ needs and sooner rather thanlater will be left on the shelf, or maintained at a very high cost but with little real use.Intended AudienceReader Recommended ReadingExecutive Synopsis and SummaryBusiness Users Entire DocumentIT Management Entire DocumentIT Strategy Entire DocumentIT Project Management Entire DocumentIT Developers Entire DocumentAbout Data Management & WarehousingData Management & Warehousing is a specialist consultancy in data warehousing based inWokingham, Berkshire in the United Kingdom. Founded in 1995 by David M Walker, ourconsultants have worked for major corporations around the world including the US, Europe,Africa and the Middle East. Our clients are invariably large organisations with a pressing needfor business intelligence. We have worked in many industry sectors but have specialists inTelco’s, manufacturing, retail, financial and transport as well as technical expertise in many ofthe leading technologies.For further information visit our website at: http://www.datamgmt.com1 Agile Software Development, Cockburn, A., Addison-Wesley, 2002 © 2007 Data Management & Warehousing Page 3
    • White Paper - Data Warehouse GovernanceIntroductionData warehouse governance defines the model the organisation will use to ensure optimaluse and re-use of the data warehouse and enforcement of corporate policies (e.g. businessdesign, technical design and application security).Governance structures must be in place as early as possible, ideally before the project startsand before development begins. They are implemented in parallel with the initial developmentto ensure that they work together from the outset.A data warehouse is, by definition, a system that interacts with large parts of the organisation.At a technical level this involves the number of systems feeding data to and being fed datafrom the data warehouse. At a human level it relates to the people who interact with thesystem. These include those querying the system, those operating the technical environmentand those directing the business who may not use the system directly but will mandate thatothers do so.The published literature on the subject falls into three categories: • Home grown – what a specific project did within a specific environment • Vendor specific – what to do when using the vendors tool set • System Integrator proprietary – how an SI say they will do it if they are given the contractThe problem is that the data warehouse interacts with and across so much of theorganisation. A programme must deploy standards and processes that integrate into theorganisations’ existing policies and procedures.This paper looks at the areas that must be covered by governance and discusses some of thestandards, options and issues. These ideas should be incorporated into the governance of thedata warehouse. © 2007 Data Management & Warehousing Page 4
    • White Paper - Data Warehouse GovernanceWhat is governance?In the introduction governance is defined as ‘the model the organisation will use to ensureoptimal use and re-use of the data warehouse and enforcement of corporate policies’, butwhat does it mean?Data Warehouses are not a single short-term project. They are long-term programmes ofwork that includes a number of projects and a significant amount of maintenance. Thedecision to deploy a data warehouse commits the business to a programme that needs to becontrolled to ensure that it provides business benefit and value for money.The role of governance is to provide the policies, processes and procedures necessary toensure that the programme of work is effective. These must be clearly communicated toeveryone involved. Governance covers the integrated management of the programme from itsinitial development, through production running to end-of-life close down. The governancestructures must also be reviewed over the lifetime of the programme. This is to ensure thatthe governance is neither too little to allow effective control or so much such that programmework is stifled.Different organisations also have different requirements for the level of detail and breadth ofscope that should be covered by Data Warehouse governance. For example, doesgovernance imply coding standards? For some people this is far too much detail, for others itis essential that the governance of the programme ensures that specific standards aredeveloped and that staff adhered to them.The key to designing a suitable level of governance within an organisation is in understandingthe following: • What events affect the data warehouse environment? • What is affected in the data warehouse by the impacting event? • What type of organisation is needed to support the data warehouse? • What are the policies and procedures needed to control the data warehouse?From this it is possible to develop the standards, processes and communications that will bothdrive the programme forward and culturally fit the specific organisation.The processes, used in conjunction with the standards, are the way in which decisions aremade. Good processes enabling quick, effective and well-communicated decisions andconsequently effective management of the solution. The converse is also true as poorstandards and processes lead to delays and unresolved issues that discredit the datawarehouse and create significant cost or overheads that eventually destroy the programme. © 2007 Data Management & Warehousing Page 5
    • White Paper - Data Warehouse GovernanceWhat affects the data warehouse?Data Management & Warehousing has identified five types of change that impact a datawarehouse: • Demand Demand is the change needed to the system to give the users what they want. This may include loading new data, possibly from new sources or restructuring existing information. It can also include adjusting system availability or the time at which data is loaded or any other user requirement. • External Change A data warehouse has a large number of systems feeding it. The systems operate in a complex and integrated environment. For example, a data warehouse might have three source systems and each source system is only allowed to perform upgrades and patches once a quarter. As a result the data warehouse is faced with twelve changes a year or one a month. In an ideal environment these changes will be developed and regression tested in line with the change in the source system. In practice most data warehouses have many more sources and more frequent changes. When these changes are not communicated they become issues. Managing external change prevents issues and therefore reduces ‘emergency fixes’ and ‘tactical solutions’ • Maintenance All computer system have to manage routine maintenance issues such as software patches, upgrades to software or hardware, special backups for year end, etc. It is easy to ignore upgrades - the system is currently working so why bother? There comes a point when the software or hardware vendor moves the product out of support. At that point any system failure can have critical consequences. Routine maintenance should be regarded as a critical part of “business as usual” and allowance made for its impact. • Risks Risks to the data warehouse can be identified by those in control. These risks need to be addressed before they affect the system. Over recent years typical risks have included changes in regulatory policies, Year 2000 issues, currency changes (e.g. 2 3 revaluations, moving to the Euro or removing trailing zeros ). There are also risks from corporate mergers and acquisitions and from the bankruptcy of other companies that provide data to supplement the data warehouse (e.g. with market share information). These risks will create significant work with tight deadlines.2 Currently 13 countries are in the Euro zone(http://en.wikipedia.org/wiki/Eurozone#Official_members)3 49 countries including Turkey, Greece, Israel, Brazil and Argentina(http://www.allaboutturkey.com/ytl.htm) © 2007 Data Management & Warehousing Page 6
    • White Paper - Data Warehouse Governance • Issues In an ideal world all of the above would ensure that the data warehouse would never have any issues. The reality is a continuous flow of low-level problems and occasionally critical issues that need to be handled. In some ways this can be a measure of the success of the solution that people care enough to report issues and 4 need support. There are cost and resource implications in handling these issues.What is affected within the data warehouse?5Once a data warehouse development has begun these areas can be affected: • Requirements Catalogue All data warehouse projects should have a comprehensive catalogue of requirements. These will be developed at the start of the system and maintained when new or changed requirements emerge. Requirements are not the same as the functionality of the system. This is because not all requirements may be met, however they provide a catalogue of work that the business needs to be delivered. This also provides a check that new demands have not already been met by another means. • Technical Infrastructure The technical infrastructure is the hardware, software and network used to build the system. It can be affected by many different factors including upgrades for better performance, platform re-hosting or a re-architecting exercise (e.g. moving from desktop to web-based clients) • Configuration Management Configuration management covers all aspect of managing the developed code and any system configuration files that exist outside the code. Suitable tools must be used to ensure that the people responsible for the maintenance of the system can look at all the different versions of code over time. This enables the deployment of new code. It also can help when there is a need to roll back changes, for example reverting to older code after a failed upgrade or to read an old format source file. It is also important that the code can be related to data models for the data warehouse and to the code versions and data models of the external systems that either supply or use data from the data warehouse.4 One Data Management & Warehousing client organisation with 2000 users and a 20Tbsystem raised 10,000 issues in one year post implementation – or about 40 per day5 This section refers to various documents such as requirements etc. Depending on who hasdeveloped the system they will have their own names and templates for these documents.For consistency this document refers to the templates and tools used in the DataManagement & Warehousing (http://www.datamgmt.com) Documentation Roadmap whichcan be found at: http://www.datamgmt.com/index.php?module=article&view=77 © 2007 Data Management & Warehousing Page 7
    • White Paper - Data Warehouse Governance• Test Management Test management is about asking questions to ensure that the impact of any change is well understood and that preventable errors are avoided. Questions like: o After a change is made, how is it tested? o Is there sufficient data to be considered a representative sample? o Is the whole system regression tested, or are changes isolated and therefore separately testable? o How can changes or testing be modified so that tests can be isolated? o How long does it take to test a change and will this impact emergency fixes to the system? o Does the change affect the performance or the data quality of the solution? o Can the figures be calculated by an alternative independent method to ensure that the numbers produced are correct? o Does the project have sufficient resources (e.g. size of machine, people, time, etc.) to perform the tests?• Data Stewardship Data stewardship is about understanding who is responsible for the data within the system. Again the role is about asking questions: o Who decides which data quality rules should be applied? o Where is data cleansed? o Who is responsible for cleaning the data? o Can systems or processes be changed to improve the quality of the data before it is sent to the data warehouse? o Is data cleansed in the source system and if so is this work also done on historical data? o What is the impact if historical data that is already in the data warehouse is cleansed in the source? o What is the relationship between time and data e.g. is 95% of the data available after one day good enough or must 100% of the data be there before it can be used?• Service Level Agreement The service level agreement defines much about the performance and availability of the system. It will include the times set aside for tasks such as backups and maintenance. It also defines the times that the system will be available to the users. It should also include agreements on how to respond in case of emergencies and how to manage systems failures. The Service Level Agreement also defines how any change that affects the system (either positively or negatively) needs to be communicated to the user community.• Support Model The support model describes the processes and escalation paths for any user support request. This model must be updated to reflect any changes to the system or any changes in the organisation. The support model should be modified to improve the responsiveness of support by understanding the frequently asked questions and improving the response time explicitly on these items. © 2007 Data Management & Warehousing Page 8
    • White Paper - Data Warehouse Governance • Training Plans Users and operators will need to be trained to use the system when it is first deployed. There is need for on-going training as the system changes, when new staff arrive or when parts of the solution are either commissioned or de-commissioned. Users will also need refresher courses to reduce the number and cost of calls to the support desk. • Schedules The system will have an operational schedule that will determine not only when users can be on the system (also covered in part by the service level agreement). The schedule defines which jobs to extract, transform or load data can be run, when they are run and the dependencies between them. • Monitoring The system must be monitored and alerts directed to some function that can prioritise them, act and if necessary escalate appropriately. The process by which this is managed and its efficiency is critical to providing a viable service to the users. Monitoring also ensures that service level agreements are met. Analysing the system events allows technical support to improve the system in the same way that analysing support calls assists the support team improve the support. The analysis must look for ways in which to bring about significant improvement to the system by either changing the system or improving the response to frequently occurring events.How often will change occur?The ten areas affected by change and the five types of change described above result in thetable below along with the likely frequency of any given type of change on a specific area ofthe data warehouse: External Change Maintenance Demand Issues Risks Requirements Catalogue 3 1 1 2 1 Technical Infrastructure 2 1 2 1 1 Configuration Management 1 3 3 1 3 Test Management 2 3 3 1 3 Data Stewardship 2 3 1 1 3 Service Level Agreement 2 2 2 2 1 Support Model 1 1 2 1 3 Training Plans 1 1 1 1 3 Schedules 2 3 2 1 2 Monitoring 1 3 3 1 1 1: Infrequently 2: Occasionally 3: FrequentlyThis table can be used throughout the life cycle of the programme to prioritise governancedevelopment to ensure that the most frequent changes are under the tightest controls. © 2007 Data Management & Warehousing Page 9
    • White Paper - Data Warehouse GovernanceWhat organisational structures are needed?This paper has identified a large amount of potential change from different sources ondifferent processes on a long-term programme. It is necessary to have sufficientorganisational structure in place no control the changes and the communication of thatchange. The basic model organisation for this can be described as follows:Figure 1 - Data Warehouse Organisational Structure Executive Sponsor(s) An executive sponsor is a senior (executive) member of the organisation with an active interest and an understanding of the business uses of the data warehouse. A system of this magnitude and with such cross-functional reach needs to be sponsored at the very highest level. It is likely that the executive sponsor for the solution as a whole will be committed to the idea of developing better strategic information. A number of executive sponsors for individual functional developments will assist this person. The sponsors of individual functional developments are responsible for the delivery of the benefits promised for any given development. The executive sponsor(s) also need to ensure that the steering committee is directing the development in line with the vision, strategic direction and priorities for the organisation. © 2007 Data Management & Warehousing Page 10
    • White Paper - Data Warehouse GovernanceSteering CommitteeThe steering committee is the hub of the on-going data warehouse solution from amanagement perspective. It is here that the development is aligned with the businessobjectives. Monitoring ensures that the programme is delivering the right projects at theright time and at fair value.By setting the principles and policies the steering committee can control the directionthat the development goes in. This maintains an enterprise wide business perspectivefor the data warehouse.The steering committee is also the centre of communication. It takes input from theuser forums and the certification committee as to what is needed. In return thecommittee manages the expectations of both the business and IT departments as towhat is possible. The steering committee is most effective when it is empowered tomake quick, effective, cross-functional decisions and communicate to all thestakeholders.Programme ManagementProgramme management is the co-ordinated management of a portfolio of projects toachieve a set of business objectives. It delivers the co-ordinated support, planning,prioritisation and monitoring of projects to meet changing business needs.To achieve the business objectives the programme manager defines a series ofprojects with quantifiable benefits that together will meet the long-term objectives of theorganisation. These projects may run simultaneously or at least overlap with each otherand they may share resources. Such projects might have some resources devoted tothe project for a period of time. In addition projects will require a range of specialistsavailable from the programme team whose services are used for shorter periods oftime. Projects within the programme can also be linked. Delays within one project willthen cause knock on effects in other projects due to logical links between tasks in bothof them. Resource and timing conflict resolution is also an integral part of the functionof programme management.The programme is usually reflected in the management structure with a programmemanager to whom the project managers will report. The programme manager will beconcerned with recruiting and maintaining their project management teams, allocationof key, shared resources and on the integration of the deliverables from each projectinto the overall programme.In this environment every project plays its part towards the organisation’s ultimate aimsand objectives. Often, as projects are completed, this translates back into a revised setof corporate objectives.© 2007 Data Management & Warehousing Page 11
    • White Paper - Data Warehouse Governance Implementation Teams The implementation teams are those that will actually do the work. They will consist of a team of people (either for some or the entire project) that will fulfil the following roles: • Project Manager • Technical Architect • Systems Administrator • Network Administrator • Database Administrator • Metadata Administrator • Data Modeller • ETL Developer • Front End Tool / Report Developer • Product Specialist • Test Manager 6 Many organisations may have outsourced some or all of this functionality . The resources may have the skills and the time to fulfil more than one role, or for some projects more than one resource may be required to perform a given role. The management and resource levels must be determined by the project scope. Exploitation Teams Whilst the implementation teams focus on building the system the exploitation team are trying to ensure that the business is extracting the most value from the solution. Exploitation teams work on the current version of the system to help the business use the current system and develop new requirements to exploit the system further. Typical roles for the teams will include: • Exploitation Team Leader • Business Analyst • Business Requirements Specialist • Technical Author / Documentation Specialist • Trainer • End User Support Specialist • Communications Specialist • Helpdesk Support Specialist As with the implementation teams the resource level is determined by the project scope. User Forums The programme should also consider setting up a number of user forums that involve end users, subject matter specialists and staff from the exploitation teams. These forums are useful to allow various teams to express their issues and aspirations for the system and expand on the art of the possible. These forums should also appoint representatives on the steering committee to represent their perspective on the system.6 Data Warehouses benefit from people with detailed knowledge of the subject andoutsourced development to coding shops with no experience is often more costly than theinitial perceived savings. When choosing resources Data Management & Warehousingrecommend that organisations look at the skills of named individuals on the project © 2007 Data Management & Warehousing Page 12
    • White Paper - Data Warehouse Governance Certification Committee A number of groups within the organisation will also assess the data warehouse to ensure that it is fit for purpose. These groups can either be consulted individually or brought together as a committee to advise the programme. They include: • Audit Most organisations will have some internal audit function. The audit function will need to ensure that the system meets the required standards and has sufficient checks and balances especially if the system is used for any form or statutory or fiscal reporting. The internal audit function should also examine how the system is managed. Where there is no (required) internal audit function an individual or team representing those with a stake in the quality of the information should perform the role of auditor. • Regulatory Compliance Depending on the organisation and industry there may be a need to comply with requirements from government, local administration or industry regulators over the data that is held and who has access to that data. Current examples 7 include data protection law , financial management laws such as Sarbanes- 8 9 10 Oxley Act and Mifid and telecoms regulators such as Ofcom . • IT Strategy & Architecture Many large organisations have a team responsible for the strategy and architecture of all IT systems. This team ensures inter-operability, re-use and cost reduction of IT systems. The team will need to review and assess any technical infrastructure and changes to the solution that are proposed. • Security Security of information is a growing concern for many organisations. Either a security team, if one exists in the organisation, or a nominated individual should review the system periodically to ensure its security.7 DPA Law website: http://www.dpalaw.info/8 Sarbanes Oxley Act: http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act9 Mifid: http://www.fsa.gov.uk/Pages/About/What/International/EU/fsap/mifid/index.shtml10 Ofcom website: http://www.ofcom.org.uk/ © 2007 Data Management & Warehousing Page 13
    • White Paper - Data Warehouse GovernanceProject Development MethodologiesThere are four main methodologies for the development itself: Waterfall The waterfall model has the following phases that are followed in order: • Requirements specification • Design • Build • Integration • Testing • Deployment • Maintenance To follow the waterfall model, one proceeds from one phase to the next in a purely sequential manner, starting each phase once the previous one has been completed. Phases of development in the waterfall model are thus discrete and there is no jumping back and forth or overlap between them, however there are practical variations on this model. Iterative11 The basic idea behind iterative development is to develop a software system incrementally. This allows the developer to take advantage of what was being learned during the development of earlier, incremental, deliverable versions of the system. Learning comes from both the development and use of the system where possible. Key steps in the process are to start with a simple implementation of a subset of the software requirements. The system is then iteratively enhanced in a sequence of evolving versions until fully implemented. At each iteration design modifications are made and new functional capabilities are added. The process itself consists of the initialization step, the iteration step and the project control list. The initialization step creates a base version of the system. The goal for this initial implementation is to create a product to which the user can react. It should offer a sample of the key aspects of the problem and provide a solution that is simple enough to understand and implement easily. To guide the iteration process, a project control list is created that contains a record of all tasks that need to be performed. It includes such items as new features to be implemented and areas of redesign of the existing solution. The project control list is constantly being revised because of the analysis phase. The iteration step involves the redesign and implementation of a task from the project control list and the analysis of the current version of the system. The goal for the design and implementation of any iteration is to be simple, straightforward and modular, supporting redesign at that stage or as a task added to the project control list.11 Description is an edited form of the text from Wikipedia:http://en.wikipedia.org/wiki/Iterative_development © 2007 Data Management & Warehousing Page 14
    • White Paper - Data Warehouse Governance The code can represent the major source of documentation of the system. The analysis of an iteration is based upon user feedback and the programme analysis facilities available. It involves analysis of the structure, modularity, usability, reliability, efficiency and achievement of goals. The project control list is modified in light of the analysis results. Agile12 Agile methods are a family of development processes that build on iterative development, not a single approach to software development. Agile evolved in the mid 1990s as part of a reaction against "heavyweight" methods such as the waterfall model of development that often became heavily regulated, regimented and micro-managed. The processes originating from this use of the waterfall model were seen as bureaucratic, slow, demeaning and inconsistent with the ways that software engineers 13 actually perform effective work. In 2001 seventeen prominent figures in the field of agile development (then called "light-weight methodologies") came together to discuss ways of creating software in a lighter, faster, more people-centric way. They created the Agile Manifesto and 14 accompanying agile principles: • The highest priority is to satisfy the customer through early and continuous delivery of valuable software. • Welcome changing requirements, even late in development. Agile processes harness change for the customers competitive advantage. • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. • Business people and developers must work together daily throughout the project. • Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done. • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. • Working software is the primary measure of progress. • Agile processes promote sustainable development. The sponsors, developers and users should be able to maintain a constant pace indefinitely. • Continuous attention to technical excellence and good design enhances agility. • Simplicity--the art of maximizing the amount of work not done--is essential. • The best architectures, requirements and designs emerge from self-organising teams. • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly12 Taken from The Agile Manifesto (http://agilemanifesto.org/)13 Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development14 Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development © 2007 Data Management & Warehousing Page 15
    • White Paper - Data Warehouse Governance Cowboy Coding15 Cowboy coding is a form of software development method without an actual defined method – team members do whatever they feel is right. Typical cowboy coding will involve no initial definition of the purpose or scope of the project, no formal description 16 of the project and will often involve only one developer . The developer will often be working from his own idea of what the software should do. It is often characterised by a lack of any documentation for either the requirements of the project or the design of the software overall.Within a data warehouse programme it is strongly advised never to allow the cowboymethodology to appear, however any of the other three can be used. There are two decidingfactors as to the approach that will be used. The first is the approach that is already beingused within the organisation and this may dominate over all other factors. The second is thepoint in the lifecycle of the warehouse. In the early part of the programme projects using agilemethodologies offer fast development and quick delivery for new applications built by expertteams that engage the users. Towards the end of the lifecycle a more waterfall like approachto maintenance and de-commissioning operations run by systems management teams islikely to be more successful for individual projects. This highlights the need for thegovernance itself to be adaptive to the changing environment.All formal methodologies (lightweight and heavyweight) can still lead to failure as themethodology team members attempt to operate within the social/political environments of theorganisation. The probability of such a failure is related to the degree of processes inhibitingthe user(s) from deviating from the organisation standard on one hand and the cost in termsof lost efficiency and lost creativity of implementing such processes on the other. Success istherefore reliant on the management choosing appropriate processes that find the correctbalance between these two competing aspects.15 Cowboy coding definition: Wikipedia: http://en.wikipedia.org/wiki/Cowboy_coding16 Cowboy coding can occur even within projects that are using more appropriatedevelopment methods. Warning signals that cowboy coding may be occurring include: • Secrecy about what a developer is working on. • The inability to describe the functionality of current development. • The tendency to do lots of quick fixes. • Working hard and not working clever, working without prioritisation. © 2007 Data Management & Warehousing Page 16
    • White Paper - Data Warehouse GovernanceBest practicesThe following items are some of the key best practices that should be implemented: Momentum Get the project going fast and then keep it moving by keeping the scope and deadlines for delivery very tight. This avoids analysis paralysis and ensures that there is space to take corrective actions if necessary. Monitor and Measure17 Large long-term projects are difficult but if the management team cannot see what is happening then they do not know when things are going wrong. Ask questions about whether teams deliver on time, to budget and scope. Track the number of issues raised in development, test and production, monitor the service level agreements, design metrics for the quality of the code and the user satisfaction, etc. Much of this is built in to approaches such as agile. Prioritise Everyone will want their component now, but it cannot all be done at once. Have a clear, transparent process for deciding the priories and then stick to them. One useful technique is to say that (provided the project uses small scopes) once started a phase cannot be stopped but that before the next phase is started all priorities can be re- assessed. This forces low value priorities to the back but rapidly brings higher priorities to the fore. The steering committee should be responsible for these priorities and the project methodology should support the prioritisation process. Light touch This document has covered many aspects of governance however the programme should implement the minimum necessary because otherwise the project will become about governance and not delivery. Review the processes regularly to add process where required and do not be afraid to remove un-necessary process. Communication The data warehouse is a long-term project for which the perception will shift over time from all parts of the business. Clear communications will help people understand what is available for them, what the priorities are going forward and how to access the information. Publish these widely to encourage involvement and interaction with the system.17 A useful book on this subject is “Controlling Software Projects: Management, Measurementand Estimates” by Tom DeMarco © 2007 Data Management & Warehousing Page 17
    • White Paper - Data Warehouse Governance Standards Develop and enforce clear standards for all aspects of the warehouse such as naming conventions for code and documents, data ownership, data cleaning, access to data, 18 service levels, performance etc. Track processes Ensure that there are clear formal processes for handling change requests, risks and issues. These are the aspects of the project that cause divergence from the baseline and therefore it is critical to know what is happening, when and why. It allows scarce resource to be focused on the correct actions. Use key design decision templates to ensure that design topics are not constantly revisited. Ensure that all change related processes have powerful change management routings to back them up. Understand cost and value Define methods for determining the cost and value of any action, some apparently low cost (often called quick-win or tactical) solutions deliver little value and their true cost is disproportionately higher as they have to be backed out after a short life. Do not forget to include end of life cost in the calculations and make budget holders responsible for demonstrating the benefit that has been derived because of the effort. Continuous Learning and Improvement Carry out stage point audits and post-mortems on all aspects of the system, use the lessons learnt from these to improve the processes to ensure that the next development will not make the same mistakes. Pilot & Prototype Pilot and prototype high risk or complex aspects of the development using methodologies like ‘Agile’ but ensure that the pilots and prototypes do not go into production until the normal quality thresholds have been met. Authority to act Ensure that the steering committee has the authority to act cross functionally using sufficient information to make good decisions. A steering committee that wants a change to a source system that has a critical effect on the data quality is ineffective if they are told that it will be put in a queue and should be delivered in two years time. In the mean time is it acceptable that the data warehouse will just have to make do with poor data? Conversely a lack of sufficient information might be making the same demand of a system that is about to be de-commissioned or for data for which the business user cannot articulate a use.18 See the Data Management & Warehousing Documentation Roadmap athttp://www.datamgmt.com © 2007 Data Management & Warehousing Page 18
    • White Paper - Data Warehouse Governance Plan for the long term Ensure that the business and users understand that whilst aspects of the data warehouse will be available quickly the system is a long-term commitment. It will need funding and maintenance across its entire life span. During that lifetime the system will also grow in size and complexity, increasing the financial burden.Governance and SuccessGovernance is the control aspect of management and should promote a situation wheresenior management are in control of the situation much like the maestro conductor of a fineorchestra. They need to understand what is going on and where to go for the next action andhow to respond to a changing situation.Sometimes the reality is that managers are working in a suppressed panic, not believing whatpeople are telling them or what they themselves are communicating to their management.When managers know that people have to work outside the process to get things done andnot knowing what is actually going on then governance has failed.Governance is therefore not the documentation, the processes or the formality, but is aboutdeveloping a culture where understanding, discipline and skill are regarded as virtues inteams that have leaders with strong technical skills, initiative, communications skills andpersonal authority.Organisations that ‘buy the book’, be it from a bookshop or as a formal methodology from avendor and follow it rigidly are guaranteed to achieve the lowest common denominatorsolution, one that checks all the boxes but underwhelms the management and disappoints theusers. Those organisations that use governance and methodologies as enablers and allowsystems to fulfil their potential by meeting the constantly changing requirements of the userssucceed.Creating effective governance for an organisation requires imagination:“What we need is imagination, but imagination in a terrible strait-jacket. We have to find a new view of the world that has to agree with everything that is known, but disagree in its predictions somewhere .... And in that disagreement it must agree with nature. If you can find any other view of the world which agrees over the entire range where things have already been observed, but disagrees somewhere else, you have made a great discovery. ...A new 19 idea is extremely difficult to think of. It takes a fantastic imagination.”19 Richard Feynman, The Character of Physical Law, 1965, Chapter 7, "Seeking New Laws.” © 2007 Data Management & Warehousing Page 19
    • White Paper - Data Warehouse GovernanceSummaryAn organisation that is embarking on a data warehousing project is undertaking a long-termdevelopment and maintenance programme of a computer system. This system will cost asignificant amount of money and should be well controlled. Governance defines the modelthat the organisation will use to ensure optimal use and re-use of the data warehouse. It willalso enforce corporate policies (e.g. business design, technical design and applicationsecurity) that ultimately derive value for money.This paper has identified five sources of change to the system: • Demand • External Change • Maintenance • Risks • IssuesIt has also looked at the aspects of the system that they will affect: • Requirements Catalogue • Technical Infrastructure • Configuration Management • Test Management • Data Stewardship • Service Level Agreement • Support Model • Training Plans • Schedules • MonitoringUsing these it has been able to highlight standards and best practices that should beemployed by the organisation to deliver effective governance. The paper has also highlightedthe fact that because governance is about fitting into the existing organisation. It alsohighlights the fact that the system grows significantly over time. Consequently there is nosingle right solution for governance but instead there are ways of working and adapting overtime that will ensure success. © 2007 Data Management & Warehousing Page 20
    • White Paper - Data Warehouse GovernanceAppendices Appendix 1 - Programme or Project? This document has several times differentiated between programmes and projects. The 20 table below outlines the differing characteristics of the two: Programmes: Projects: Address the entire business change Deliver a specific change component Focus on strategic goals Focus on tactical delivery May have imprecise definition Have a precise objective May have uncertain timing Are defined with a specific timeline and budget Evolve over a period of time to derive Try to avoid change to the defined optimum benefit for the organisation scope in order to ensure delivery Require much senior management Require management communication attention, often including strategic and primarily at an operational level political debate across organisational concerning operational details boundaries Produce an overall improvement in the Produce specific pre-defined business that may be multi-faceted and deliverables not fully defined at the outset of the programme Require a manager who is high- Require a manager who pays attention powered, high-level, visionary, strategic, to detail, has good team leadership, political, sales-oriented and works with plans in detail, follows a disciplined people at the top and across the approach and delivers the goods. organisation20 The ePMBook by Simon Wallace (http://www.epmbook.com/) © 2007 Data Management & Warehousing Page 21
    • White Paper - Data Warehouse GovernanceAppendix 2 – The "Declaration of Interdependence"Throughout this document reference has been made to the agile developmentmethodology. Because of the development of agile, a set of principles characterised bystatements “We accomplish X by doing Y.” was developed for project managers. Theseprinciples are of value outside the agile methodology itself and should be used byproject managers in any data warehouse programme.The “Declaration of Interdependence” for modern (agile/adaptive) (product/project) 21management"We ... • increase return on investment by making continuous flow of value our focus. • deliver reliable results by engaging customers in frequent interactions and shared ownership. • expect uncertainty and manage for it through iterations, anticipation and adaptation. • unleash creativity and innovation by recognizing that individuals are the ultimate source of value and creating an environment where they can make a difference. • boost performance through group accountability for results and shared responsibility for team effectiveness. • improve effectiveness and reliability through situationally specific strategies, processes and practices."21 2005 David Anderson, Sanjiv Augustine, Christopher Avery, Alistair Cockburn, MikeCohn, Doug DeCarlo, Donna Fitzgerald, Jim Highsmith, Ole Jepsen, Lowell Lindstrom,Todd Little, Kent McDonald, Pollyanna Pixton, Preston Smith and Robert Wysocki:http://alistair.cockburn.us/index.php/The_declaration_of_interdependence_for_modern_management© 2007 Data Management & Warehousing Page 22
    • White Paper - Data Warehouse GovernanceAppendix 3 - Team Values and PrinciplesThe author of this white paper used to work at Sequent Computer Systems. Thecompany had a set of values and principles that the author feels have always proveduseful to teams building data warehouses by creating a culture in which governancecan work. Our Values: • Easy to Do Business With o The Customer ALWAYS Comes First o Do All We Can to Ensure Customer Success - Does Not Always Mean Saying Yes o Three Keys -- Listen, Consider and Act o Go the "Extra Mile" for the Customer • Profitability o Required for Our Success o Allows for Growth & Investments for Future o Everyone is Responsible for Profitability o Follow the "Spend Smart and Invest Smart" Concept • Teamwork o Strong & Balanced Teams o Put Group Goals & Interests Ahead of Personal Reward o Open & Honest Communication o Youre Accountable for Your Commitments o Nobody Wins Unless the Team Wins • Quality o Exceed the Customer Expectations o Strive for Continuous Improvement in All You Do o Know Who Your Customers Are o Establish Clear Mutually Acceptable Expectations o Managed Expectations, Consistently Achieved • People o Assimilating & Integrating New Members is Critical o Open, Honest & Timely Communication o Treat Each Other With Respect o Take Time to Say "Thank You" for a Good Job o Strive for Win-Win Relationships Our Principles: • Act With Honesty & Uncompromising Integrity • Take Responsibility for Our Customers Success • Strive to be the Best • Have a "Can Do" Attitude • Respect Each Other • Exhibit Team Pride • Take Calculated Risks • Be Active Community Citizens • Urgently Do the Right Thing • Make Consultative Decisions© 2007 Data Management & Warehousing Page 23
    • White Paper - Data Warehouse GovernanceReferencesThe section below represents some useful resources for those considering building a datawarehouse solution. Web resourcesOrganisation WebsiteData Management & Warehousing http://www.datamgmt.comDM Review – http://www.dmreview.com/Data Warehouse Project ManagementThe Data Warehouse Institute – http://www.tdwi.org/Data Warehouse GovernanceData warehouse governance – http://www.amazon.com/Best practices at Blue Cross & Blue Shield of North CarolinaData Warehouse Project Management http://www.amazon.com/The Seven Pillars of Data Warehouse Governance http://www.itweb.co.za/The ePM Book http://www.epmbook.com/Software Testing Resources http://www.testingfaqs.org/From Here to Agility: The Physics of Speed http://www.stickyminds.com/Agile Manifesto http://agilemanifesto.org/Agile Alliance http://www.agilealliance.org/e-Programme http://www.e-programme.com/Data Protection Law http://www.dpalaw.info/Alistair Cockburn http://alistair.cockburn.us/Wikipedia http://en.wikipedia.comAcknowledgementsThe author and Data Management & Warehousing would like to acknowledge and thank allthe clients and colleagues who have taken the time to review and comment on this documentprior to publication. Special thanks to Ron Ballard for the most detailed of reviews and proofreading.Copyright© 2007 Data Management & Warehousing. All rights reserved. Reproduction not permittedwithout written authorisation. References to other companies and their products usetrademarks owned by the respective companies and are for reference purposes only. © 2007 Data Management & Warehousing Page 24