Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Migration: A White Paper by Bloor Research


Published on

This paper is about using information management software from Business Objects, an SAP company, for SAP data migration projects, either for upgrades from one version of SAP to a newer one, or from other environments to SAP. In practice, many of the considerations that apply to SAP data migrations are no different from those that pertain generally to non-SAP environments.

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Data Migration: A White Paper by Bloor Research

  1. 1. SAP Data Migration Using Business Objects Information Management Software A White Paper by Bloor Research Author: Philip Howard Review Date: 17 July 2008
  2. 2. Executive summary This paper is about using information management software from Business Objects, an SAP company, for SAP data migration projects, either for upgrades from one version of SAP to a newer one, or from other environments to SAP. In practice, many of the considerations that apply to SAP data migrations are no different from those that pertain generally to non-SAP environments. However, there are some particular requirements that apply to SAP implementations, which we will discuss. A 2007 survey conducted by Kognitio found that 72% of UK financial services organizations believe data migration projects to be too risky and too costly, and the report found similar levels of trepidation across the retail, manufacturing, and transport sectors. This is true regardless of the reason for the migration – whether the migration is based on a change of application (such as moving from Oracle Applications to SAP), a change of database provider, a consolidation of multiple applications of the same type, or even when upgrading from one version of an application to another. Organisations fear migrations because experience has taught them that migrations often overrun their budgets, are delivered late, do not meet user expectations, and, in some cases, fail completely. Moreover, it is not simply a question of a project running over time or over budget – there are also significant business risks associated with overruns in terms of customer relations and user morale, as well as application-specific risks, such as potential delays in introducing new products. In order to understand why these overruns occur, it is important to understand the various elements that make up a migration from, say, one version of SAP R/3 to another. The first part consists of the two versions of R/3 themselves. Since SAP R/3 software is well known and understood, there is little application development to be done at this level other than customisation. However, there is a significant amount of work to be done at the data level, both in the movement and cleansing of the data that has to be migrated, and in the process involved in the migration. It is necessary to understand the relationships that exist between different data elements so that you can migrate business entities (such as a customer with his orders) rather than treating these separately, which runs the risk that these become disconnected. Of course, if you are developing a new application or re-developing an existing one, then there is considerable risk associated with this work at the application level. However, very many migrations do not involve much more than customisation of the new application and yet still companies continue to experience overruns, delays, and failures. The conclusion has to be that the migration of the data is the issue. Research bears this out. In 2006/2007, Bloor Research conducted a survey with Global 2000 companies and found that more 80% of data migration projects ran over time and/or over budget. What is more, The Standish Group conducted a similar study in 1999 and got almost identical results. In other words, the ability of the industry to properly conduct data migrations has not improved in seven years! However, this does not mean that it is impossible to undertake data migration projects that are successful and are delivered on time and on budget. Rather, successful data migration requires the use of appropriate tools and methodologies, best practices, and expertise, and it is on these factors that we focus in this white paper. While there are some specific requirements for SAP environments that go beyond those that might be necessary for other data migration scenarios, the details and discussion points in this paper otherwise apply to any data migration project. © Bloor Research July 17, 2008 Page 2
  3. 3. In practice, many migration projects are delayed longer due to the fear of project failure. As a result, this only makes them complex and time-consuming. It is our view that with the appropriate processes, tools, and people in place (perhaps through a migration competency centre), data migration should become routine, so that migrations can be planned when they are most useful for the business and without the concern over potential failures that characterise them today. © Bloor Research July 17, 2008 Page 3
  4. 4. Before you begin In our research, we found that around 20% of companies had no separate budget or timescale for data migration as opposed to the application element of their projects – a recipe for disaster if ever there was one. So, the first thing is that there must be a separate budget and timeline for data migration. Moreover, an appropriate person with data management expertise should be on the overall project steering committee from the outset. If data management issues are left to users and/or application developers, it is all too easy for the project committee to make unfounded and unwarranted assumptions about the data elements of the project, which can lead to precisely the difficulties you want to avoid. There remains the question of how you estimate the timescales and resources that will be required to achieve a successful migration. There are three things that you need to know: 1. Where is the data coming from to populate your new application? The answer to this may be simple, in that all the data is coming from an existing application, or it may be more complex in that the data is coming from multiple existing applications. In the latter case, if some of the same data (for example, customer data) currently exists in multiple applications, then you have to decide which version of the customer data you are going to use or how you are going to combine data from those sources. In the worst case scenario, you may have data required for the new application that does not exist in current applications but does exist in spreadsheets, Access databases, and other end-user resources spread around the organisation that are not known about by the IT department. In this case, you will need to find out about these resources by talking to relevant end users. Finally, the new application may be able to exploit data that you do not currently have at all and you will need to understand how to use this new data. If the data is not mandatory then it should not affect initial implementation, but otherwise you will need to know whether the data can be created manually or sourced from a third party. 2. What sort of state is this data in? Once you know where the data is coming from you need to understand it both in terms of the structure of the data and its quality. The former is important because the number and complexity of the mappings that you will construct to take the data from their existing sources to the intended target systems will depend, in part, on the structure of that data. The latter is important both in business terms (you do not want to be working with missing, invalid, duplicated, or incorrect data) and for technical reasons. It is important to understand this last reason. With a different data model in the source and target systems, there may be constraints upon the data in the new system that did not apply in the original application. For example, you might have a customer ID field that requires a numeric entry; if the previous application did not have the same constraint, then any non-numeric customer ID will either not load into the new application or, if you can load it, it will not run with the new application – it may even crash the new application. 3. How will the data be structured in the new environment? This is the inverse of knowing the structure of the existing system and, combined with that information, can be used to derive the mappings and transformations that will be required. Once you have this information, you need someone who can interpret it. That is, you need someone who can estimate how long it will take (and what resources will be required) to create the relevant mappings from the source to the target system, and how long it will take to clean up the data from the source systems so they are in a suitable format for loading into the © Bloor Research July 17, 2008 Page 4
  5. 5. new application. It is worth noting at this point that this is not a pipedream: some organizations specializing in data migration will give you a fixed price quote and fixed delivery schedule, provided that they have the information just outlined. Note that we are not necessarily advocating such an approach; we are simply pointing out that once you know all about the data, and given relevant expertise, then you know everything you need to know to about the likely costs and resources that will be involved in the migration. In practice, points 1) and 3) are relatively easy – particularly when you are migrating between known environments, such as an SAP R/3 upgrade or a migration from a third-party application to SAP R/3, because both sides of the equation are well-known and documented. Where there is an issue is with point 2), because most companies do not know what state their data is in – and this is complicated further if there are multiple source systems such as when consolidating several applications of the same type. The process of understanding your data is known as profiling. Tools As we have intimated in the previous section, managing a successful migration project is not simply about the tools that you use but also about the skills and expertise of the people involved. Our research indicates that the average company in the Global 2000 undertakes 4.5 migration projects each year – in which case it might be worth considering the establishment of a migration competency centre or, at the least, requiring that data migration specialists get recognition and rewards commensurate with the importance of accomplishing successful and timely migrations. The skills possessed by such individuals will mitigate the risk involved in migrations and reduce any worries associated with them. This should mean that businesses can choose to migrate when it best serves the business rather than being constrained by fear of failure. Nevertheless, while expertise is important, so too are tools and the facilities they provide. The appropriate use of tools will not only assist in project estimation and scoping but also in cleansing and transforming the data, documenting the processes involved, and in ensuring that data lineage is provided for compliance reasons. The main tools required are data profiling, data cleansing and matching, and transformation and data movement tools. These information management tools are provided by Business Objects in the form of: o BusinessObjects Data Insight, which provides facilities for profiling and analysing the data in your source systems. To put it simply, BusinessObjects Data Insight allows you to discover the extent of the problem you will have when migrating the data. Historically, and even today in very many companies, profiling has been done manually (that is, by visually inspecting the data and comparing it with what you expect to see). There are a number of problems with this approach: it is easy to miss things, it is time-consuming, it is labour-intensive, and it is boring. In particular, the fact that it is boring means that nobody wants to do it, so it tends to be given to less experienced personnel who are less likely to do a good job. In other words, you get a vicious circle. In particular, companies who use manual profiling never do enough profiling – and when we say never, we mean never! We have never, ever met a company that did an adequate job of manual profiling. Thus, we consider the use of automated data profiling to be mandatory. o BusinessObjects Data Quality Management, which provides data cleansing, matching, de-duplication, and similar capabilities, as well as the ability to enrich the data from outside sources such as Dun & Bradstreet. BusinessObjects Data Quality Management © Bloor Research July 17, 2008 Page 5
  6. 6. includes metrics management and dashboard capabilities so that you can monitor the quality of your data as you proceed through the data migration project. o BusinessObjects Data Integrator, which is the company’s tool for accessing any data source, defining appropriate transformations, and scheduling and running the actual process of moving the data. o BusinessObjects Data Services, which is a single product providing all the functionality of BusinessObjects Data Integrator and BusinessObjects Data Quality Management using a common development, runtime, and management environment. In addition, Business Objects offers an extraction, transformation, and load (ETL) mapping tool, BusinessObjects Composer, and the SAP NetWeaver Master Data Management component. While we do not intend to spend time discussing these individually, since the technologies involved are well-known, it is worth briefly commenting on the application of master data management (MDM) to migration scenarios, in that the data consistency provided by MDM should make future data migrations much easier. More broadly, the implementation of data governance should mean that data sources are understood, and of good quality, prior to the commencement of any migration project, thus significantly reducing the efforts required in migration projects. Business Objects is one of the leading vendors of information management software for data profiling, data quality, and data integration products. Here we highlight the key features that are especially appropriate in data migration environments in general, and in SAP environments in particular: o Connectivity – The ability to connect to a variety of sources and targets is important for any data migration project. This includes databases, file systems, message queues, and so on, supporting structured, semi-structured, and unstructured data. It is also useful to be able to offer adapters that have a deep understanding of application environments, such as SAP (see next bullet), Siebel, Oracle, and so forth. The reason is that connectivity adapters facilitate the discovery of metadata during the data profiling stage and, indeed, are useful during the movement phase also because they enable understanding of the data model underlying the application. o SAP Application Interfaces – It is important to shield the data migration developer from having to know about specific SAP constructs such as ABAP, BAPI, and iDOCS, both because this reduces the skill level required of the developer and also because it reduces the complexity of the project and should, therefore, speed up the delivery of the project. Business Objects provides appropriate application interfaces that have been certified by SAP. o Normalisation – SAP application environments are strictly normalised. That is, they adhere closely to formal relational standards. However, many third-party applications do not have that requirement and it is often the case that databases are de-normalised for a variety of reasons. Thus, it is important when you are migrating to SAP that the software you use has the ability to normalise the data. This is a facility within BusinessObjects Data Integrator. o Data Auditing and Validation – The ability to check if your data successfully loaded into the target environment is necessary for any data migration job. With BusinessObjects Data Integrator, an auditing feature allows you to check the sum and row count of data leaving the source system and entering the target environment. Also, the ability to ensure that unwanted data does not enter the target system is crucial to ensuring the success of a data migration load. BusinessObjects Data Integrator also provides the ability to set up © Bloor Research July 17, 2008 Page 6
  7. 7. rules that filter unwanted data. It also provides a dashboard for you view statistics on the percentage of data which passed and didn’t pass your rules. o Templates – What would be especially useful to support SAP migrations would be a template of the target application, with the relevant data model built-in to the template. This would significantly improve migration timescales. In practice, no vendor currently supplies such a starter for migration, even though SAP has encouraged them to do deliver such a capability. However, SAP and Business Objects are jointly developing such a facility. Managing the migration A migration project may consist of multiple elements. For example, you may be migrating across operating systems, hardware platforms, databases, and applications as well as migrating the data. However, regardless of the complexity of the migration environment (and, actually, it is a good idea to simplify it, migrating as few elements as possible within a single project). The overall process can be likened to the design of a car: whatever the chassis, bodywork, or interior trim, it is the engine that makes a car go; and it is the data that makes applications go. It is a good idea to bear this analogy in mind. For example, it is poor practice to reallocate staff from the data migration team to other groups within a project just because the application part of the migration is running behind schedule – this gives the impression that the application is more important than the data and those that work on the latter are less important. This kind of practice will be counterproductive if you wish to establish a data migration competency centre or its equivalent. More generally, you will need conventional project management tools for this purpose, either from a third-party supplier or using built-in features, such as those provided by BusinessObjects Composer. Note that just as data migration should have its own budget and timescales, so it should be managed as its own subproject. Conclusion Our research shows that many IT professionals delay application migration and upgrade decisions due to fear of failure, even when a delay may not be in the best interest of the business. However, with the appropriate tools, methodology, and expertise, data migrations can proceed smoothly and on time. Of the above three requirements, all are important – but, perhaps most especially, expertise is important. Companies need to build up a core of experienced data migration staff that are recognised and rewarded as such, so that the business determines the pace of upgrades and migrations, rather than worries about IT failure. In this context, it is worth noting that Business Objects offers experienced data migration staff and consultants as well as an established migration methodology, so the company can provide exactly the required expertise either on an on-going basis or for knowledge transfer. With respect to SAP in particular, a couple of technical requirements may mean that the environment differs from some others. Clearly, as an SAP company, Business Objects is well- placed to service these needs. Moreover, we can expect to see even closer links between the data integration facilities provided by Business Objects and the application needs of SAP now that they form a single company. In time, we expect Business Objects to offer data migration © Bloor Research July 17, 2008 Page 7
  8. 8. facilities (such as templates) for SAP environments that are not available elsewhere, making BusinessObjects Data Services software the de facto standard for SAP data migrations. © Bloor Research July 17, 2008 Page 8