Establishing a Robust Data Readiness Methodology          Prepared by:           James Chi
Our Recommended Solution Summary                                                                   The data readiness stra...
Data Input Sources                                                         These data are subsequently provided to the    ...
Data Staging                                           Comprehensive Data Readiness Process All master and transactional d...
application tables and/or spreadsheet rows and           source data from its original record format to a columns are impo...
SAP system in order to load approved staged             Project Preparation – This phase is to provide target data without...
Continual Improvement Iterative Process                                                       Business                    ...
Final Preparation – Development of the central        About the Author Data Staging & Transformation tool is completed and...
Upcoming SlideShare
Loading in …5

Establishing A Robust Data Migration Methodology - White Paper


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Establishing A Robust Data Migration Methodology - White Paper

  1. 1. Establishing a Robust Data Readiness Methodology Prepared by: James Chi
  2. 2. Our Recommended Solution Summary The data readiness strategy and methodology The planning, execution, verification, and described below is the result of an evolutionary documentation of the migration of application data process developed over many SAP from legacy or source systems to SAP are critical implementations with multiple clients in various components to any successful SAP project industry verticals. This methodology is intended to implementation. SAP requires and expects master not only deliver repeatable, predictable, and and transactional data of high quality for the demonstrate results; but also bring visibility to intended process integration benefits to be data quality issues early enough in the project to realized. mitigate them. Data Readiness is, however, one of the most overlooked aspects of an implementation project. Data Readiness Components This is partly because so much emphasis is placed Let us first introduce the distinct components that on re-engineering the business processes that the make up a data migration landscape. As quality and accuracy of data often takes a lesser illustrated in Figure 1, our recommended priority. However, based on our experience, we methodology follows the traditional Extract, would suggest that many SAP implementation Transform, and Load (ETL) data migration projects simply lack the tools and methodologies component model. to systematically identify and perform data readiness and conversion activities and resolve data quality issues. Data Conversion Component Overview Data Export Data Input Source Data Staging Destination Source Applications LSMW Manual Data Collection BDC BDC Direct / iMac via data construction application CATT Manual Data Collection - via Excel and Flat File Custom ABAP Central Data Staging And Transformation Tool Manual Data Collection via SAP Manual Input SAP Systems Extract Transform Load Figure 1Confidential GROM Associates, Inc. -2
  3. 3. Data Input Sources These data are subsequently provided to the central Data Staging & Transformation tool Data for the project implementation come from sources as identified in the functional Manual Data Collection in Excel and Flat File – In specifications. The data for loading into SAP either some cases the need to collect data manually that already exists in an electronic format or are does not exist in the source system(s) is served by manually captured in an approved electronic MS-Excel Spreadsheets or Flat Text File. Based on format. Import programs need to be kept as the complexity of the data that is needed, the simple as possible for faster implementation and project team develops and distributes an Excel easier traceability. Import data can come from the spreadsheet application to help facilitate the following sources: manual data collection process. The data is subsequently uploaded to the central Data Staging Source Application Data – Data from source & Transformation tool. systems are either exported into a comma delimited text file or copied tables when ODBC Manual Data Collection in SAP – In certain database connections are available. Data are functional areas, the project can manually collect extracted out of source applications following the data for SAP where data do not exist in source principle of “all data and records” without data systems directly in the SAP system. It is filtering, filtering, translation, or formatting. sometimes advantageous to build SAP data directly in the SAP environment and take advantage of Manual Data Collection – Data may be manually existing pre-defined data value tables and collected in situations where source data does not validation logic. The data is subsequently exist. Based on the complexity and referential extracted from SAP and provided to the central dependency of the collected data, a data Data Staging & Transformation tool. construction application can be developed to help facilitate the manual data collection and validation process. Data Readiness Process Overview Extract Transform Load 1 1 7 2 4 SOURCE Staged s Proces Target TARGET SYSTEMS Process Source Data SYSTEMS Data DATA STAGING Uploaded Up Up APPLICATION Target Data rt rt da po dat po 5 te Re Re e 8 Target Data Source Data Kickouts 3 Kickouts Configuration Team Team Data Owner Referential & Target Data Supplemential Kickouts Data Data Owner 6 Figure 2Confidential GROM Associates, Inc. -3
  4. 4. Data Staging Comprehensive Data Readiness Process All master and transactional data loaded into the Let us now describe the steps involved in a robust SAP system should be staged in a central Data and comprehensive data readiness process. The Staging & Transformation tool. This repository overall process is illustrated in Figure 2. receives source data and outputs transformed In order to ensure ongoing execution, target data. It contains source data in its troubleshooting, and problem resolution originally supplied form, all the rules to convert, throughout the data conversion test cycles translate, supplement and format this data into the described in the next section “Data Conversion destination format, and intermediate tables Approach and Methodology”, the Systematic Data required for data readiness processing. The output Readiness Process is followed for each data test from the central Data Staging & Transformation run. Following is a high-level overview of the tool is used as the source of data loads into SAP. process. Commercial ETL tools are designed for the purpose of extracting, transforming, and loading data. These tools should be leveraged on projects where available. On projects where a commercial ETL Step 1: Extraction of Source Data tool is not available, native database tools such as The conversion starts with the extraction of source Microsoft’s DTS or Oracle’s Warehouse Builder can data. This extraction, depending upon its source be used as well. may be a direct ODBC connection, a spreadsheet or flat file created programmatically, or a manually Once staged in their original or approved collection loaded spreadsheet. Original spreadsheets and format, all data is filtered, translated, and flat files must be secured in a centralized location formatted in a traceable and reportable fashion via for audit and validation purposes. In all cases, the execution of individual data rules in the central extract of source data must be accompanied by a Data Staging & Transformation Tool. Exceptions to report that details the contents. A Source Data this rule should only be permitted for manually Reconciliation Report should be produced for each entered data objects. extract and must indicate the total number of records contained in the source. Other metrics Data Export Destination Programs should be supplied for key data fields such as sums, totals, or hash totals of data columns Data is exported from the central Data Staging & contained in the source. This information will be Transformation tool into SAP via standard SAP very important in demonstrating that the source data conversion methods and tools. Data data has been completely and accurately imported programs must be kept as simple as possible to into the central Data Staging & Transformation ensure quick development and better traceability tool. for troubleshooting and reconciliation purposes. These conversion methods and tools are: LSMW – Legacy System Migration Step 2–3: Upload, Process, and Workbench Verification of Extracted Data & Data BDC Programs – Binary Direct Connection Quality Checkpoint One CATT – Computer Aided Test Tool Post Load Custom ABAP The next step in the process begins the upload of Post Load Manual Input data from source applications and manual collection repositories in their native format into the central Data Staging & Transformation tool. It is critical for all data to be imported into the staging tool in an “as-is” format. All sourceConfidential GROM Associates, Inc. -4
  5. 5. application tables and/or spreadsheet rows and source data from its original record format to a columns are imported into the staging tool without format that can be read by the SAP data upload any filtering and manipulation. This ensures that programs for loading into SAP. These data staging all data record filtering, translation, harmonization, rules, define the main transformation of the and formatting operations are performed in the filtered source data into data that is coded and staging tool in an approved, auditable, traceable, formatted for SAP upload purposes. All data and reportable fashion via execution of business formatting, filtering, and translation rules are rules at individual source level. based on criteria documented in the functional specifications. Data reconciliation activities are Once the data has been successfully extracted into performed to verify that all required business rules the central Data Staging & Transformation tool, defined in the functional specifications have been the source data is modified according to data completely and accurately applied. filtering rules. Data filtering refers to reducing the dataset based upon rules documented in the Step 5: Data Quality Checkpoint Two functional specifications and business relevancy parameters. This filtering is performed in order to Once the data has been successfully filtered, ensure that only active and relevant data are translated, and formatted, the resulting dataset loaded into SAP. Additionally, source data can can be subject to another set of quality and now be subject to a variety of quality and integrity integrity checks aimed at identifying target data checks to identify source data issues that can integrity and completeness issues. These issues either be resolved in the staging tool as a can be resolved in the staging tool as a transformation rule, resolved in SAP, resolved in transformation rule or be resolved back in the source system. Data records that do not pass key the data construction application, or resolved back quality or integrity checks should be flagged as in the source system. Data records which do not such and omitted from subsequent transformation pass key quality or integrity checks should be and loading steps, and directed to Data Owners for flagged as such and omitted from subsequent correction or clarification. loading steps, and directed to data owners and configuration team for correction or clarification. Data reconciliation activities are also performed. All results are gathered and compared to the Data reconciliation activities are also performed Source Data Reconciliation Report. Results and from the target SAP environment perspective. All Kickouts are provided to Data Owners for review, results are gathered and compared to verify that approval and correction. all required business rules defined in the functional specifications have been completely and accurately Step 4: Transformation of Staged Data applied. Results and kickouts reports are provided to Data Owners and Configuration Team for review Once the source data has been filtered, all source and correction. data are combined into a single staged target SAP data for translation, supplementation and Step 6: Data Supplementation formatting rules specifically designed for the target environment per Design Specifications. Data Following review of target data results and translation refers to replacing source system kickouts reports, data owners have the opportunity coding, groupings, and other source system to inject additional data into the transformation application data characteristics to corresponding process of staged data. Additional data refers to SAP coding, groupings, and data characteristics. missing data component that is required according to functional or SAP system specifications and Supplementation refers to supplying additional referential or required data according to Design cross reference data that mapping legacy data into Specifications that are not available from source new SAP data per Design Specifications. data. Data formatting refers to converting the Configuration team has the opportunity to verify, validate and correct data value needed in targetConfidential GROM Associates, Inc. -5
  6. 6. SAP system in order to load approved staged Project Preparation – This phase is to provide target data without errors. initial preparation and planning for the SAP implementation project, the important data Step 7-8: Loading of Target Data into SAP readiness issues addressed during the project & Final Verification preparation phase are: Subsequent to the successful completion of data Finalization of data migration scope and data quality checks, translated and formatted data will readiness strategy be loaded into SAP via any of the mechanism On-boarding of data team described under the “Data Export Destination Installation of ETL toolset Programs” section of this document and verified Initiation of legacy system connection and for accuracy and completeness. This verification extraction will involve a combination of visual inspection and technical checks including record counts, sums, Business Blueprint – Define the business and or hash totals of data columns contained in processes to be supported by the SAP system and the export files and SAP tables. the functional requirements, data conversion and readiness activities begins with the identification of Data Readiness Approach and data objects which require conversion from the Methodology source application to the SAP system. During this phase, all data and records will be extracted and profiled from source systems, business and SAP Now that we have introduced both data readiness readiness requirements will be defined, and landscape components and process, we can finally Mapping Documentation completed in order for position how this all fits in the lifecycle of an SAP data quality report development. The quality and implementation project. integrity of the source data will assessed What follows is a description of the various data repeatedly during this period. readiness activities as they are executed Realization (Build) – Build the system based throughout the Grom’s Best Practice Data upon the requirements described in the functional Readiness Approach. Grom’s Data Readiness specifications, included in this phase are several Approach is an enhanced, refined and data readiness process development and individual complementary to ASAP methodology that SAP data object testing cycles. During the early part of implementation project is typically followed. realization, functional specifications are developed for the data conversion objects identified during requirements gathering. These design Project Definition – The purpose of this phase is specifications serve as the basis for determining to understand and define data quality baseline and which conversion mechanisms are used and a path forward with respect to data readiness for provide additional functional conversion program SAP implementation. Once data quality baseline development and testing details for a given data has been defined and understood, data migration object. The project team develops all required data and readiness scope can be derived and estimated conversion rules and programs. These conversion in alignment with business objectives of SAP rules and programs are tested repeatedly in the implementation. Toolset selection can be Q/A or Unit Test environments as illustrated in accomplished based on scope of the conversion. Figure 3. Finally, the effort and cost of the conversion can be estimated for approval.Confidential GROM Associates, Inc. -6
  7. 7. Continual Improvement Iterative Process Business Blueprint Unit Test Environments Integration Test Test Environments ing E Pull On Demand entsv Sources Data Staging Application Go-Live Cutover Rehearsal Environments Results Resolutions User Reports SAP Production Figure 3 Realization (Test) – The purpose of this phase is Level prior to Go-Live as illustrated below in figure dedicated for testing and refinement of conversion 4. By the end of this realization test phase, the rules and programs of the central Data Staging central Data Staging & Transformation tool will be and Transformation tool. As source data evolves tested with full data conversions in 2 to 3 rounds in the course of normal business operation over of Unit Testing and 2 to 3 rounds of Integration the project timeline, new data issues may surface Testing. and conversion rules may need to be updated or refined through the Continual Improvement Data Quality with Continual Improvement Process Interactive Process. As the target SAP system in Transactionable Data Quality Level High each environments continue to mature into “To- Data Go-Live Be” production system, data readiness will be Qua lity measured and reported against environment to confirm alignment of design and functional Business Blueprint Install/Run/Support specifications. Through this iterative testing and Final Preparation Da Act repeatable process, data quality with respect to ta ivit Re ies readiness will elevate closer toward ad Transactionable Data Quality ine ss Realization Realization (Build) (Test) Low Project Time Line Figure 4Confidential GROM Associates, Inc. -7
  8. 8. Final Preparation – Development of the central About the Author Data Staging & Transformation tool is completed and cutover activities will be rehearsed 2 to 3 James Chi is the Director of the GROM’s Business rounds during this phase. As part of final Consulting Group Enterprise Solutions Practice and production cutover, final source data extractions has overall delivery responsibilities for all GROM- and preparations will be performed and all master led projects. James joined GROM after spending and transactional data will be loaded into the the last seventeen years delivering SAP solutions production environment. Production data in the pharmaceutical, medical device, and reconciliation and validation reports will be consumer products industries. James’ strong prepared to ensure all records are accounted for. functional background in Supply Chain Planning Any additional manual data conversion activities and Manufacturing Execution has blended to create and manual configuration steps in SAP will a well-rounded business expert with more than executed according to conversion plan. Finally, fifteen years of Project Management experience. data owners sign-off the production load and James has a BE in Electrical Engineering from validation reports as required by the SAP Stevens Institute of Technology. implementation project. Install / Run / Support – As the purpose of this phase is the transition from the pre-production environment to live production operation, this phase is used to closely monitor system transactions, and to optimize system performance. From a data conversion perspective, any post go- live issues related to data should be investigated, resolved, and closed.Confidential GROM Associates, Inc. -8