Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Solution Blueprint - Customer 360

2,603 views

Published on

Solution Blueprint - Customer 360

  1. 1. We operate as John Hancock in the United States, and Manulife in other parts of the world. Customer 360 Solution Blueprint Final January 30th, 2015
  2. 2. 1 Table of Contents # Topic Page 1 Revision History 02 2 Document Distribution List 03 3 References 04 4 CCF Solution Architecture 05 5 Key Assumptions 06 6 Technical Risk and Mitigation 07 7 Technical/Non-Functional Requirements 08 8 Data Flow 15 # Topic Page 9 Conceptual Architecture 24 10 Extract, Transform and Load (ETL) 28 11 Data Quality 59 12 Master Data Management (MDM) 72 13 Data Architecture 109 14 Data Governance 116 15 Summary & Next Steps 129 16 Appendix 131
  3. 3. 2 Customer Centricity Foundation (CCF) As-Is Solution Architecture
  4. 4. 3 Key Assumptions No. Assumption 1 The Data elements identified for customer 360 are the critical data elements that will be measured for data quality and data governance. 2 Data owners and data stewards have been identified for critical data elements for customer domain within the Canadian Division. 3 Out of the box Data Quality Information Analyzer reporting tool will be used for measuring customer critical data. 4 The Data Governance workflows will be provided as swim lane diagrams in Visio and will not be automated using AWD by the Data Governance work stream. 5 The IBM Standard Data Model for Party, Accounts and Product will be leverage with possibility of extension up to 10-15 custom entities. The estimated entities for this phase are around 40-45. 6 Account/Policy Data will be hosted in MDM. For this phase, Account/policy Data will not be mastered. The Account/Policy Data is maintained with relationship to Customer. 7 MDM will house product data to support customer to product and product to line of business relationships but is not considered master data. 8 Virtual and physical MDM will be leveraged for distinct match, merge, and persistence capabilities in a hybrid model. 9 The out of the box MDM artifacts will be the basis for any customization or configuration (matching algorithms, UI components ,and services). 10 There will be multiple points of human interaction within the MDM tool suite based on a given task (linking, data quality, reference data management, golden record management, or product data configuration). 11 Customer 360 will leverage existing extract files from admin systems, except for MPW where new extract process will be developed
  5. 5. 4 Technical Risk and Mitigation No. Risk Description Risk Category Mitigation Strategy 1 Quality of data is not being high enough to effectively match and merge client data before exposing it to the client via the Customer Portal and CCT. High Execute Data Profiling, Remediation and Monitoring 2 Access to unmasked data for the purposes of effectively profiling data, data quality and identifying fields to match on, and for respective probability and thresholds. High Identity individuals who need access to unmasked data and restrict access to others. 3 Aggregating entire Customer data from across the Canadian division in one place increases risk of Confidential Customer data being lost, stolen and exposed in case of a Security Breach. High Implementing high Security measures and protocols to protect data e.g. using SFTP and HTTPS. 4 Several Technology components will be implemented for the first time at Manulife, raising a risk of integration challenges which can lead to schedule and cost impacts High Conduct a Proof of Technology for Customer 360 architecture components
  6. 6. 5 Technical/Non-Functional Requirements No. Criteria Requirement Criteria Count 1 Availability Criteria for determining the availability of the system for service when required by the end users. 6 2 Maintainability Criteria pertaining to the ease of maintenance of the system with respect to needed replacement of technology and rectifying defects. 1 3 Operability The criteria related to the day to day ease of Operation of the System. 2 4 Performance Criteria related to the speed and response of the system. 4 5 Recoverability Criteria to determine how soon the system would recover to the original state after a failure. 8 6 Scalability Criteria to determine the ability of the system to increase throughput under increased load when additional resources are added. 2 7 Security Criteria to determine Security measures to protect the system from internal and external threats. 4
  7. 7. 6 Technical/Non-Functional Requirements Req No. Work Stream Requirement Owner Status 1 MDM Q. Determine what time the data will be available ? A. The data should be cleansed, matched and synchronized with Salesforce by 7am. Jamie Complete 2 ETL Q. Determine when do the source files arrive by? A. All source files expected to arrive by 5am. Steven Complete 3 Web Service Q. Determine how often will ePresentment pull data from ePresentment stage? A. Large runs monthly and annually. Otherwise, small runs occurring nightly for notices, letters, confirms, etc. Jamie Complete 4 ETL Q. Determine does ePresentment stage expect full data everyday? A. ePresentment would expect full data everyday. Deltas would suffice, but any document delivery preference changes during the day should be reflected in the ePreferences staging database. Jamie Complete 5 ETL Q. Determine when something fails within ETL, how soon should someone be notified of the errors? A. Recommendation : ETL error notification should be sent at least once a day. Steven Open 6 IVR Q. Determine is it ok for the IVR vocal password to be down for the day? A. 12 hours to align with overnight batch schedules. Jamie Open Availability
  8. 8. 7 Technical/Non-Functional Requirements Req No. Work Stream Requirement Owner Status 1 ETL Q. Determine how are deltas being identified / captured within the current processing? A.. For Dataphile, the source and target files are being compared to capture the deltas. For the other source files the process is still to be determined. Vishal/ Steven Open Maintainability Operability Req No. Work Stream Requirement Owner Status 1 ETL Q. What is the existing File validation processes? Jamie Open 2 ETL Q. Determine what happens when the source files per system do not arrive on time? A. Given that 360 should process files as they come in, there should not be holding of any files. Jamie Complete
  9. 9. 8 Technical/Non-Functional Requirements Performance Req No. Work Stream Requirement Owner Status 1 ETL Q. Determine if all the files are processed as the file arrives or is there a queue in process? A. The files should be processed as they arrive, preference for a real-time processing option. However if there are cost or delivery date related issues, then files will be processed when all files are available, or at an arbitrary time. Jamie Open 2 ETL Q. Determine what volume are we expecting from each of the source systems (initial and incremental)? Steven Open 3 Tech Arch Q. Determine what is the daily expected volume from each of the sources/within ePresentment stage? A. Total of new customers + preference changes. Expect less than 5000 per day, ongoing. (Rough estimate) Jamie Complete 4 Tech Arch Q. Determine what are the archiving requirements for the ePresentment stage? A. Archiving requirements are not necessary for ePresentment stage. It is not source of record. Data is primarily transient; being staged for performance reasons. Jamie Complete
  10. 10. 9 Technical/Non-Functional Requirements Req No. Work Stream Requirement Owner Status 1 Tech Arch Q. Determine the fail over time? A. Fail over to alternate site should be immediate; utilizing a cross-data center clustered WAS architecture with our global load balancer. Jamie Complete 2 Tech Arch Q. Determine the data loss time? A. For data where 360 is not source of record, 24 hour data loss is probably acceptable as data can be re-run from the admin systems. For data where 360 is the source of record (preferences, for example) then acceptable data loss is very small. However would still probably be hours’ worth, given that Salesforce would capture the data and presumably we can resend the messages. Jamie Complete 3 Tech Arch Q. Determine what happens when the virtual MDM is lost? How soon can it be recovered? A. Virtual repository would have to be back up within a 24 hour period. Jamie Complete 4 Tech Arch Q. How often will the system be backed up? A. System should be backed up nightly. Tape backup would be the option. Jamie Complete 5 Tech Arch Q. Who will be responsible for the database back up? A. DBAs would be responsible for the database backup. Jamie Complete 6 Tech Arch Q. What data must be saved in case of a disaster? A. Recommendation: Jamie Open 7 Tech Arch Q. How quickly after a major disaster must the system be up and running? A. System should be back up and running within 24 hours after disaster. Jamie Complete 8 Tech Arch Q. What is the acceptable system downtime per 24-hour period? A. Acceptable system downtime would largely be driven by dependent systems. Jamie Open Disaster Recovery and Business Continuity
  11. 11. 10 Technical/Non-Functional Requirements Req No. Work Stream Requirement Owner Status 1 Tech Arch Q. Determine the expected growth rate? A. Expected Growth Rate for the first year is 150% as result of IIS business being added and Standard Life Acquisition. Year over year, after that, it is expected that the growth rate will be between 10% - 50%. Jamie Complete 2 ETL Q. Determine what files are batch / real-time? A. All files are batch, including MPW. However, feeds should be processed as they come in. There will be interdependencies e.g. MLIA gets data from both iFast Seg and PPlus Jamie Complete Scalability
  12. 12. 11 Technical/Non-Functional Requirements Req No. Work Stream Requirement Owner Status 1 MDM Q. How should web services be secured? A. Services being exposed by DataPower or IIB should be secured using 2-way SSL certificates. Services on the WAS or MDM servers should be exposed to DataPower/IIB over HTTPS and utilize a user id/password. Additionally, IP restrictions should be in place such that the services can only be called by IIB/DataPower. Jamie Complete 2 Data Protection Q. How should SIN be protected? A. SIN number should not be visible to users of the MDM/DataQuality platforms (DGO members). Only the last 3 digits should be stored in Salesforce. Jamie Complete 3 Data Protection Q. How should Customer data who are also employees be handled? A. Employee customer data should only be visible to those in roles that have access to staff plan. Additionally, there are probably requirements about how board member data should be handled and High Net Worth data. Jamie Open 4 Tech Arch Q. Determine if FTP/SFTP is being used in the Current State and the Future State Requirement? A. Current State – FTP, Future State – SFTP. Jamie Complete Security
  13. 13. 12 Data Flow
  14. 14. 13 Systematic Data Flow Admin Systems FTP Server DMO Staging ETL DQ Repository IBM MDM Server MDM Stewardship UI IBM WebSphere DataPower WebSphere cast Iron Cloud Synchronization Source Files Dependency and Scheduling ePresentment Contact Center Portal Manual Data Check? Systematic Customer updates No Yes 1 2 3 4 5 6 7 8 9 9 10 1111 OOTB XML 14 Dox Delivery Preferences Staging Area DQ Stewardship UI Existing Process/Tools/Technology Out of Scope of Customer 360 Waiting Decisions or Discussions In Scope for Customer 360 IIB 16 IVR Staging Area SFTP 12 Adhoc Systems Data Management Processes: Data Security, Metadata Management, Data Quality Management, Data Lineage 15 Reference Data Updates 13 13
  15. 15. 14 Systematic Data Flow Description (1 of 3) Step No. Step Summary Details of the Step 1 Data Extraction from Admin Systems For Phase 1, existing data feeds from admin system for DMO will be leveraged for MDM There will be additional Admin System feeds required for MDM namely “MPW” for wealth management. 2 Data Files are moved to a FTP Server The extracted data Flies will land on a common FTP server as depicted in the systematic data flow. 3 Source File Dependency and Scheduling Multiple files will arrive from some admin system which have dependencies upon each other. The CA scheduler will wait for all the files to arrive from an admin system before calling the Informatica ETL jobs to load the staging tables. 4 Standardization and Transformation of the Data Informatica PowerCenter performs a series of ETL transformations on the data within staging tables while moving it to a Pre-DQ tables within the DQ repository. IBM Quality Stage will then perform standardization and enrichment tasks while moving the enriched data to the finalized DQ repository 5 Standardized Data is sent to MDM in the format OOTB XML In this step the data will be prepared and sent to the IBM MDM Advance for ingestion into matching and linking processes. This format will be OOTB XML interface specified by the MDM platform. The Virtual (Initiate) part of MDM received the data. The customer updates are then matched against existing customer records in MDM. 6 Data Stewardship UI or a workflow to review the updates in MDM Few scenarios could occur such as “False Negative” or “ False Postive” where systematic match and merger results are not fully reliable and scenarios such as updates that may have larger impact on cross- business or cross –system level. These scenarios may need to be reviewed and approved before the changes are applied to the system. It can be achieved by triggering a workflow for data stewards or for the roles as defined in Data Governance Operating Model to look at the data and take necessary steps to validate or update data changes.
  16. 16. 15 Step No. Step Summary Details of the Step 7 Workflow is triggered As mentioned in the step 6, a workflow may be triggered to review and approve such scenarios. There is an opportunity to set up Data Governance processes to review and approve very critical scenarios. 8 Record is approved and updated In this step, the updates are reviewed/updated and approved and the updates are persisted in MDM Repository . 9 Updated Data propagated to Salesforce As customer updates happen in MDM, relevant information is sent to Salesforce through websphere Cast Iron Cloud Synchronization. 10 Salesforce Updates are now available for Portal and CCT Portal and CCT will have updated customer records from Sales Force . 11 Portal/CCT Updates to MDM via Salesforce For data integration from Salesforce.com, triggered events may call exposed service calls from Salesforce to Manulife. These services would be exposed through the IBM Datapower appliance and will be serviced by IIB (IBM integration Bus), integrating with either the staging area or Quality Stage, in order to have the data written to the MDM platform on-demand. 12 Salesforce, CCT and Portal updates synchronization with Admin Systems Any Customer updates in Salesforce that are relevant to Admin Systems will be directly sent to Admin Systems through Systematic interface. Those change as described in steps 1 through 8 will flow in to MDM system on a daily basis as defined in batch. There are few exceptions to this rule such as portal username, preferences, portal updated email id and phone number, Portal Status and so on. These exception data elements will be updated in MDM from Portal through Salesforce. Systematic Data Flow Description (2 of 3)
  17. 17. 16 Step No. Step Summary Details of the Step 13 Life Cycle Management of Master Data/Reference Data In Manulife Landscape, Customer updates and Reference Data updates will take place in Admin or in some cases, in consuming systems (systems that received updates from MDM). These updates are to be consistently applied across landscape in order to seamless movement of data. Analysis of the information that will be hosted in MDM for this phase revealed that the frequency of the reference data updates is minimal. Hence, automatic reference data updates across system is ruled out. There will be an overarching process established to monitor and control life cycle management of master data and reference data. The data governance process will be designed based on these scenarios. 14 Updating Preferences in MDM MDM will be the repository for customer preference data. The stored customer preferences will be pushed into a messaging queue using XML, and picked up by Informatica PowerCenter to load into the “Document Delivery Preferences Staging Area” for other systems (ePresentment) to consume. The purpose of this staging area is to avoid intensive resource consumption of MDM server when customer preferences need to be sent to DSTO. 15 Propagating Preferences from supplementary Database to ePresentment The ‘Document Delivery Preferences Staging Area’ will host printing preferences for ePresentment and other systems to consume. 16 IVR Id will be updated from IVR As Voice Print Id Is updated in IVR, it will be synchronized with MDM through IBM Integration BUS Systematic Data Flow Description (3 of 3)
  18. 18. 17 Conceptual Architecture
  19. 19. 18 Customer 360 To-Be Architecture 1 2 3 4 5 6 7
  20. 20. 19 Customer 360 – Conceptual Architecture Description Component Functional Area Description 1 ETL Existing data feeds from the admin system will be leveraged for Customer 360. The extracted data Flies will land on a common FTP server and further loaded into a staging database using Informatica PowerCenter for consumption. 2 Data Quality Once PowerCenter transfers the data from the file(s) in to a database Staging Area. IBM Information Analyzer and Quality Stage will parse, cleanse, enrich, and standardize the data; based on defined business rules. 3 MDM IBM Infosphere MDM will then ingest the data. The records are matched, merged, and ultimately stored in the physical MDM repository where they become available for consumption by downstream systems. 4 Softlayer Consuming applications, proxy, and compound services will use a combination of IBM tools and custom services (Cast Iron, IIB, DataPower, and Java Services) to access MDM repository data. 5 Dox Deliver & Preferences Staging The Dox delivery and preferences staging database will be loaded from MDM using PowerCenter (ETL) to allow for quick and easy access by the customer preferences user interface. 6 Data Governance The Data governance organization and tools will cross the customer 360 architecture and will touch all phases of data management lifecycle. 7 Downstream Applications Salesforce will be the primary consumer of MDM data supporting two customer applications: CCT and Customer Portal.
  21. 21. 20 Key Decision: Separate Customer 360 Stage Accenture will Stage the incoming feed data 1 to 1 without applying any transformation / business rules or filtering rules. The only logic embedded within these load processes would be file validation checks and corresponding Exception Handling / ABC logic. Benefits of STG table(s): Risk Mitigation Scalability Measuring Integrity Development Complexity Current operational systems will not be affected, however, existing business rules will be leveraged for Customer 360 design. Provides an opportunity to leverage Stage to design future projects around Data Profiling and Data Quality Services. Enables a true view of the source system data providing accurate Data Quality measurements. Replacing transformation hardcoding with business managed reference tables and conforming data in DQ is an easier operation to be handled within DQ.
  22. 22. 21 Extract, Transform and Load (ETL)
  23. 23. 22 Accenture Data Integration Framework Accenture Data Integration Framework encompasses combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution encompasses discovery, cleansing, monitoring, transforming and delivery of data from a variety of sources.
  24. 24. 23 Requirements for File Validation, Change Data Identification, and transformation business rules will be defined for each Admin system feed as it is loaded into the Customer 360 staging database Acquisition Filtering General Rules defined for DMO will be analyzed and leveraged when appropriate. Additional rules will be defined specifically to meet the Customer 360 requirements. Change Data Identification Changed Data Identification (CDI) is a process to identify data records which have been newly created, updated, or deleted. Some files may contain one or more of the following fields within the header or record for CDI. • Time stamps • Status (I / U / D flags) • Create Date • Combination of fields / date field Other files will be compared against historic or current data sets to determine the delta for CDI. Source Data Acquisition Existing source data interface mappings and process will be leveraged for Customer 360. A copy of the existing Admin data files will be created for Customer 360. Admin data files will be loaded into its own Staging (STG) table with little to no transformations to reflect the integrity of the supplied data feed. Various File validation rules will be used to verify that each Admin feed is received with no data leakage before proceeding with the STG process. Supplied Data files will be archived on the FTP Server for determined amount of time. SourceData DataStaging(STG) Acquisition 21
  25. 25. 24 Source Data Acquisition File Validations Process Description Exception Type Header Anomalies Instance where business date within the header is incorrect. Warning Trailer Anomalies Instances where either the trailer record is missing or the count specified in the trailer does not match the total count of records received. Fatal / Critical Error Missing header Instance where the header is missing. Fatal / Critical Error File validation errors indicate issues in the source feed which may prevent processing of a source data file. These errors will be classified as either ‘Warnings’ or ‘Fatal / Critical Errors’. These will be handled by a defined Exception Handling and ABC (Audit Balance Control) process. Below are some of the typical file validation processes that will be implemented: 1
  26. 26. 25 CDI Options Change Indicators Receive deltas DB Minus Pre-DQ table Match Match against MDM Change Data Identification (CDI) Customer 360 receives a mix of source data files populated with only delta or changed data will other data files contain the full contents of the customer population. Multiple strategies will be considered when determining the appropriate changed data identification strategy to be applied for received data file. Description Can the source send only deltas? Could a hash compare be performed between the STG (from current feed) and Pre-DQ table? Could a hash compare be performed between the STG (from current feed) and Pre-DQ table? Does the source file provide Changed Data Indicators (I / U / D) ? Can the source include Changed Data Indicators ? Could ETL be used to match incoming feeds against MDM ? 2
  27. 27. 26 CDI Scenario Database Subtraction (Minus) This approach performs a quick database MINUS operation to identify updates / new records. INTERSECT operation should be used in conjunction (UNION between MINUS and INTERSECT) to capture similar records between both feeds and identify Deletes. Perform Hash of each record Perform Hash of each record Compare Hashes to identify I / U / D ETL Transformations Pre-DQ table Pre-DQ table(s) STG table(s) Pro(s) Con(s) Provides a simplistic approach, performance efficiency, and ease of implementation. None yet identified. Does not require retaining history within Pre-DQ table(s). None yet identified. 2
  28. 28. 27 CDI Scenario Hash Compare against Pre-DQ This approach calculates a hash of each record within Pre-DQ table(s) and each record in the STG table(s) having most recent data. Both set of hashes are them compared to identify Inserts / Updates / Deletes. Perform Hash of each record Perform Hash of each record Compare Hashes to identify I / U / D ETL Transformations Pre-DQ table Pre-DQ table(s) STG table(s) Pro(s) Con(s) Prevent from processing full load each time. Requires retaining history in Pre-DQ table(s). NA Calculating record hash and performing hash comparison may impact performance depending on record volume / size. 2
  29. 29. 28 Pro(s) Con(s) Prevent from processing full load each time. Calculating record hash and performing hash comparison may impact performance depending on record volume / size. Prevents from having to store history in Pre-DQ table(s) in addition to MDM table(s). Increased complexity in ETL, due to completely different data model. Perform Hash of each record Perform Hash of each record Compare Hashes to identify I / U / D ETL Transformations Pre-DQ table STG table(s) MDM table(s) CDI Scenario Match against MDM This approach performs a hash of each record within MDM table(s) and each record from the incoming feed. Both set of hashes are then compared to identify Inserts / Updates / Deletes. 2
  30. 30. 29 CDI Options Change Indicators Receive deltas DB Minus Pre-DQ table Match Match against MDM Change Data Identification (CDI) Recommendation Based on current understanding of what information the client receives from the source feeds, Accenture recommends some of the below CDI approaches. Depending on source feed, either of the below recommended CDI approaches would be implemented. Description Can the source send only deltas? Could a hash compare be performed between the STG (from current feed) and Pre-DQ table? Could a hash compare be performed between the STG (from current feed) and Pre-DQ table? Does the source file provide Changed Data Indicators (I / U / D) ? Can the source include Changed Data Indicators ? Could ETL be used to match incoming feeds against MDM ? Source data does not provide any Changed Data Indicators. Hence, CDI is performed within existing DMO processing. The source data cannot make changes. Source data cannot send deltas, especially now that many difference processes use these full feeds. Possible option. This option is preferred over other methodologies due to simplistic approach, performance efficiency, and ease of implementation. Possible option Performance impact based on expected volume would need to be considered. Avoid as much as possible. Performance impact based on expected volume would need to be considered. Understanding / High Level Recommendation    2
  31. 31. 30 Exception Handling Existing exception handling / warning rules for DMO process will be leveraged. Customer 360 exception handling process, will capture and log exceptions: • As they occur during Stage (STG) load. • As data is pushed using ETL to DQ. Batch Services Audit & Control Source system traceability will be enabled through: • Audit (A): Audits will provide traceability of data and all the batch processes. • Balance (B): Enables performing checks and balances on the processed records. • Control (C): Provides workflow management throughout the batch. ETL Parameters / Variables Existing parameters / variables will be analyzed to evaluate what could be leveraged. Additional parameters / variables might be defined to simplify development efforts. Scheduler CA’s scheduler will enable the orchestration of batch program executions from source system feeds through MDM and SFDC. ETL Services Service Components 1 2 34
  32. 32. 31 What is Audit Balance Control (ABC)? ABC will provide business users: 1. Visibility into audit information including ‘Who’ updated a record, ‘What’ changes were made and ‘When’ the record was updated. 2. Needed metrics for operational reporting of batch processes. 3. Ability to balance back to the source feed. Audit • Audit provides Manulife traceability of data and processes. • Audit tables should be tagged with sufficient data to: – validate success – assess performance – research issues • Enables linking each data record to the process which created or modified it. Balance • Independent comparison of checks and sums – checks for lost records – checks for computation and other processing errors Control • Provides workflow management to ensure that: – processes are run at the right times – exceptions are caught – exception notification occurs – exception recovery occurs 1
  33. 33. 32 Audit Balance Control (ABC) Methodology Audit, Balance and Control (ABC) process will be implemented at 2 levels. At batch level, information about the batch will be captured, while at an individual job (segment) level detailed execution information associated with each child job will be captured. Batch Group • Process will create a unique BATCH_GRP_ID in Batch Group table for every batch initiated by scheduler or “kick-off” script with status “In Progress”. • Ex: iFast workflow might include many processes within. Hence, iFast workflow will have BATCH_GRP_ID = 1 and BATCH_GRP_STATUS_CD = 0 (In Progress). Batch Segment • The process will create a unique BATCH_SEG_ID in Batch Segment table for every job within the Batch Group. • Ex: Within iFast workflow, might have 2 file validation processes, and 1 staging process. Hence, within BATCH_GRP_ID =1 for 2 file validation processes, BATCH_SEG_ID = 1, 2 and for 1 Staging process BATCH_SEG_ID = 3 and SEG_STATUS_CD = 0 (In Progress). 0 = In Progress 1 = Completed 2 = Failed 1
  34. 34. 33 Audit Fields Each Audit / Exception Handling table will capture some common set of metadata, often identified as Audit fields. Audit Attributes Description Create Date Standard audit date time stamp record was created. Created By Standard audit user id that created record. Update Date Standard audit date time stamp record was updated. Updated By Standard audit user id that created record. Audit Attributes (populated with every ETL Batch execution). 1
  35. 35. 34 ABC Batch Group Metadata At batch group level, batch level metadata information related to the batch and not individual jobs (segments) will be captured. Business requirements / design considerations for Customer 360, might result in modification of recommended attributes for ABC. Attributes Description Sample Values Batch Group ID Primary key. 1000 Batch Descriptive Name Descriptive name of the job package. Runs iFAST MMF process Batch Name Job name of the job package. Wf_iFastMMF_Account_STG Start Date The date/time stamp the batch process started. 01/12/2015 21:59:55 End Date The date/time stamp the batch process ended. 1/12/2015 22:59:00 Batch Status Code The batch group process status code. C MLAC Source System ID MLAC’s source system ID. 23 Batch Group attributes 1
  36. 36. 35 ABC Batch Segment Metadata Column Name Description Sample Values Segment ID Primary key for child level audit records. 1100 Batch Group ID Foreign key of Job Package (parent) table. 1000 Process Type Name Name of the type of process (STG, DQ, etc.). STAGING File_Date Business Date of the file. 01/11/2015 09:00:00 Process Type Start Date/Time Start date/time of the process type. 01/12/2015 22:00:00 Process Type End Date/Time End date/time of the process type. 01/12/2015 22:45:00 Segment Name Exact name of the job (session / script name, etc.). s_iFastMMF_Account_STG Source table / file Name Name of the source table. iFast_MMF_Account Target table / file name Name of the target table. iFastMMF_Account_STG # Re-processed Errors 2 # Read records 1000 # Inserted records 980 # Critical Error records 20 # Warning records 4 Net Balance Records (Read Ct + Error_Reprocessing_Ct) – (Insert Ct + Error Ct ). (Read Ct) – (Insert Ct + Error Ct ) Balance Indicator If Net Balance Records = 0 Then ‘Y’ Else ‘N’. ‘Y’ Segment Status Code The segment process status code. Success AUDIT fields 4 ETL AUDIT fields (already defined). 4 ETL AUDIT fields (already defined) Existing Fields New FieldsLegend: 1
  37. 37. 36 What is Exception Handling? Exceptions refer to any error which occur when reading or processing a data file or mapping. Examples of potential errors include, but are limited to the following: Incorrect data format, duplicate values, unknown codes not defined in the business dictionaries, file issues, etc. Exceptions may lead to data rejections and even an abort of an ETL process. Exceptions will be capturing for reporting in order to conduct root cause, remediation options and impact analysis. While informational / warnings are technically not errors, they will be captured for trend analysis. The following are examples of error types, and reporting. Warnings: Processing of record / file may continue with a warning message but require attention. Look into the root cause to determine whether to take action. Informational: Not an error or a warning (acceptable data) or processing conditions that do not impact output. This is an informational event; no action needs to be taken. 2000 1000 1009 – SIN is non-numeric or < 9 Examples: 1010 – Active record fell off feed Examples: Critical: Exceptions that cause an ETL interface to abort or violate business rules. Take action to resolve the issue. 3000 2010 – Input value is not unique 4014 – Record already exists File Validation Errors Examples: 2
  38. 38. 37 Exception Handling Metadata Attributes Description Sample Values EXCEPTION ID Unique ID for exception record. 1 BATCH ID Unique ID for the current Batch. 100 Source table / file name The name of the source table / file. iFast-MMF MLAC System ID MLAC Source System ID. 23 Exception Code Reason Code for the exception. 2010 Exception Comments Generic description of the exception. Value ‘23’ already present in target table ‘iFastMMF_Account_STG’ Exception Type ID Type of Exception (E = Critical Error, W = Warning). C Record Field Field / file column that caused the error / warning. Account_ID Field data Incoming field / file column data associated with the critical error / warning. 22 Session Name Informatica session name. s_iFastMMF_STG Data Stream Exception record from the source file. 1 | 100 | 23| ….n Exception Record Status Run status of the exception record (New, Processed, Re-try failed). New Re-process Indicator Flag to determine if error record should be re-processed or not. Y Re-processed Date Date the error record was re-processed. 01/12/2015 12:00:00 Re-process Count # of times the exception record has been re-processed. 1 ETL AUDIT fields 4 ETL Audit fields (already defined). -- ETL Exception table(s) will be created to capture error types and codes. These tables will be built using Accenture common standards and extended to capture further business requirement metadata. Existing Fields New Fields 2
  39. 39. 38 Conceptual Data Model Batch Group Batch Group ID Batch Grp Job Name … n Batch Segment Batch Segment ID Batch Seg Job Name … n Batch Grp ID Exception Log Exception ID Batch ID Exception Src Field ….n Exception Code Exception Codes Code Description ABC Exception Handling One Many Exception Type Exception Class ID Class Description
  40. 40. 39 Exception Handling / ABC - Data Flow Informational Warning Fatal / Critical Auto Retry Manual Err Correction Input File (14 sources) CA Scheduler (Re-start ability / Dependency) 1 Data Governance 2 1 to 1 Mapping Transformations Non-required missing field Invalid SIN number Missing Key Re-ProcessErrors 4 3 Re-ProcessErrors Extract Transform Load (ETL) Staging Table(s) Pre-DQ Table(s) Audit & Error Table(s) 2 Data Quality (DQ) DQ Table(s) Data Cleansing Missing Phone Number Missing Lookup Value Match Field Error N Y N Y File Validation Process Duplicate filename Header Anomalies Trailer Anomalies Missing Header
  41. 41. 40 Exception Handling / ABC - Data Flow Steps Step No. Step Summary Details of the Step 1 Load Source Data. The file validation processes will be applied to each of the 13 Admin system files. 2 Determine if any exceptions were noted within source feeds. 2a. If no exceptions were encountered, load the records into STG tables 2b. If exceptions were encountered, fail the file and abort the process. Record the issue within the Error table(s). 2c. Log the process details into the Audit table(s). 3 While pushing STG data to DQ through ETL, verify if an exception occurred. 3a. If no fatal errors were encountered, process the records, along with any ‘Informational’ or ‘Warnings’ error types through the ETL for DQ process. 3b. If fatal errors were encountered, fail the record and log the details within the Error table(s). 3c. Log the process details into the Audit table(s). 4 To perform DQ process, check is performed to see if any exceptions occurred. 4a. If no fatal errors were encountered, load the records along with ‘Informational’ or ‘Warnings’ types into the final DQ tables. 4b. If fatal errors were encountered, fail the record and log the details within the Error table(s). 4c. Log the process details into the Audit table(s).
  42. 42. 41 ETL Parameters & Variables Parameters Parameters represent a constant value and help retain the same value throughout the process run. Parameters could be used to change values such as database connections or file names from job to job. Parameters provide: 1. Portability across environments (Dev, SIT, UAT, Prod). 2. Simplification and automation of code promotion. 3. Removal of manual step of updating the database connection(s) in a session. Variables Variables represent a value that can be change during run-time. In addition to simplifying development, some of the value they add include: 1. Defining of general properties for Integration Services such as email address, log file. counts, etc. 2. Evaluating task conditions to record information for downstream job execution. Specific parameters / variables will be defined during Gate IV / Design phase. 3
  43. 43. 42 A one-to-one design will be followed such that there is one CA Scheduler job for each unique batch session. All ETL batch jobs will be developed such that they are fully re-star table. Unique approach to restarting and recovering a job will be defined depending on volume of data and other requirements defined during Gate IV. Below would be considered while defining the restart and recovery methodology: 1. Job dependencies. 2. Data Duplications. 3. Guaranteed delivery of data. Scheduler4
  44. 44. 43 Source File Metadata (1 of 4) To define File Processing Strategy and CDC Strategy for Customer 360 , a detailed understanding of each file feed is needed. Additionally, more information will likely be needed before Gate IV (before kicking-off Design). Growth rate is expected to be 150% for all sources during the 1st year and 10%-50% (average of 25%) subsequent years. Sources / Src_Sys_ID Files / Physical Filename File – Data Streams / Record Types Vendor File Type Avg. File Volume (# of Days) Expected Growth Rate (150% + 25%) Freq. Feed Type File Arrival Time iFAST-Seg Fund (18) Account Relationship / MMIF_ACRL.DT (ask developer) NA IFDS (Timeshare) Flat File (Fixed POS) 4,725 (7 days) 14,766 Daily (M – F) Delta Earliest: 11:41PM Latest: 04:55 AM Account / MMIF_ACCT.DT NA Flat File (Fixed POS) 5,879 (8 days) 18,372 Daily (M – F) Delta Holdings (may be out of scope) MMIF_ACCS.DT / VENDOR IS: IFDS NA Flat File (Fixed POS) 4,365 (7 days) 13,641 Daily (M – F) Delta iFAST-MMF (36) Account Relationship / MMMF_ACRL.DT (ask developer) NA IFDS (Timeshare) Flat File (Fixed POS) 324 (6 days) 1,013 Daily (M – F) Delta Earliest: 10:59 PM Latest: 05:25 AM Account / MMMF_ACCT.DT NA Flat File (Fixed POS) 7,365 (7 days) 23,016 Daily (M – F) Delta Holdings / (may be out of scope) MMMF_ACCS.DT NA Flat File (Fixed POS) 3,519 (6 days) 10,997 Daily (M – F) Delta
  45. 45. 44 Source File Metadata (2 of 4) Sources / Src_Sys_ID Files / Physical Filename File – Data Streams / Record Types Vendor File Type Avg. File Volume (# of Days) Expected Growth Rate (150% + 25%) Freq. Feed Type File Arrival Time Dataphile-IIROC (28) DP_7585_DMO.DT CIFDTL, PLNDTL, BNDHLD, EQTHLD, FNDHLD, TRMHLD (may be out of scope) Broadridge Flat File (Fixed POS) 1,444,637 (5 days) 4,514,491 Daily (M-F) Full Earliest: 03:01 AM Latest: 07:41 AM Dataphile-MFDA (30) DP_7584_DMO.DT CIFDTL, PLNDTL, BNDHLD, EQTHLD, FNDHLD, TRMHLD (may be out of scope) Flat File (Fixed POS) 1,296,394 (4 days) 4,051,231 Daily (M-F) Full Dataphile-Seg (35) DP_3495_DMO.DT CIFDTL, PLNDTL, FNDHLD (may be out of scope) Flat File (Fixed POS) 35,149 (2 days) 109,841 Daily (M-F) Full PS (16) Client / pwb.pwbdpccl60.client Customer, Customer Address Flat File (Manulife Mainframe) Flat File (Fixed POS) 515,409 (231 days) 1,610,653 Daily (M-F) Delta Earliest: 00:44 AM Latest: 05:08 AM Coverage (may be out of scope) / Pwb.pwbdpcco.dicco60.cover age Coverage Customer Flat File (Fixed POS) Daily (M-F) Delta Policy / pwb.pwbdpcpo.dipo60.policy Policy, Policy Customer Flat File (Fixed POS) Daily (M-F) Delta CAPSIL 4.2 (8) Flat File (Manulife Mainframe) Flat File (Fixed POS) 253,568 (251 days) 792,400 Daily Delta Earliest: 10:16 AM Latest: 12:42 AM VAPS (37) VAPS_DATA.txt Account, Client Manulife Mainframe Flat File (Manulife Mainframe) 392,263 (2 days) 1,225,822 Daily Full Earliest: 09:18 PM Latest: 02:47 AM
  46. 46. 45 Source File Metadata (3 of 4) Sources / Src_Sys_ID Files / Physical Filename File – Data Streams / Record Types Vendor File Type Avg. File Volume (# of Days) Expected Growth Rate (150% + 25%) Freq. Feed Type File Arrival Time CATS (15) Client Customer, Customer Address Manulife Mainframe Flat File (Fixed POS) 1,989,064 (252 days) 6,215,825 Daily (M-F) Delta Earliest 00:02 AM Latest: 11:23 PM Coverage (may be out of scope) Coverage Customer Flat File (Fixed POS) Daily (M-F) Delta Policy Policy, Policy Customer Flat File (Fixed POS) Daily (M-F) Delta CLAS (14) Client Customer, Customer Address Manulife Mainframe Flat File (Fixed POS) 1,068,435 (252 days) 3,338,860 Daily (M-F) Delta Earliest 00:44 AM Latest: 06:05 AM Coverage (may be out of scope) Coverage Customer Flat File (Fixed POS) Daily (M-F) Delta Policy Policy, Policy Customer Flat File (Fixed POS) Daily (M-F) Delta PPlus-GIC (20) PPLUSDATA.txt Plan, Customer, Investments (may be out of scope) SIT Flat File (Fixed POS) Daily (M-Su) Full Earliest: 02:51 AM Latest: 10:09 PM PPlus-Bank DMO.DT Plan, Customer, Loan, Retail, Term (may be out of scope) SIT Flat File (Fixed POS) Daily (M-F) Full
  47. 47. 46 Source File Metadata (4 of 4) Sources / Src_Sys_ID Files / Physical Filename File – Data Streams / Record Types Vendor File Type Avg. File Volume (# of Days) Expected Growth Rate (150% + 25%) Freq. Feed Type File Arrival Time MPW Customers, Plan NA Wealth Manager Oracle DB 250 customers 1000 plans 20% / yr HR Hr.txt SQL Server DB Sometimes 0 and sometimes few records 150% in 1st year due to expected acquisition Daily (M-F) Delta Estimated: 12:00 AM
  48. 48. 47 File Processing Strategy Business requirement calls for attempting to process file as they arrive in an attempt to keep the process to as real-time as possible. To define file processing strategy for Customer 360, based on existing source file feed information, an understanding of below requirements were considered. ID Requirements Existing Process 1 Are there dependencies between files amongst source system feeds ? For ex: Is there a dependency between files from iFast feeds and PPlus feeds ? There certainly exists dependencies between files amongst source system feeds (ex: dependencies exist between iFast files and PPlus files). Other dependencies are yet to be identified. It is yet to be analyzed if the files needed for Customer 360 amongst source system feeds have dependencies. 2 Are there dependencies amongst files within the same source system feed ? There might be dependencies amongst files from the same source feed. Whether or not those dependencies would exist amongst files within Customer 360’s scope, is yet to be identified. 3 If a file within a source system does not arrive on time, how does DMO currently handle this ? Currently, if a file does not arrive, the file watch fails that batch and all other dependent batches. For example, if iFast-MMF Account file does not arrive, Account Relationship and other files for iFast-MMF will not be processed. Operations team works with required teams to resolve issue ASAP. In the meantime, batch status is manually updated to ‘Complete’ status. 4 If a file is not received, does the source send 2 files the next day ? Sources have been requested to only send 1 file. Hence, if iFast-MMF was not processed, all other files next day will include records from the day prior. 5 Does the next day’s file include records from previous day ? Yes. 6 Does the source system send all their feeds all together ? Yes.
  49. 49. 48 File Processing Strategy – Data Flow Based on current understanding of various scenarios (described in the previous slide) and existing file delivery agreements with source systems, our recommendation is to continue existing file processing strategy. As we look further into file dependencies and analyze file arrival times during Gate IV, this recommendation might go through revisions. Source System 1 Source System 2 Source System n FTP Server Copy files for Customer 360 Leverage Existing Process 1 Did all files per source system arrive? Audit & Error Table(s) 2a ETL Process Critical File Validation Errors? Y N Fail Entire Batch N Y 2c 3a 3b 3c Watch for all files from each source system to arrive 2b New Process for Customer 360 for Customer 360 File 2 File 1 Source System 1 Source System 2 File 1 for DMO File 2 File 1 Source System 1 Source System 2 File 1 For Customer 360
  50. 50. 49 File Processing Strategy – Data Flow Description Step No. Step Summary Details of the Step 1 Did the file arrive? File Watch process evaluates if all required files for each source has been received. Since, each source systems do no have dependencies upon each other, upon presence of all files for the source system, ETL process for that source system will start. 2 Did the file arrive? Is the file valid? 2a. If any file within a source system is not received, entire batch for that source system should fail. 2b. If all required files for the source system has been received, the batch should kick off. 2c. Log Audit & Error table(s). 3 Are there any Critical File Validation Errors? 3a. If any critical error was encountered within the received source files, fail the batch and record Audit & Error table(s). 3b. If no critical errors were encountered in the received source files, proceed to ETL Process. 3c. Log Audit & Error table(s).
  51. 51. 50 ETL Development Cycle – Single Track Create workspace and project Layout job by arranging steps, data sources, and targets Validate Job Generate job package Finalize project in repository Deploy job artifacts Validate JobExport Job Deploy to next environment 1) Developers • Developers creates connections to repositories and folders needed for development. • The developer lays out the mapping (data flow) by arranging data sources, transformations, and targets. The developer defines the links and configures the mapping. • The developer tests the job until it is ready for promotion. • The job code is then exported in an XML format ready to be imported to another environment’s repository. 2) Manulife Tech • Client team verifies and approves the job. • The job is then moved into the test environment. • These changes can be deployed by importing the XML into the repository / environment. 3) Functional Team • The functional test team, aided by job owners will validate the results or recommend changes. • Changes may include modifications to joins / transformation rules or new transformations and error conditions. • If there are unexpected results, the process restarts in the developer’s workspace. 4) Manulife Tech • Once the functional test team validates the results, the client teams promote the necessary artifacts forward. • The promotion process will iterate between promote and test cycles until the artifacts reach production. 1 1) 234
  52. 52. 51 Development Standards & Guidelines These standards will encompass: 1. Naming conventions for ETL jobs. 2. Naming conventions for objects used within each ETL. 3. Project and project object directory structures. 4. Preferred and recommended development practices. 5. Guidelines around preferred use of re-useable components to streamline development efforts. Accenture will be adopting Manulife guidelines and standards and may be enriched using Accenture Methodologies and leading practices. Development standards will be defined to capture ETL development guidelines for Customer 360 that will be applicable to design and development efforts of the project.
  53. 53. 52 Summary & Next Steps
  54. 54. 53 Summary and Next Steps  Completed the Data Flow and Solution Architecture for Customer 360.  Gathered and documented high level business requirements and non functional requirements.  Developed the conceptual data model for customer 360.  Drafted initial MDM component architecture and customer matching rules.  Produced directional data quality profiling results on masked DMO data.  Defined the structure for the data management and governance organization structures.  Create the Detailed Design Inventory and data mapping for the design phase.  Run Data Quality profiling reports on unmasked source system data.  Develop the component dependency and design order.  Support the design and build of Technical Architecture.  Start Identifying the gaps in governance processes and defining the formal roles and processes for data governance. Summary of Plan & Analyze Outcome Next Steps

×