This document provides an overview of the conceptual data flow and architecture for a Customer 360 solution. Key components include extracting data from various admin systems, transforming and loading it into a data quality repository, matching and merging records in MDM, propagating updates to downstream systems like Salesforce, and enabling data steward review of matches and merges. The data flows both systematically and in response to user changes in various applications and portals.
Customer Event Hub - the modern Customer 360° viewGuido Schmutz
Today, companies are using various channels to communicate with their customers. As a consequence, a lot of data is created, more and more also outside of the traditional IT infrastructure of an enterprise. This data often does not have a common format and they are continuously created with ever increasing volume. With Internet of Things (IoT) and their sensors, the volume as well as the velocity of data just gets more extreme.
To achieve a complete and consistent view of a customer, all these customer-related information has to be included in a 360 degree view in a real-time or near-real-time fashion. By that, the Customer Hub will become the Customer Event Hub. It constantly shows the actual view of a customer over all his interaction channels and provides an enterprise the basis for a substantial and effective customer relation.
In this presentation the value of such a platform is shown and how it can be implemented.
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker
Past, present and future of data mesh at Intuit. This deck describes a vision and strategy for improving data worker productivity through a Data Mesh approach to organizing data and holding data producers accountable. Delivered at the inaugural Data Mesh Leaning meetup on 5/13/2021.
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Master data management (mdm) & plm in context of enterprise product managementTata Consultancy Services
The presentation discusses the classical features and advantages of Master Data Management (MDM) system along with appropriate situations to use it. How do companies apply MDM who design, manufacture and sell their products in several geographies facing challenges in making appropriate decisions on their investment in PLM & MDM space?
Another important aspect covers the comparison/relation between a MDM system (or Product Master System) and Enterprise PLM system. How can you maximize your ROI on both PLM and MDM investments? With examples from different industries the key takeaways include whether your organization requires an MDM solution or not.
Describes what Enterprise Data Architecture in a Software Development Organization should cover and does that by listing over 200 data architecture related deliverables an Enterprise Data Architect should remember to evangelize.
Creating an Effective MDM Strategy for SalesforcePerficient, Inc.
As Salesforce has grown from a simple, standalone tool to a platform that touches every customer interaction, the data has grown more complex. This problem happens for many reasons including user error, adding other cloud apps requiring data integration, and business mergers and acquisitions that create multiple instances of Salesforce within an organization.
A master data management (MDM) strategy is critical to helping companies solve challenges like providing enterprise analytics and creating a 360-degree view of the customer. With Informatica Cloud, companies are learning to address the challenges and explore alternatives including a cost-effective cloud MDM versus a full-blown MDM solution.
During this webinar, our experts demonstrated the Informatica cloud MDM solution in action and showed how with an effective strategy, you can:
-Support the business case for MDM consolidation of multiple instances
-Create a customer 360-degree view in the cloud
-Understand the use case, reference architecture, and why companies are choosing cloud-based MDM
Gartner: Master Data Management FunctionalityGartner
Gartner will further examine key trends shaping the future MDM market during the Gartner MDM Summit 2011, 2-3 February in London. More information at www.europe.gartner.com/mdm
Glossaries, Dictionaries, and Catalogs Result in Data GovernanceDATAVERSITY
Data catalogs, business glossaries, and data dictionaries house metadata that is important to your organization’s governance of data. People in your organization need to be engaged in leveraging the tools, understanding the data that is available, who is responsible for the data, and knowing how to get their hands on the data to perform their job function. The metadata will not govern itself.
Join Bob Seiner for the webinar where he will discuss how glossaries, dictionaries, and catalogs can result in effective Data Governance. People must have confidence in the metadata associated with the data that you need them to trust. Therefore, the metadata in your data catalog, business glossary, and data dictionary must result in governed data. Learn how glossaries, dictionaries, and catalogs can result in Data Governance in this webinar.
Bob will discuss the following subjects in this webinar:
- Successful Data Governance relies on value from very important tools
- What it means to govern your data catalog, business glossary, and data dictionary
- Why governing the metadata in these tools is important
- The roles necessary to govern these tools
- Governance expected from metadata in catalogs, glossaries, and dictionaries
Customer Event Hub - the modern Customer 360° viewGuido Schmutz
Today, companies are using various channels to communicate with their customers. As a consequence, a lot of data is created, more and more also outside of the traditional IT infrastructure of an enterprise. This data often does not have a common format and they are continuously created with ever increasing volume. With Internet of Things (IoT) and their sensors, the volume as well as the velocity of data just gets more extreme.
To achieve a complete and consistent view of a customer, all these customer-related information has to be included in a 360 degree view in a real-time or near-real-time fashion. By that, the Customer Hub will become the Customer Event Hub. It constantly shows the actual view of a customer over all his interaction channels and provides an enterprise the basis for a substantial and effective customer relation.
In this presentation the value of such a platform is shown and how it can be implemented.
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker
Past, present and future of data mesh at Intuit. This deck describes a vision and strategy for improving data worker productivity through a Data Mesh approach to organizing data and holding data producers accountable. Delivered at the inaugural Data Mesh Leaning meetup on 5/13/2021.
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Master data management (mdm) & plm in context of enterprise product managementTata Consultancy Services
The presentation discusses the classical features and advantages of Master Data Management (MDM) system along with appropriate situations to use it. How do companies apply MDM who design, manufacture and sell their products in several geographies facing challenges in making appropriate decisions on their investment in PLM & MDM space?
Another important aspect covers the comparison/relation between a MDM system (or Product Master System) and Enterprise PLM system. How can you maximize your ROI on both PLM and MDM investments? With examples from different industries the key takeaways include whether your organization requires an MDM solution or not.
Describes what Enterprise Data Architecture in a Software Development Organization should cover and does that by listing over 200 data architecture related deliverables an Enterprise Data Architect should remember to evangelize.
Creating an Effective MDM Strategy for SalesforcePerficient, Inc.
As Salesforce has grown from a simple, standalone tool to a platform that touches every customer interaction, the data has grown more complex. This problem happens for many reasons including user error, adding other cloud apps requiring data integration, and business mergers and acquisitions that create multiple instances of Salesforce within an organization.
A master data management (MDM) strategy is critical to helping companies solve challenges like providing enterprise analytics and creating a 360-degree view of the customer. With Informatica Cloud, companies are learning to address the challenges and explore alternatives including a cost-effective cloud MDM versus a full-blown MDM solution.
During this webinar, our experts demonstrated the Informatica cloud MDM solution in action and showed how with an effective strategy, you can:
-Support the business case for MDM consolidation of multiple instances
-Create a customer 360-degree view in the cloud
-Understand the use case, reference architecture, and why companies are choosing cloud-based MDM
Gartner: Master Data Management FunctionalityGartner
Gartner will further examine key trends shaping the future MDM market during the Gartner MDM Summit 2011, 2-3 February in London. More information at www.europe.gartner.com/mdm
Glossaries, Dictionaries, and Catalogs Result in Data GovernanceDATAVERSITY
Data catalogs, business glossaries, and data dictionaries house metadata that is important to your organization’s governance of data. People in your organization need to be engaged in leveraging the tools, understanding the data that is available, who is responsible for the data, and knowing how to get their hands on the data to perform their job function. The metadata will not govern itself.
Join Bob Seiner for the webinar where he will discuss how glossaries, dictionaries, and catalogs can result in effective Data Governance. People must have confidence in the metadata associated with the data that you need them to trust. Therefore, the metadata in your data catalog, business glossary, and data dictionary must result in governed data. Learn how glossaries, dictionaries, and catalogs can result in Data Governance in this webinar.
Bob will discuss the following subjects in this webinar:
- Successful Data Governance relies on value from very important tools
- What it means to govern your data catalog, business glossary, and data dictionary
- Why governing the metadata in these tools is important
- The roles necessary to govern these tools
- Governance expected from metadata in catalogs, glossaries, and dictionaries
Reference matter data management:
Two categories of structured data :
Master data: is data associated with core business entities such as customer, product, asset, etc.
Transaction data: is the recording of business transactions such as orders in manufacturing, loan and credit card payments in banking, and product sales in retail.
Reference data: is any kind of data that is used solely to categorize other data found in a database, or solely for relating data in a database to information beyond the boundaries of the enterprise .
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how data architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Data Governance Takes a Village (So Why is Everyone Hiding?)DATAVERSITY
Data governance represents both an obstacle and opportunity for enterprises everywhere. And many individuals may hesitate to embrace the change. Yet if led well, a governance initiative has the potential to launch a data community that drives innovation and data-driven decision-making for the wider business. (And yes, it can even be fun!). So how do you build a roadmap to success?
This session will gather four governance experts, including Mary Williams, Associate Director, Enterprise Data Governance at Exact Sciences, and Bob Seiner, author of Non-Invasive Data Governance, for a roundtable discussion about the challenges and opportunities of leading a governance initiative that people embrace. Join this webinar to learn:
- How to build an internal case for data governance and a data catalog
- Tips for picking a use case that builds confidence in your program
- How to mature your program and build your data community
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace, from digital transformation to marketing, customer centricity, population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Data protection and privacy regulations such as the EU’s General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Singapore’s Personal Data Protection Act (PDPA) have been major drivers for data governance initiatives and the emergence of data catalog solutions. Organizations have an ever-increasing appetite to leverage their data for business advantage, either through internal collaboration, data sharing across ecosystems, direct commercialization, or as the basis for AI-driven business decision-making. This requires data governance and especially data asset catalog solutions to step up once again and enable data-driven businesses to leverage their data responsibly, ethically, compliantly, and accountably.
This presentation explores how data catalog has become a key technology enabler in overcoming these challenges.
Business Intelligence & Data Analytics– An Architected ApproachDATAVERSITY
Business intelligence (BI) and data analytics are increasing in popularity as more organizations are looking to become more data-driven. Many tools have powerful visualization techniques that can create dynamic displays of critical information. To ensure that the data displayed on these visualizations is accurate and timely, a strong Data Architecture is needed. Join this webinar to understand how to create a robust Data Architecture for BI and data analytics that takes both business and technology needs into consideration.
Data Catalog for Better Data Discovery and GovernanceDenodo
Watch full webinar here: https://buff.ly/2Vq9FR0
Data catalogs are en vogue answering critical data governance questions like “Where all does my data reside?” “What other entities are associated with my data?” “What are the definitions of the data fields?” and “Who accesses the data?” Data catalogs maintain the necessary business metadata to answer these questions and many more. But that’s not enough. For it to be useful, data catalogs need to deliver these answers to the business users right within the applications they use.
In this session, you will learn:
*How data catalogs enable enterprise-wide data governance regimes
*What key capability requirements should you expect in data catalogs
*How data virtualization combines dynamic data catalogs with delivery
In the past few years, the term "data lake" has leaked into our lexicon. But what exactly IS a data lake? Some IT managers confuse data lakes with data warehouses. Some people think data lakes replace data warehouses. Both of these conclusions are false. Their is room in your data architecture for both data lakes and data warehouses. They both have different use cases and those use cases can be complementary.
Todd Reichmuth, Solutions Engineer with Snowflake Computing, has spent the past 18 years in the world of Data Warehousing and Big Data. He spent that time at Netezza and then later at IBM Data. Earlier in 2018 making the jump to the cloud at Snowflake Computing.
Mike Myer, Sales Director with Snowflake Computing, has spent the past 6 years in the world of Security and looking to drive awareness to better Data Warehousing and Big Data solutions available! Was previously at local tech companies FireMon and Lockpath and decided to join Snowflake due to the disruptive technology that's truly helping folks in the Big Data world on a day to day basis.
DAS Slides: Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key inter-relationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how Data Architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Activate Data Governance Using the Data CatalogDATAVERSITY
Data Governance programs depend on the activation of data stewards that are held formally accountable for how they manage data. The data catalog is a critical tool to enable your stewards to contribute and interact with an inventory of metadata about the data definition, production, and usage. This interaction is active Data Governance in the truest sense of the word.
In this RWDG webinar, Bob Seiner will share tips and techniques focused on activating your data stewards through a data catalog. Data Governance programs that involve stewards in daily activities are more likely to demonstrate value from their data-intensive investments.
Bob will address the following in this webinar:
- A comparison of active and passive Data Governance
- What it means to have an active Data Governance program
- How a data catalog tool can be used to activate data stewards
- The role a data catalog plays in Data Governance
- The metadata in the data catalog will not govern itself
To take a “ready, aim, fire” tactic to implement Data Governance, many organizations assess themselves against industry best practices. The process is not difficult or time-consuming and can directly assure that your activities target your specific needs. Best practices are always a strong place to start.
Join Bob Seiner for this popular RWDG topic, where he will provide the information you need to set your program in the best possible direction. Bob will walk you through the steps of conducting an assessment and share with you a set of typical results from taking this action. You may be surprised at how easy it is to organize the assessment and may hear results that stimulate the actions that you need to take.
In this webinar, Bob will share:
- The value of performing a Data Governance best practice assessment
- A practical list of industry Data Governance best practices
- Criteria to determine if a practice is best practice
- Steps to follow to complete an assessment
- Typical recommendations and actions that result from an assessment
Presentation on Data Mesh: The paradigm shift is a new type of eco-system architecture, which is a shift left towards a modern distributed architecture in which it allows domain-specific data and views “data-as-a-product,” enabling each domain to handle its own data pipelines.
Data-Ed Webinar: Best Practices with the DMMDATAVERSITY
The Data Management Maturity (DMM) model is a framework for the evaluation and assessment of an organization’s data management capabilities. The model allows an organization to evaluate its current state data management capabilities, discover gaps to remediate, and strengths to leverage. The assessment method reveals priorities, business needs, and a clear, rapid path for process improvements. This webinar will describe the DMM, its evolution, and illustrate its use as a roadmap guiding organizational data management improvements.
Takeaways:
•Our profession is advancing its knowledge and has a wide-spread basis for partnerships
•New industry assessment standard is based on successful CMM/CMMI foundation
•Clear need for data strategy
•A clear and unambiguous call for participation
Master Data Management – Aligning Data, Process, and GovernanceDATAVERSITY
Master Data Management (MDM) provides organizations with an accurate and comprehensive view of their business-critical data such as customers, products, vendors, and more. While mastering these key data areas can be a complex task, the value of doing so can be tremendous – from real-time operational integration to data warehousing and analytic reporting. This webinar will provide practical strategies for gaining value from your MDM initiative, while at the same time assuring a solid architectural and governance foundation that will ensure long-term, enterprise-wide success.
resentation of use cases of Master Data Management for Customer Data. It presents the business drivers and how Talend platform for MDM can adress them.
Data Governance Best Practices, Assessments, and RoadmapsDATAVERSITY
When starting or evaluating the present state of your Data Governance program, it is important to focus on best practices such that you don’t take a ready, fire, aim approach. Best practices need to be practical and doable to be selected for your organization, and the program must be at risk if the best practice is not achieved.
Join Bob Seiner for an important webinar focused on industry best practice around standing up formal Data Governance. Learn how to assess your organization against the practices and deliver an effective roadmap based on the results of conducting the assessment.
In this webinar, Bob will focus on:
- Criteria to select the appropriate best practices for your organization
- How to define the best practices for ultimate impact
- Assessing against selected best practices
- Focusing the recommendations on program success
- Delivering a roadmap for your Data Governance program
Key Elements of a Successful Data Governance ProgramDATAVERSITY
At its core, Data Governance (DG) is all about managing data with guidance. This immediately provokes the question: Would you tolerate any of your assets to be managed without guidance? (In all likelihood, your organization has been managing data without adequate guidance and this accounts for its current, less-than-optimal state.) This program provides a practical guide to implementing DG or recharging your existing program. It provides an understanding of what Data Governance functions are required and how they fit with other Data Management disciplines. Understanding these aspects is a prerequisite to eliminate the ambiguity that often surrounds initial discussions and implement effective Data Governance/Stewardship programs that manage data in support of organizational strategy. Delegates will understand why Data Governance can be tricky for organizations due to data’s confounding characteristics. This webinar will focus on four key DG elements:
- Keeping DG practically focused
- DG must exist at the same level as HR
- Gradually add ingredients (practicing and getting better)
- Data Governance in action: storytelling
The concept of a 360° view, especially of customers, although it potentially applies to other things too, has been around for a substantial period of time. The idea behind the 360° view of customers is that the more you know about your customers the easier it will be to meet their needs, both in terms of products and aftersales care, and to market additional goods and services to them in the most efficient fashion. Thus a 360° view helps both in terms of customer retention and acquisition, as well as up-sell and cross-sell.
In this presentation which complements Bloor Whitepaper on the "Extended 360 degree view" we will discuss why we believe that extending the traditional 360° view makes sense and we will give some uses that demonstrate why the extended 360° view represents an opportunity, both for those that have already implemented a 360° view and for those that have not.
Reference matter data management:
Two categories of structured data :
Master data: is data associated with core business entities such as customer, product, asset, etc.
Transaction data: is the recording of business transactions such as orders in manufacturing, loan and credit card payments in banking, and product sales in retail.
Reference data: is any kind of data that is used solely to categorize other data found in a database, or solely for relating data in a database to information beyond the boundaries of the enterprise .
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how data architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Data Governance Takes a Village (So Why is Everyone Hiding?)DATAVERSITY
Data governance represents both an obstacle and opportunity for enterprises everywhere. And many individuals may hesitate to embrace the change. Yet if led well, a governance initiative has the potential to launch a data community that drives innovation and data-driven decision-making for the wider business. (And yes, it can even be fun!). So how do you build a roadmap to success?
This session will gather four governance experts, including Mary Williams, Associate Director, Enterprise Data Governance at Exact Sciences, and Bob Seiner, author of Non-Invasive Data Governance, for a roundtable discussion about the challenges and opportunities of leading a governance initiative that people embrace. Join this webinar to learn:
- How to build an internal case for data governance and a data catalog
- Tips for picking a use case that builds confidence in your program
- How to mature your program and build your data community
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace, from digital transformation to marketing, customer centricity, population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Data protection and privacy regulations such as the EU’s General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Singapore’s Personal Data Protection Act (PDPA) have been major drivers for data governance initiatives and the emergence of data catalog solutions. Organizations have an ever-increasing appetite to leverage their data for business advantage, either through internal collaboration, data sharing across ecosystems, direct commercialization, or as the basis for AI-driven business decision-making. This requires data governance and especially data asset catalog solutions to step up once again and enable data-driven businesses to leverage their data responsibly, ethically, compliantly, and accountably.
This presentation explores how data catalog has become a key technology enabler in overcoming these challenges.
Business Intelligence & Data Analytics– An Architected ApproachDATAVERSITY
Business intelligence (BI) and data analytics are increasing in popularity as more organizations are looking to become more data-driven. Many tools have powerful visualization techniques that can create dynamic displays of critical information. To ensure that the data displayed on these visualizations is accurate and timely, a strong Data Architecture is needed. Join this webinar to understand how to create a robust Data Architecture for BI and data analytics that takes both business and technology needs into consideration.
Data Catalog for Better Data Discovery and GovernanceDenodo
Watch full webinar here: https://buff.ly/2Vq9FR0
Data catalogs are en vogue answering critical data governance questions like “Where all does my data reside?” “What other entities are associated with my data?” “What are the definitions of the data fields?” and “Who accesses the data?” Data catalogs maintain the necessary business metadata to answer these questions and many more. But that’s not enough. For it to be useful, data catalogs need to deliver these answers to the business users right within the applications they use.
In this session, you will learn:
*How data catalogs enable enterprise-wide data governance regimes
*What key capability requirements should you expect in data catalogs
*How data virtualization combines dynamic data catalogs with delivery
In the past few years, the term "data lake" has leaked into our lexicon. But what exactly IS a data lake? Some IT managers confuse data lakes with data warehouses. Some people think data lakes replace data warehouses. Both of these conclusions are false. Their is room in your data architecture for both data lakes and data warehouses. They both have different use cases and those use cases can be complementary.
Todd Reichmuth, Solutions Engineer with Snowflake Computing, has spent the past 18 years in the world of Data Warehousing and Big Data. He spent that time at Netezza and then later at IBM Data. Earlier in 2018 making the jump to the cloud at Snowflake Computing.
Mike Myer, Sales Director with Snowflake Computing, has spent the past 6 years in the world of Security and looking to drive awareness to better Data Warehousing and Big Data solutions available! Was previously at local tech companies FireMon and Lockpath and decided to join Snowflake due to the disruptive technology that's truly helping folks in the Big Data world on a day to day basis.
DAS Slides: Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key inter-relationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how Data Architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Activate Data Governance Using the Data CatalogDATAVERSITY
Data Governance programs depend on the activation of data stewards that are held formally accountable for how they manage data. The data catalog is a critical tool to enable your stewards to contribute and interact with an inventory of metadata about the data definition, production, and usage. This interaction is active Data Governance in the truest sense of the word.
In this RWDG webinar, Bob Seiner will share tips and techniques focused on activating your data stewards through a data catalog. Data Governance programs that involve stewards in daily activities are more likely to demonstrate value from their data-intensive investments.
Bob will address the following in this webinar:
- A comparison of active and passive Data Governance
- What it means to have an active Data Governance program
- How a data catalog tool can be used to activate data stewards
- The role a data catalog plays in Data Governance
- The metadata in the data catalog will not govern itself
To take a “ready, aim, fire” tactic to implement Data Governance, many organizations assess themselves against industry best practices. The process is not difficult or time-consuming and can directly assure that your activities target your specific needs. Best practices are always a strong place to start.
Join Bob Seiner for this popular RWDG topic, where he will provide the information you need to set your program in the best possible direction. Bob will walk you through the steps of conducting an assessment and share with you a set of typical results from taking this action. You may be surprised at how easy it is to organize the assessment and may hear results that stimulate the actions that you need to take.
In this webinar, Bob will share:
- The value of performing a Data Governance best practice assessment
- A practical list of industry Data Governance best practices
- Criteria to determine if a practice is best practice
- Steps to follow to complete an assessment
- Typical recommendations and actions that result from an assessment
Presentation on Data Mesh: The paradigm shift is a new type of eco-system architecture, which is a shift left towards a modern distributed architecture in which it allows domain-specific data and views “data-as-a-product,” enabling each domain to handle its own data pipelines.
Data-Ed Webinar: Best Practices with the DMMDATAVERSITY
The Data Management Maturity (DMM) model is a framework for the evaluation and assessment of an organization’s data management capabilities. The model allows an organization to evaluate its current state data management capabilities, discover gaps to remediate, and strengths to leverage. The assessment method reveals priorities, business needs, and a clear, rapid path for process improvements. This webinar will describe the DMM, its evolution, and illustrate its use as a roadmap guiding organizational data management improvements.
Takeaways:
•Our profession is advancing its knowledge and has a wide-spread basis for partnerships
•New industry assessment standard is based on successful CMM/CMMI foundation
•Clear need for data strategy
•A clear and unambiguous call for participation
Master Data Management – Aligning Data, Process, and GovernanceDATAVERSITY
Master Data Management (MDM) provides organizations with an accurate and comprehensive view of their business-critical data such as customers, products, vendors, and more. While mastering these key data areas can be a complex task, the value of doing so can be tremendous – from real-time operational integration to data warehousing and analytic reporting. This webinar will provide practical strategies for gaining value from your MDM initiative, while at the same time assuring a solid architectural and governance foundation that will ensure long-term, enterprise-wide success.
resentation of use cases of Master Data Management for Customer Data. It presents the business drivers and how Talend platform for MDM can adress them.
Data Governance Best Practices, Assessments, and RoadmapsDATAVERSITY
When starting or evaluating the present state of your Data Governance program, it is important to focus on best practices such that you don’t take a ready, fire, aim approach. Best practices need to be practical and doable to be selected for your organization, and the program must be at risk if the best practice is not achieved.
Join Bob Seiner for an important webinar focused on industry best practice around standing up formal Data Governance. Learn how to assess your organization against the practices and deliver an effective roadmap based on the results of conducting the assessment.
In this webinar, Bob will focus on:
- Criteria to select the appropriate best practices for your organization
- How to define the best practices for ultimate impact
- Assessing against selected best practices
- Focusing the recommendations on program success
- Delivering a roadmap for your Data Governance program
Key Elements of a Successful Data Governance ProgramDATAVERSITY
At its core, Data Governance (DG) is all about managing data with guidance. This immediately provokes the question: Would you tolerate any of your assets to be managed without guidance? (In all likelihood, your organization has been managing data without adequate guidance and this accounts for its current, less-than-optimal state.) This program provides a practical guide to implementing DG or recharging your existing program. It provides an understanding of what Data Governance functions are required and how they fit with other Data Management disciplines. Understanding these aspects is a prerequisite to eliminate the ambiguity that often surrounds initial discussions and implement effective Data Governance/Stewardship programs that manage data in support of organizational strategy. Delegates will understand why Data Governance can be tricky for organizations due to data’s confounding characteristics. This webinar will focus on four key DG elements:
- Keeping DG practically focused
- DG must exist at the same level as HR
- Gradually add ingredients (practicing and getting better)
- Data Governance in action: storytelling
The concept of a 360° view, especially of customers, although it potentially applies to other things too, has been around for a substantial period of time. The idea behind the 360° view of customers is that the more you know about your customers the easier it will be to meet their needs, both in terms of products and aftersales care, and to market additional goods and services to them in the most efficient fashion. Thus a 360° view helps both in terms of customer retention and acquisition, as well as up-sell and cross-sell.
In this presentation which complements Bloor Whitepaper on the "Extended 360 degree view" we will discuss why we believe that extending the traditional 360° view makes sense and we will give some uses that demonstrate why the extended 360° view represents an opportunity, both for those that have already implemented a 360° view and for those that have not.
The role of Big Data and Modern Data Management in Driving a Customer 360 fro...Cloudera, Inc.
Organizations spanning all industries are in pursuit of Customer 360, which aims to integrate and enrich customer information across multiple channels, systems, devices and products in order to improve the interaction experience and maximize the value delivered. To achieve this real-time integration requires a modern approach to working with data and the Cloud is providing a differentiating strategic platform for many organisations. Discover how you can strategically structure your data environment leveraging the Cloud to empower analytical deployment, create next generation customer applications whilst also saving costs and realising greater efficiencies.
Delvinia CEO Adam Froman was among the speakers at the Canadian Marketing Association's CMA Summit 2012, held at the Westin Harbour Castle Hotel on May 16 and 17.
Adam shared how Delvinia is marrying real time feedback from customers with deep profiling data on their digital behaviours to help organizations capture and act on the Voice of the Customer to create better customer experiences.
A Customer-Centric Banking Platform Powered by MongoDB MongoDB
Speaker: Alan Reyes Vilchis, Technical Lead, Banco Azteca
Level: 200 (Intermediate)
Track: Developer
Business apps powered by single customer views (SCV’s) are one of the most predominant uses of MongoDB. However, each view presents unique challenges such as distilling data from providers, removing duplication, matching records in different systems, and more.
The team at Banco Azteca (part of Grupo Salinas, a holding company of enterprises in media, telecommunications and financial services) in Mexico City has launched various customer-centric banking services that are powered by an SCV built on MongoDB, which has not only allowed the expansion into mobile-first consumer markets, but has also helped in identifying and preventing fraud across the group’s enterprises. This talk will explore the overall initiative, and place emphasis on the technical innovations regarding design, serialization and transactions across multiple systems.
What You Will Learn:
- “Serialization magic” using the MongoDB Java driver and Jackson
- Implementing transactional-like logic across different systems
- Conceptual and physical design for building a Single-View
Apache Kafka Scalable Message Processing and more! Guido Schmutz
media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Customer Event Hub – a modern Customer 360° view with DataStax Enterprise (DSE) Guido Schmutz
Today, companies are using various channels to communicate with their customers. As a consequence, a lot of data is created, more and more also outside of the traditional IT infrastructure of an enterprise. This data often does not have a common format and they are continuously created with ever increasing volume. With Internet of Things (IoT) and their sensors, the volume as well as the velocity of data just gets more extreme.
To achieve a complete and consistent view of a customer, all these customer-related information has to be included in a 360 degree view in a real-time or near-real-time fashion. By that, the Customer Hub will become the Customer Event Hub. It constantly shows the actual view of a customer over all his interaction channels and provides an enterprise the basis for a substantial and effective customer relation.
In this presentation the value of such a platform is shown and how it can be implemented using DataStax Enterprise as the backend.
The Connected Consumer – Real-time Customer 360Capgemini
With Business Data Lake technologies based on EMC’s Big Data portfolio it becomes possible to move away from channel specific analytics towards a 360 customer view.
This presentation will show how technologies like Spark, Hadoop, and Kafka help companies gain a real-time view of everything their customers do and make changes to customer touch points whether mobile, web, in-store, direct marketing or existing transactional systems.
Presented by Steve Jones, Vice President, Insights & Data, Capgemini at EMC World 2016
http://www.capgemini.com/emc
Customer Event Hub – a modern Customer 360° view with DataStax Enterprise (DSE)Guido Schmutz
Today, companies are using various channels to communicate with their customers. As a consequence, a lot of data is created, more and more also outside of the traditional IT infrastructure of an enterprise. This data often does not have a common format and they are continuously created with ever increasing volume. With Internet of Things (IoT) and their sensors, the volume as well as the velocity of data just gets more extreme.
To achieve a complete and consistent view of a customer, all these customer-related information has to be included in a 360 degree view in a real-time or near-real-time fashion. By that, the Customer Hub will become the Customer Event Hub. It constantly shows the actual view of a customer over all his interaction channels and provides an enterprise the basis for a substantial and effective customer relation.
In this presentation the value of such a platform is shown and how it can be implemented using DataStax Enterprise as the backend.
How to build an effective omni-channel CRM & Marketing Strategy & 360 custome...Comarch
How to tackle current market trends regarding CRM & Loyalty strategies? How to be relevant, make a difference, synchronize marketing channels and profile customers. Demo screens of Comarch CRM & Marketing platform and introduction to the Loyalty 3.0 approach. How to engage and retain customers these days?
ANTS - 360 view of your customer - bigdata innovation summit 2016
Dữ liệu lớn, theo nghĩa đen, không phải là một cái gì đó mới, nó đã hiện hữu trong một thời gian dài. Bất kỳ một công ty đã kinh doanh trong một vài năm đều sở hữu một khối lượng dữ liệu từ thông tin khách hàng thông qua hồ sơ giao dịch và sử dụng sản phẩm. Những doanh nghiệp khác nhau sẽ có những khả năng khai thác và sử dụng dữ liệu lớn khác nhau.
Một số doanh nghiệp đã đạt đến độ chín muồi, trong khi đó một số doanh nghiệp khác chỉ vừa bắt đầu cuộc hành trình. Tại Việt Nam các doanh nghiệp viễn thông, các hãng hàng không, các ngân hàng, các tập đoàn bán lẻ, các cơ quan chính phủ đang sở hữu một khối lượng dữ liệu khổng lồ, tuy nhiên, việc xử lý, phân tích dữ liệu lớn còn đang trong giai đoạn rất sơ khai.
Diễn đàn dữ liệu lớn sẽ tập trung chia sẻ những kiến thức thực tế và mang tính chất ứng dụng để người tham dự có thể tích luỹ và áp dụng.
Graph in Customer 360 - StampedeCon Big Data Conference 2017StampedeCon
Enterprises typically have many data silos of partial customer data and a common theme in big data projects to use big data tools and pipelines to unify all siloed customer data into a single, queryable, platform for improving all future customer interactions. This data often comes from billing, website traffic, logistics, and marketing; all in different formats with different properties. Graph provides a way to unify all of the data into a single place for use in tracking the flow of a user through the various silos. Graph can also be used for visualizations and analytics that are difficult in other systems.
In this talk we will explore the ways in which Graph can be leveraged in a customer 360 use case. What it can add to a more conventional system and what the approach to developing a graph based Customer 360 system should be.
Organizations across diverse industries are in pursuit of Customer 360, by integrating customer information across multiple channels, systems, devices and products. Having a 360-degree view of the customer enables enterprises to improve the interaction experience, drive customer loyalty and improve retention. However delivering a true Customer 360 can be very challenging.
The 5S Approach to Performance Tuning by Chuck EzellDatavail
If your organization relies on data, optimizing the performance of your database can increase your earnings and savings. Many factors large and small can affect performance, so fine-tuning your database is essential. Performance Tuning expert and Senior Applications Tuner for Datavail, Chuck Ezell, sheds light on the right questions to get the answers that will help you move forward by using a defined approach, refered to as 5S.
This performance tuning white paper addresses each stage of this novel approach, as well as key performance issues: SQL, Space, Sessions, Statistics, and Scheduled Processes.
"It can always get worse!" – Lessons Learned in over 20 years working with Or...Markus Michalewicz
First presented during the DOAG 2022 Conference and Exhibition, this presentation discusses and reviews the most significant lessons learned in over 20 years of working with Oracle Maximum Availability Architecture. It explains why documentation is good, but automated checks are better, and why standardization can help increase the availability of nearly all systems, including database systems.
• Associate Consultant pursuing Executive MBA with 3+ years of experience in Healthcare ,Banking domain & software development, implementation in the areas of Data warehousing using IBM web sphere Data stage 8.1 tool and IBM Info Sphere Data stage 8.7,ETL Architecture, enhancement, maintenance, Production support, Data Modeling, Data profiling, Reporting including Business requirement, system requirement gathering.
Data Engineer's Lunch #60: Series - Developing Enterprise ConsciousnessAnant Corporation
In Data Engineer's Lunch #60, Rahul Singh, CEO here at Anant, will discuss modern data processing/pipeline approaches.
Want to learn about modern data engineering patterns & practices for global data platforms? A high-level overview of different types, frameworks, and workflows in data processing and pipeline design.
9 Hyperion Performance Myths and How to Debunk ThemDatavail
In this presentation, we explore and debunk 9 misconceptions of Hyperion Performance Management that are keeping you from effectively managing your platform.
Oracle Log Analytics Cloud Services solution monitors, aggregates, indexes, and analyzes all log data from your applications and infrastructure (running on-premises or in the cloud). It enables users to search, explore, and correlate this data to troubleshoot problems faster and derive operational insight to make better decisions.
Oracle Log Analytics Cloud Services solution monitors, aggregates, indexes, and analyzes all log data from your applications and infrastructure (running on-premises or in the cloud). It enables users to search, explore, and correlate this data to troubleshoot problems faster and derive operational insight to make better decisions.
IT327-1404A-01 Implementing, Managing and Mainta.docxpriestmanmable
IT327-1404A-01 Implementing, Managing and Maintaining a Network I
Phase 1 Individual Project
Michael F. Ryan Jr.
10/14/14
Contents
Week 1: Target Organization 3
Target Organization 3
Analysis and Recommendations 3
Week 2: Transmission Control Protocol/Internet Protocol (TCP/IP) Addressing and Management 4
Week 3: Fault Tolerance Plan 5
Week 4: Security Plan 6
Week 5: Network Monitoring 7
All Weeks: Network Pro Prep Toolkit 8
Week 1: Target OrganizationTarget Organization
About Ryan Computer Enterprises
Ryan Computer Enterprises formed in 2000, and provides leading business solutions. Currently there are over 8000 employees that are located throughout five states with three locations in Pennsylvania, Florida, California, Texas and New York. The administrative office currently is located in Pennsylvania. Currently there are over 500 additional employees located in the corporate headquarters. The company provides cutting edge solutions and needs USAA, JP Morgan, Goldman Sacks and other financial institutions. In 2002, Forbes magazine rated Ryan Computer Enterprises as the number one business organization for the field of technical consultation.
Major clients of Ryan Computer Enterprises are all in the field of investment banking and insurance. Some major factors of focus for the company will be improvement of business process the clientele have set in place, legacy systems and to have a set solution in place to be utilized thought their organization. Ryan Computer Enterprises has stated that our goal is align both the company and client goals.
Ryan Computer Enterprises has slowly replaced existing infrastructure over the years focusing on workstation, printer, switch and routers. Some areas of interest that go beyond the physical infrastructure are application upgrades. The applications that are in questions for upgrades are DBMS and other various apps such as Office 365. It has been over 2 years since the last upgrade to the IT system has been conducted and in order to provide better security and stability it has been decided that specific facets that have been listed must be upgraded as well as servers and the use of cloud technology must be implemented to allow for virtual storage of information. Other technology that should be visited is the addition of WAPs through out the facilities this allows the vendors and employees to use their mobile devices and other wireless items as well as the clientele.
The last item for discussion that will be addressed as an upgrade will be the addition of remote access. Having VPN capabilities allows Ryan Computer Enterprises to better facilitate troubleshooting and adds to the functionality of its employees. All email services as well as server access allow an employee to better assist with technical work as well as giving the clients the ability to conduct business wherever they might be located.
Metrics that are used to identify some of the issues are based on multiple factors but some ...
November 2014 Webinar - Disaster Recovery Worthy of a Zombie ApocalypseRapidScale
80% of companies that do not recover from a data loss within one month are likely to go out of business in the immediate future (Bernstein Crisis Management). With Disaster Recovery and Business Continuity, a business is able to survive and thrive after a disaster has struck.
Feature Store as a Data Foundation for Machine LearningProvectus
Looking to design and build a centralized, scalable Feature Store for your Data Science & Machine Learning teams to take advantage of? Come and learn from experts of Provectus and Amazon Web Services (AWS) how to!
Feature Store is a key component of the ML stack and data infrastructure, which enables feature engineering and management. By having a Feature Store, organizations can save massive amounts of resources, innovate faster, and drive ML processes at scale. In this webinar, you will learn how to build a Feature Store with a data mesh pattern and see how to achieve consistency between real-time and training features, to improve reproducibility with time-traveling for data.
Agenda
- Modern Data Lakes & Modern ML Infrastructure
- Existing and Emerging Architectural Shifts
- Feature Store: Overview and Reference Architecture
- AWS Perspective on Feature Store
Intended Audience
Technology executives & decision makers, manager-level tech roles, data architects & analysts, data engineers & data scientists, ML practitioners & ML engineers, and developers
Presenters
- Stepan Pushkarev, Chief Technology Officer, Provectus
- Gandhi Raketla, Senior Solutions Architect, AWS
- German Osin, Senior Solutions Architect, Provectus
Feel free to share this presentation with your colleagues and don't hesitate to reach out to us at info@provectus.com if you have any questions!
REQUEST WEBINAR: https://provectus.com/webinar-feature-store-as-data-foundation-for-ml-nov-2020/
4. 3
Key Assumptions
No. Assumption
1
The Data elements identified for customer 360 are the critical data elements that will be measured for data quality and data
governance.
2
Data owners and data stewards have been identified for critical data elements for customer domain within the Canadian
Division.
3 Out of the box Data Quality Information Analyzer reporting tool will be used for measuring customer critical data.
4
The Data Governance workflows will be provided as swim lane diagrams in Visio and will not be automated using AWD by the
Data Governance work stream.
5
The IBM Standard Data Model for Party, Accounts and Product will be leverage with possibility of extension up to 10-15 custom
entities. The estimated entities for this phase are around 40-45.
6
Account/Policy Data will be hosted in MDM. For this phase, Account/policy Data will not be mastered. The Account/Policy Data
is maintained with relationship to Customer.
7
MDM will house product data to support customer to product and product to line of business relationships but is not considered
master data.
8 Virtual and physical MDM will be leveraged for distinct match, merge, and persistence capabilities in a hybrid model.
9
The out of the box MDM artifacts will be the basis for any customization or configuration (matching algorithms, UI components
,and services).
10
There will be multiple points of human interaction within the MDM tool suite based on a given task (linking, data quality,
reference data management, golden record management, or product data configuration).
11
Customer 360 will leverage existing extract files from admin systems, except for MPW where new extract process will be
developed
5. 4
Technical Risk and Mitigation
No. Risk Description
Risk
Category
Mitigation Strategy
1
Quality of data is not being high enough to effectively match and
merge client data before exposing it to the client via the
Customer Portal and CCT.
High
Execute Data Profiling, Remediation
and Monitoring
2
Access to unmasked data for the purposes of effectively profiling
data, data quality and identifying fields to match on, and for
respective probability and thresholds.
High
Identity individuals who need access
to unmasked data and restrict access
to others.
3
Aggregating entire Customer data from across the Canadian
division in one place increases risk of Confidential Customer data
being lost, stolen and exposed in case of a Security Breach.
High
Implementing high Security measures
and protocols to protect data e.g.
using SFTP and HTTPS.
4
Several Technology components will be implemented for the first
time at Manulife, raising a risk of integration challenges which
can lead to schedule and cost impacts
High
Conduct a Proof of Technology for
Customer 360 architecture
components
6. 5
Technical/Non-Functional Requirements
No. Criteria Requirement Criteria Count
1 Availability
Criteria for determining the availability of the system for service when required
by the end users.
6
2 Maintainability
Criteria pertaining to the ease of maintenance of the system with respect to
needed replacement of technology and rectifying defects.
1
3 Operability The criteria related to the day to day ease of Operation of the System. 2
4 Performance Criteria related to the speed and response of the system. 4
5 Recoverability
Criteria to determine how soon the system would recover to the original state
after a failure.
8
6 Scalability
Criteria to determine the ability of the system to increase throughput under
increased load when additional resources are added.
2
7 Security
Criteria to determine Security measures to protect the system from internal and
external threats.
4
7. 6
Technical/Non-Functional Requirements
Req No.
Work
Stream
Requirement Owner Status
1 MDM
Q. Determine what time the data will be available ?
A. The data should be cleansed, matched and synchronized with Salesforce by 7am.
Jamie Complete
2 ETL
Q. Determine when do the source files arrive by?
A. All source files expected to arrive by 5am.
Steven Complete
3 Web Service
Q. Determine how often will ePresentment pull data from ePresentment stage?
A. Large runs monthly and annually. Otherwise, small runs occurring nightly for notices,
letters, confirms, etc.
Jamie Complete
4 ETL
Q. Determine does ePresentment stage expect full data everyday?
A. ePresentment would expect full data everyday. Deltas would suffice, but any document
delivery preference changes during the day should be reflected in the ePreferences staging
database.
Jamie Complete
5 ETL
Q. Determine when something fails within ETL, how soon should someone be notified of the
errors?
A. Recommendation : ETL error notification should be sent at least once a day.
Steven Open
6 IVR
Q. Determine is it ok for the IVR vocal password to be down for the day?
A. 12 hours to align with overnight batch schedules.
Jamie Open
Availability
8. 7
Technical/Non-Functional Requirements
Req No.
Work
Stream
Requirement Owner Status
1 ETL
Q. Determine how are deltas being identified / captured within the current processing?
A.. For Dataphile, the source and target files are being compared to capture the deltas. For
the other source files the process is still to be determined.
Vishal/
Steven
Open
Maintainability
Operability
Req No.
Work
Stream
Requirement Owner Status
1 ETL
Q. What is the existing File validation processes?
Jamie Open
2 ETL
Q. Determine what happens when the source files per system do not arrive on time?
A. Given that 360 should process files as they come in, there should not be holding of any
files.
Jamie Complete
9. 8
Technical/Non-Functional Requirements
Performance
Req No.
Work
Stream
Requirement Owner Status
1 ETL
Q. Determine if all the files are processed as the file arrives or is there a queue in process?
A. The files should be processed as they arrive, preference for a real-time processing option.
However if there are cost or delivery date related issues, then files will be processed when
all files are available, or at an arbitrary time.
Jamie Open
2 ETL
Q. Determine what volume are we expecting from each of the source systems (initial and
incremental)?
Steven Open
3 Tech Arch
Q. Determine what is the daily expected volume from each of the sources/within
ePresentment stage?
A. Total of new customers + preference changes. Expect less than 5000 per day, ongoing.
(Rough estimate)
Jamie Complete
4 Tech Arch
Q. Determine what are the archiving requirements for the ePresentment stage?
A. Archiving requirements are not necessary for ePresentment stage. It is not source of
record. Data is primarily transient; being staged for performance reasons.
Jamie Complete
10. 9
Technical/Non-Functional Requirements
Req No.
Work
Stream
Requirement Owner Status
1 Tech Arch
Q. Determine the fail over time?
A. Fail over to alternate site should be immediate; utilizing a cross-data center clustered
WAS architecture with our global load balancer.
Jamie Complete
2 Tech Arch
Q. Determine the data loss time?
A. For data where 360 is not source of record, 24 hour data loss is probably acceptable as
data can be re-run from the admin systems. For data where 360 is the source of record
(preferences, for example) then acceptable data loss is very small. However would still
probably be hours’ worth, given that Salesforce would capture the data and presumably we
can resend the messages.
Jamie Complete
3 Tech Arch
Q. Determine what happens when the virtual MDM is lost? How soon can it be recovered?
A. Virtual repository would have to be back up within a 24 hour period.
Jamie Complete
4 Tech Arch
Q. How often will the system be backed up?
A. System should be backed up nightly. Tape backup would be the option.
Jamie Complete
5 Tech Arch
Q. Who will be responsible for the database back up?
A. DBAs would be responsible for the database backup.
Jamie Complete
6 Tech Arch
Q. What data must be saved in case of a disaster?
A. Recommendation:
Jamie Open
7 Tech Arch
Q. How quickly after a major disaster must the system be up and running?
A. System should be back up and running within 24 hours after disaster.
Jamie Complete
8 Tech Arch
Q. What is the acceptable system downtime per 24-hour period?
A. Acceptable system downtime would largely be driven by dependent systems.
Jamie Open
Disaster Recovery and Business Continuity
11. 10
Technical/Non-Functional Requirements
Req No.
Work
Stream
Requirement Owner Status
1 Tech Arch
Q. Determine the expected growth rate?
A. Expected Growth Rate for the first year is 150% as result of IIS business being added
and Standard Life Acquisition. Year over year, after that, it is expected that the growth rate
will be between 10% - 50%.
Jamie Complete
2 ETL
Q. Determine what files are batch / real-time?
A. All files are batch, including MPW. However, feeds should be processed as they come in.
There will be interdependencies e.g. MLIA gets data from both iFast Seg and PPlus
Jamie Complete
Scalability
12. 11
Technical/Non-Functional Requirements
Req No.
Work
Stream
Requirement Owner Status
1 MDM
Q. How should web services be secured?
A. Services being exposed by DataPower or IIB should be secured using 2-way SSL
certificates. Services on the WAS or MDM servers should be exposed to DataPower/IIB
over HTTPS and utilize a user id/password. Additionally, IP restrictions should be in place
such that the services can only be called by IIB/DataPower.
Jamie Complete
2
Data
Protection
Q. How should SIN be protected?
A. SIN number should not be visible to users of the MDM/DataQuality platforms (DGO
members). Only the last 3 digits should be stored in Salesforce.
Jamie Complete
3
Data
Protection
Q. How should Customer data who are also employees be handled?
A. Employee customer data should only be visible to those in roles that have access to staff
plan. Additionally, there are probably requirements about how board member data should
be handled and High Net Worth data.
Jamie Open
4 Tech Arch
Q. Determine if FTP/SFTP is being used in the Current State and the Future State
Requirement?
A. Current State – FTP, Future State – SFTP.
Jamie Complete
Security
14. 13
Systematic Data Flow
Admin
Systems
FTP
Server
DMO
Staging
ETL
DQ
Repository
IBM
MDM
Server
MDM
Stewardship
UI
IBM WebSphere
DataPower
WebSphere
cast Iron
Cloud
Synchronization
Source Files
Dependency
and
Scheduling
ePresentment
Contact
Center Portal
Manual
Data
Check?
Systematic Customer updates
No
Yes
1
2
3
4
5
6
7
8
9
9
10
1111
OOTB
XML
14
Dox Delivery
Preferences
Staging
Area
DQ
Stewardship
UI
Existing Process/Tools/Technology Out of Scope of Customer 360 Waiting Decisions or
Discussions
In Scope for Customer 360
IIB
16
IVR
Staging
Area
SFTP
12
Adhoc
Systems
Data Management Processes: Data Security, Metadata Management, Data Quality Management, Data Lineage
15
Reference
Data
Updates
13
13
15. 14
Systematic Data Flow Description (1 of 3)
Step
No.
Step Summary Details of the Step
1
Data Extraction from Admin
Systems
For Phase 1, existing data feeds from admin system for DMO will be leveraged for MDM There will be
additional Admin System feeds required for MDM namely “MPW” for wealth management.
2
Data Files are moved to a FTP
Server
The extracted data Flies will land on a common FTP server as depicted in the systematic data flow.
3
Source File Dependency and
Scheduling
Multiple files will arrive from some admin system which have dependencies upon each other. The CA
scheduler will wait for all the files to arrive from an admin system before calling the Informatica ETL jobs
to load the staging tables.
4
Standardization and
Transformation of the Data
Informatica PowerCenter performs a series of ETL transformations on the data within staging tables
while moving it to a Pre-DQ tables within the DQ repository. IBM Quality Stage will then perform
standardization and enrichment tasks while moving the enriched data to the finalized DQ repository
5
Standardized Data is sent to
MDM in the format OOTB XML
In this step the data will be prepared and sent to the IBM MDM Advance for ingestion into matching and
linking processes. This format will be OOTB XML interface specified by the MDM platform. The Virtual
(Initiate) part of MDM received the data. The customer updates are then matched against existing
customer records in MDM.
6
Data Stewardship UI or a
workflow to review the updates
in MDM
Few scenarios could occur such as “False Negative” or “ False Postive” where systematic match and
merger results are not fully reliable and scenarios such as updates that may have larger impact on cross-
business or cross –system level. These scenarios may need to be reviewed and approved before the
changes are applied to the system. It can be achieved by triggering a workflow for data stewards or for the
roles as defined in Data Governance Operating Model to look at the data and take necessary steps to
validate or update data changes.
16. 15
Step
No.
Step Summary Details of the Step
7 Workflow is triggered
As mentioned in the step 6, a workflow may be triggered to review and approve such scenarios.
There is an opportunity to set up Data Governance processes to review and approve very critical
scenarios.
8
Record is approved and
updated
In this step, the updates are reviewed/updated and approved and the updates are persisted in
MDM Repository .
9
Updated Data propagated to
Salesforce
As customer updates happen in MDM, relevant information is sent to Salesforce through
websphere Cast Iron Cloud Synchronization.
10
Salesforce Updates are now
available for Portal and CCT
Portal and CCT will have updated customer records from Sales Force .
11
Portal/CCT Updates to MDM
via Salesforce
For data integration from Salesforce.com, triggered events may call exposed service calls from
Salesforce to Manulife. These services would be exposed through the IBM Datapower appliance
and will be serviced by IIB (IBM integration Bus), integrating with either the staging area or Quality
Stage, in order to have the data written to the MDM platform on-demand.
12
Salesforce, CCT and Portal
updates synchronization with
Admin Systems
Any Customer updates in Salesforce that are relevant to Admin Systems will be directly sent to
Admin Systems through Systematic interface. Those change as described in steps 1 through 8 will
flow in to MDM system on a daily basis as defined in batch. There are few exceptions to this rule
such as portal username, preferences, portal updated email id and phone number, Portal Status
and so on. These exception data elements will be updated in MDM from Portal through
Salesforce.
Systematic Data Flow Description (2 of 3)
17. 16
Step
No.
Step Summary Details of the Step
13
Life Cycle Management of
Master Data/Reference Data
In Manulife Landscape, Customer updates and Reference Data updates will take place in Admin
or in some cases, in consuming systems (systems that received updates from MDM). These
updates are to be consistently applied across landscape in order to seamless movement of data.
Analysis of the information that will be hosted in MDM for this phase revealed that the frequency of
the reference data updates is minimal. Hence, automatic reference data updates across system is
ruled out. There will be an overarching process established to monitor and control life cycle
management of master data and reference data. The data governance process will be designed
based on these scenarios.
14 Updating Preferences in MDM
MDM will be the repository for customer preference data. The stored customer preferences will be
pushed into a messaging queue using XML, and picked up by Informatica PowerCenter to load
into the “Document Delivery Preferences Staging Area” for other systems (ePresentment) to
consume.
The purpose of this staging area is to avoid intensive resource consumption of MDM server when
customer preferences need to be sent to DSTO.
15
Propagating Preferences from
supplementary Database to
ePresentment
The ‘Document Delivery Preferences Staging Area’ will host printing preferences for ePresentment
and other systems to consume.
16
IVR Id will be updated from
IVR
As Voice Print Id Is updated in IVR, it will be synchronized with MDM through IBM Integration BUS
Systematic Data Flow Description (3 of 3)
20. 19
Customer 360 – Conceptual Architecture
Description
Component Functional Area Description
1 ETL
Existing data feeds from the admin system will be leveraged for Customer 360. The extracted data
Flies will land on a common FTP server and further loaded into a staging database using
Informatica PowerCenter for consumption.
2 Data Quality
Once PowerCenter transfers the data from the file(s) in to a database Staging Area. IBM
Information Analyzer and Quality Stage will parse, cleanse, enrich, and standardize the data;
based on defined business rules.
3 MDM
IBM Infosphere MDM will then ingest the data. The records are matched, merged, and ultimately
stored in the physical MDM repository where they become available for consumption by
downstream systems.
4 Softlayer
Consuming applications, proxy, and compound services will use a combination of IBM tools and
custom services (Cast Iron, IIB, DataPower, and Java Services) to access MDM repository data.
5
Dox Deliver &
Preferences Staging
The Dox delivery and preferences staging database will be loaded from MDM using PowerCenter
(ETL) to allow for quick and easy access by the customer preferences user interface.
6 Data Governance
The Data governance organization and tools will cross the customer 360 architecture and will
touch all phases of data management lifecycle.
7
Downstream
Applications
Salesforce will be the primary consumer of MDM data supporting two customer applications: CCT
and Customer Portal.
21. 20
Key Decision: Separate Customer 360 Stage
Accenture will Stage the incoming feed data 1 to 1 without applying any transformation / business
rules or filtering rules. The only logic embedded within these load processes would be file validation
checks and corresponding Exception Handling / ABC logic.
Benefits of STG table(s):
Risk
Mitigation
Scalability
Measuring
Integrity
Development
Complexity
Current operational systems will not be affected, however, existing business rules will be
leveraged for Customer 360 design.
Provides an opportunity to leverage Stage to design future projects around Data Profiling
and Data Quality Services.
Enables a true view of the source system data providing accurate Data Quality
measurements.
Replacing transformation hardcoding with business managed reference tables and
conforming data in DQ is an easier operation to be handled within DQ.
23. 22
Accenture Data Integration Framework
Accenture Data Integration Framework encompasses combination of
technical and business processes used to combine data from disparate
sources into meaningful and valuable information. A complete data
integration solution encompasses discovery, cleansing, monitoring,
transforming and delivery of data from a variety of sources.
24. 23
Requirements for File Validation, Change Data Identification, and transformation business rules will
be defined for each Admin system feed as it is loaded into the Customer 360 staging database
Acquisition
Filtering
General Rules defined for DMO will
be analyzed and leveraged when
appropriate. Additional rules will be
defined specifically to meet the
Customer 360 requirements.
Change Data Identification
Changed Data Identification (CDI) is
a process to identify data records
which have been newly created,
updated, or deleted.
Some files may contain one or more
of the following fields within the
header or record for CDI.
• Time stamps
• Status (I / U / D flags)
• Create Date
• Combination of fields / date field
Other files will be compared against
historic or current data sets to
determine the delta for CDI.
Source Data Acquisition
Existing source data interface
mappings and process will be
leveraged for Customer 360.
A copy of the existing Admin data
files will be created for Customer 360.
Admin data files will be loaded into its
own Staging (STG) table with little to
no transformations to reflect the
integrity of the supplied data feed.
Various File validation rules will be
used to verify that each Admin feed is
received with no data leakage before
proceeding with the STG process.
Supplied Data files will be
archived on the FTP Server for
determined amount of time.
SourceData
DataStaging(STG)
Acquisition
21
25. 24
Source Data Acquisition
File
Validations
Process Description Exception Type
Header
Anomalies
Instance where business date within the header is incorrect. Warning
Trailer
Anomalies
Instances where either the trailer record is missing or the count specified in
the trailer does not match the total count of records received.
Fatal / Critical Error
Missing header Instance where the header is missing. Fatal / Critical Error
File validation errors indicate issues in the source feed which may prevent processing of a source
data file. These errors will be classified as either ‘Warnings’ or ‘Fatal / Critical Errors’. These will be
handled by a defined Exception Handling and ABC (Audit Balance Control) process.
Below are some of the typical file validation processes that will be implemented:
1
26. 25
CDI Options
Change Indicators
Receive deltas
DB Minus
Pre-DQ table Match
Match against MDM
Change Data Identification (CDI)
Customer 360 receives a mix of source data files populated with only delta or changed data will
other data files contain the full contents of the customer population. Multiple strategies will be
considered when determining the appropriate changed data identification strategy to be applied for
received data file.
Description
Can the source send only deltas?
Could a hash compare be performed between the STG (from current feed) and Pre-DQ table?
Could a hash compare be performed between the STG (from current feed) and Pre-DQ table?
Does the source file provide Changed Data Indicators (I / U / D) ?
Can the source include Changed Data Indicators ?
Could ETL be used to match incoming feeds against MDM ?
2
27. 26
CDI Scenario
Database Subtraction (Minus)
This approach performs a quick database MINUS operation to identify updates / new records.
INTERSECT operation should be used in conjunction (UNION between MINUS and INTERSECT) to
capture similar records between both feeds and identify Deletes.
Perform Hash of each
record
Perform Hash of each
record
Compare Hashes to
identify I / U / D
ETL
Transformations Pre-DQ table
Pre-DQ
table(s)
STG
table(s)
Pro(s) Con(s)
Provides a simplistic approach, performance
efficiency, and ease of implementation.
None yet identified.
Does not require retaining history within Pre-DQ
table(s).
None yet identified.
2
28. 27
CDI Scenario
Hash Compare against Pre-DQ
This approach calculates a hash of each record within Pre-DQ table(s) and each record in the STG
table(s) having most recent data. Both set of hashes are them compared to identify Inserts / Updates
/ Deletes.
Perform Hash of each
record
Perform Hash of each
record
Compare Hashes to
identify I / U / D
ETL
Transformations Pre-DQ table
Pre-DQ
table(s)
STG
table(s)
Pro(s) Con(s)
Prevent from processing full load each time. Requires retaining history in Pre-DQ table(s).
NA Calculating record hash and performing hash
comparison may impact performance depending on
record volume / size.
2
29. 28
Pro(s) Con(s)
Prevent from processing full load each time. Calculating record hash and performing hash comparison
may impact performance depending on record volume /
size.
Prevents from having to store history in Pre-DQ table(s) in
addition to MDM table(s).
Increased complexity in ETL, due to completely different
data model.
Perform Hash of each
record
Perform Hash of each
record
Compare Hashes to
identify I / U / D
ETL
Transformations Pre-DQ table
STG
table(s)
MDM
table(s)
CDI Scenario
Match against MDM
This approach performs a hash of each record within MDM table(s) and each record from the
incoming feed. Both set of hashes are then compared to identify Inserts / Updates / Deletes.
2
30. 29
CDI Options
Change Indicators
Receive deltas
DB Minus
Pre-DQ table Match
Match against MDM
Change Data Identification (CDI)
Recommendation
Based on current understanding of what information the client receives from the source feeds,
Accenture recommends some of the below CDI approaches. Depending on source feed, either of the
below recommended CDI approaches would be implemented.
Description
Can the source send only deltas?
Could a hash compare be performed between the
STG (from current feed) and Pre-DQ table?
Could a hash compare be performed between the
STG (from current feed) and Pre-DQ table?
Does the source file provide Changed Data
Indicators (I / U / D) ?
Can the source include Changed Data Indicators ?
Could ETL be used to match incoming feeds
against MDM ?
Source data does not provide any Changed
Data Indicators. Hence, CDI is performed
within existing DMO processing.
The source data cannot make changes.
Source data cannot send deltas, especially
now that many difference processes use
these full feeds.
Possible option.
This option is preferred over other methodologies
due to simplistic approach, performance efficiency,
and ease of implementation.
Possible option
Performance impact based on expected volume
would need to be considered.
Avoid as much as possible.
Performance impact based on expected volume
would need to be considered.
Understanding / High Level Recommendation
2
31. 30
Exception
Handling
Existing exception handling /
warning rules for DMO
process will be leveraged.
Customer 360 exception
handling process, will
capture and log exceptions:
• As they occur during
Stage (STG) load.
• As data is pushed using
ETL to DQ.
Batch Services
Audit & Control
Source system traceability
will be enabled through:
• Audit (A): Audits will
provide traceability of data
and all the batch
processes.
• Balance (B): Enables
performing checks and
balances on the
processed records.
• Control (C): Provides
workflow management
throughout the batch.
ETL Parameters
/ Variables
Existing parameters /
variables will be analyzed to
evaluate what could be
leveraged.
Additional parameters /
variables might be defined to
simplify development efforts.
Scheduler
CA’s scheduler will enable
the orchestration of batch
program executions from
source system feeds through
MDM and SFDC.
ETL Services
Service Components
1 2 34
32. 31
What is Audit Balance Control (ABC)?
ABC will provide business users:
1. Visibility into audit information including ‘Who’ updated a record, ‘What’ changes were made
and ‘When’ the record was updated.
2. Needed metrics for operational reporting of batch processes.
3. Ability to balance back to the source feed.
Audit
• Audit provides Manulife traceability
of data and processes.
• Audit tables should be tagged with
sufficient data to:
– validate success
– assess performance
– research issues
• Enables linking each data record to
the process which created or
modified it.
Balance
• Independent comparison of checks
and sums
– checks for lost records
– checks for computation and
other processing errors
Control
• Provides workflow management to
ensure that:
– processes are run at the right
times
– exceptions are caught
– exception notification occurs
– exception recovery occurs
1
33. 32
Audit Balance Control (ABC) Methodology
Audit, Balance and Control (ABC) process will be implemented at 2 levels. At batch level,
information about the batch will be captured, while at an individual job (segment) level detailed
execution information associated with each child job will be captured.
Batch
Group
• Process will create a unique BATCH_GRP_ID in Batch Group table for every batch
initiated by scheduler or “kick-off” script with status “In Progress”.
• Ex: iFast workflow might include many processes within. Hence, iFast workflow will have
BATCH_GRP_ID = 1 and BATCH_GRP_STATUS_CD = 0 (In Progress).
Batch
Segment
• The process will create a unique BATCH_SEG_ID in Batch Segment table for every
job within the Batch Group.
• Ex: Within iFast workflow, might have 2 file validation processes, and 1 staging
process. Hence, within BATCH_GRP_ID =1 for 2 file validation processes,
BATCH_SEG_ID = 1, 2 and for 1 Staging process BATCH_SEG_ID = 3 and
SEG_STATUS_CD = 0 (In Progress).
0 = In Progress
1 = Completed 2 = Failed
1
34. 33
Audit Fields
Each Audit / Exception Handling table will capture some common set of metadata, often identified as
Audit fields.
Audit Attributes Description
Create Date Standard audit date time stamp record was created.
Created By Standard audit user id that created record.
Update Date Standard audit date time stamp record was updated.
Updated By Standard audit user id that created record.
Audit Attributes (populated with every ETL Batch execution).
1
35. 34
ABC Batch Group Metadata
At batch group level, batch level metadata information related to the batch and not individual jobs
(segments) will be captured.
Business requirements / design considerations for Customer 360, might result in modification of
recommended attributes for ABC.
Attributes Description Sample Values
Batch Group ID Primary key. 1000
Batch
Descriptive
Name
Descriptive name of the job package. Runs iFAST MMF process
Batch Name Job name of the job package. Wf_iFastMMF_Account_STG
Start Date The date/time stamp the batch process started. 01/12/2015 21:59:55
End Date The date/time stamp the batch process ended. 1/12/2015 22:59:00
Batch Status
Code
The batch group process status code. C
MLAC Source
System ID
MLAC’s source system ID. 23
Batch Group attributes
1
36. 35
ABC Batch Segment Metadata
Column Name Description Sample Values
Segment ID Primary key for child level audit records. 1100
Batch Group ID Foreign key of Job Package (parent) table. 1000
Process Type Name Name of the type of process (STG, DQ, etc.). STAGING
File_Date Business Date of the file. 01/11/2015 09:00:00
Process Type Start Date/Time Start date/time of the process type. 01/12/2015 22:00:00
Process Type End Date/Time End date/time of the process type. 01/12/2015 22:45:00
Segment Name Exact name of the job (session / script name, etc.). s_iFastMMF_Account_STG
Source table / file Name Name of the source table. iFast_MMF_Account
Target table / file name Name of the target table. iFastMMF_Account_STG
# Re-processed Errors 2
# Read records 1000
# Inserted records 980
# Critical Error records 20
# Warning records 4
Net Balance Records (Read Ct + Error_Reprocessing_Ct) – (Insert Ct + Error Ct ). (Read Ct) – (Insert Ct + Error Ct )
Balance Indicator If Net Balance Records = 0 Then ‘Y’ Else ‘N’. ‘Y’
Segment Status Code The segment process status code. Success
AUDIT fields 4 ETL AUDIT fields (already defined). 4 ETL AUDIT fields (already defined)
Existing Fields New FieldsLegend:
1
37. 36
What is Exception Handling?
Exceptions refer to any error which occur when reading or processing a data file or mapping.
Examples of potential errors include, but are limited to the following: Incorrect data format,
duplicate values, unknown codes not defined in the business dictionaries, file issues, etc.
Exceptions may lead to data rejections and even an abort of an ETL process. Exceptions will be
capturing for reporting in order to conduct root cause, remediation options and impact analysis.
While informational / warnings are technically not errors, they will be captured for trend analysis.
The following are examples of error types, and reporting.
Warnings: Processing of record / file may continue with
a warning message but require attention. Look into the root
cause to determine whether to take action.
Informational: Not an error or a warning (acceptable
data) or processing conditions that do not impact output.
This is an informational event; no action needs to be taken.
2000
1000
1009 – SIN is non-numeric or < 9
Examples:
1010 – Active record fell off feed
Examples:
Critical: Exceptions that cause an ETL interface to abort
or violate business rules. Take action to resolve the issue.
3000
2010 – Input value is not unique
4014 – Record already exists
File Validation Errors
Examples:
2
38. 37
Exception Handling Metadata
Attributes Description Sample Values
EXCEPTION ID Unique ID for exception record. 1
BATCH ID Unique ID for the current Batch. 100
Source table / file name The name of the source table / file. iFast-MMF
MLAC System ID MLAC Source System ID. 23
Exception Code Reason Code for the exception. 2010
Exception Comments Generic description of the exception.
Value ‘23’ already present in target
table ‘iFastMMF_Account_STG’
Exception Type ID Type of Exception (E = Critical Error, W = Warning). C
Record Field Field / file column that caused the error / warning. Account_ID
Field data Incoming field / file column data associated with the critical error / warning. 22
Session Name Informatica session name. s_iFastMMF_STG
Data Stream Exception record from the source file. 1 | 100 | 23| ….n
Exception Record Status Run status of the exception record (New, Processed, Re-try failed). New
Re-process Indicator Flag to determine if error record should be re-processed or not. Y
Re-processed Date Date the error record was re-processed. 01/12/2015 12:00:00
Re-process Count # of times the exception record has been re-processed. 1
ETL AUDIT fields 4 ETL Audit fields (already defined). --
ETL Exception table(s) will be created to capture error types and codes. These tables
will be built using Accenture common standards and extended to capture further
business requirement metadata.
Existing Fields
New Fields
2
39. 38
Conceptual Data Model
Batch Group
Batch Group ID
Batch Grp Job Name
… n
Batch Segment
Batch Segment ID
Batch Seg Job Name
… n
Batch Grp ID
Exception Log
Exception ID
Batch ID
Exception Src Field
….n
Exception Code
Exception Codes
Code Description
ABC
Exception
Handling
One
Many
Exception Type
Exception Class ID
Class Description
40. 39
Exception Handling / ABC - Data Flow
Informational
Warning
Fatal / Critical
Auto Retry
Manual Err
Correction
Input File
(14 sources)
CA Scheduler (Re-start ability / Dependency)
1
Data Governance
2
1 to 1 Mapping
Transformations
Non-required
missing field
Invalid SIN
number
Missing Key
Re-ProcessErrors
4
3
Re-ProcessErrors
Extract Transform Load (ETL)
Staging
Table(s)
Pre-DQ
Table(s)
Audit &
Error Table(s)
2
Data Quality (DQ)
DQ
Table(s)
Data Cleansing
Missing
Phone
Number
Missing
Lookup
Value
Match Field
Error
N
Y
N
Y
File Validation Process
Duplicate filename
Header Anomalies
Trailer Anomalies
Missing Header
41. 40
Exception Handling / ABC - Data Flow Steps
Step
No.
Step Summary Details of the Step
1 Load Source Data. The file validation processes will be applied to each of the 13 Admin system files.
2
Determine if any
exceptions were noted
within source feeds.
2a. If no exceptions were encountered, load the records into STG tables
2b. If exceptions were encountered, fail the file and abort the process. Record the
issue within the Error table(s).
2c. Log the process details into the Audit table(s).
3
While pushing STG data to
DQ through ETL, verify if
an exception occurred.
3a. If no fatal errors were encountered, process the records, along with any
‘Informational’ or ‘Warnings’ error types through the ETL for DQ process.
3b. If fatal errors were encountered, fail the record and log the details within the
Error table(s).
3c. Log the process details into the Audit table(s).
4
To perform DQ process,
check is performed to see
if any exceptions occurred.
4a. If no fatal errors were encountered, load the records along with ‘Informational’ or
‘Warnings’ types into the final DQ tables.
4b. If fatal errors were encountered, fail the record and log the details within the Error
table(s).
4c. Log the process details into the Audit table(s).
42. 41
ETL Parameters & Variables
Parameters
Parameters represent a constant value and help retain the same value throughout the process run.
Parameters could be used to change values such as database connections or file names from job to job.
Parameters provide:
1. Portability across environments (Dev, SIT, UAT, Prod).
2. Simplification and automation of code promotion.
3. Removal of manual step of updating the database connection(s) in a session.
Variables
Variables represent a value that can be change during run-time. In addition to simplifying development,
some of the value they add include:
1. Defining of general properties for Integration Services such as email address, log file.
counts, etc.
2. Evaluating task conditions to record information for downstream job execution.
Specific parameters / variables will be defined during Gate IV / Design phase.
3
43. 42
A one-to-one design will be followed such that there is one CA Scheduler job for each unique
batch session.
All ETL batch jobs will be developed such that they are fully re-star table. Unique approach to
restarting and recovering a job will be defined depending on volume of data and other
requirements defined during Gate IV.
Below would be considered while defining the restart and recovery methodology:
1. Job dependencies.
2. Data Duplications.
3. Guaranteed delivery of data.
Scheduler4
44. 43
Source File Metadata (1 of 4)
To define File Processing Strategy and CDC Strategy for Customer 360 , a detailed understanding of
each file feed is needed. Additionally, more information will likely be needed before Gate IV (before
kicking-off Design).
Growth rate is expected to be 150% for all sources during the 1st year and 10%-50% (average of 25%) subsequent years.
Sources /
Src_Sys_ID
Files /
Physical Filename
File – Data Streams /
Record Types
Vendor File Type Avg. File
Volume
(# of Days)
Expected Growth
Rate
(150% + 25%)
Freq. Feed
Type
File Arrival
Time
iFAST-Seg Fund
(18)
Account Relationship /
MMIF_ACRL.DT
(ask developer)
NA
IFDS
(Timeshare)
Flat File
(Fixed POS)
4,725
(7 days)
14,766 Daily
(M – F)
Delta
Earliest:
11:41PM
Latest:
04:55 AM
Account /
MMIF_ACCT.DT
NA Flat File
(Fixed POS)
5,879
(8 days)
18,372 Daily
(M – F)
Delta
Holdings
(may be out of scope)
MMIF_ACCS.DT /
VENDOR IS: IFDS
NA Flat File
(Fixed POS)
4,365
(7 days)
13,641 Daily
(M – F)
Delta
iFAST-MMF
(36)
Account Relationship /
MMMF_ACRL.DT
(ask developer)
NA
IFDS
(Timeshare)
Flat File
(Fixed POS)
324
(6 days)
1,013 Daily
(M – F)
Delta
Earliest:
10:59 PM
Latest:
05:25 AM
Account /
MMMF_ACCT.DT
NA Flat File
(Fixed POS)
7,365
(7 days)
23,016 Daily
(M – F)
Delta
Holdings /
(may be out of scope)
MMMF_ACCS.DT
NA Flat File
(Fixed POS)
3,519
(6 days)
10,997 Daily
(M – F)
Delta
45. 44
Source File Metadata (2 of 4)
Sources /
Src_Sys_ID
Files /
Physical Filename
File – Data Streams /
Record Types
Vendor File Type Avg. File
Volume
(# of Days)
Expected
Growth Rate
(150% + 25%)
Freq. Feed
Type
File Arrival
Time
Dataphile-IIROC
(28)
DP_7585_DMO.DT CIFDTL, PLNDTL,
BNDHLD, EQTHLD,
FNDHLD, TRMHLD
(may be out of scope)
Broadridge
Flat File
(Fixed POS)
1,444,637
(5 days)
4,514,491 Daily
(M-F)
Full
Earliest:
03:01 AM
Latest:
07:41 AM
Dataphile-MFDA
(30)
DP_7584_DMO.DT CIFDTL, PLNDTL,
BNDHLD, EQTHLD,
FNDHLD, TRMHLD
(may be out of scope)
Flat File
(Fixed POS)
1,296,394
(4 days)
4,051,231 Daily
(M-F)
Full
Dataphile-Seg
(35)
DP_3495_DMO.DT CIFDTL, PLNDTL, FNDHLD
(may be out of scope)
Flat File
(Fixed POS)
35,149
(2 days)
109,841 Daily
(M-F)
Full
PS
(16)
Client /
pwb.pwbdpccl60.client
Customer, Customer
Address
Flat File
(Manulife
Mainframe)
Flat File
(Fixed POS)
515,409
(231 days)
1,610,653
Daily
(M-F)
Delta
Earliest:
00:44 AM
Latest:
05:08 AM
Coverage
(may be out of scope) /
Pwb.pwbdpcco.dicco60.cover
age
Coverage Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
Policy /
pwb.pwbdpcpo.dipo60.policy
Policy, Policy Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
CAPSIL 4.2
(8)
Flat File
(Manulife
Mainframe)
Flat File
(Fixed POS)
253,568
(251 days)
792,400 Daily Delta Earliest:
10:16 AM
Latest:
12:42 AM
VAPS
(37)
VAPS_DATA.txt Account, Client Manulife
Mainframe
Flat File
(Manulife
Mainframe)
392,263
(2 days)
1,225,822 Daily Full Earliest:
09:18 PM
Latest:
02:47 AM
46. 45
Source File Metadata (3 of 4)
Sources /
Src_Sys_ID
Files /
Physical Filename
File – Data Streams /
Record Types
Vendor File Type Avg. File
Volume
(# of Days)
Expected
Growth Rate
(150% + 25%)
Freq. Feed
Type
File Arrival
Time
CATS
(15)
Client Customer, Customer
Address
Manulife
Mainframe
Flat File
(Fixed POS)
1,989,064
(252 days)
6,215,825
Daily
(M-F)
Delta
Earliest
00:02 AM
Latest:
11:23 PM
Coverage
(may be out of scope)
Coverage Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
Policy Policy, Policy Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
CLAS
(14)
Client Customer, Customer
Address
Manulife
Mainframe
Flat File
(Fixed POS)
1,068,435
(252 days)
3,338,860
Daily
(M-F)
Delta
Earliest
00:44 AM
Latest:
06:05 AM
Coverage
(may be out of scope)
Coverage Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
Policy Policy, Policy Customer Flat File
(Fixed POS)
Daily
(M-F)
Delta
PPlus-GIC
(20)
PPLUSDATA.txt Plan, Customer,
Investments
(may be out of scope)
SIT Flat File
(Fixed POS)
Daily
(M-Su)
Full Earliest:
02:51 AM
Latest:
10:09 PM
PPlus-Bank DMO.DT Plan, Customer, Loan,
Retail, Term
(may be out of scope)
SIT Flat File
(Fixed POS)
Daily
(M-F)
Full
47. 46
Source File Metadata (4 of 4)
Sources /
Src_Sys_ID
Files /
Physical Filename
File – Data Streams /
Record Types
Vendor File Type Avg. File Volume
(# of Days)
Expected
Growth Rate
(150% + 25%)
Freq. Feed
Type
File Arrival
Time
MPW Customers,
Plan
NA Wealth
Manager
Oracle DB 250 customers
1000 plans
20% / yr
HR Hr.txt SQL Server DB Sometimes 0 and
sometimes few
records
150% in 1st
year due to
expected
acquisition
Daily
(M-F)
Delta Estimated:
12:00 AM
48. 47
File Processing Strategy
Business requirement calls for attempting to process file as they arrive in an attempt to keep the
process to as real-time as possible. To define file processing strategy for Customer 360, based on
existing source file feed information, an understanding of below requirements were considered.
ID Requirements Existing Process
1
Are there dependencies between files amongst source system feeds ?
For ex: Is there a dependency between files from iFast feeds and
PPlus feeds ?
There certainly exists dependencies between files amongst source system
feeds (ex: dependencies exist between iFast files and PPlus files). Other
dependencies are yet to be identified.
It is yet to be analyzed if the files needed for Customer 360 amongst
source system feeds have dependencies.
2
Are there dependencies amongst files within the same source system
feed ?
There might be dependencies amongst files from the same source feed.
Whether or not those dependencies would exist amongst files within
Customer 360’s scope, is yet to be identified.
3
If a file within a source system does not arrive on time, how does DMO
currently handle this ?
Currently, if a file does not arrive, the file watch fails that batch and all
other dependent batches. For example, if iFast-MMF Account file does not
arrive, Account Relationship and other files for iFast-MMF will not be
processed.
Operations team works with required teams to resolve issue ASAP. In the
meantime, batch status is manually updated to ‘Complete’ status.
4
If a file is not received, does the source send 2 files the next day ? Sources have been requested to only send 1 file. Hence, if iFast-MMF
was not processed, all other files next day will include records from the
day prior.
5 Does the next day’s file include records from previous day ? Yes.
6 Does the source system send all their feeds all together ? Yes.
49. 48
File Processing Strategy – Data Flow
Based on current understanding of various scenarios (described in the previous slide) and existing
file delivery agreements with source systems, our recommendation is to continue existing file
processing strategy. As we look further into file dependencies and analyze file arrival times during
Gate IV, this recommendation might go through revisions.
Source
System 1
Source
System 2
Source
System n
FTP Server
Copy files for
Customer 360
Leverage Existing Process
1
Did all files per
source system
arrive?
Audit &
Error Table(s)
2a
ETL Process
Critical File
Validation
Errors?
Y
N
Fail Entire Batch
N
Y
2c
3a
3b
3c
Watch for all files from
each source system
to arrive
2b
New Process for Customer 360
for Customer 360
File 2
File 1
Source System 1
Source System 2
File 1
for DMO
File 2
File 1
Source System 1
Source System 2
File 1
For Customer 360
50. 49
File Processing Strategy – Data Flow Description
Step
No.
Step Summary Details of the Step
1 Did the file arrive?
File Watch process evaluates if all required files for each source has been received.
Since, each source systems do no have dependencies upon each other, upon
presence of all files for the source system, ETL process for that source system will
start.
2
Did the file arrive?
Is the file valid?
2a. If any file within a source system is not received, entire batch for that source
system should fail.
2b. If all required files for the source system has been received, the batch should kick
off.
2c. Log Audit & Error table(s).
3
Are there any Critical File
Validation Errors?
3a. If any critical error was encountered within the received source files, fail the batch
and record Audit & Error table(s).
3b. If no critical errors were encountered in the received source files, proceed to ETL
Process.
3c. Log Audit & Error table(s).
51. 50
ETL Development Cycle – Single Track
Create
workspace and
project
Layout job by
arranging steps,
data sources,
and targets
Validate Job Generate job
package
Finalize project
in repository
Deploy job
artifacts
Validate JobExport Job
Deploy to next
environment
1) Developers
• Developers creates connections to repositories
and folders needed for development.
• The developer lays out the mapping (data flow)
by arranging data sources, transformations, and
targets. The developer defines the links and
configures the mapping.
• The developer tests the job until it is ready for
promotion.
• The job code is then exported in an XML format
ready to be imported to another environment’s
repository.
2) Manulife Tech
• Client team verifies and approves the job.
• The job is then moved into the test environment.
• These changes can be deployed by importing the
XML into the repository / environment.
3) Functional Team
• The functional test team, aided by job owners will
validate the results or recommend changes.
• Changes may include modifications to joins /
transformation rules or new transformations and
error conditions.
• If there are unexpected results, the process
restarts in the developer’s workspace.
4) Manulife Tech
• Once the functional test team validates the
results, the client teams promote the necessary
artifacts forward.
• The promotion process will iterate between
promote and test cycles until the artifacts reach
production.
1
1)
234
52. 51
Development Standards & Guidelines
These standards will encompass:
1. Naming conventions for ETL jobs.
2. Naming conventions for objects used within each ETL.
3. Project and project object directory structures.
4. Preferred and recommended development practices.
5. Guidelines around preferred use of re-useable components to streamline development efforts.
Accenture will be adopting Manulife guidelines and standards and may be enriched using Accenture
Methodologies and leading practices.
Development standards will be defined to capture ETL development guidelines for Customer 360
that will be applicable to design and development efforts of the project.
54. 53
Summary and Next Steps
Completed the Data Flow and Solution Architecture for Customer 360.
Gathered and documented high level business requirements and non functional requirements.
Developed the conceptual data model for customer 360.
Drafted initial MDM component architecture and customer matching rules.
Produced directional data quality profiling results on masked DMO data.
Defined the structure for the data management and governance organization structures.
Create the Detailed Design Inventory and data mapping for the design phase.
Run Data Quality profiling reports on unmasked source system data.
Develop the component dependency and design order.
Support the design and build of Technical Architecture.
Start Identifying the gaps in governance processes and defining the formal roles and processes for data governance.
Summary of Plan & Analyze Outcome
Next Steps