The data architecture of solutions is frequently not given the attention it deserves or needs. Frequently, too little attention is paid to designing and specifying the data architecture within individual solutions and their constituent components. This is due to the behaviours of both solution architects ad data architects.
Solution architecture tends to concern itself with functional, technology and software components of the solution
Data architecture tends not to get involved with the data aspects of technology solutions, leaving a data architecture gap. Combined with the gap where data architecture tends not to get involved with the data aspects of technology solutions, there is also frequently a solution architecture data gap. Solution architecture also frequently omits the detail of data aspects of solutions leading to a solution data architecture gap. These gaps result in a data blind spot for the organisation.
Data architecture tends to concern itself with post-individual solutions. Data architecture needs to shift left into the domain of solutions and their data and more actively engage with the data dimensions of individual solutions. Data architecture can provide the lead in sealing these data gaps through a shift-left of its scope and activities as well providing standards and common data tooling for solution data architecture
The objective of data design for solutions is the same as that for overall solution design:
• To capture sufficient information to enable the solution design to be implemented
• To unambiguously define the data requirements of the solution and to confirm and agree those requirements with the target solution consumers
• To ensure that the implemented solution meets the requirements of the solution consumers and that no deviations have taken place during the solution implementation journey
Solution data architecture avoids problems with solution operation and use:
• Poor and inconsistent data quality
• Poor performance, throughput, response times and scalability
• Poorly designed data structures can lead to long data update times leading to long response times, affecting solution usability, loss of productivity and transaction abandonment
• Poor reporting and analysis
• Poor data integration
• Poor solution serviceability and maintainability
• Manual workarounds for data integration, data extract for reporting and analysis
Data-design-related solution problems frequently become evident and manifest themselves only after the solution goes live. The benefits of solution data architecture are not always evident initially.
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY
A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape. Join this webinar to demystify the various architecture styles and understand how they can add value to your organization.
Data Architecture Strategies: Data Architecture for Digital TransformationDATAVERSITY
MDM, data quality, data architecture, and more. At the same time, combining these foundational data management approaches with other innovative techniques can help drive organizational change as well as technological transformation. This webinar will provide practical steps for creating a data foundation for effective digital transformation.
Data Catalogs Are the Answer – What is the Question?DATAVERSITY
Organizations with governed metadata made available through their data catalog can answer questions their people have about the organization’s data. These organizations get more value from their data, protect their data better, gain improved ROI from data-centric projects and programs, and have more confidence in their most strategic data.
Join Bob Seiner for this lively webinar where he will talk about the value of a data catalog and how to build the use of the catalog into your stewards’ daily routines. Bob will share how the tool must be positioned for success and viewed as a must-have resource that is a steppingstone and catalyst to governed data across the organization.
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how Data Architecture is a key component of an overall Enterprise Architecture for enhanced business value and success.
To take a “ready, aim, fire” tactic to implement Data Governance, many organizations assess themselves against industry best practices. The process is not difficult or time-consuming and can directly assure that your activities target your specific needs. Best practices are always a strong place to start.
Join Bob Seiner for this popular RWDG topic, where he will provide the information you need to set your program in the best possible direction. Bob will walk you through the steps of conducting an assessment and share with you a set of typical results from taking this action. You may be surprised at how easy it is to organize the assessment and may hear results that stimulate the actions that you need to take.
In this webinar, Bob will share:
- The value of performing a Data Governance best practice assessment
- A practical list of industry Data Governance best practices
- Criteria to determine if a practice is best practice
- Steps to follow to complete an assessment
- Typical recommendations and actions that result from an assessment
You Need a Data Catalog. Do You Know Why?Precisely
The data catalog has become a popular discussion topic within data management and data governance circles. A data catalog is a central repository that contains metadata for describing data sets, how they are defined, and where to find them. TDWI research indicates that implementing a data catalog is a top priority among organizations we survey. The data catalog can also play an important part in the governance process. It provides features that help ensure data quality, compliance, and that trusted data is used for analysis. Without an in-depth knowledge of data and associated metadata, organizations cannot truly safeguard and govern their data.
Join this on-demand webinar to learn more about the data catalog and its role in data governance efforts.
Topics include:
· Data management challenges and priorities
· The modern data catalog – what it is and why it is important
· The role of the modern data catalog in your data quality and governance programs
· The kinds of information that should be in your data catalog and why
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY
A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape. Join this webinar to demystify the various architecture styles and understand how they can add value to your organization.
Data Architecture Strategies: Data Architecture for Digital TransformationDATAVERSITY
MDM, data quality, data architecture, and more. At the same time, combining these foundational data management approaches with other innovative techniques can help drive organizational change as well as technological transformation. This webinar will provide practical steps for creating a data foundation for effective digital transformation.
Data Catalogs Are the Answer – What is the Question?DATAVERSITY
Organizations with governed metadata made available through their data catalog can answer questions their people have about the organization’s data. These organizations get more value from their data, protect their data better, gain improved ROI from data-centric projects and programs, and have more confidence in their most strategic data.
Join Bob Seiner for this lively webinar where he will talk about the value of a data catalog and how to build the use of the catalog into your stewards’ daily routines. Bob will share how the tool must be positioned for success and viewed as a must-have resource that is a steppingstone and catalyst to governed data across the organization.
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how Data Architecture is a key component of an overall Enterprise Architecture for enhanced business value and success.
To take a “ready, aim, fire” tactic to implement Data Governance, many organizations assess themselves against industry best practices. The process is not difficult or time-consuming and can directly assure that your activities target your specific needs. Best practices are always a strong place to start.
Join Bob Seiner for this popular RWDG topic, where he will provide the information you need to set your program in the best possible direction. Bob will walk you through the steps of conducting an assessment and share with you a set of typical results from taking this action. You may be surprised at how easy it is to organize the assessment and may hear results that stimulate the actions that you need to take.
In this webinar, Bob will share:
- The value of performing a Data Governance best practice assessment
- A practical list of industry Data Governance best practices
- Criteria to determine if a practice is best practice
- Steps to follow to complete an assessment
- Typical recommendations and actions that result from an assessment
You Need a Data Catalog. Do You Know Why?Precisely
The data catalog has become a popular discussion topic within data management and data governance circles. A data catalog is a central repository that contains metadata for describing data sets, how they are defined, and where to find them. TDWI research indicates that implementing a data catalog is a top priority among organizations we survey. The data catalog can also play an important part in the governance process. It provides features that help ensure data quality, compliance, and that trusted data is used for analysis. Without an in-depth knowledge of data and associated metadata, organizations cannot truly safeguard and govern their data.
Join this on-demand webinar to learn more about the data catalog and its role in data governance efforts.
Topics include:
· Data management challenges and priorities
· The modern data catalog – what it is and why it is important
· The role of the modern data catalog in your data quality and governance programs
· The kinds of information that should be in your data catalog and why
Describes what Enterprise Data Architecture in a Software Development Organization should cover and does that by listing over 200 data architecture related deliverables an Enterprise Data Architect should remember to evangelize.
Data Governance and Metadata ManagementDATAVERSITY
Metadata is a tool that improves data understanding, builds end-user confidence, and improves the return on investment in every asset associated with becoming a data-centric organization. Metadata’s use has expanded beyond “data about data” to cover every phase of data analytics, protection, and quality improvement. Data Governance and metadata are connected at the hip in every way possible. As the song goes, “You can’t have one without the other.”
In this RWDG webinar, Bob Seiner will provide a way to renew your energy by focusing on the valuable asset that can make or break your Data Governance program’s success. The truth is metadata is already inherent in your data environment, and it can be leveraged by making it available to all levels of the organization. At issue is finding the most appropriate ways to leverage and share metadata to improve data value and protection.
Throughout this webinar, Bob will share information about:
- Delivering an improved definition of metadata
- Communicating the relationship between successful governance and metadata
- Getting your business community to embrace the need for metadata
- Determining the metadata that will provide the most bang for your bucks
- The importance of Metadata Management to becoming data-centric
Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY
Digital Transformation is a top priority for many organizations, and a successful digital journey requires a strong data foundation. Creating this digital transformation requires a number of core data management capabilities such as MDM, With technological innovation and change occurring at an ever-increasing rate, it’s hard to keep track of what’s hype and what can provide practical value for your organization. Join this webinar to see the results of a recent DATAVERSITY survey on emerging trends in Data Architecture, along with practical commentary and advice from industry expert Donna Burbank.
Data Catalog for Better Data Discovery and GovernanceDenodo
Watch full webinar here: https://buff.ly/2Vq9FR0
Data catalogs are en vogue answering critical data governance questions like “Where all does my data reside?” “What other entities are associated with my data?” “What are the definitions of the data fields?” and “Who accesses the data?” Data catalogs maintain the necessary business metadata to answer these questions and many more. But that’s not enough. For it to be useful, data catalogs need to deliver these answers to the business users right within the applications they use.
In this session, you will learn:
*How data catalogs enable enterprise-wide data governance regimes
*What key capability requirements should you expect in data catalogs
*How data virtualization combines dynamic data catalogs with delivery
Data Architecture - The Foundation for Enterprise Architecture and GovernanceDATAVERSITY
Organizations are faced with an increasingly complex data landscape, finding themselves unable to cope with exponentially increasing data volumes, compounded by additional regulatory requirements with increased fines for non-compliance. Enterprise architecture and data governance are often discussed at length, but often with different stakeholder audiences. This can result in complementary and sometimes conflicting initiatives rather than a focused, integrated approach. Data governance requires a solid data architecture foundation in order to support the pillars of enterprise architecture. In this session, IDERA’s Ron Huizenga will discuss a practical, integrated approach to effectively understand, define and implement an cohesive enterprise architecture and data governance discipline with integrated modeling and metadata management.
Data Architecture Best Practices for Advanced AnalyticsDATAVERSITY
Many organizations are immature when it comes to data and analytics use. The answer lies in delivering a greater level of insight from data, straight to the point of need.
There are so many Data Architecture best practices today, accumulated from years of practice. In this webinar, William will look at some Data Architecture best practices that he believes have emerged in the past two years and are not worked into many enterprise data programs yet. These are keepers and will be required to move towards, by one means or another, so it’s best to mindfully work them into the environment.
Introduction to Data Management Maturity ModelsKingland
Jeff Gorball, the only individual accredited in the EDM Council Data Management Capability Model and the CMMI Institute Data Management Maturity Model, introduces audiences to both models and shares how you can choose which one is best for your needs.
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how data architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
This is Part 4 of the GoldenGate series on Data Mesh - a series of webinars helping customers understand how to move off of old-fashioned monolithic data integration architecture and get ready for more agile, cost-effective, event-driven solutions. The Data Mesh is a kind of Data Fabric that emphasizes business-led data products running on event-driven streaming architectures, serverless, and microservices based platforms. These emerging solutions are essential for enterprises that run data-driven services on multi-cloud, multi-vendor ecosystems.
Join this session to get a fresh look at Data Mesh; we'll start with core architecture principles (vendor agnostic) and transition into detailed examples of how Oracle's GoldenGate platform is providing capabilities today. We will discuss essential technical characteristics of a Data Mesh solution, and the benefits that business owners can expect by moving IT in this direction. For more background on Data Mesh, Part 1, 2, and 3 are on the GoldenGate YouTube channel: https://www.youtube.com/playlist?list=PLbqmhpwYrlZJ-583p3KQGDAd6038i1ywe
Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/)
Mr. Pollock is an expert technology leader for data platforms, big data, data integration and governance. Jeff has been CTO at California startups and a senior exec at Fortune 100 tech vendors. He is currently Oracle VP of Products and Cloud Services for Data Replication, Streaming Data and Database Migrations. While at IBM, he was head of all Information Integration, Replication and Governance products, and previously Jeff was an independent architect for US Defense Department, VP of Technology at Cerebra and CTO of Modulant – he has been engineering artificial intelligence based data platforms since 2001. As a business consultant, Mr. Pollock was a Head Architect at Ernst & Young’s Center for Technology Enablement. Jeff is also the author of “Semantic Web for Dummies” and "Adaptive Information,” a frequent keynote at industry conferences, author for books and industry journals, formerly a contributing member of W3C and OASIS, and an engineering instructor with UC Berkeley’s Extension for object-oriented systems, software development process and enterprise architecture.
Data Governance Takes a Village (So Why is Everyone Hiding?)DATAVERSITY
Data governance represents both an obstacle and opportunity for enterprises everywhere. And many individuals may hesitate to embrace the change. Yet if led well, a governance initiative has the potential to launch a data community that drives innovation and data-driven decision-making for the wider business. (And yes, it can even be fun!). So how do you build a roadmap to success?
This session will gather four governance experts, including Mary Williams, Associate Director, Enterprise Data Governance at Exact Sciences, and Bob Seiner, author of Non-Invasive Data Governance, for a roundtable discussion about the challenges and opportunities of leading a governance initiative that people embrace. Join this webinar to learn:
- How to build an internal case for data governance and a data catalog
- Tips for picking a use case that builds confidence in your program
- How to mature your program and build your data community
DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...DATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task. The opportunity in getting it right can be significant, however, as data drives many of the key initiatives in today’s marketplace: digital transformation, marketing, customer centricity, and more. This webinar will help de-mystify Data Strategy and Data Architecture and will provide concrete, practical ways to get started.
Tackling Data Quality problems requires more than a series of tactical, one-off improvement projects. By their nature, many Data Quality problems extend across and often beyond an organization. Addressing these issues requires a holistic architectural approach combining people, process, and technology. Join Nigel Turner and Donna Burbank as they provide practical ways to control Data Quality issues in your organization.
Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY
With technological innovation and change occurring at an ever-increasing rate, it’s hard to keep track of what’s hype and what can provide practical value for your organization. Join this webinar to see the results of a recent DATAVERSITY survey on emerging trends in Data Architecture, along with practical commentary and advice from industry expert Donna Burbank.
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Alan McSweeney
These notes describe a generalised data integration architecture framework and set of capabilities.
With many organisations, data integration tends to have evolved over time with many solution-specific tactical approaches implemented. The consequence of this is that there is frequently a mixed, inconsistent data integration topography. Data integrations are often poorly understood, undocumented and difficult to support, maintain and enhance.
Data interoperability and solution interoperability are closely related – you cannot have effective solution interoperability without data interoperability.
Data integration has multiple meanings and multiple ways of being used such as:
- Integration in terms of handling data transfers, exchanges, requests for information using a variety of information movement technologies
- Integration in terms of migrating data from a source to a target system and/or loading data into a target system
- Integration in terms of aggregating data from multiple sources and creating one source, with possibly date and time dimensions added to the integrated data, for reporting and analytics
- Integration in terms of synchronising two data sources or regularly extracting data from one data sources to update a target
- Integration in terms of service orientation and API management to provide access to raw data or the results of processing
There are two aspects to data integration:
1. Operational Integration – allow data to move from one operational system and its data store to another
2. Analytic Integration – move data from operational systems and their data stores into a common structure for analysis
Describes what Enterprise Data Architecture in a Software Development Organization should cover and does that by listing over 200 data architecture related deliverables an Enterprise Data Architect should remember to evangelize.
Data Governance and Metadata ManagementDATAVERSITY
Metadata is a tool that improves data understanding, builds end-user confidence, and improves the return on investment in every asset associated with becoming a data-centric organization. Metadata’s use has expanded beyond “data about data” to cover every phase of data analytics, protection, and quality improvement. Data Governance and metadata are connected at the hip in every way possible. As the song goes, “You can’t have one without the other.”
In this RWDG webinar, Bob Seiner will provide a way to renew your energy by focusing on the valuable asset that can make or break your Data Governance program’s success. The truth is metadata is already inherent in your data environment, and it can be leveraged by making it available to all levels of the organization. At issue is finding the most appropriate ways to leverage and share metadata to improve data value and protection.
Throughout this webinar, Bob will share information about:
- Delivering an improved definition of metadata
- Communicating the relationship between successful governance and metadata
- Getting your business community to embrace the need for metadata
- Determining the metadata that will provide the most bang for your bucks
- The importance of Metadata Management to becoming data-centric
Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY
Digital Transformation is a top priority for many organizations, and a successful digital journey requires a strong data foundation. Creating this digital transformation requires a number of core data management capabilities such as MDM, With technological innovation and change occurring at an ever-increasing rate, it’s hard to keep track of what’s hype and what can provide practical value for your organization. Join this webinar to see the results of a recent DATAVERSITY survey on emerging trends in Data Architecture, along with practical commentary and advice from industry expert Donna Burbank.
Data Catalog for Better Data Discovery and GovernanceDenodo
Watch full webinar here: https://buff.ly/2Vq9FR0
Data catalogs are en vogue answering critical data governance questions like “Where all does my data reside?” “What other entities are associated with my data?” “What are the definitions of the data fields?” and “Who accesses the data?” Data catalogs maintain the necessary business metadata to answer these questions and many more. But that’s not enough. For it to be useful, data catalogs need to deliver these answers to the business users right within the applications they use.
In this session, you will learn:
*How data catalogs enable enterprise-wide data governance regimes
*What key capability requirements should you expect in data catalogs
*How data virtualization combines dynamic data catalogs with delivery
Data Architecture - The Foundation for Enterprise Architecture and GovernanceDATAVERSITY
Organizations are faced with an increasingly complex data landscape, finding themselves unable to cope with exponentially increasing data volumes, compounded by additional regulatory requirements with increased fines for non-compliance. Enterprise architecture and data governance are often discussed at length, but often with different stakeholder audiences. This can result in complementary and sometimes conflicting initiatives rather than a focused, integrated approach. Data governance requires a solid data architecture foundation in order to support the pillars of enterprise architecture. In this session, IDERA’s Ron Huizenga will discuss a practical, integrated approach to effectively understand, define and implement an cohesive enterprise architecture and data governance discipline with integrated modeling and metadata management.
Data Architecture Best Practices for Advanced AnalyticsDATAVERSITY
Many organizations are immature when it comes to data and analytics use. The answer lies in delivering a greater level of insight from data, straight to the point of need.
There are so many Data Architecture best practices today, accumulated from years of practice. In this webinar, William will look at some Data Architecture best practices that he believes have emerged in the past two years and are not worked into many enterprise data programs yet. These are keepers and will be required to move towards, by one means or another, so it’s best to mindfully work them into the environment.
Introduction to Data Management Maturity ModelsKingland
Jeff Gorball, the only individual accredited in the EDM Council Data Management Capability Model and the CMMI Institute Data Management Maturity Model, introduces audiences to both models and shares how you can choose which one is best for your needs.
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
Enterprise Architecture (EA) provides a visual blueprint of the organization, and shows key interrelationships between data, process, applications, and more. By abstracting these assets in a graphical view, it’s possible to see key interrelationships, particularly as they relate to data and its business impact across the organization. Join us for a discussion on how data architecture is a key component of an overall enterprise architecture for enhanced business value and success.
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
This is Part 4 of the GoldenGate series on Data Mesh - a series of webinars helping customers understand how to move off of old-fashioned monolithic data integration architecture and get ready for more agile, cost-effective, event-driven solutions. The Data Mesh is a kind of Data Fabric that emphasizes business-led data products running on event-driven streaming architectures, serverless, and microservices based platforms. These emerging solutions are essential for enterprises that run data-driven services on multi-cloud, multi-vendor ecosystems.
Join this session to get a fresh look at Data Mesh; we'll start with core architecture principles (vendor agnostic) and transition into detailed examples of how Oracle's GoldenGate platform is providing capabilities today. We will discuss essential technical characteristics of a Data Mesh solution, and the benefits that business owners can expect by moving IT in this direction. For more background on Data Mesh, Part 1, 2, and 3 are on the GoldenGate YouTube channel: https://www.youtube.com/playlist?list=PLbqmhpwYrlZJ-583p3KQGDAd6038i1ywe
Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/)
Mr. Pollock is an expert technology leader for data platforms, big data, data integration and governance. Jeff has been CTO at California startups and a senior exec at Fortune 100 tech vendors. He is currently Oracle VP of Products and Cloud Services for Data Replication, Streaming Data and Database Migrations. While at IBM, he was head of all Information Integration, Replication and Governance products, and previously Jeff was an independent architect for US Defense Department, VP of Technology at Cerebra and CTO of Modulant – he has been engineering artificial intelligence based data platforms since 2001. As a business consultant, Mr. Pollock was a Head Architect at Ernst & Young’s Center for Technology Enablement. Jeff is also the author of “Semantic Web for Dummies” and "Adaptive Information,” a frequent keynote at industry conferences, author for books and industry journals, formerly a contributing member of W3C and OASIS, and an engineering instructor with UC Berkeley’s Extension for object-oriented systems, software development process and enterprise architecture.
Data Governance Takes a Village (So Why is Everyone Hiding?)DATAVERSITY
Data governance represents both an obstacle and opportunity for enterprises everywhere. And many individuals may hesitate to embrace the change. Yet if led well, a governance initiative has the potential to launch a data community that drives innovation and data-driven decision-making for the wider business. (And yes, it can even be fun!). So how do you build a roadmap to success?
This session will gather four governance experts, including Mary Williams, Associate Director, Enterprise Data Governance at Exact Sciences, and Bob Seiner, author of Non-Invasive Data Governance, for a roundtable discussion about the challenges and opportunities of leading a governance initiative that people embrace. Join this webinar to learn:
- How to build an internal case for data governance and a data catalog
- Tips for picking a use case that builds confidence in your program
- How to mature your program and build your data community
DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...DATAVERSITY
Developing a Data Strategy for your organization can seem like a daunting task. The opportunity in getting it right can be significant, however, as data drives many of the key initiatives in today’s marketplace: digital transformation, marketing, customer centricity, and more. This webinar will help de-mystify Data Strategy and Data Architecture and will provide concrete, practical ways to get started.
Tackling Data Quality problems requires more than a series of tactical, one-off improvement projects. By their nature, many Data Quality problems extend across and often beyond an organization. Addressing these issues requires a holistic architectural approach combining people, process, and technology. Join Nigel Turner and Donna Burbank as they provide practical ways to control Data Quality issues in your organization.
Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY
With technological innovation and change occurring at an ever-increasing rate, it’s hard to keep track of what’s hype and what can provide practical value for your organization. Join this webinar to see the results of a recent DATAVERSITY survey on emerging trends in Data Architecture, along with practical commentary and advice from industry expert Donna Burbank.
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Alan McSweeney
These notes describe a generalised data integration architecture framework and set of capabilities.
With many organisations, data integration tends to have evolved over time with many solution-specific tactical approaches implemented. The consequence of this is that there is frequently a mixed, inconsistent data integration topography. Data integrations are often poorly understood, undocumented and difficult to support, maintain and enhance.
Data interoperability and solution interoperability are closely related – you cannot have effective solution interoperability without data interoperability.
Data integration has multiple meanings and multiple ways of being used such as:
- Integration in terms of handling data transfers, exchanges, requests for information using a variety of information movement technologies
- Integration in terms of migrating data from a source to a target system and/or loading data into a target system
- Integration in terms of aggregating data from multiple sources and creating one source, with possibly date and time dimensions added to the integrated data, for reporting and analytics
- Integration in terms of synchronising two data sources or regularly extracting data from one data sources to update a target
- Integration in terms of service orientation and API management to provide access to raw data or the results of processing
There are two aspects to data integration:
1. Operational Integration – allow data to move from one operational system and its data store to another
2. Analytic Integration – move data from operational systems and their data stores into a common structure for analysis
Efforts to improve computer software have led to the general use of certain methodologies, such as the Agile System Development Lifecycle, that are extremely focused on software coding. Even common technologies used for Big Data analytics, such as Hadoop and commodity disc storage, require additional programmer attention to implement capabilities that used to be handled by relational database management systems and (the) smart disc. How are organizations that are successfully “data driven” changing to focus on data-centric development?
Decoding the Role of a Data Engineer.pdfDatavalley.ai
A data engineer is a crucial player in the field of big data. They are responsible for designing, building, and maintaining the systems that manage and process vast amounts of data. This requires a unique combination of technical skills, including programming, database management, and data warehousing. The goal of a data engineer is to turn raw data into valuable insights and information that can be used to support decision-making and drive business outcomes.
The slide aids to understand and provide insights on the following topics,
* Overview for Data Science
* Definition of Data and Information
* Types of Data and Representation
* Data Value Chain - [ Data Acquisition; Data Analysis; Data Curating; Data Storage; Data Usage ]
* Basic concepts of Big Data
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?DATAVERSITY
With technological innovation and change occurring at an ever-increasing rate, it’s hard to keep track of what’s hype and what can provide practical value for your organization. Join this webinar to see the results of a recent DATAVERSITY survey on emerging trends in data architecture, along with practical commentary and advice from industry expert Donna Burbank.
Data-Ed Webinar: Data Modeling FundamentalsDATAVERSITY
Every organization produces and consumes data. Because data is so important to day to day operations, data trends are hitting the mainstream and businesses are adopting buzzwords such as Big Data, NoSQL, data scientist, etc., to seek solutions for their fundamental issues. Few realize that the importance of any solution, regardless of platform or technology, relies on the data model supporting it. Data modeling is not an optional task for an organization’s data effort. It is a vital activity that supports the solutions driving your business.
This webinar will address fundamental data modeling methodologies, as well as trends around the practice of data modeling itself. We will discuss abstract models and entity frameworks, as well as the general shift from data modeling being segmented to becoming more integrated with business practices.
Learning Objectives:
How are anchor modeling, data vault, etc. different and when should I apply them?
Integrating data models to business models and the value this creates
Application development (Data first, code first, object first)
Data Con LA 2022 - Self-Service Success and Data ProductsData Con LA
Chirag Katbamna, Senior Manager, Accenture
We have grown past the traditional reporting off of centralized EDW data store. Each department wants to be empowered to do their own analytics, but this creates new challenges in areas of governance, security, access and monitoring. How do we do this the right way? - Explore the concept of Data Mesh - Explore what is Data Product - Explore how to implement successful Self-Service - Pushing the limits of new capabilities
Five Things to Consider About Data Mesh and Data GovernanceDATAVERSITY
Data mesh was among the most discussed and controversial enterprise data management topics of 2021. One of the reasons people struggle with data mesh concepts is we still have a lot of open questions that we are not thinking about:
Are you thinking beyond analytics? Are you thinking about all possible stakeholders? Are you thinking about how to be agile? Are you thinking about standardization and policies? Are you thinking about organizational structures and roles?
Join data.world VP of Product Tim Gasper and Principal Scientist Juan Sequeda for an honest, no-bs discussion about data mesh and its role in data governance.
Big Data Analytics Architecture Powerpoint Presentation SlidesSlideTeam
"You can download this product from SlideTeam.net"
Select our content ready Big Data Analytics Architecture PowerPoint Presentation Slides to showcase the process of data curation and analysis. This big data analysis framework PowerPoint complete deck comprises of professionally designed PPT slides like conceptual view of big data reference, different types of data, important aspects, unified information management, real-time analytics, intelligent process, architecture principles, all forms of data, consistent information and object model, integrated analysis, insight to action, etc. Demonstrate how to connect information from different areas using the big data management presentation design. Big data analytics framework presentation deck also goes well with various related topics such as big data processing, data science, data warehouse, data storage, data analysis, data virtualization, modern data architecture and many more. Data analytics platform PPT design is a helpful tool to simplify the data discovery process. Showcase unified approach of data management with ready to use big data storage architecture PowerPoint template. Address feelings of inferiority with our Big Data Analytics Architecture Powerpoint Presentation Slides. Enhance the confidence they have in their ability. https://bit.ly/3sLV07g
Conceptual vs. Logical vs. Physical Data ModelingDATAVERSITY
A model is developed for a purpose. Understanding the strengths of each of the three Data Modeling types will prepare you with a more robust analyst toolkit. The program will describe modeling characteristics shared by each modeling type. Using the context of a reverse engineering exercise, delegates will be able to trace model components as they are used in a common data reengineering exercise that is also tied to a Data Governance exercise.
Learning objectives:
-Understanding the role played by models
-Differentiate appropriate use among conceptual, logical, and physical data models
- Understand the rigor of the round-trip data reengineering analyses
- Apply appropriate use of various Data Modeling types
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...DATAVERSITY
Because every organization produces and propagates data as part of their day-to-day operations, data trends are becoming more and more important in the mainstream business world’s consciousness. For many organizations in various industries, though, comprehension of this development begins and ends with buzzwords: “Big Data”, “NoSQL”, “data scientist”, and so on. Few realize that any and all solutions to their business problems, regardless of platform or relevant technology, rely to a critical extent on the data model supporting them. As such, data modeling is not an optional task for an organization’s data effort, but rather a vital activity that facilitates the solutions driving your business.
Instead of the technical minutiae of data modeling, this webinar will focus on its value and practicality for your organization. In doing so, we will:
- Address fundamental data modeling methodologies, their differences and various practical applications, and trends around the practice of data modeling itself
- Discuss abstract models and entity frameworks, as well as some basic tenets for application development
- Examine the general shift from segmented data modeling to more business-integrated practices
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016Caserta
Caserta Concepts Founder and President, Joe Caserta, gave this presentation at Strata + Hadoop World 2016 in New York, NY. His session covers path-to-purchase analytics using a data lake and spark.
For more information, visit http://casertaconcepts.com/
Building the Artificially Intelligent EnterpriseDatabricks
This session looks at where we are today with data and analytics and what is needed to transition to the Artificially Intelligent Enterprise.
How do you mobilise developers to exploit what data scientists and business analysts have built? How do you align it all with business strategy to maximise business outcomes? How do you combine BI, predictive and prescriptive analytics, automation and reinforcement learning to get maximum value across the enterprise? What is the blueprint for building the artificially intelligent enterprise?
•Data and analytics – Where are we?
•Why is the journey only half-way done?
•2021 and beyond – The new era of AI usage and not just build
•The requirement – event-driven, on-demand and automated analytics
•Operationalising what you build – DataOps, MLOps and RPA
•Mobilising the masses to integrate AI into processes – what needs to be done?
•Business strategy alignment – the guiding light to AI utilisation for high reward
•Agility step change – the shift to no-code integration of AI by citizen developers
•Recording decisions, and analysing business impact
•Reinforcement-learning – transitioning to continuous reward
This is a slide deck that was assembled as a result of months of Project work at a Global Multinational. Collaboration with some incredibly smart people resulted in content that I wish I had come across prior to having to have assembled this.
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...Denodo
Watch full webinar here: https://bit.ly/3xj6fnm
Presented at Chief Data Officer Live 2021 A/NZ
The world is changing faster than ever. And for companies to compete and succeed they need to be agile in order to respond quickly to market changes and emerging opportunities. Data plays an integral role in achieving this business agility. However, given the complex nature of the enterprise data architecture finding and analysing data is an increasingly challenging task. Data virtualization is a modern data integration technique that integrates data in real-time, without having to physically replicate it.
Watch on-demand this session to understand what data virtualization is and how it:
- Delivers data in real-time, and without replication
- Creates a logical architecture to provide a single view of truth
- Centralises the data governance and security framework
- Democratises data for faster decision making and business agility
Similar to Data Architecture for Solutions.pdf (20)
Solution Architecture and Solution Estimation.pdfAlan McSweeney
Solution architects and the solution architecture function are ideally placed to create solution delivery estimates
Solution architects have the knowledge and understanding of the solution constituent component and structure that is needed to create solution estimate:
• Knowledge of solution options
• Knowledge of solution component structure to define a solution breakdown structure
• Knowledge of available components and the options for reuse
• Knowledge of specific solution delivery constraints and standards that both control and restrain solution options
Accurate solution delivery estimates are need to understand the likely cost/resources/time/options needed to implement a new solution within the context of a range of solutions and solution options. These estimates are a key input to investment management and making effective decisions on the portfolio of solutions to implement. They enable informed decision-making as part of IT investment management.
An estimate is not a single value. It is a range of values depending on a number of conditional factors such level of knowledge, certainty, complexity and risk. The range will narrow as the level of knowledge and uncertainty decreases
There is no easy or magic way to create solution estimates. You have to engage with the complexity of the solution and its components. The more effort that is expended the more accurate the results of the estimation process will be. But there is always a need to create estimates (reasonably) quickly so a balance is needed between effort and quality of results.
The notes describe a structured solution estimation process and an associated template. They also describe the wider context of solution estimates in terms of IT investment and value management and control.
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...Alan McSweeney
This analysis seeks to validate published COVID-19 mortality statistics using mortality data derived from general mortality statistics, mortality estimated from population size and mortality rates and death notice data
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...Alan McSweeney
This analysis looks at the changes in the numbers of priests and nuns in Ireland for the years 1926 to 2016. It combines data from a range of sources to show the decline in the numbers of priests and nuns and their increasing age profile.
This analysis consists of the following sections:
• Summary - this highlights some of the salient points in the analysis.
• Overview of Analysis - this describes the approach taken in this analysis.
• Context – this provides background information on the number of Catholics in Ireland as a context to this analysis.
• Analysis of Census Data 1926 – 2016 - this analyses occupation age profile data for priests and nuns. It also includes sample projections on the numbers of priests and nuns.
• Analysis of Catholic Religious Mortality 2014-2021 - this analyses death notice data from RIP.ie to shows the numbers of priests and nuns that have died in the years 2014 to 2021. It also looks at deaths of Irish priests and nuns outside Ireland and at the numbers of countries where Irish priests and nuns have worked.
• Analysis of Data on Catholic Clergy From Other Sources - this analyses data on priests and nuns from other sources.
• Notes on Data Sources and Data Processing - this lists the data sources used in this analysis.
IT Architecture’s Role In Solving Technical Debt.pdfAlan McSweeney
Technical debt is an overworked term without an effective and common agreed understanding of what exactly it is, what causes it, what are its consequences, how to assess it and what to do about it.
Technical debt is the sum of additional direct and indirect implementation and operational costs incurred and risks and vulnerabilities created because of sub-optimal solution design and delivery decisions.
Technical debt is the sum of all the consequences of all the circumventions, budget reduction, time pressure, lack of knowledge, manual workarounds, short-cuts, avoidance, poor design and delivery quality and decisions to remove elements from solution scope and failure to provide foundational and backbone solution infrastructure.
Technical debt leads to a negative feedback cycle with short solution lifespan, earlier solution replacement and short-term tactical remedial actions.
All the disciplines within IT architecture have a role to play in promoting an understanding of and in the identification of how to resolve technical debt. IT architecture can provide the leadership in both remediating existing technical debt and preventing future debt.
Failing to take a complete view of the technical debt within the organisation means problems and risks remained unrecognised and unaddressed. The real scope of the problem is substantially underestimated. Technical debt is always much more than poorly written software.
Technical debt can introduce security risks and vulnerabilities into the organisation’s solution landscape. Failure to address technical debt leaves exploitable security risks and vulnerabilities in place.
Shadow IT or ghost IT is a largely unrecognised source of technical debt including security risks and vulnerabilities. Shadow IT is the consequence of a set of reactions by business functions to an actual or perceived inability or unwillingness of the IT function to respond to business needs for IT solutions. Shadow IT is frequently needed to make up for gaps in core business solutions, supplementing incomplete solutions and providing omitted functionality.
Solution Architecture And Solution SecurityAlan McSweeney
This describes an approach to embedding security within the technology solution landscape. It describes a security model that encompasses the range of individual solution components up to the entire solution landscape. The solution security model allows the security status of a solution and its constituent delivery and operational components to be tracked wherever those components are located. This provides an integrated approach to solution security across all solution components and across the entire organisation topology of solutions. It allows the solution architect to validate the security of an individual solution. It enables the security status of the entire solution landscape to be assessed and recorded. Solution security is a wicked problem because there is no certainly about when the problem has been resolved and a state of security has been achieved. The security state of a solution can just be expressed along a subjective spectrum of better or worse rather than a binary true or false. Solution security can have negative consequences: prevents types of access, limits availability in different ways, restricts functionality provided, makes solution harder to use, lengthens solution delivery times, increases costs along the entire solution lifecycle, leads to loss of usability, utility and rate of use.
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Alan McSweeney
This paper describes how technologies such as data pseudonymisation and differential privacy technology enables access to sensitive data and unlocks data opportunities and value while ensuring compliance with data privacy legislation and regulations.
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Alan McSweeney
Your data has value to your organisation and to relevant data sharing partners. It has been expensively obtained. It represents a valuable asset on which a return must be generated. To achieve the value inherent in the data you need to be able to make it appropriately available to others, both within and outside the organisation.
Organisations are frequently data rich and information poor, lacking the skills, experience and resources to convert raw data into value.
These notes outline technology approaches to achieving compliance with data privacy regulations and legislation while providing access to data.
There are different routes to making data accessible and shareable within and outside the organisation without compromising compliance with data protection legislation and regulations and removing the risk associated with allowing access to personal data:
• Differential Privacy – source data is summarised and individual personal references are removed. The one-to-one correspondence between original and transformed data has been removed
• Anonymisation – identifying data is destroyed and cannot be recovered so individual cannot be identified. There is still a one-to-one correspondence between original and transformed data
• Pseudonymisation – identifying data is encrypted and recovery data/token is stored securely elsewhere. There is still a one-to-one correspondence between original and transformed data
These technologies and approaches are not mutually exclusive – each is appropriate to differing data sharing and data access use cases
The data privacy regulatory and legislative landscape is complex and getting even more complex so an approach to data access and sharing that embeds compliance as a matter of course is required.
Appropriate technology appropriately implemented and operated is a means of managing and reducing risks of re-identification by making the time, skills, resources and money necessary to achieve this unrealistic.
Technology is part of a risk management approach to data privacy. There is wider operational data sharing and data privacy framework that includes technology aspects, among other key areas. Using these technologies will embed such compliance by design into your data sharing and access facilities. This will allow you to realise value from your data successfully.
Solution architects must be aware of the need for solution security and of the need to have enterprise-level controls that solutions can adopt.
The sets of components that comprise the extended solution landscape, including those components that provide common or shared functionality, are located in different zones, each with different security characteristics.
The functional and operational design of any solution and therefore its security will include many of these components, including those inherited by the solution or common components used by the solution.
The complete solution security view should refer explicitly to the components and their controls.
While each individual solution should be able to inherit the security controls provided by these components, the solution design should include explicit reference to them for completeness and to avoid unvalidated assumptions.
There is a common and generalised set of components, many of which are shared, within the wider solution topology that should be considered when assessing overall solution architecture and solution security.
Individual solutions must be able to inherit security controls, facilities and standards from common enterprise-level controls, standards, toolsets and frameworks.
Individual solutions must not be forced to implement individual infrastructural security facilities and controls. This is wasteful of solution implementation resources, results in multiple non-standard approaches to security and represents a security risk to the organisation.
The extended solution landscape potentially consists of a large number of interacting components and entities located in different zones, each with different security profiles, requirements and concerns. Different security concerns and therefore controls apply to each of these components.
Solution security is not covered by a single control. It involves multiple overlapping sets of controls providing layers of security.
Solution Architecture And (Robotic) Process Automation SolutionsAlan McSweeney
Automation is a technology trend IT architects should be aware of and know how to respond to business requests as well as recommend automation technologies and solutions where appropriate. Automation is a bigger topic than just RPA (Robotic Process Automation).
Automation solutions, like all other technology solutions, should be subject to an architecture and design process. There are many approaches to and options for the automation of business activities. Too often automation solutions are tactical applications layered over existing business systems
The objective of all IT solutions is to automate manual business processes and their activities to a certain extent. The requirement for RPA-type applications arises in part because of automation failures within existing applications or the need to automate the interactions with or integrations between separate, possibly legacy, applications.
One of the roles of IT architecture is to always seek to take the wider architectural view and to ensure that solutions are designed and delivered within a strategic framework to avoid, as much as is practical and realistic, short-term tactical solutions and approaches that lead to an accumulation of design, operations and support debt. Tactical solutions will always play a part in the organisation’s solution landscape.
The objective of these notes is to put automation into its wider and larger IT architecture context while accepting the need for tactical approaches in some instances.
These notes cover the following topics:
• Solution And Process Automation – The Wider Technology And Approach Landscape
• Business Processes, Business Solutions And Automation
• Organisation Process Model
• Strategic And Tactical Automation
• Deciding On The Scope Of Automation
• Digital Strategy, Digital Transformation And Automation
• Specifying The Automation Solution
• Business Process Model and Notation (BPMN)
• Sample Business Process – Order To Cash
• RPA (Robotic Process Automation)
Data Profiling, Data Catalogs and Metadata HarmonisationAlan McSweeney
These notes discuss the related topics of Data Profiling, Data Catalogs and Metadata Harmonisation. It describes a detailed structure for data profiling activities. It identifies various open source and commercial tools and data profiling algorithms. Data profiling is a necessary pre-requisite activity in order to construct a data catalog. A data catalog makes an organisation’s data more discoverable. The data collected during data profiling forms the metadata contained in the data catalog. This assists with ensuring data quality. It is also a necessary activity for Master Data Management initiatives. These notes describe a metadata structure and provide details on metadata standards and sources.
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...Alan McSweeney
This document compares published COVID-19 mortality statistics for Ireland with publicly available mortality data extracted from informal public data sources. This mortality data is taken from published death notices on the web site www.rip.ie. This is used a substitute for poor quality and long-delayed officially published mortality statistics.
Death notice information on the web site www.rip.ie is available immediately and contains information at a greater level of detail than published statistics. There is a substantial lag in officially published mortality data and the level of detail is very low. However, the extraction of death notice data and its conversion into a usable and accurate format requires a great deal of processing.
The objective of this analysis is to assess the accuracy of published COVID-19 mortality statistics by comparing trends in mortality over the years 2014 to 2020 with both numbers of deaths recorded from 2020 to 2021 and the COVID-19 statistics. It compares number of deaths for the seven 13-month intervals:
1. Mar 2014 - Mar 2015
2. Mar 2015 - Mar 2016
3. Mar 2016 - Mar 2017
4. Mar 2017 - Mar 2018
5. Mar 2018 - Mar 2019
6. Mar 2019 - Mar 2020
7. Mar 2020 - Mar 2021
It focuses on the seventh interval which is when COVID-19 deaths have occurred. It combines an analysis of mortality trends with details on COVID-19 deaths. This is a fairly simplistic analysis that looks to cross-check COVID-19 death statistics using data from other sources.
The subject of what constitutes a death from COVID-19 is controversial. This analysis is not concerned with addressing this controversy. It is concerned with comparing mortality data from a number of sources to identify potential discrepancies. It may be the case that while the total apparent excess number of deaths over an interval is less than the published number of COVID-19 deaths, the consequence of COVID-19 is to accelerate deaths that might have occurred later in the measurement interval.
Accurate data is needed to make informed decisions. Clearly there are issues with Irish COVID-19 mortality data. Accurate data is also needed to ensure public confidence in decision-making. Where this published data is inaccurate, this can lead of a loss of this confidence that can exploited.
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...Alan McSweeney
This analysis looks at the potential impact that large numbers of electric vehicles could have on electricity demand, electricity generation capacity and on the electricity transmission and distribution grid in Ireland. It combines data from a number of sources – electricity usage patterns, vehicle usage patterns, electric vehicle current and possible future market share – to assess the potential impact of electric vehicles.
It then analyses a possible approach to electric vehicle charging where the domestic charging unit has some degree of decentralised intelligence and decision-making capability in deciding when to start vehicle charging to minimise electricity usage impact and optimise electricity generation usage.
The potential problem to be addressed is that if large numbers of electric cars are plugged-in and charging starts immediately when the drivers of those cars arrive home, the impact on demand for electricity will be substantial.
Operational Risk Management Data Validation ArchitectureAlan McSweeney
This describes a structured approach to validating data used to construct and use an operational risk model. It details an integrated approach to operational risk data involving three components:
1. Using the Open Group FAIR (Factor Analysis of Information Risk) risk taxonomy to create a risk data model that reflects the required data needed to assess operational risk
2. Using the DMBOK model to define a risk data capability framework to assess the quality and accuracy of risk data
3. Applying standard fault analysis approaches - Fault Tree Analysis (FTA) and Failure Mode and Effect Analysis (FMEA) - to the risk data capability framework to understand the possible causes of risk data failures within the risk model definition, operation and use
Ireland 2019 and 2020 Compared - Individual ChartsAlan McSweeney
This analysis compares some data areas - Economy, Crime, Aviation, Energy, Transport, Health, Mortality. Housing and Construction - for Ireland for the years 2019 and 2020, illustrating the changes that have occurred between the two years. It shows some of the impacts of COVID-19 and of actions taken in response to it, such as the various lockdowns and other restrictions.
The first lockdown clearly had major changes on many aspects of Irish society. The third lockdown which began at the end of the period analysed will have as great an impact as the first lockdown.
The consequences of the events and actions that have causes these impacts could be felt for some time into the future.
Analysis of Irish Mortality Using Public Data Sources 2014-2020Alan McSweeney
This describes the use of published death notices on the web site www.rip.ie as a substitute to officially published mortality statistics. This analysis uses data from RIP.ie for the years 2014 to 2020.
Death notice information is available immediately and contains information at a greater level of detail than published statistics. There is a substantial lag in officially published mortality data.
This analysis compares some data areas - Economy, Crime, Aviation, Energy, Transport, Health, Mortality. Housing and Construction - for Ireland for the years 2019 and 2020, illustrating the changes that have occurred between the two years. It shows some of the impacts of COVID-19 and of actions taken in response to it, such as the various lockdowns and other restrictions.
The first lockdown clearly had major changes on many aspects of Irish society. The third lockdown which began at the end of the period analysed will have as great an impact as the first lockdown.
The consequences of the events and actions that have causes these impacts could be felt for some time into the future.
Review of Information Technology Function Critical Capability ModelsAlan McSweeney
IT Function critical capabilities are key areas where the IT function needs to maintain significant levels of competence, skill and experience and practise in order to operate and deliver a service. There are several different IT capability frameworks. The objective of these notes is to assess the suitability and applicability of these frameworks. These models can be used to identify what is important for your IT function based on your current and desired/necessary activity profile.
Capabilities vary across organisation – not all capabilities have the same importance for all organisations. These frameworks do not readily accommodate variability in the relative importance of capabilities.
The assessment approach taken is to identify a generalised set of capabilities needed across the span of IT function operations, from strategy to operations and delivery. This generic model is then be used to assess individual frameworks to determine their scope and coverage and to identify gaps.
The generic IT function capability model proposed here consists of five groups or domains of major capabilities that can be organised across the span of the IT function:
1. Information Technology Strategy, Management and Governance
2. Technology and Platforms Standards Development and Management
3. Technology and Solution Consulting and Delivery
4. Operational Run The Business/Business as Usual/Service Provision
5. Change The Business/Development and Introduction of New Services
In the context of trends and initiatives such as outsourcing, transition to cloud services and greater platform-based offerings, should the IT function develop and enhance its meta-capabilities – the management of the delivery of capabilities? Is capability identification and delivery management the most important capability? Outsourced service delivery in all its forms is not a fire-and-forget activity. You can outsource the provision of any service except the management of the supply of that service.
The following IT capability models have been evaluated:
• IT4IT Reference Architecture https://www.opengroup.org/it4it contains 32 functional components
• European e-Competence Framework (ECF) http://www.ecompetences.eu/ contains 40 competencies
• ITIL V4 https://www.axelos.com/best-practice-solutions/itil has 34 management practices
• COBIT 2019 https://www.isaca.org/resources/cobit has 40 management and control processes
• APQC Process Classification Framework - https://www.apqc.org/process-performance-management/process-frameworks version 7.2.1 has 44 major IT management processes
• IT Capability Maturity Framework (IT-CMF) https://ivi.ie/critical-capabilities/ contains 37 critical capabilities
The following model has not been evaluated
• Skills Framework for the Information Age (SFIA) - http://www.sfia-online.org/ lists over 100 skills
Critical Review of Open Group IT4IT Reference ArchitectureAlan McSweeney
This reviews the Open Group’s IT4IT Reference Architecture (https://www.opengroup.org/it4it) with respect to other operational frameworks to determine its suitability and applicability to the IT operating function.
IT4IT is intended to be a reference architecture for the management of the IT function. It aims to take a value chain approach to create a model of the functions that IT performs and the services it provides to assist organisations in the identification of the activities that contribute to business competitiveness. It is intended to be an integrated framework for the management of IT that emphasises IT service lifecycles.
This paper reviews what is meant by a value-chain, with special reference to the Supply Chain Operations Reference (SCOR) model (https://www.apics.org/apics-for-business/frameworks/scor). the most widely used and most comprehensive such model.
The SCOR model is part of wider set of operations reference models that describe a view of the critical elements in a value chain:
• Product Life Cycle Operations Reference model (PLCOR) - Manages the activities for product innovation and product and portfolio management
• Customer Chain Operations Reference model (CCOR) - Manages the customer interaction processes
• Design Chain Operations Reference model (DCOR) - Manages the product and service development processes
• Managing for Supply Chain Performance (M4SC) - Translates business strategies into supply chain execution plans and policies
It also compares the IT4IT Reference Architecture and its 32 functional components to other frameworks that purport to identify the critical capabilities of the IT function:
• IT Capability Maturity Framework (IT-CMF) https://ivi.ie/critical-capabilities/ contains 37 critical capabilities
• Skills Framework for the Information Age (SFIA) - http://www.sfia-online.org/ lists over 100 skills
• European e-Competence Framework (ECF) http://www.ecompetences.eu/ contains 40 competencies
• ITIL IT Service Management https://www.axelos.com/best-practice-solutions/itil
• COBIT 2019 https://www.isaca.org/resources/cobit has 40 management and control processes
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020Alan McSweeney
This analysis seeks to determine if there are excess deaths that occurred in Ireland in the interval Jan – Jun 2020 that can be attributed to COVID-19. Excess deaths means deaths in excess of the number of expected deaths plus the number of deaths directly attributed to COVID-19. On the other hand a deficiency of deaths would occur when the number of expected deaths plus the number of deaths directly attributed to COVID-19 is less than the actual deaths.
This analysis uses number of deaths taken from the web site RIP.ie to generate an estimate of the number of deaths in Jan – Jun 2020 in the absence of any other official source. The last data extract from the RIP.ie web site was taken on 3 Jul 2020.
The analysis uses historical data from RIP.ie from 2018 and 2019 to assess its accuracy as a data source.
The analysis then uses the following three estimation approaches to assess the excess or deficiency of deaths:
1. The pattern of deaths in 2020 can be compared to previous comparable year or years. The additional COVID-19 deaths can be added to the comparable year and the difference between the expected, actual from RIP.ie and actual COVID-19 deaths can be analysed to generate an estimate of any excess or deficiency.
2. The age-specific mortality rates described on page 16 can be applied to estimates of population numbers to generates an estimate of expected deaths. This can be compared to the actual RIP.ie and actual COVID-19 deaths to generate an estimate of any excess or deficiency.
3. The range of death rates per 1,000 of population as described in Figure 10 on page 16 can be applied to estimates of population numbers to generates an estimate of expected deaths. This can be compared to the actual RIP.ie and actual COVID-19 deaths to generate an estimate of any excess or deficiency.
This presentation describes systematic, repeatable and co-ordinated approach to agile solution architecture and design. It is intended to describe a set of practical steps and activities embedded within a framework to allow an agile method to be adopted and used for solution design and delivery. This approach ensures consistency in the assessment of solution design options and in subsequent solution design and solution delivery activities. This process leads to the rapid design and delivery of realistic and achievable solutions that meet real solution consumer needs. The approach provides for effective solution decision-making. It generates options and results quickly and consistently. Implementing a framework such as this provides for the creation of a knowledgebase of previous solution design and delivery exercises that leads to an accumulated body of knowledge within the organisation.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Data Architecture for Solutions.pdf
1. Data Architecture For
Solutions
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
https://www.researchgate.net/profile/Alan-Mcsweeney
https://www.amazon.com/dp/1797567616
2. Introduction
• These notes discuss how overall organisation data
architecture can positively impact solution design and how
solution data architecture competence within solution
architecture can contribute to solution design success
January 9, 2023 2
Data
Architecture
Solution
Architecture
Solution Data
Architecture
Can
Contribute to
Common Data
Infrastructure
Tools and
Data
Standards
Can
Contribute to
Overall
Organisation
Data Quality
Can Ensure
that the Data
Aspects of
Solution
Design Are
Covered in
Solution
Designs
Can Ensure
that Solution
Data Concerns
Are Addressed
in Solution
Designs
3. Topics
• Data Architecture For Solutions
• Traditional Scope Of Data Architecture
• Solution Data Architecture
• What Do We Mean By Data Architecture?
• Data Architecture And Common Data Tooling And
Standards
• Data Design And Modelling For Solutions
January 9, 2023 3
4. Data Architecture For Solutions
• Data breathes life into solutions
• Solutions get data, use data, share data, process data and create data
• There will be many different types of data used by a solution
− Master data
− Reference data
− Input data
− Interim data
− Generated data
− Solution activity and usage data
• Any solution will consist of many different components of different types
• Solution components and their data will be deployed and operated across a solution
landscape that can span multiple zones and platforms
• Within the solution, each data type will have a different lifecycle
• The solutions within the organisation solution landscape will have both shared and
private data
− Shared - common data (master or reference) or upstream data from other solutions or data sent
downstream
− Private – data held locally within the solution
January 9, 2023 4
5. Solution And Data
• All IT solutions support, implement and operate business
processes that take data inputs, process data, generate result
and create primary and supporting data output
− Direct data outputs – what the process in intended to create
− Indirect data outputs – logs, audit trails, reports, analyses
• Data outputs are then used in different ways
− Generated results
− As a record that the work was performed
− As inputs into other processes and solutions
− To report on the operation of the process or as an audit log
• Data breathes life into and activates the static components of a
solution
• The data architecture of solutions is frequently not given the
attention it deserves or needs
January 9, 2023 5
6. Data Architecture For Solutions
• Frequently, too little attention is paid to designing and
specifying the data architecture within individual solutions
and their constituent components
• This is due to the behaviours of both solution architects ad
data architects
• Solution architecture tends to concern itself with
functional, technology and software components of the
solution
• Data architecture tends to concern itself with post-
individual solutions
January 9, 2023 6
7. Traditional Scope Of Data Architecture
January 9, 2023 7
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Data Sources
(Internal, External) Extract, Transform, Load Data Platform Access and Usage
Merge,
Aggregate,
Transform
Data
Sources
(from
Solutions)
8. Traditional Scope Of Data Architecture
January 9, 2023 8
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Merge,
Aggregate,
Transform
Scope Of Data Architecture
Data
Sources
(from
Solutions)
9. Traditional Scope Of Data Architecture
• Traditional approaches to data architecture effectively
appends or layers newer technologies on top on existing
solutions and data sources and their data structures
• Data architecture largely ignores data architectures within
individual solutions
• Data architecture needs to shift left into the domain of
solutions and their data and more actively engage with the
data dimensions of individual solutions
January 9, 2023 9
10. Traditional Scope Of Data Architecture
January 9, 2023 10
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Merge,
Aggregate,
Transform
This Is Not A Modern Data Architecture
Data
Sources
(from
Solutions)
11. Not A Modern Data Architecture
• You are fooling yourself if you
believe that enveloping
existing data solutions,
sources and structures with a
skin of modernity comprises a
data architecture
• If you put lipstick on a pig, it is
still a pig
January 9, 2023 11
12. Common and
Shared Data
Processes and
Standards
Data
Architecture
Operation,
Measurement
Data
Architecture
Review,
Improvement,
Update
Common and
Shared Data
Infrastructural
Components
Data
Solutions,
Sources and
Structures
This Is A Data Architecture
January 9, 2023 12
Data Architecture Overview
Data Management, Governance, Supporting Processes
Data Infrastructure, Storage and Operations Software, Hardware and Processes
Data Security, Protection, Compliance, Access Control, Authentication, Authorisation
Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract
Content, Unstructured Data, Records and Document Management
Master and Reference Data Management
Data Warehouse, Data Marts, Data Lakes
Data Reporting and Analytics, Visualisation Tools and Facilities
Data Discovery, Analysis, Design and Modelling
External Data Sources and Interacting Parties Data Transfer/Exchange/Integration/Publication
Metadata Data Management
Data Quality
Data Solution Design
13. Traditional Scope Of Data Architecture And Data Architecture
Gap
Data
architecture
tends not to
get involved
with the
data aspects
of
technology
solutions,
leaving a
data
architecture
gap
January 9, 2023 13
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data
Processing
Workflow
Data
Sources
(from
Solutions)
Merge,
Aggregate,
Transform
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
…
…
Data Architecture Gap Data Architecture
14. Solution Data Aspects Across The Solution Landscape
January 9, 2023 14
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Span Of Organisation Solution Landscape
Changes to Existing
Systems and Their
Data Models
15. Solution Components And Their Types
January 9, 2023 15
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Changes to Existing Systems
New Custom Developed Applications
Acquired and Customised Software
Products
System Integrations/ Data Transfers/
Exchanges
Reporting and Analysis Facilities
Sets of Installation and
Implementation Services
Information Storage Facilities
Existing Data Conversions/
Migrations
New Data Loads
Central, Distributed and
Communications Infrastructure
Cutover/ Transfer to Production And
Support
Operational Functions and Processes
Parallel Runs
Enhanced Support/ Hypercare
Sets of Maintenance, Service
Management and Support Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes, Knowledge
Management
Training and Documentation
Component Type Solution Components
16. Reduced Scope Of Traditional Solution Architecture
Scope
• The solution is the sum
of the components
needed to deliver and
operate it
• Solution architecture
tends not to concern
itself with some key
aspects of the
complete solution,
including some of
those related to data
• Solution architecture
tends to focus on
technology aspects of
a solution, omitting
business and data
facets
• The data dimensions
of other solution
components also
tends to be omitted
partially or completely
by solution
architecture
January 9, 2023 16
Complete
Solution
17. Data Related Solution Components And Their Types
January 9, 2023 17
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Changes to Existing Systems
New Custom Developed Applications
Acquired and Customised Software
Products
System Integrations/ Data Transfers/
Exchanges
Reporting and Analysis Facilities
Sets of Installation and
Implementation Services
Information Storage Facilities
Existing Data Conversions/
Migrations
New Data Loads
Central, Distributed and
Communications Infrastructure
Cutover/ Transfer to Production And
Support
Operational Functions and Processes
Parallel Runs
Enhanced Support/ Hypercare
Sets of Maintenance, Service
Management and Support Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes, Knowledge
Management
Training and Documentation
Component Type Solution Components
18. Data Dimensions Of Solution Component Types
January 9, 2023 18
Changes to
Existing
Systems
Possible new data stores
and their design and data
models
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
New Custom
Developed
Applications
New data stores and their
design and data models
Data tools
Data processes
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data design
Metadata design
Data quality design
Acquired
and
Customised
Software
Products
New data stores and their
design and data models
Data tools
Data processes
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
System
Integrations/
Data
Transfers/
Exchanges
Source and target data
stores and their design
and data models
Data transformations and
aggregations
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
Reporting
and Analysis
Facilities
Reporting data stores and
their design and data
models
Reporting tools
Reporting processes
Data security, encryption
and access control
Master and reference
data
Metadata
Data quality
Information
Storage
Facilities
Performance, capacity
and throughput
Data management and
governance
Existing Data
Conversions/
Migrations
Source and target data
stores and their design
and data models
Data transformations and
aggregations
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
New Data
Loads
Target data stores and
their design and data
models
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
19. Data Dimensions Of Solution Component Types
• The following solution component types involve data
architecture and design activities:
− Changes to Existing Systems
− New Custom Developed Applications
− Acquired and Customised Software Products
− System Integrations/ Data Transfers/ Exchanges
− Reporting and Analysis Facilities
− Information Storage Facilities
− Existing Data Conversions/ Migrations
− New Data Loads
• The components of each of these types will potentially involve
data and therefore data design work across a range of areas
that will need to be included in the solution design and
subsequent solution implementation activities
• Solution architecture can fail t include some of the data
dimensions of the solution components of these types
January 9, 2023 19
20. Solution Components And Their Types
• Any technology solution will consist of a potentially large
number of components, each of a give type
• Each solution component type belongs to one of three
classes
1. Time-Bounded Delivery Entity Types
• Time-bounded solution component types required to get the solution fully
operational
2. Enduring Functional and Operational Technology Entity Types
• Operational instrumentation and functional component types required for
the solution to operate and be usable by its target consumers
3. Enduring Organisational, Process, Procedure and Structural
Entity Types
• Organisation and process changes and other supporting activities and sets
of effort required to use the solution optimally
January 9, 2023 20
21. Solution Components Classes And Types
January 9, 2023 21
Solution Component Classes
and Types
Time-Bounded Delivery
Entity Types
Sets of Installation and
Implementation Services
Existing Data Conversions/
Migrations
New Data Loads
Parallel Runs
Enhanced Support/ Hypercare
Enduring Functional and
Operational Technology
Entity Types
Changes to Existing Systems
New Custom Developed
Applications
Acquired and Customised Software
Products
System Integrations/ Data
Transfers/ Exchanges
Reporting and Analysis Facilities
Information Storage Facilities
Central, Distributed and
Communications Infrastructure
Application Hosting and
Management Services
Enduring Organisational,
Process, Procedure and
Structural Entity Types
Cutover/ Transfer to Production
And Support
Operational Functions and
Processes
Sets of Maintenance, Service
Management and Support Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes,
Knowledge Management
Training and Documentation
22. Solution Architecture Data Gap
• Combined with the gap where data architecture tends not to
get involved with the data aspects of technology solutions,
there is also frequently a solution architecture data gap
• Solution architecture also frequently omits the detail of data
aspects of solutions across the various components of the
types:
− Changes to Existing Systems
− New Custom Developed Applications
− Acquired and Customised Software Products
− System Integrations/ Data Transfers/ Exchanges
− Reporting and Analysis Facilities
− Information Storage Facilities
− Existing Data Conversions/ Migrations
− New Data Loads
January 9, 2023 22
23. Solution Architecture Data Gap
• These gaps result in a data blind spot for the organisation
January 9, 2023 23
Central,
Distributed and
Communications
Infrastructure
Changes to
Existing Systems
New Custom
Developed
Applications
Information
Storage Facilities
Acquired and
Customised
Software Products
System
Integrations/ Data
Transfers/
Exchanges
Changes to
Existing Business
Processes
New Business
Processes
Organisational
Changes,
Knowledge
Management
Reporting and
Analysis Facilities
Existing Data
Conversions/
Migrations
New Data Loads
Training and
Documentation
Sets of
Installation and
Implementation
Services
Operational
Functions and
Processes
Parallel Runs
Cutover/ Transfer
to Production
Sets of
Maintenance,
Service
Management and
Support Services
Application
Hosting and
Management
Services
Enhanced
Support/
Hypercare
Data Architecture
Solution Gap Data Architecture
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data
Processing
Workflow
Merge,
Aggregate,
Transform
…
Solution Architecture
Data Gap
Solution Architecture
24. Solution Architecture Data Gap
• Data architecture can provide the lead in sealing these data
gaps through a shift-left of its scope and activities as well
providing standards and common data tooling for solution
data architecture
January 9, 2023 24
25. Shift Left Of The Scope Of Data Architecture
January 9, 2023 25
Source
Systems
Extract,
Transform,
Load
Data
Platform
Access
and
Usage
Changes to
Existing
Systems
and Their
Data
Models
New Custom
Developed
Applications
and Their
Data Models
Acquired and
Customised
Software
Products and
Their Data
Models
System
Integrations
/ Data
Transfers/
Exchanges
Reporting
and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions
/ Migrations
New Data
Loads
• Shifting data architecture to
the left means getting involved
in the data aspects of solution
design, specification, selection
and implementation at the
earliest opportunity
• This then needs to be
repeated for each solution
within the organisation
solution landscape
• The data aspects of solutions
should be closely integrated
within the organisation’s data
architecture
Shift Left of Scope of Data
Architecture
26. Generalised Data Lifecycle
• Each data type within a
solution will have a lifecycle
from design and creation to
ultimate archival and
possible deletion
January 9, 2023 26
Enter, Create, Acquire,
Derive, Update,
Integrate, Capture
Secure, Store,
Replicate and
Distribute
Preserve, Protect and
Recover
Archive and Recall
Delete/Remove
Implement Underlying
Technology
Architect, Budget,
Plan, Design and
Specify
Present, Report,
Analyse, Model
• A set of data lifecycle view
for solutions can assist in
solution data architecture
27. Generalised Data Lifecycle Stages
• Architect, Budget, Plan, Design and Specify - This relates to the design and specification of the data
storage and management and their supporting processes
− This establishes the data management framework
• Implement Underlying Technology - This is concerned with implementing the data-related hardware and
software technology components
− This relates to database components, data storage hardware, backup and recovery software, monitoring and control software and other
items
• Enter, Create, Acquire, Derive, Update, Integrate, Capture - This stage is where data originated, such as
data entry or data capture and acquired from other systems or sources
• Secure, Store, Replicate and Distribute - In this stage, data is stored with appropriate security and access
controls including data access and update audit
− It may be replicated to other applications and distributed
• Present, Report, Analyse, Model - This stage is concerned with the presentation of information, the
generation of reports and analysis and the created of derived information
• Preserve, Protect and Recover - This stage relates to the management of data in terms of availability,
backup, recovery and retention/preservation
• Archive and Recall - This stage is where information that is no longer active but still required in archived
to secondary data storage platforms and from which the information can be recovered if required
• Delete/Remove - The stage is concerned with the deletion of data that cannot or does not need to be
retained any longer
− Data has to be able to be disposed of in a managed, systematic and auditable way
• Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer, Standards,
Governance, Fund - This is not a single stage but a set of processes and procedures that cross all stages
and is concerned with ensuring that the processes associated with each of the lifestyle stages are
operated correctly and that data assurance, quality and governance procedures exist and are operated
January 9, 2023 27
28. Solution Data Types And Lifecycles
• Every solution will
have one or more
types of data it
reads, processes
or creates
• Each data type will
have a separate
lifecycle that
reflects how it is
processed and
how its attributes
need to be
reflected in its
governance and
management
January 9, 2023 28
Solution
Data Type 1
Data Type 2
…
Data Type N
29. Data Lifecycle Stages And Solution Component Types
January 9, 2023 29
Central, Distributed
and Communications
Infrastructure
Changes to Existing
Systems
New Custom
Developed
Applications
System Integrations/
Data Transfers/
Exchanges
Changes to Existing
Business Processes
Organisational
Changes, Knowledge
Management
Training and
Documentation
Sets of Installation
and Implementation
Services
Parallel Runs
Enhanced Support/
Hypercare
Information Storage
Facilities
Acquired and
Customised Software
Products
New Business
Processes
Reporting and
Analysis Facilities
Existing Data
Conversions/
Migrations
New Data Loads
Operational
Functions and
Processes
Cutover/ Transfer to
Production
Sets of Maintenance,
Service Management
and Support Services
Application Hosting
and Management
Services
Secure, Store,
Replicate and
Distribute
Archive and Recall
Delete/Remove
Implement Underlying
Technology
Architect, Budget,
Plan, Design and
Specify
Present, Report,
Analyse, Model
Enter, Create, Acquire,
Derive, Update,
Integrate, Capture
Preserve, Protect and
Recover
30. Data Lifecycle Stages And Solution Components
• Each stage within the lifecycle of a solution data type will be realised by a solution component
• Mapping the stages within the lifecycle of solution data types and identifying the impact on solution
component types can contribute to effective solution data architecture design
− This provides traceability to ensure the data is being handled correctly
• For example, the Preserve, Protect and Recover data stage involving solution activities such as backup
and recovery, replication, business continuity and disaster recovery may require solution components of
the types:
− Information Storage Facilities
− Sets of Maintenance, Service Management and Support Services
− Operational Functions and Processes
January 9, 2023 30
Operational
Functions and
Processes
Sets of Maintenance,
Service Management
and Support Services
Information Storage
Facilities
Preserve, Protect and
Recover
31. So, What Do We Mean By Data Architecture?
• If data architecture can contribute to solution architecture
then the scope of data architecture should be defined and
agreed to ensure this is possible
January 9, 2023 31
32. Data Architecture And Data Strategy
• Data architecture defines the target data structures,
operations, principles, standards, organisation, tools,
management, governance that the organisation is aiming
to define, implement and operate
− The data architecture is designed to be implemented and
operated
• Data strategy defines how the organisation intends to use
data to deliver on its business strategy
− Data strategy precedes and feeds into the data architecture
January 9, 2023 32
33. Data Strategy And Data Architecture In A Wider
Business And Technology Context
January 9, 2023 33
Business
Objectives
Business
Architecture
Enterprise
Architecture
Solution
Implementation
and
Delivery
Support,
Management
and
Operations
Business
Processes
Required
Operational
Business
Solutions
Business
Strategy
Business
Solution
Analysis and
Design/
Selection
Business IT
Strategy
IT Function
Strategy
Required
Operational
Processes
Required
Supporting and
Enabling
Business
Solutions
Support
Solution
Analysis and
Design/
Selection
Required
Structure,
Capabilities
and
Resources
Digital
Strategy
Digital IT
Architecture
Solution
Portfolio
Design And
Specification
Solution
Portfolio
Management
Solution
Change and
Evolution
Business
Structure and
Operational
Model
Data
Strategy
Data
Architecture
34. Data Strategy And Data Architecture In A Wider
Business And Technology Context
• Data strategy
follows from
business strategy
and business
objectives
• Data architecture
translates the
conceptual nature of
the data strategy
into a more
implementation-
specific and –
oriented view
January 9, 2023 34
Business
Architecture
Enterprise
Architecture
Required
Operational
Business
Solutions
Business
Solution
Analysis and
Design/
Selection
Business IT
Strategy
IT Function
Strategy
Digital
Strategy
Digital IT
Architecture
Data
Strategy
Data
Architecture
Business
Objectives
35. Data Architecture
• A Data Architecture
exists to support the
objectives and the
operations of the
organisation
• This includes enabling
individual functional
solutions to be
designed and
implemented in
accordance with the
wider organisation data
architecture
January 9, 2023 35
Organisation Data
Architecture
Data
Infrastructure
Tools and
Facilities
Functional
Solutions
Data Standards
36. Data Architecture Structure
• For each set of subject arears within the data architecture
design and specification process, create an activity
breakdown based on the phases
− Research, Design, Define, Plan
− Implement, Operate
− Administer, Manage, Monitor, Improve
• Data architecture cannot be separated from its
implementation, operation and subsequent measurement
and improvement
• Architecture without execution and employment is
incomplete
January 9, 2023 36
37. Data Architecture Evolution And Development
• The data architecture is not static – it must be responsive to and accommodating of change
• It needs to evolve and develop in response to:
− Changing organisation needs and direction, driven by internal and/or external demands
− Changing organisation business strategy
− New technologies and capabilities that the organisation can usefully avail of
− Experience from implementation and operation
• The architecture should embed within itself explicitly the ability to assess its implementation
and operation and to grow, change, improve in response to these factors
January 9, 2023 37
Research,
Design,
Define,
Plan
Implement,
Operate
Administer,
Manage,
Monitor,
Improve
Define
Measurement
Framework,
Results and
Performance
Indicators
Review Delivery
and Operation
of Architecture
Experience and
Lessons from
Implementation
and Operation
Changing
Organisation
Needs and
Direction
Changes to
Organisation
Business
Strategy
New Data
Technologies
and
Capabilities
Data
Architecture
Changes
38. Data Architecture Subject Areas
Data
Architecture
Data Architecture Overall data architecture and data technology standards and design and implement data infrastructural
technology solutions
Data Management, Governance,
Supporting Processes
Standards, processes and their enforcement, planning, supervision, control and usage of data resources and
the design and implementation of data management processes, data ownership
Data Infrastructure, Storage and
Operations Software, Hardware and
Processes
Infrastructure hardware and software required to store and provide access to data, either on-premises or
hosted and facilities and processes required to operate and support the infrastructure, approach to analysis,
design, implementation, testing, deployment, maintenance and data storage structures
Data Security, Protection, Compliance,
Access Control, Authentication,
Authorisation
Approach to ensuring data security and protection, designing and implementing data security model covering
data, tools and infrastructure, ensuring compliance with regulatory standards, controlling access to data,
designing and implementing data authorisation model
Data Integration, Access, Flow, Exchange,
Transfer, Transformation, Load And
Extract
Data resource integration, extraction, transformation, movement, delivery, replication, transfer, sharing,
federation, virtualisation and operational support and approach to implementing a common approach and
providing a common set of tools
Content, Unstructured Data, Records and
Document Management
Approach to the implementation and management of acquisition, storage, indexing of and access to
unstructured data resources such as files and digitised paper records and the integration of these resources
with structured data resources
Master and Reference Data Management
Approach to the implementation and management of master versions of shared data resources to reduce
redundancy and maintain data quality through standardised data definitions and use of common data lookup
values including data dictionaries
Data Warehouse, Data Marts, Data Lakes Facilities for storing data extracted from operational systems for long-term storage and to enable access for
reporting and analysis
Data Reporting and Analytics,
Visualisation Tools and Facilities
Approach to providing a common approach and providing a common set of tools, facilities and supporting
technologies and standards for data reporting, decision support, analysis and visualisation
Data Discovery, Analysis, Design and
Modelling
Approach to the implementation and management of data description standards and the collection,
categorisation, maintenance, integration, application, use and management of data descriptions including
data catalogs
External Data Sources and Interacting
Parties Data Transfer/Exchange/
Integration/Publication
Management of data sources and targets outside the organisation and the parties that provide that data or to
whim the data is made available including contracts and agreement, service levels, access approaches
Metadata Data Management
Approach to the implementation and management of data description standards and the collection,
categorisation, maintenance, integration, application, use and management of data descriptions including
data catalogs
Data Quality Designing, implementing and operating approach, processes and standards to ensure and maintain data
quality
Data Solution Design Defining and implementing standards relating to the use of data within solutions
January 9, 2023 38
39. Data Architecture Subject Areas
• This is intended to represent a comprehensive view of data
architecture
January 9, 2023 39
40. Data Architecture Subject Areas
• The proposed subject areas do not exist in isolation
• They are interrelated areas on which to focus analysis,
planning and design effort and attention while maintaining
a higher level and more complete and integrated view
• The individual topics allow each subject area to be
analysed and specified in detail that is appropriate for the
organisation
• The topics are designed to be independent of any specific
hardware, software or platform technology
January 9, 2023 40
41. Relationships Between Data Architecture Topics
January 9, 2023 41
Data Solution Design
Data Quality
External Data Sources and
Interacting Parties Data
Transfer/Exchange/
Integration/Publication
Data Discovery, Analysis,
Design and Modelling
Data Infrastructure, Storage
and Operations Software,
Hardware and Processes
Data Security, Protection,
Compliance, Access Control,
Authentication,
Authorisation
Content, Unstructured Data,
Records and Document
Management
Master and Reference Data
Management
Data Architecture
Data Management,
Governance, Supporting
Processes
Data Reporting and
Analytics, Visualisation
Tools and Facilities
Data Warehouse, Data
Marts, Data Lakes
Metadata Data
Management
Data Integration, Access,
Flow, Exchange, Transfer,
Transformation, Load And
Extract
42. Data Architecture Topic Scope
Data Architecture
Research, Design, Define, Plan
Data Architecture Strategy and Scope
Definition
Data Architecture Capability Establishment
Define Current Data Architecture Baseline,
Inventory, Gaps, Issues, Concerns
Define Architecture Supporting Tools and
Processes Definition
Data Architecture Scope and Activities
Data Architecture Strategy and Scope
Definition
Data Architecture Capability Establishment
Define Current Data Architecture Baseline,
Inventory, Gaps, Issues, Concerns
Define Architecture Supporting Tools and
Processes Definition
Data Architecture Scope and Activities
Data Architecture Implementation Planning
Implement, Operate
Data Architecture Supporting Tools and
Processes Implementation and Operation
Data Architecture Team Formation
Data Architecture Implementation
Data Architecture Performance and Results
Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Data Architecture Review and Improvement
Data Architecture Management
Data Architecture Operation Assessment
January 9, 2023 42
43. Data Management, Governance, Supporting
Processes Topic Scope
Data Management, Governance,
Supporting Processes
Research, Design, Define, Plan
Data Governance Capability Establishment
Define Governance Strategy
Define Current Data Governance Baseline
Define Governance Supporting Tools and Processes
Definition
Data Governance Scope and Activities
Define Governance Policies Define Governance Standards
Define Governance Compliance, Monitoring and
Reporting Data Persistence Standards
Data Lifecycle Definition and Management Create Data Asset Inventory
Create Business Glossary Perform Data Value Assessment
Data Governance Implementation Planning
Data Governance Process Definition
Implement, Operate
Data Governance Supporting Tools and Processes
Implementation and Operation
Data Governance Team Formation
Data Governance Implementation
Data Governance Performance and Results
Indicators and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Governance Review and Improvement
Data Governance Management
Data Governance Operation Assessment
Data Governance Implementation and Operation
Reporting
January 9, 2023 43
44. Data Infrastructure, Storage and Operations Software,
Hardware and Processes Topic Scope
Data Infrastructure, Storage and Operations Software,
Hardware and Processes
Research, Design, Define, Plan
Data Infrastructure, Storage and Operations Capability
Establishment
Data Infrastructure and Storage Hardware, Software and
Platform Inventory
Data Operations and Process Inventory
Data Infrastructure, Storage and Operations Existing
Processes and Standards Inventory and Review
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Definition
Data Infrastructure, Storage and Operations Software and
Hardware Scope and Activities
Data Storage Hardware Technology Target Definition Data Storage Software Technology Target Definition
Data Storage Platform Technology Target Definition
Data Product, Platform and Vendor Selection and
Management
Data Backup and Recovery Data Infrastructure Performance Monitoring Tools
Data Archival and Purge Tools
Data Infrastructure, Storage and Operations Availability,
Business Continuity, Disaster Recovery and Replication
Definition
Data Performance Testing and Validation Approach
Data Infrastructure, Storage and Operations Standards
Definition
Data Infrastructure, Storage and Operations Performance
and Capacity Planning Standards and Data Collection and
Analysis
Data Infrastructure, Storage and Operations
Implementation Planning
Implement, Operate
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Supporting Tools and Processes
Implementation and Operation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Team Formation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Implementation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Performance and Results Indicators
and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Infrastructure, Storage and Operations Review and
Improvement
Data Infrastructure, Storage and Operations Management
Data Infrastructure, Storage and Operations Operation
Assessment
January 9, 2023 44
45. Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Topic Scope
Data Security, Protection, Compliance, Access
Control, Authentication, Authorisation
Research, Design, Define, Plan
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Capability Establishment
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Existing Approach
Inventory and Baseline
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Supporting Tools and
Processes Definition
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Scope and Activities
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Architecture Definition
Compliance, Regulatory and Data Protection
Requirements Across All Data Types
Security Information, Event and Alert Logging and
Auditing
Data Loss Prevention
Data Security Product, Platform and Vendor Selection
and Management
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Standards Definition
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Monitoring, Data
Collection and Analysis
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Implementation Planning
Implement, Operate
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Supporting Tools and
Processes Implementation and Operation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Team Formation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Implementation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Performance and Results
Indicators and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Review and Improvement
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Management
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Operation Assessment
January 9, 2023 45
46. Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Topic Scope
Data Integration, Access, Flow, Exchange,
Transfer, Transformation, Load And Extract
Research, Design, Define, Plan
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Capability
Establishment
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Existing Approach
Inventory and Baseline
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Supporting Tools and
Processes Definition
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Scope and Activities
Data Integration Security, Authentication, Authorisation
Data Integration Product, Platform and Vendor Selection
and Management
Data Integration Scheduler and Rules Engine
Internal and External Data Sources, Targets and Channels
Definition
Data Integration Development, Testing and Deployment
Data Integration Operations Management,
Administration
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Standards Definition
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Monitoring, Data
Collection and Analysis
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Implementation
Planning
Implement, Operate
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Supporting Tools and
Processes Implementation and Operation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Team Formation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Implementation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Performance and
Results Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Review and
Improvement
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Management
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Operation Assessment
January 9, 2023 46
47. Content, Unstructured Data, Records and Document
Management Topic Scope
Content, Unstructured Data, Records
and Document Management
Research, Design, Define, Plan
Content, Unstructured Data, Records and Document
Management Capability Establishment
Content, Unstructured Data, Records and Document
Management Existing Approach Inventory and Baseline
Content, Unstructured Data, Records and Document
Management Supporting Tools and Processes Definition
Content, Unstructured Data, Records and Document
Management Scope and Activities
Content, Unstructured Data, Records and Document
Management Security, Authentication, Authorisation
Data Integration Product, Platform and Vendor Selection
and Management
Records Management Strategy Metadata Management
Content, Unstructured Data, Records and Document
Lifecycle Management
Content, Unstructured Data, Records and Document
Management Standards Definition
Content, Unstructured Data, Records and Document
Management Monitoring, Data Collection and Analysis
Content, Unstructured Data, Records and Document
Management Implementation Planning
Implement, Operate
Content, Unstructured Data, Records and Document
Management Supporting Tools and Processes
Implementation and Operation
Content, Unstructured Data, Records and Document
Management Team Formation
Content, Unstructured Data, Records and Document
Management Implementation
Content, Unstructured Data, Records and Document
Management Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor, Improve
Content, Unstructured Data, Records and Document
Management Review and Improvement
Content, Unstructured Data, Records and Document
Management Management
Content, Unstructured Data, Records and Document
Management Operation Assessment
January 9, 2023 47
48. Master and Reference Data Management Topic Scope
Master and Reference Data
Management
Research, Design, Define, Plan
Master and Reference Data Management Capability
Establishment
Master and Reference Data Management Existing
Approach Inventory and Baseline
Master and Reference Data Management Supporting
Tools and Processes Definition
Master and Reference Data Management Scope and
Activities
Industry Data Standards Data Glossaries and Taxonomies
Business Rules Analysis and Definition
Master and Reference Data Management Product,
Platform and Vendor Selection and Management
Master Data Stores Reference Data Stores
Master and Reference Data Management Standards
Definition
Master and Reference Data Management Monitoring,
Data Collection and Analysis
Master and Reference Data Management
Implementation Planning
Implement, Operate
Master and Reference Data Management Supporting
Tools and Processes Implementation and Operation
Master and Reference Data Management Team
Formation
Master and Reference Data Management
Implementation
Master and Reference Data Management Performance
and Results Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Master and Reference Data Management Review and
Improvement
Master and Reference Data Management Management
Master and Reference Data Management Operation
Assessment
January 9, 2023 48
49. Data Warehouse, Data Marts, Data Lakes Topic Scope
Data Warehouse, Data Marts, Data
Lakes
Research, Design, Define, Plan
Data Warehouse, Data Marts, Data Lakes Capability
Establishment
Data Warehouse, Data Marts, Data Lakes Existing
Approach Inventory and Baseline
Data Warehouse, Data Marts, Data Lakes Supporting
Tools and Processes Definition
Data Warehouse, Data Marts, Data Lakes Scope and
Activities
Data Models Creation Long-term Data Storage Architecture
Data Integration and Population
Data Warehouse, Data Marts, Data Lakes Product,
Platform and Vendor Selection and Management
Data Access Metadata Management
Data Virtualisation
Data Warehouse, Data Marts, Data Lakes Standards
Definition
Data Warehouse, Data Marts, Data Lakes
Monitoring, Data Collection and Analysis
Data Warehouse, Data Marts, Data Lakes
Performance and Capacity Planning Standards and
Data Collection and Analysis
Data Warehouse, Data Marts, Data Lakes
Implementation Planning
Implement, Operate
Data Warehouse, Data Marts, Data Lakes Supporting
Tools and Processes Implementation and Operation
Data Warehouse, Data Marts, Data Lakes Team
Formation
Data Warehouse, Data Marts, Data Lakes
Implementation
Data Warehouse, Data Marts, Data Lakes
Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor,
Improve
Data Warehouse, Data Marts, Data Lakes Review
and Improvement
Data Warehouse, Data Marts, Data Lakes
Management
Data Warehouse, Data Marts, Data Lakes Operation
Assessment
January 9, 2023 49
50. Data Reporting and Analytics, Visualisation Tools and
Facilities Topic Scope
Data Reporting and Analytics,
Visualisation Tools and Facilities
Research, Design, Define, Plan
Data Reporting and Analytics, Visualisation Tools
and Facilities Capability Establishment
Data Reporting and Analytics, Visualisation Tools
and Facilities Existing Approach Inventory and
Baseline
Data Reporting and Analytics, Visualisation Tools
and Facilities Supporting Tools and Processes
Definition
Data Reporting and Analytics, Visualisation Tools
and Facilities Scope and Activities
Reporting and Visualisation Architecture and
Approach
Analytics Architecture and Approach
Data Integration, Access and Security
Data Reporting and Analytics, Visualisation Facility
Access and Security
Data Reporting and Analytics, Visualisation Product,
Platform and Vendor Selection and Management
Data Reporting and Analytics, Visualisation
Development, Testing and Deployment
Data Reporting and Analytics, Visualisation
Distribution and Security
Data Reporting and Analytics, Visualisation Tools
and Facilities Standards Definition
Data Reporting and Analytics, Visualisation Tools
and Facilities Monitoring, Data Collection and
Analysis
Data Reporting and Analytics, Visualisation Tools
and Facilities Performance and Capacity Planning
Standards and Data Collection and Analysis
Data Reporting and Analytics, Visualisation Tools
and Facilities Implementation Planning
Implement, Operate
Data Reporting and Analytics, Visualisation Tools
and Facilities Supporting Tools and Processes
Implementation and Operation
Data Reporting and Analytics, Visualisation Tools
and Facilities Team Formation
Data Reporting and Analytics, Visualisation Tools
and Facilities Implementation
Data Reporting and Analytics, Visualisation Tools
and Facilities Performance and Results Indicators
and Measurement Framework Definition
Administer, Manage, Monitor,
Improve
Data Reporting and Analytics, Visualisation Tools
and Facilities Review and Improvement
Data Reporting and Analytics, Visualisation Tools
and Facilities Management
Data Reporting and Analytics, Visualisation Tools
and Facilities Operation Assessment
January 9, 2023 50
51. Data Discovery, Analysis, Design and Modelling Topic
Scope
Data Discovery, Analysis, Design and Modelling
Research, Design, Define, Plan
Data Discovery, Analysis, Design and Modelling
Capability Establishment
Data Discovery, Analysis, Design and Modelling Existing
Approach Inventory and Baseline
Data Discovery, Analysis, Design and Modelling
Supporting Tools and Processes Definition
Data Discovery, Analysis, Design and Modelling Scope
and Activities
Data Modelling Definition Data Profiling Approach Definition
Data Discovery and Profiling Tool Selection and
Implementation
Data Lineage Definition Including Tool Selection
Data Catalog Definition Including Tool Selection Data Dictionary Definition Including Tool Selection
Semantic Layer Definition Including Tool/Platform
Selection
Data Discovery, Analysis, Design and Modelling
Standards Definition
Data Discovery, Analysis, Design and Modelling
Monitoring, Data Collection and Analysis
Data Discovery, Analysis, Design and Modelling
Performance and Capacity Planning Standards and Data
Collection and Analysis
Data Discovery, Analysis, Design and
Modelling Implementation Planning
Implement, Operate
Data Discovery, Analysis, Design and Modelling
Supporting Tools and Processes Implementation and
Operation
Data Discovery, Analysis, Design and Modelling Team
Formation
Data Discovery, Analysis, Design and Modelling
Implementation
Data Discovery, Analysis, Design and Modelling
Performance and Results Indicators and Measurement
Framework Definition
Administer, Manage, Monitor, Improve
Data Discovery, Analysis, Design and Modelling Review
and Improvement
Data Discovery, Analysis, Design and Modelling
Management
Data Discovery, Analysis, Design and Modelling
Operation Assessment
January 9, 2023 51
52. External Data Sources and Interacting Parties Data
Transfer/ Exchange/ Integration/ Publication Topic Scope
External Data Sources and Interacting Parties
Data Transfer/Exchange/Integration/Publication
Research, Design, Define, Plan
External Data Sources and Interacting Parties Capability
Establishment
External Data Sources and Interacting Parties Existing
Approach Inventory and Baseline
External Data Sources and Interacting Parties Supporting
Tools and Processes Definition
External Data Sources and Interacting Parties Scope and
Activities
Data Transfer/Exchange/Integration/Publication
Request Review and Approval Process
Data Transfer/Exchange/Integration/Publication
Implementation Process Definition
Data Transfer/Exchange/Integration/Publication Toolset
Definition and Acquisition
Data Transfer/Exchange/Integration/Publication Activity
Monitoring and Review
Data Transfer/Exchange/Integration/Publication Access
Standards
Data Transfer/Exchange/Integration/Publication Open
Data Approach
Data Transfer/Exchange/Integration/Publication
Security Definition
External Data Sources and Interacting Parties Standards
Definition
External Data Sources and Interacting Parties
Monitoring, Data Collection and Analysis
External Data Sources and Interacting Parties
Implementation
Implement, Operate
External Data Sources and Interacting Parties Supporting
Tools and Processes Implementation and Operation
External Data Sources and Interacting Parties Team
Formation
External Data Sources and Interacting Parties
Implementation
External Data Sources and Interacting Parties
Performance and Results Indicators and Measurement
Framework Definition
Administer, Manage, Monitor, Improve
External Data Sources and Interacting Parties Review
and Improvement
External Data Sources and Interacting Parties
Management
External Data Sources and Interacting Parties Operation
Assessment
January 9, 2023 52
53. Metadata Data Management Topic Scope
Metadata Data Management
Research, Design, Define, Plan
Metadata Data Management Capability
Establishment
Metadata Data Management Existing Approach
Scope Definition, Inventory and Baseline
Metadata Data Management Supporting Tools
and Processes Definition
Metadata Data Management Scope and Activities
Define Metadata Architecture and Approach Metadata Standard Review
Metadata Data Management Standards Definition Define Metadata Management Tools
Metadata Creation and Maintenance Approach Metadata Repositories Definition
Metadata Integration and Usage Approach
Metadata Data Management Monitoring, Data
Collection and Analysis
Metadata Data Management Performance and
Capacity Planning Standards and Data Collection
and Analysis
Metadata Data Management Implementation
Planning
Implement, Operate
Metadata Data Management Supporting Tools
and Processes Implementation and Operation
Metadata Data Management Team Formation
Metadata Data Management Implementation
Metadata Data Management Performance and
Results Indicators and Measurement Framework
Definition
Baseline Metadata Creation
Administer, Manage, Monitor,
Improve
Metadata Data Management Review and
Improvement
Metadata Data Management Management
Metadata Data Management Operation
Assessment
January 9, 2023 53
54. Data Quality Topic Scope
Data Quality
Research, Design, Define, Plan
Data Quality Capability Establishment
Data Quality Existing Approach Inventory, Profile and
Baseline
Data Quality Supporting Tools and Processes Definition
Data Quality Scope and Activities
Data Quality Requirements Data Quality Rules Approach
Data Quality Service Level Management Approach Data Quality Analysis and Reporting
Data Quality Standards Definition
Data Quality Monitoring, Data Collection and Analysis
Data Quality Implementation Planning
Implement, Operate
Data Quality Supporting Tools and Processes
Implementation and Operation
Data Quality Team Formation
Data Quality Implementation
Data Quality Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Quality Review and Improvement
Data Quality Management
Data Quality Operation Assessment
January 9, 2023 54
55. Data Solution Design Topic Scope
Data Solution Design
Research, Design, Define, Plan
Data Solution Design Advisory Capability
Establishment
Data Solution Design Existing Approach Review,
Inventory and Baseline
Data Solution Design Scope and Activities
Data Management and Governance Standards Data Modelling and Design Standards
Operational and Archival Data Data Storage and Persistence Standards
Data Infrastructure Standards Data Transfer/Exchange/Integration Standards
Data Reporting and Analysis Standards Data Performance and Throughput Standards
Data Security Standards
Data Solution Design Monitoring, Data Collection
and Analysis
Data Solution Design Implementation Planning
Implement, Operate
Data Solution Design Supporting Tools and
Processes Implementation and Operation
Data Solution Design Team Formation
Data Solution Design Implementation
Data Solution Design Performance and Results
Indicators and Measurement Framework
Definition
Administer, Manage, Monitor,
Improve
Data Solution Design Review and Improvement
Data Solution Design Management
Data Infrastructure, Storage and Operations
Operation Assessment
January 9, 2023 55
56. Data Architecture Coverage Of Solution Component
Types
January 9, 2023 56
Solution
Component
Types
Data Architecture Subject Areas
Data
Architecture
Data
Management,
Governance,
Supporting
Processes
Data
Infrastructure,
Storage and
Operations
Software,
Hardware and
Processes
Data Security,
Protection,
Compliance,
Access Control,
Authentication,
Authorisation
Data
Integration,
Access, Flow,
Exchange,
Transfer,
Transformation,
Load And
Extract
Content,
Unstructured
Data, Records
and Document
Management
Master and
Reference Data
Management
Data
Warehouse,
Data Marts,
Data Lakes
Data Reporting
and Analytics,
Visualisation
Tools and
Facilities
Data Discovery,
Analysis,
Design and
Modelling
External Data
Sources and
Interacting
Parties Data
Transfer/
Exchange/
Integration/
Publication
Metadata Data
Management
Data Quality Data Solution
Design
Changes to Existing Systems
X X X X X X X X X X X X
New Custom Developed
Applications X X X X X X X X X X X X
Acquired and Customised
Software Products X X X X X X X X X X X X
System Integrations/ Data
Transfers/ Exchanges X X X
Reporting and Analysis
Facilities X X X X X X X X
Sets of Installation and
Implementation Services
Information Storage
Facilities X
Existing Data Conversions/
Migrations X X X X X X
New Data Loads
X X X X X X
Central, Distributed and
Communications
Infrastructure
X X
Cutover/ Transfer to
Production And Support
Operational Functions and
Processes
Parallel Runs
Enhanced Support/
Hypercare
Sets of Maintenance, Service
Management and Support
Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes,
Knowledge Management
Training and Documentation
57. Data Architecture Coverage Of Solution Component
Types
• The data architecture subject areas impact the data
aspects of many solution component types within solutions
• An effective data architecture can contribute to effective
solution architecture and solution data architecture
January 9, 2023 57
58. Data Architecture And Common Data Tooling And
Standards
• Data architecture needs to provide common infrastructural data tools and
common data standards for solutions
− Tools
• Data storage infrastructure – hardware, software and platforms
• Data warehouse platform
• Data reporting, visualisation and analysis
• Data transfer/exchange/integration/extract/transform/load
• Data operations – backup/recovery/replication/business continuity/disaster recovery
• Data anonymisation/pseudonymisation/encryption
• Data monitoring/performance/capacity planning
• Master data management platform
• Reference data platform
• Data catalog/semantic layer
• Document management
• Data analysis
− Standards
• Data security
• Data quality
• Metadata
• Data discovery
• Data modelling
• Data management – classification/retention/archive/deletion
January 9, 2023 58
59. Organisation Data Architecture And Solution Data
Architecture
January 9, 2023 59
Organisation Data
Architecture
Common Data
Standards
Common Data
Infrastructure
Tools and
Facilities
Common Data
Operations
Individual
Solutions
Individual
Solutions
Individual
Solutions
Common Data
Design and
Implementation
Approaches
Common Data
Model
60. Organisation Data Architecture And Solution Data
Architecture
• Common data infrastructure tools allows reuse, reduces
decision-making overhead and delays, reduces cost,
accelerates individual deployment and achieves
standardisation
• Toolset will need to change in response to changing
business needs and technology landscape
January 9, 2023 60
61. Common Data Plumbing Infrastructure
• Data architecture should take on the projects required to deliver the
common data plumbing infrastructure
• These are foundational components
• Individual solution delivery activities should not have to be responsible for
their implementation
January 9, 2023 61
62. Common Data Plumbing Infrastructure
January 9, 2023 62
Common ETL
Common Data
Transfer/
Exchange
Common API
Layer
Common API
Layer
Common Data
Infrastructure/
Platform
Common Data
Warehouse
Common Data
Reporting
Common Data
Analytics
Common Data
Backup and
Recovery
Common
Business
Continuity and
Disaster Recovery
Common
Document
Management
Common Data
Analytics
Common
Performance
Monitoring and
Capacity Planning
Common Audit
Logging Data
Management
Common Data
Catalog
Common
Reference and
Master Data
Individual
Solutions
63. Common Data Plumbing Infrastructure
• This ideal is regularly not fully in place
• Individual solutions often have to implement some of
these capabilities that are not available centrally
• The leads to sub-optimal solutions with point resolutions
to specific requirements
January 9, 2023 63
64. Data Design And Modelling For Solutions
• The objective of data design for solutions is the same as
that for overall solution design:
− To capture sufficient information to enable the solution design to
be implemented
− To unambiguously define the data requirements of the solution
and to confirm and agree those requirements with the target
solution consumers
− To ensure that the implemented solution meets the requirements
of the solution consumers and that no deviations have taken place
during the solution implementation journey
January 9, 2023 64
65. Why Pay Attention To Solution Data Architecture?
• Solution data architecture avoids problems with solution
operation and use:
− Poor and inconsistent data quality
− Poor performance, throughput, response times and scalability
• Poorly designed data structures can lead to long data update times leading to
long response times, affecting solution usability, loss of productivity and
transaction abandonment
− Poor reporting and analysis
− Poor data integration
− Poor solution serviceability and maintainability
− Manual workarounds for data integration, data extract for reporting and
analysis
• Data-design-related solution problems frequently become
evident and manifest themselves only after the solution goes
live
• The benefits of solution data architecture are not always
evident initially
January 9, 2023 65
66. New Technology And Impact on Solution Data
Architecture
• New solution deployment and operating models affects
solution data architecture
− New solution design, deployment and operating models
− Greater use of platform-based solution implementation and
deployment
− Wider range of complex data technology options, especially in
terms of data analysis
− Distributed solution components, distributed solution consumer
base, distributed access with many interfaces, integration points
and data flows
− Complexity with multiple data integrations
January 9, 2023 66
67. Solution Design – From …
January 9, 2023 67
Solution
Central Data
Store
Solution
Central
Application
Component
Solution API
Solution
Central
Infrastructure
Solution
Hosted
Infrastructure
Solution
Internal
Consumers
Solution
External
Private
Consumers
Solution
Hosted Data
Store
Solution
Hosted
Application
Component
Solution
Hosted
Analytics
Access and
Security
Infrastructure
Central To
Hosting
Facility
Connectivity
Solution
External Public
Consumers
Solution
Mobile App
68. To …
• Increasing solution landscape complexity and diversity gives rise to greater data
design complexity
January 9, 2023 68
Solution
Central Data
Store
Solution
Central
Application
Component
Solution API
Solution
Central
Infrastructure
Solution
Hosted
Infrastructure
Solution
Internal
Consumers
Solution
External
Private
Consumers
Solution
Hosted Data
Store
Solution
Hosted
Application
Component
Solution
Hosted
Analytics
Access and
Security
Infrastructure
Central To
Hosting
Facility
Connectivity
Solution
External Public
Consumers
Solution
Mobile App
69. Solution Entity Model
January 9, 2023 69
Solution
Component
Types
Solution
Components
Solution
Solution Zones
Solution
Zone Types
Solution
Topology
Solution Consists Of
Multiple Components
Each Solution
Component
Has A Type
Solution Exists
Within A
Topology Of
Many Solutions
Solution Components
Are Located In Solution
Zones
Each Solution
Zone Has A Type
Solution
Operational
Entity
Solution
Operational
Entity Type
Deployed
Solution
Consists Of
Multiple
Operational
Entities
Each Solution
Operational
Entity Has A Type
Solution Operational Entities
Are Located In Solution Zones
Some Solution
Components
Become
Deployed
Operational
Entities
70. Solution Zone Types and Zones
January 9, 2023 70
Solution
Component
Types
Solution
Components
Solution
Solution Zones
Solution
Zone Types
Solution
Topology
Solution Consists Of
Multiple Components
Each Solution
Component
Has A Type
Solution Exists
Within A
Topology Of
Many Solutions
Solution Components
Are Located In Solution
Zones
Each Solution
Zone Has A Type
Solution
Operational
Entity
Solution
Operational
Entity Type
Deployed
Solution
Consists Of
Multiple
Operational
Entities
Each Solution
Operational
Entity Has A Type
Solution Operational Entities
Are Located In Solution Zones
Some Solution
Components
Become
Deployed
Operational
Entities
71. Solution Zones
• Solution zones are locations where groups of closely related solution
components reside
• They represent containers for solution components
• Zones are located within the wider physical solution landscape
• Each zone and the components it holds have different security
requirements
• Not all solutions will have components in all zone and not all
organisations will have all the zone types
• The solution and its constituent components can span multiple
different zones of the same type
• The zone approach is useful way of representing the entirety of a
solution, its constituent components, their connectivity, linkages and
interactions, especially data storage, processing and interactions
• You will have different levels of control over different solution zones
(including no control) – this impacts data design considerations
January 9, 2023 71
74. Sample Solution Zone Types
Zone Description
Insecure External Organisation
Presentation And Access
Where publicly accessible or accessing entities reside. These entities are regarded
as insecure and/or untrusted.
Secure External Organisation
Participation and Collaboration
Outside the physical organisation boundary where entities that are provided by or
to trusted external parties reside
Secure External Organisation Access Contain entities that enable secure access or are securely accessible from outside
the organisation
Organisation Contain the entities within the organisation boundary and contains all the
locations, business units and functions within it
Central Solutions and Access Contains the solution entities and their data
Solution Zone Contains the solution entities
Data Zone Zone within the organisation where data is segregated for security
Remote Business Unit Solutions and
Access
Remotely located organisation business unit or location and the entities it
contains
Workstation Zone Zone within the organisation where users accessing data and solutions are
segregated for security
Outsourced Service Provider Solutions
and Access
Contains solutions provided by and located in facilities provided by outsourced
partners
Cloud Service Provider Solutions and
Access
Contains solutions - platform, infrastructure and service - provided by and located
in cloud service providers
Co-Located Solutions and Access Contains solutions the organisation has located in facilities provided by co-
location providers
January 9, 2023 74
75. Solution Consumers
• Any solution with have different sets of consumers of different
types:
− Controlled Consumers – typically organisation personnel over whom the
solution owner has substantial control
− Partially Controlled Consumers – typically external business partners
and other interacting parties over whom the solution owner has some
control and influence
− Uncontrolled Consumers – typically members of the public at whom the
solution is targeted and whose needs must be inferred through groups
of proxy consumers
• Each consumer type will have different data-related needs and
expectations
• Classifying and understanding the target solution consumers
will contribute to solution data architecture design
January 9, 2023 75
76. Data Design And Modelling For Solutions Activities
Data
Modelling
Data Journeys
Design
Data
Processing
Design
January 9, 2023 76
Logical
Data Model
Physical Data
Model
Conceptual
Data Model
77. Data Design And Modelling For Solutions Activities
• This includes the following activities:
− Data Modelling – define the data entities, structures, attributes
and contents
• Conceptual Data Model (CDM) – create an initial high-level view of solution
• Logical Data Model (LDM) – expand the CDM with detailed data
requirements
• Physical Data Model (PDM) – translate the LDM into implementation-
specific details
− Data Journeys Design – create an inventory of data journeys and
identify the steps within the journeys and the data entities
involved
− Data Processing Design – define the detail of the processing
performed on data entities
January 9, 2023 77
78. Data Design And Modelling For Solutions Activities
• The data analysis and design activities are not linear or
sequential
• As the analysis progresses, earlier work may need to be
revisited to be elaborated and expanded on
January 9, 2023 78
Conceptual Data
Model
Logical Data
Model
Physical Data
Model
Data Journeys
Design
Data Processing
Design
Data Modelling
79. Packaged Solutions And Platforms
• Packaged solution components and platforms on which
solutions components are implemented and deployed will
have pre-defined data models with a greater or lesser
degree of configuration and customisation
• The data design activities should still be performed for
these solution components
• The inherent data limitations and restrictions of the
packages and platforms should be clearly defined and
understood
January 9, 2023 79
80. Solutions And Shared/Private Data
January 9, 2023 80
• Solutions and
their
components
within the
organisation
solution
landscape will
have both local
data and data
that is shared
with or between
other solutions,
either for
upstream/
downstream
processing or as
shared data
repositories
81. Solutions And Shared/Private Data
• Data design activities are different for shared and private
solution data
• Shared solution data includes reference and master data
− Reference data consists of common code, structural and identifier
values
• Their purpose is to ensure consistency across data values
• Reference data is static or slowly changing
− Master data relates to common transaction identifiers such interacting
parties (customer, partner, etc,) details
• Having a single version of master data ensures single view of all interactions with
the party across the organisation can be identified
• Solution data modelling activities should identify the
occurrences of reference and master data to maximise reuse
and contribute to maintaining a single version of the truth
January 9, 2023 81
82. Shared/Common Data Issues
• Because shared data is shared, the main concerns and
issues relate to:
− Ownership – who owns and is responsible
− Maintenance – who maintains it and keeps it current, who is
responsible for allowing updates, what is the process for applying
updates and changes
− Quality – who is responsible for maintaining data quality
• Individual solutions should not have to solve these
problems, but commonly and unfortunately have to
January 9, 2023 82
83. Common Data Formats
• Master data will have a common format
• Individual solutions need to adhere to the common master
data format
January 9, 2023 83
84. Data Modelling – Conceptual Data Model
• The Conceptual Data Model (CDM) represents concepts,
entities and their relationships within the scope of the
solution
• It is used to create a common understanding among all the
solution stakeholders
• The CDM defines the scope of the solution
• The functional requirements of the solution provide an
input to the CDM and its constituent entities
January 9, 2023 84
85. Data Modelling – Logical Data Model
• The Logical Data Model (LDM) expands on the agreed CDM
• Detailed data requirements for the specified data entities
are defined, including solution zones
January 9, 2023 85
86. Data Modelling – Physical Data Model
• The Physical Data Model (PDM) translates the LDM into
technology-specific implementation details and a
technology structure across the solution zones
• The LDM may be updated to reflect and accommodate
technology and platform specific features, limitations,
capabilities and restrictions
January 9, 2023 86
87. Data Journeys Design
• Solutions have journeys as consumers use the solution to
achieve results
• Solution journeys reflect solution consumer experiences
• Data journeys represent the data exchanges and transfers
that occur to support the solution journeys
• Solution data architecture should first create an inventory
of data journeys
• The processing data journeys can then be expanded to
reflect the lifecycle for the data type(s) associated with the
data journeys
January 9, 2023 87
88. Data Processing Design
• Data processing design describes the detailed processing
that is performed on data
• It defines the business rules that are applied to data within
the scope of the solution
January 9, 2023 88
89. Data Processing Design
Identify the
data
processing
performed on
data entities
and objects
by each
solution
component
for each
solution
journeys
January 9, 2023 89
Data
Entity/Object
Solution
Component
Data
Entity/Object
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
D1
D2
D3
D4 D5
90. Summary
• The data architecture of solutions is frequently not given the attention it deserves or needs
• Frequently, too little attention is paid to designing and specifying the data architecture within
individual solutions and their constituent components
• This is due to the behaviours of both solution architects ad data architects
• Solution architecture tends to concern itself with functional, technology and software
components of the solution
• Data architecture tends not to get involved with the data aspects of technology solutions,
leaving a data architecture gap
• Combined with the gap where data architecture tends not to get involved with the data aspects
of technology solutions, there is also frequently a solution architecture data gap
• Solution architecture also frequently omits the detail of data aspects of solutions leading to a
solution data architecture gap
• These gaps result in a data blind spot for the organisation
• Data architecture tends to concern itself with post-individual solutions
• Data architecture needs to shift left into the domain of solutions and their data and more
actively engage with the data dimensions of individual solutions
• Data architecture can provide the lead in sealing these data gaps through a shift-left of its scope
and activities as well providing standards and common data tooling for solution data
architecture
January 9, 2023 90