SlideShare a Scribd company logo
1 of 91
Download to read offline
Data Architecture For
Solutions
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
https://www.researchgate.net/profile/Alan-Mcsweeney
https://www.amazon.com/dp/1797567616
Introduction
• These notes discuss how overall organisation data
architecture can positively impact solution design and how
solution data architecture competence within solution
architecture can contribute to solution design success
January 9, 2023 2
Data
Architecture
Solution
Architecture
Solution Data
Architecture
Can
Contribute to
Common Data
Infrastructure
Tools and
Data
Standards
Can
Contribute to
Overall
Organisation
Data Quality
Can Ensure
that the Data
Aspects of
Solution
Design Are
Covered in
Solution
Designs
Can Ensure
that Solution
Data Concerns
Are Addressed
in Solution
Designs
Topics
• Data Architecture For Solutions
• Traditional Scope Of Data Architecture
• Solution Data Architecture
• What Do We Mean By Data Architecture?
• Data Architecture And Common Data Tooling And
Standards
• Data Design And Modelling For Solutions
January 9, 2023 3
Data Architecture For Solutions
• Data breathes life into solutions
• Solutions get data, use data, share data, process data and create data
• There will be many different types of data used by a solution
− Master data
− Reference data
− Input data
− Interim data
− Generated data
− Solution activity and usage data
• Any solution will consist of many different components of different types
• Solution components and their data will be deployed and operated across a solution
landscape that can span multiple zones and platforms
• Within the solution, each data type will have a different lifecycle
• The solutions within the organisation solution landscape will have both shared and
private data
− Shared - common data (master or reference) or upstream data from other solutions or data sent
downstream
− Private – data held locally within the solution
January 9, 2023 4
Solution And Data
• All IT solutions support, implement and operate business
processes that take data inputs, process data, generate result
and create primary and supporting data output
− Direct data outputs – what the process in intended to create
− Indirect data outputs – logs, audit trails, reports, analyses
• Data outputs are then used in different ways
− Generated results
− As a record that the work was performed
− As inputs into other processes and solutions
− To report on the operation of the process or as an audit log
• Data breathes life into and activates the static components of a
solution
• The data architecture of solutions is frequently not given the
attention it deserves or needs
January 9, 2023 5
Data Architecture For Solutions
• Frequently, too little attention is paid to designing and
specifying the data architecture within individual solutions
and their constituent components
• This is due to the behaviours of both solution architects ad
data architects
• Solution architecture tends to concern itself with
functional, technology and software components of the
solution
• Data architecture tends to concern itself with post-
individual solutions
January 9, 2023 6
Traditional Scope Of Data Architecture
January 9, 2023 7
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Data Sources
(Internal, External) Extract, Transform, Load Data Platform Access and Usage
Merge,
Aggregate,
Transform
Data
Sources
(from
Solutions)
Traditional Scope Of Data Architecture
January 9, 2023 8
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Merge,
Aggregate,
Transform
Scope Of Data Architecture
Data
Sources
(from
Solutions)
Traditional Scope Of Data Architecture
• Traditional approaches to data architecture effectively
appends or layers newer technologies on top on existing
solutions and data sources and their data structures
• Data architecture largely ignores data architectures within
individual solutions
• Data architecture needs to shift left into the domain of
solutions and their data and more actively engage with the
data dimensions of individual solutions
January 9, 2023 9
Traditional Scope Of Data Architecture
January 9, 2023 10
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data Encryption,
Anonymisation,
Pseudonymisation
Security
and Access
Control
Data
Processing
Workflow
Data Model
and Data
Store
API
Interface
Data
Interrogation
and Analysis
Data
Visualisation
Data
Extract
Management
and
Administration
Data
Publication
and Sharing
Existing and
New
Reports
Data Storage
Platform/
Infrastructure
Usage and
Performance
Monitoring
Semantic
Layer
Merge,
Aggregate,
Transform
This Is Not A Modern Data Architecture
Data
Sources
(from
Solutions)
Not A Modern Data Architecture
• You are fooling yourself if you
believe that enveloping
existing data solutions,
sources and structures with a
skin of modernity comprises a
data architecture
• If you put lipstick on a pig, it is
still a pig
January 9, 2023 11
Common and
Shared Data
Processes and
Standards
Data
Architecture
Operation,
Measurement
Data
Architecture
Review,
Improvement,
Update
Common and
Shared Data
Infrastructural
Components
Data
Solutions,
Sources and
Structures
This Is A Data Architecture
January 9, 2023 12
Data Architecture Overview
Data Management, Governance, Supporting Processes
Data Infrastructure, Storage and Operations Software, Hardware and Processes
Data Security, Protection, Compliance, Access Control, Authentication, Authorisation
Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract
Content, Unstructured Data, Records and Document Management
Master and Reference Data Management
Data Warehouse, Data Marts, Data Lakes
Data Reporting and Analytics, Visualisation Tools and Facilities
Data Discovery, Analysis, Design and Modelling
External Data Sources and Interacting Parties Data Transfer/Exchange/Integration/Publication
Metadata Data Management
Data Quality
Data Solution Design
Traditional Scope Of Data Architecture And Data Architecture
Gap
Data
architecture
tends not to
get involved
with the
data aspects
of
technology
solutions,
leaving a
data
architecture
gap
January 9, 2023 13
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data
Processing
Workflow
Data
Sources
(from
Solutions)
Merge,
Aggregate,
Transform
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom Developed
Applications and Their
Data Models
Acquired and Customised
Software Products and
Their Data Models
System Integrations/
Data Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
…
…
Data Architecture Gap Data Architecture
Solution Data Aspects Across The Solution Landscape
January 9, 2023 14
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Changes to Existing
Systems and Their
Data Models
New Custom
Developed
Applications and Their
Data Models
Acquired and
Customised Software
Products and Their Data
Models
System
Integrations/ Data
Transfers/
Exchanges
Reporting and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions/
Migrations
New Data
Loads
Span Of Organisation Solution Landscape
Changes to Existing
Systems and Their
Data Models
Solution Components And Their Types
January 9, 2023 15
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Changes to Existing Systems
New Custom Developed Applications
Acquired and Customised Software
Products
System Integrations/ Data Transfers/
Exchanges
Reporting and Analysis Facilities
Sets of Installation and
Implementation Services
Information Storage Facilities
Existing Data Conversions/
Migrations
New Data Loads
Central, Distributed and
Communications Infrastructure
Cutover/ Transfer to Production And
Support
Operational Functions and Processes
Parallel Runs
Enhanced Support/ Hypercare
Sets of Maintenance, Service
Management and Support Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes, Knowledge
Management
Training and Documentation
Component Type Solution Components
Reduced Scope Of Traditional Solution Architecture
Scope
• The solution is the sum
of the components
needed to deliver and
operate it
• Solution architecture
tends not to concern
itself with some key
aspects of the
complete solution,
including some of
those related to data
• Solution architecture
tends to focus on
technology aspects of
a solution, omitting
business and data
facets
• The data dimensions
of other solution
components also
tends to be omitted
partially or completely
by solution
architecture
January 9, 2023 16
Complete
Solution
Data Related Solution Components And Their Types
January 9, 2023 17
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Changes to Existing Systems
New Custom Developed Applications
Acquired and Customised Software
Products
System Integrations/ Data Transfers/
Exchanges
Reporting and Analysis Facilities
Sets of Installation and
Implementation Services
Information Storage Facilities
Existing Data Conversions/
Migrations
New Data Loads
Central, Distributed and
Communications Infrastructure
Cutover/ Transfer to Production And
Support
Operational Functions and Processes
Parallel Runs
Enhanced Support/ Hypercare
Sets of Maintenance, Service
Management and Support Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes, Knowledge
Management
Training and Documentation
Component Type Solution Components
Data Dimensions Of Solution Component Types
January 9, 2023 18
Changes to
Existing
Systems
Possible new data stores
and their design and data
models
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
New Custom
Developed
Applications
New data stores and their
design and data models
Data tools
Data processes
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data design
Metadata design
Data quality design
Acquired
and
Customised
Software
Products
New data stores and their
design and data models
Data tools
Data processes
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
System
Integrations/
Data
Transfers/
Exchanges
Source and target data
stores and their design
and data models
Data transformations and
aggregations
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
Reporting
and Analysis
Facilities
Reporting data stores and
their design and data
models
Reporting tools
Reporting processes
Data security, encryption
and access control
Master and reference
data
Metadata
Data quality
Information
Storage
Facilities
Performance, capacity
and throughput
Data management and
governance
Existing Data
Conversions/
Migrations
Source and target data
stores and their design
and data models
Data transformations and
aggregations
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
New Data
Loads
Target data stores and
their design and data
models
Data tools
Data processes
Changes to existing data
stores and their design
and data models
Performance, capacity
and thoughput
Data security, encryption
and access control
Data management and
governance
Master and reference
data
Metadata
Data quality
Data Dimensions Of Solution Component Types
• The following solution component types involve data
architecture and design activities:
− Changes to Existing Systems
− New Custom Developed Applications
− Acquired and Customised Software Products
− System Integrations/ Data Transfers/ Exchanges
− Reporting and Analysis Facilities
− Information Storage Facilities
− Existing Data Conversions/ Migrations
− New Data Loads
• The components of each of these types will potentially involve
data and therefore data design work across a range of areas
that will need to be included in the solution design and
subsequent solution implementation activities
• Solution architecture can fail t include some of the data
dimensions of the solution components of these types
January 9, 2023 19
Solution Components And Their Types
• Any technology solution will consist of a potentially large
number of components, each of a give type
• Each solution component type belongs to one of three
classes
1. Time-Bounded Delivery Entity Types
• Time-bounded solution component types required to get the solution fully
operational
2. Enduring Functional and Operational Technology Entity Types
• Operational instrumentation and functional component types required for
the solution to operate and be usable by its target consumers
3. Enduring Organisational, Process, Procedure and Structural
Entity Types
• Organisation and process changes and other supporting activities and sets
of effort required to use the solution optimally
January 9, 2023 20
Solution Components Classes And Types
January 9, 2023 21
Solution Component Classes
and Types
Time-Bounded Delivery
Entity Types
Sets of Installation and
Implementation Services
Existing Data Conversions/
Migrations
New Data Loads
Parallel Runs
Enhanced Support/ Hypercare
Enduring Functional and
Operational Technology
Entity Types
Changes to Existing Systems
New Custom Developed
Applications
Acquired and Customised Software
Products
System Integrations/ Data
Transfers/ Exchanges
Reporting and Analysis Facilities
Information Storage Facilities
Central, Distributed and
Communications Infrastructure
Application Hosting and
Management Services
Enduring Organisational,
Process, Procedure and
Structural Entity Types
Cutover/ Transfer to Production
And Support
Operational Functions and
Processes
Sets of Maintenance, Service
Management and Support Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes,
Knowledge Management
Training and Documentation
Solution Architecture Data Gap
• Combined with the gap where data architecture tends not to
get involved with the data aspects of technology solutions,
there is also frequently a solution architecture data gap
• Solution architecture also frequently omits the detail of data
aspects of solutions across the various components of the
types:
− Changes to Existing Systems
− New Custom Developed Applications
− Acquired and Customised Software Products
− System Integrations/ Data Transfers/ Exchanges
− Reporting and Analysis Facilities
− Information Storage Facilities
− Existing Data Conversions/ Migrations
− New Data Loads
January 9, 2023 22
Solution Architecture Data Gap
• These gaps result in a data blind spot for the organisation
January 9, 2023 23
Central,
Distributed and
Communications
Infrastructure
Changes to
Existing Systems
New Custom
Developed
Applications
Information
Storage Facilities
Acquired and
Customised
Software Products
System
Integrations/ Data
Transfers/
Exchanges
Changes to
Existing Business
Processes
New Business
Processes
Organisational
Changes,
Knowledge
Management
Reporting and
Analysis Facilities
Existing Data
Conversions/
Migrations
New Data Loads
Training and
Documentation
Sets of
Installation and
Implementation
Services
Operational
Functions and
Processes
Parallel Runs
Cutover/ Transfer
to Production
Sets of
Maintenance,
Service
Management and
Support Services
Application
Hosting and
Management
Services
Enhanced
Support/
Hypercare
Data Architecture
Solution Gap Data Architecture
Data
Ingestion
and
Integration
Data Validation
and Error
Handling
Data
Processing
Workflow
Merge,
Aggregate,
Transform
…
Solution Architecture
Data Gap
Solution Architecture
Solution Architecture Data Gap
• Data architecture can provide the lead in sealing these data
gaps through a shift-left of its scope and activities as well
providing standards and common data tooling for solution
data architecture
January 9, 2023 24
Shift Left Of The Scope Of Data Architecture
January 9, 2023 25
Source
Systems
Extract,
Transform,
Load
Data
Platform
Access
and
Usage
Changes to
Existing
Systems
and Their
Data
Models
New Custom
Developed
Applications
and Their
Data Models
Acquired and
Customised
Software
Products and
Their Data
Models
System
Integrations
/ Data
Transfers/
Exchanges
Reporting
and
Analysis
Facilities
Information
Storage
Facilities
Existing Data
Conversions
/ Migrations
New Data
Loads
• Shifting data architecture to
the left means getting involved
in the data aspects of solution
design, specification, selection
and implementation at the
earliest opportunity
• This then needs to be
repeated for each solution
within the organisation
solution landscape
• The data aspects of solutions
should be closely integrated
within the organisation’s data
architecture
Shift Left of Scope of Data
Architecture
Generalised Data Lifecycle
• Each data type within a
solution will have a lifecycle
from design and creation to
ultimate archival and
possible deletion
January 9, 2023 26
Enter, Create, Acquire,
Derive, Update,
Integrate, Capture
Secure, Store,
Replicate and
Distribute
Preserve, Protect and
Recover
Archive and Recall
Delete/Remove
Implement Underlying
Technology
Architect, Budget,
Plan, Design and
Specify
Present, Report,
Analyse, Model
• A set of data lifecycle view
for solutions can assist in
solution data architecture
Generalised Data Lifecycle Stages
• Architect, Budget, Plan, Design and Specify - This relates to the design and specification of the data
storage and management and their supporting processes
− This establishes the data management framework
• Implement Underlying Technology - This is concerned with implementing the data-related hardware and
software technology components
− This relates to database components, data storage hardware, backup and recovery software, monitoring and control software and other
items
• Enter, Create, Acquire, Derive, Update, Integrate, Capture - This stage is where data originated, such as
data entry or data capture and acquired from other systems or sources
• Secure, Store, Replicate and Distribute - In this stage, data is stored with appropriate security and access
controls including data access and update audit
− It may be replicated to other applications and distributed
• Present, Report, Analyse, Model - This stage is concerned with the presentation of information, the
generation of reports and analysis and the created of derived information
• Preserve, Protect and Recover - This stage relates to the management of data in terms of availability,
backup, recovery and retention/preservation
• Archive and Recall - This stage is where information that is no longer active but still required in archived
to secondary data storage platforms and from which the information can be recovered if required
• Delete/Remove - The stage is concerned with the deletion of data that cannot or does not need to be
retained any longer
− Data has to be able to be disposed of in a managed, systematic and auditable way
• Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer, Standards,
Governance, Fund - This is not a single stage but a set of processes and procedures that cross all stages
and is concerned with ensuring that the processes associated with each of the lifestyle stages are
operated correctly and that data assurance, quality and governance procedures exist and are operated
January 9, 2023 27
Solution Data Types And Lifecycles
• Every solution will
have one or more
types of data it
reads, processes
or creates
• Each data type will
have a separate
lifecycle that
reflects how it is
processed and
how its attributes
need to be
reflected in its
governance and
management
January 9, 2023 28
Solution
Data Type 1
Data Type 2
…
Data Type N
Data Lifecycle Stages And Solution Component Types
January 9, 2023 29
Central, Distributed
and Communications
Infrastructure
Changes to Existing
Systems
New Custom
Developed
Applications
System Integrations/
Data Transfers/
Exchanges
Changes to Existing
Business Processes
Organisational
Changes, Knowledge
Management
Training and
Documentation
Sets of Installation
and Implementation
Services
Parallel Runs
Enhanced Support/
Hypercare
Information Storage
Facilities
Acquired and
Customised Software
Products
New Business
Processes
Reporting and
Analysis Facilities
Existing Data
Conversions/
Migrations
New Data Loads
Operational
Functions and
Processes
Cutover/ Transfer to
Production
Sets of Maintenance,
Service Management
and Support Services
Application Hosting
and Management
Services
Secure, Store,
Replicate and
Distribute
Archive and Recall
Delete/Remove
Implement Underlying
Technology
Architect, Budget,
Plan, Design and
Specify
Present, Report,
Analyse, Model
Enter, Create, Acquire,
Derive, Update,
Integrate, Capture
Preserve, Protect and
Recover
Data Lifecycle Stages And Solution Components
• Each stage within the lifecycle of a solution data type will be realised by a solution component
• Mapping the stages within the lifecycle of solution data types and identifying the impact on solution
component types can contribute to effective solution data architecture design
− This provides traceability to ensure the data is being handled correctly
• For example, the Preserve, Protect and Recover data stage involving solution activities such as backup
and recovery, replication, business continuity and disaster recovery may require solution components of
the types:
− Information Storage Facilities
− Sets of Maintenance, Service Management and Support Services
− Operational Functions and Processes
January 9, 2023 30
Operational
Functions and
Processes
Sets of Maintenance,
Service Management
and Support Services
Information Storage
Facilities
Preserve, Protect and
Recover
So, What Do We Mean By Data Architecture?
• If data architecture can contribute to solution architecture
then the scope of data architecture should be defined and
agreed to ensure this is possible
January 9, 2023 31
Data Architecture And Data Strategy
• Data architecture defines the target data structures,
operations, principles, standards, organisation, tools,
management, governance that the organisation is aiming
to define, implement and operate
− The data architecture is designed to be implemented and
operated
• Data strategy defines how the organisation intends to use
data to deliver on its business strategy
− Data strategy precedes and feeds into the data architecture
January 9, 2023 32
Data Strategy And Data Architecture In A Wider
Business And Technology Context
January 9, 2023 33
Business
Objectives
Business
Architecture
Enterprise
Architecture
Solution
Implementation
and
Delivery
Support,
Management
and
Operations
Business
Processes
Required
Operational
Business
Solutions
Business
Strategy
Business
Solution
Analysis and
Design/
Selection
Business IT
Strategy
IT Function
Strategy
Required
Operational
Processes
Required
Supporting and
Enabling
Business
Solutions
Support
Solution
Analysis and
Design/
Selection
Required
Structure,
Capabilities
and
Resources
Digital
Strategy
Digital IT
Architecture
Solution
Portfolio
Design And
Specification
Solution
Portfolio
Management
Solution
Change and
Evolution
Business
Structure and
Operational
Model
Data
Strategy
Data
Architecture
Data Strategy And Data Architecture In A Wider
Business And Technology Context
• Data strategy
follows from
business strategy
and business
objectives
• Data architecture
translates the
conceptual nature of
the data strategy
into a more
implementation-
specific and –
oriented view
January 9, 2023 34
Business
Architecture
Enterprise
Architecture
Required
Operational
Business
Solutions
Business
Solution
Analysis and
Design/
Selection
Business IT
Strategy
IT Function
Strategy
Digital
Strategy
Digital IT
Architecture
Data
Strategy
Data
Architecture
Business
Objectives
Data Architecture
• A Data Architecture
exists to support the
objectives and the
operations of the
organisation
• This includes enabling
individual functional
solutions to be
designed and
implemented in
accordance with the
wider organisation data
architecture
January 9, 2023 35
Organisation Data
Architecture
Data
Infrastructure
Tools and
Facilities
Functional
Solutions
Data Standards
Data Architecture Structure
• For each set of subject arears within the data architecture
design and specification process, create an activity
breakdown based on the phases
− Research, Design, Define, Plan
− Implement, Operate
− Administer, Manage, Monitor, Improve
• Data architecture cannot be separated from its
implementation, operation and subsequent measurement
and improvement
• Architecture without execution and employment is
incomplete
January 9, 2023 36
Data Architecture Evolution And Development
• The data architecture is not static – it must be responsive to and accommodating of change
• It needs to evolve and develop in response to:
− Changing organisation needs and direction, driven by internal and/or external demands
− Changing organisation business strategy
− New technologies and capabilities that the organisation can usefully avail of
− Experience from implementation and operation
• The architecture should embed within itself explicitly the ability to assess its implementation
and operation and to grow, change, improve in response to these factors
January 9, 2023 37
Research,
Design,
Define,
Plan
Implement,
Operate
Administer,
Manage,
Monitor,
Improve
Define
Measurement
Framework,
Results and
Performance
Indicators
Review Delivery
and Operation
of Architecture
Experience and
Lessons from
Implementation
and Operation
Changing
Organisation
Needs and
Direction
Changes to
Organisation
Business
Strategy
New Data
Technologies
and
Capabilities
Data
Architecture
Changes
Data Architecture Subject Areas
Data
Architecture
Data Architecture Overall data architecture and data technology standards and design and implement data infrastructural
technology solutions
Data Management, Governance,
Supporting Processes
Standards, processes and their enforcement, planning, supervision, control and usage of data resources and
the design and implementation of data management processes, data ownership
Data Infrastructure, Storage and
Operations Software, Hardware and
Processes
Infrastructure hardware and software required to store and provide access to data, either on-premises or
hosted and facilities and processes required to operate and support the infrastructure, approach to analysis,
design, implementation, testing, deployment, maintenance and data storage structures
Data Security, Protection, Compliance,
Access Control, Authentication,
Authorisation
Approach to ensuring data security and protection, designing and implementing data security model covering
data, tools and infrastructure, ensuring compliance with regulatory standards, controlling access to data,
designing and implementing data authorisation model
Data Integration, Access, Flow, Exchange,
Transfer, Transformation, Load And
Extract
Data resource integration, extraction, transformation, movement, delivery, replication, transfer, sharing,
federation, virtualisation and operational support and approach to implementing a common approach and
providing a common set of tools
Content, Unstructured Data, Records and
Document Management
Approach to the implementation and management of acquisition, storage, indexing of and access to
unstructured data resources such as files and digitised paper records and the integration of these resources
with structured data resources
Master and Reference Data Management
Approach to the implementation and management of master versions of shared data resources to reduce
redundancy and maintain data quality through standardised data definitions and use of common data lookup
values including data dictionaries
Data Warehouse, Data Marts, Data Lakes Facilities for storing data extracted from operational systems for long-term storage and to enable access for
reporting and analysis
Data Reporting and Analytics,
Visualisation Tools and Facilities
Approach to providing a common approach and providing a common set of tools, facilities and supporting
technologies and standards for data reporting, decision support, analysis and visualisation
Data Discovery, Analysis, Design and
Modelling
Approach to the implementation and management of data description standards and the collection,
categorisation, maintenance, integration, application, use and management of data descriptions including
data catalogs
External Data Sources and Interacting
Parties Data Transfer/Exchange/
Integration/Publication
Management of data sources and targets outside the organisation and the parties that provide that data or to
whim the data is made available including contracts and agreement, service levels, access approaches
Metadata Data Management
Approach to the implementation and management of data description standards and the collection,
categorisation, maintenance, integration, application, use and management of data descriptions including
data catalogs
Data Quality Designing, implementing and operating approach, processes and standards to ensure and maintain data
quality
Data Solution Design Defining and implementing standards relating to the use of data within solutions
January 9, 2023 38
Data Architecture Subject Areas
• This is intended to represent a comprehensive view of data
architecture
January 9, 2023 39
Data Architecture Subject Areas
• The proposed subject areas do not exist in isolation
• They are interrelated areas on which to focus analysis,
planning and design effort and attention while maintaining
a higher level and more complete and integrated view
• The individual topics allow each subject area to be
analysed and specified in detail that is appropriate for the
organisation
• The topics are designed to be independent of any specific
hardware, software or platform technology
January 9, 2023 40
Relationships Between Data Architecture Topics
January 9, 2023 41
Data Solution Design
Data Quality
External Data Sources and
Interacting Parties Data
Transfer/Exchange/
Integration/Publication
Data Discovery, Analysis,
Design and Modelling
Data Infrastructure, Storage
and Operations Software,
Hardware and Processes
Data Security, Protection,
Compliance, Access Control,
Authentication,
Authorisation
Content, Unstructured Data,
Records and Document
Management
Master and Reference Data
Management
Data Architecture
Data Management,
Governance, Supporting
Processes
Data Reporting and
Analytics, Visualisation
Tools and Facilities
Data Warehouse, Data
Marts, Data Lakes
Metadata Data
Management
Data Integration, Access,
Flow, Exchange, Transfer,
Transformation, Load And
Extract
Data Architecture Topic Scope
Data Architecture
Research, Design, Define, Plan
Data Architecture Strategy and Scope
Definition
Data Architecture Capability Establishment
Define Current Data Architecture Baseline,
Inventory, Gaps, Issues, Concerns
Define Architecture Supporting Tools and
Processes Definition
Data Architecture Scope and Activities
Data Architecture Strategy and Scope
Definition
Data Architecture Capability Establishment
Define Current Data Architecture Baseline,
Inventory, Gaps, Issues, Concerns
Define Architecture Supporting Tools and
Processes Definition
Data Architecture Scope and Activities
Data Architecture Implementation Planning
Implement, Operate
Data Architecture Supporting Tools and
Processes Implementation and Operation
Data Architecture Team Formation
Data Architecture Implementation
Data Architecture Performance and Results
Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Data Architecture Review and Improvement
Data Architecture Management
Data Architecture Operation Assessment
January 9, 2023 42
Data Management, Governance, Supporting
Processes Topic Scope
Data Management, Governance,
Supporting Processes
Research, Design, Define, Plan
Data Governance Capability Establishment
Define Governance Strategy
Define Current Data Governance Baseline
Define Governance Supporting Tools and Processes
Definition
Data Governance Scope and Activities
Define Governance Policies Define Governance Standards
Define Governance Compliance, Monitoring and
Reporting Data Persistence Standards
Data Lifecycle Definition and Management Create Data Asset Inventory
Create Business Glossary Perform Data Value Assessment
Data Governance Implementation Planning
Data Governance Process Definition
Implement, Operate
Data Governance Supporting Tools and Processes
Implementation and Operation
Data Governance Team Formation
Data Governance Implementation
Data Governance Performance and Results
Indicators and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Governance Review and Improvement
Data Governance Management
Data Governance Operation Assessment
Data Governance Implementation and Operation
Reporting
January 9, 2023 43
Data Infrastructure, Storage and Operations Software,
Hardware and Processes Topic Scope
Data Infrastructure, Storage and Operations Software,
Hardware and Processes
Research, Design, Define, Plan
Data Infrastructure, Storage and Operations Capability
Establishment
Data Infrastructure and Storage Hardware, Software and
Platform Inventory
Data Operations and Process Inventory
Data Infrastructure, Storage and Operations Existing
Processes and Standards Inventory and Review
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Definition
Data Infrastructure, Storage and Operations Software and
Hardware Scope and Activities
Data Storage Hardware Technology Target Definition Data Storage Software Technology Target Definition
Data Storage Platform Technology Target Definition
Data Product, Platform and Vendor Selection and
Management
Data Backup and Recovery Data Infrastructure Performance Monitoring Tools
Data Archival and Purge Tools
Data Infrastructure, Storage and Operations Availability,
Business Continuity, Disaster Recovery and Replication
Definition
Data Performance Testing and Validation Approach
Data Infrastructure, Storage and Operations Standards
Definition
Data Infrastructure, Storage and Operations Performance
and Capacity Planning Standards and Data Collection and
Analysis
Data Infrastructure, Storage and Operations
Implementation Planning
Implement, Operate
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Supporting Tools and Processes
Implementation and Operation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Team Formation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Implementation
Data Infrastructure, Storage and Operations Supporting
Tools and Processes Performance and Results Indicators
and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Infrastructure, Storage and Operations Review and
Improvement
Data Infrastructure, Storage and Operations Management
Data Infrastructure, Storage and Operations Operation
Assessment
January 9, 2023 44
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Topic Scope
Data Security, Protection, Compliance, Access
Control, Authentication, Authorisation
Research, Design, Define, Plan
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Capability Establishment
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Existing Approach
Inventory and Baseline
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Supporting Tools and
Processes Definition
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Scope and Activities
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Architecture Definition
Compliance, Regulatory and Data Protection
Requirements Across All Data Types
Security Information, Event and Alert Logging and
Auditing
Data Loss Prevention
Data Security Product, Platform and Vendor Selection
and Management
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Standards Definition
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Monitoring, Data
Collection and Analysis
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Implementation Planning
Implement, Operate
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Supporting Tools and
Processes Implementation and Operation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Team Formation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Implementation
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Performance and Results
Indicators and Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Review and Improvement
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Management
Data Security, Protection, Compliance, Access Control,
Authentication, Authorisation Operation Assessment
January 9, 2023 45
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Topic Scope
Data Integration, Access, Flow, Exchange,
Transfer, Transformation, Load And Extract
Research, Design, Define, Plan
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Capability
Establishment
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Existing Approach
Inventory and Baseline
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Supporting Tools and
Processes Definition
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Scope and Activities
Data Integration Security, Authentication, Authorisation
Data Integration Product, Platform and Vendor Selection
and Management
Data Integration Scheduler and Rules Engine
Internal and External Data Sources, Targets and Channels
Definition
Data Integration Development, Testing and Deployment
Data Integration Operations Management,
Administration
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Standards Definition
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Monitoring, Data
Collection and Analysis
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Implementation
Planning
Implement, Operate
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Supporting Tools and
Processes Implementation and Operation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Team Formation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Implementation
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Performance and
Results Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Review and
Improvement
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Management
Data Integration, Access, Flow, Exchange, Transfer,
Transformation, Load And Extract Operation Assessment
January 9, 2023 46
Content, Unstructured Data, Records and Document
Management Topic Scope
Content, Unstructured Data, Records
and Document Management
Research, Design, Define, Plan
Content, Unstructured Data, Records and Document
Management Capability Establishment
Content, Unstructured Data, Records and Document
Management Existing Approach Inventory and Baseline
Content, Unstructured Data, Records and Document
Management Supporting Tools and Processes Definition
Content, Unstructured Data, Records and Document
Management Scope and Activities
Content, Unstructured Data, Records and Document
Management Security, Authentication, Authorisation
Data Integration Product, Platform and Vendor Selection
and Management
Records Management Strategy Metadata Management
Content, Unstructured Data, Records and Document
Lifecycle Management
Content, Unstructured Data, Records and Document
Management Standards Definition
Content, Unstructured Data, Records and Document
Management Monitoring, Data Collection and Analysis
Content, Unstructured Data, Records and Document
Management Implementation Planning
Implement, Operate
Content, Unstructured Data, Records and Document
Management Supporting Tools and Processes
Implementation and Operation
Content, Unstructured Data, Records and Document
Management Team Formation
Content, Unstructured Data, Records and Document
Management Implementation
Content, Unstructured Data, Records and Document
Management Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor, Improve
Content, Unstructured Data, Records and Document
Management Review and Improvement
Content, Unstructured Data, Records and Document
Management Management
Content, Unstructured Data, Records and Document
Management Operation Assessment
January 9, 2023 47
Master and Reference Data Management Topic Scope
Master and Reference Data
Management
Research, Design, Define, Plan
Master and Reference Data Management Capability
Establishment
Master and Reference Data Management Existing
Approach Inventory and Baseline
Master and Reference Data Management Supporting
Tools and Processes Definition
Master and Reference Data Management Scope and
Activities
Industry Data Standards Data Glossaries and Taxonomies
Business Rules Analysis and Definition
Master and Reference Data Management Product,
Platform and Vendor Selection and Management
Master Data Stores Reference Data Stores
Master and Reference Data Management Standards
Definition
Master and Reference Data Management Monitoring,
Data Collection and Analysis
Master and Reference Data Management
Implementation Planning
Implement, Operate
Master and Reference Data Management Supporting
Tools and Processes Implementation and Operation
Master and Reference Data Management Team
Formation
Master and Reference Data Management
Implementation
Master and Reference Data Management Performance
and Results Indicators and Measurement Framework
Definition
Administer, Manage, Monitor, Improve
Master and Reference Data Management Review and
Improvement
Master and Reference Data Management Management
Master and Reference Data Management Operation
Assessment
January 9, 2023 48
Data Warehouse, Data Marts, Data Lakes Topic Scope
Data Warehouse, Data Marts, Data
Lakes
Research, Design, Define, Plan
Data Warehouse, Data Marts, Data Lakes Capability
Establishment
Data Warehouse, Data Marts, Data Lakes Existing
Approach Inventory and Baseline
Data Warehouse, Data Marts, Data Lakes Supporting
Tools and Processes Definition
Data Warehouse, Data Marts, Data Lakes Scope and
Activities
Data Models Creation Long-term Data Storage Architecture
Data Integration and Population
Data Warehouse, Data Marts, Data Lakes Product,
Platform and Vendor Selection and Management
Data Access Metadata Management
Data Virtualisation
Data Warehouse, Data Marts, Data Lakes Standards
Definition
Data Warehouse, Data Marts, Data Lakes
Monitoring, Data Collection and Analysis
Data Warehouse, Data Marts, Data Lakes
Performance and Capacity Planning Standards and
Data Collection and Analysis
Data Warehouse, Data Marts, Data Lakes
Implementation Planning
Implement, Operate
Data Warehouse, Data Marts, Data Lakes Supporting
Tools and Processes Implementation and Operation
Data Warehouse, Data Marts, Data Lakes Team
Formation
Data Warehouse, Data Marts, Data Lakes
Implementation
Data Warehouse, Data Marts, Data Lakes
Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor,
Improve
Data Warehouse, Data Marts, Data Lakes Review
and Improvement
Data Warehouse, Data Marts, Data Lakes
Management
Data Warehouse, Data Marts, Data Lakes Operation
Assessment
January 9, 2023 49
Data Reporting and Analytics, Visualisation Tools and
Facilities Topic Scope
Data Reporting and Analytics,
Visualisation Tools and Facilities
Research, Design, Define, Plan
Data Reporting and Analytics, Visualisation Tools
and Facilities Capability Establishment
Data Reporting and Analytics, Visualisation Tools
and Facilities Existing Approach Inventory and
Baseline
Data Reporting and Analytics, Visualisation Tools
and Facilities Supporting Tools and Processes
Definition
Data Reporting and Analytics, Visualisation Tools
and Facilities Scope and Activities
Reporting and Visualisation Architecture and
Approach
Analytics Architecture and Approach
Data Integration, Access and Security
Data Reporting and Analytics, Visualisation Facility
Access and Security
Data Reporting and Analytics, Visualisation Product,
Platform and Vendor Selection and Management
Data Reporting and Analytics, Visualisation
Development, Testing and Deployment
Data Reporting and Analytics, Visualisation
Distribution and Security
Data Reporting and Analytics, Visualisation Tools
and Facilities Standards Definition
Data Reporting and Analytics, Visualisation Tools
and Facilities Monitoring, Data Collection and
Analysis
Data Reporting and Analytics, Visualisation Tools
and Facilities Performance and Capacity Planning
Standards and Data Collection and Analysis
Data Reporting and Analytics, Visualisation Tools
and Facilities Implementation Planning
Implement, Operate
Data Reporting and Analytics, Visualisation Tools
and Facilities Supporting Tools and Processes
Implementation and Operation
Data Reporting and Analytics, Visualisation Tools
and Facilities Team Formation
Data Reporting and Analytics, Visualisation Tools
and Facilities Implementation
Data Reporting and Analytics, Visualisation Tools
and Facilities Performance and Results Indicators
and Measurement Framework Definition
Administer, Manage, Monitor,
Improve
Data Reporting and Analytics, Visualisation Tools
and Facilities Review and Improvement
Data Reporting and Analytics, Visualisation Tools
and Facilities Management
Data Reporting and Analytics, Visualisation Tools
and Facilities Operation Assessment
January 9, 2023 50
Data Discovery, Analysis, Design and Modelling Topic
Scope
Data Discovery, Analysis, Design and Modelling
Research, Design, Define, Plan
Data Discovery, Analysis, Design and Modelling
Capability Establishment
Data Discovery, Analysis, Design and Modelling Existing
Approach Inventory and Baseline
Data Discovery, Analysis, Design and Modelling
Supporting Tools and Processes Definition
Data Discovery, Analysis, Design and Modelling Scope
and Activities
Data Modelling Definition Data Profiling Approach Definition
Data Discovery and Profiling Tool Selection and
Implementation
Data Lineage Definition Including Tool Selection
Data Catalog Definition Including Tool Selection Data Dictionary Definition Including Tool Selection
Semantic Layer Definition Including Tool/Platform
Selection
Data Discovery, Analysis, Design and Modelling
Standards Definition
Data Discovery, Analysis, Design and Modelling
Monitoring, Data Collection and Analysis
Data Discovery, Analysis, Design and Modelling
Performance and Capacity Planning Standards and Data
Collection and Analysis
Data Discovery, Analysis, Design and
Modelling Implementation Planning
Implement, Operate
Data Discovery, Analysis, Design and Modelling
Supporting Tools and Processes Implementation and
Operation
Data Discovery, Analysis, Design and Modelling Team
Formation
Data Discovery, Analysis, Design and Modelling
Implementation
Data Discovery, Analysis, Design and Modelling
Performance and Results Indicators and Measurement
Framework Definition
Administer, Manage, Monitor, Improve
Data Discovery, Analysis, Design and Modelling Review
and Improvement
Data Discovery, Analysis, Design and Modelling
Management
Data Discovery, Analysis, Design and Modelling
Operation Assessment
January 9, 2023 51
External Data Sources and Interacting Parties Data
Transfer/ Exchange/ Integration/ Publication Topic Scope
External Data Sources and Interacting Parties
Data Transfer/Exchange/Integration/Publication
Research, Design, Define, Plan
External Data Sources and Interacting Parties Capability
Establishment
External Data Sources and Interacting Parties Existing
Approach Inventory and Baseline
External Data Sources and Interacting Parties Supporting
Tools and Processes Definition
External Data Sources and Interacting Parties Scope and
Activities
Data Transfer/Exchange/Integration/Publication
Request Review and Approval Process
Data Transfer/Exchange/Integration/Publication
Implementation Process Definition
Data Transfer/Exchange/Integration/Publication Toolset
Definition and Acquisition
Data Transfer/Exchange/Integration/Publication Activity
Monitoring and Review
Data Transfer/Exchange/Integration/Publication Access
Standards
Data Transfer/Exchange/Integration/Publication Open
Data Approach
Data Transfer/Exchange/Integration/Publication
Security Definition
External Data Sources and Interacting Parties Standards
Definition
External Data Sources and Interacting Parties
Monitoring, Data Collection and Analysis
External Data Sources and Interacting Parties
Implementation
Implement, Operate
External Data Sources and Interacting Parties Supporting
Tools and Processes Implementation and Operation
External Data Sources and Interacting Parties Team
Formation
External Data Sources and Interacting Parties
Implementation
External Data Sources and Interacting Parties
Performance and Results Indicators and Measurement
Framework Definition
Administer, Manage, Monitor, Improve
External Data Sources and Interacting Parties Review
and Improvement
External Data Sources and Interacting Parties
Management
External Data Sources and Interacting Parties Operation
Assessment
January 9, 2023 52
Metadata Data Management Topic Scope
Metadata Data Management
Research, Design, Define, Plan
Metadata Data Management Capability
Establishment
Metadata Data Management Existing Approach
Scope Definition, Inventory and Baseline
Metadata Data Management Supporting Tools
and Processes Definition
Metadata Data Management Scope and Activities
Define Metadata Architecture and Approach Metadata Standard Review
Metadata Data Management Standards Definition Define Metadata Management Tools
Metadata Creation and Maintenance Approach Metadata Repositories Definition
Metadata Integration and Usage Approach
Metadata Data Management Monitoring, Data
Collection and Analysis
Metadata Data Management Performance and
Capacity Planning Standards and Data Collection
and Analysis
Metadata Data Management Implementation
Planning
Implement, Operate
Metadata Data Management Supporting Tools
and Processes Implementation and Operation
Metadata Data Management Team Formation
Metadata Data Management Implementation
Metadata Data Management Performance and
Results Indicators and Measurement Framework
Definition
Baseline Metadata Creation
Administer, Manage, Monitor,
Improve
Metadata Data Management Review and
Improvement
Metadata Data Management Management
Metadata Data Management Operation
Assessment
January 9, 2023 53
Data Quality Topic Scope
Data Quality
Research, Design, Define, Plan
Data Quality Capability Establishment
Data Quality Existing Approach Inventory, Profile and
Baseline
Data Quality Supporting Tools and Processes Definition
Data Quality Scope and Activities
Data Quality Requirements Data Quality Rules Approach
Data Quality Service Level Management Approach Data Quality Analysis and Reporting
Data Quality Standards Definition
Data Quality Monitoring, Data Collection and Analysis
Data Quality Implementation Planning
Implement, Operate
Data Quality Supporting Tools and Processes
Implementation and Operation
Data Quality Team Formation
Data Quality Implementation
Data Quality Performance and Results Indicators and
Measurement Framework Definition
Administer, Manage, Monitor, Improve
Data Quality Review and Improvement
Data Quality Management
Data Quality Operation Assessment
January 9, 2023 54
Data Solution Design Topic Scope
Data Solution Design
Research, Design, Define, Plan
Data Solution Design Advisory Capability
Establishment
Data Solution Design Existing Approach Review,
Inventory and Baseline
Data Solution Design Scope and Activities
Data Management and Governance Standards Data Modelling and Design Standards
Operational and Archival Data Data Storage and Persistence Standards
Data Infrastructure Standards Data Transfer/Exchange/Integration Standards
Data Reporting and Analysis Standards Data Performance and Throughput Standards
Data Security Standards
Data Solution Design Monitoring, Data Collection
and Analysis
Data Solution Design Implementation Planning
Implement, Operate
Data Solution Design Supporting Tools and
Processes Implementation and Operation
Data Solution Design Team Formation
Data Solution Design Implementation
Data Solution Design Performance and Results
Indicators and Measurement Framework
Definition
Administer, Manage, Monitor,
Improve
Data Solution Design Review and Improvement
Data Solution Design Management
Data Infrastructure, Storage and Operations
Operation Assessment
January 9, 2023 55
Data Architecture Coverage Of Solution Component
Types
January 9, 2023 56
Solution
Component
Types
Data Architecture Subject Areas
Data
Architecture
Data
Management,
Governance,
Supporting
Processes
Data
Infrastructure,
Storage and
Operations
Software,
Hardware and
Processes
Data Security,
Protection,
Compliance,
Access Control,
Authentication,
Authorisation
Data
Integration,
Access, Flow,
Exchange,
Transfer,
Transformation,
Load And
Extract
Content,
Unstructured
Data, Records
and Document
Management
Master and
Reference Data
Management
Data
Warehouse,
Data Marts,
Data Lakes
Data Reporting
and Analytics,
Visualisation
Tools and
Facilities
Data Discovery,
Analysis,
Design and
Modelling
External Data
Sources and
Interacting
Parties Data
Transfer/
Exchange/
Integration/
Publication
Metadata Data
Management
Data Quality Data Solution
Design
Changes to Existing Systems
X X X X X X X X X X X X
New Custom Developed
Applications X X X X X X X X X X X X
Acquired and Customised
Software Products X X X X X X X X X X X X
System Integrations/ Data
Transfers/ Exchanges X X X
Reporting and Analysis
Facilities X X X X X X X X
Sets of Installation and
Implementation Services
Information Storage
Facilities X
Existing Data Conversions/
Migrations X X X X X X
New Data Loads
X X X X X X
Central, Distributed and
Communications
Infrastructure
X X
Cutover/ Transfer to
Production And Support
Operational Functions and
Processes
Parallel Runs
Enhanced Support/
Hypercare
Sets of Maintenance, Service
Management and Support
Services
Application Hosting and
Management Services
Changes to Existing Business
Processes
New Business Processes
Organisational Changes,
Knowledge Management
Training and Documentation
Data Architecture Coverage Of Solution Component
Types
• The data architecture subject areas impact the data
aspects of many solution component types within solutions
• An effective data architecture can contribute to effective
solution architecture and solution data architecture
January 9, 2023 57
Data Architecture And Common Data Tooling And
Standards
• Data architecture needs to provide common infrastructural data tools and
common data standards for solutions
− Tools
• Data storage infrastructure – hardware, software and platforms
• Data warehouse platform
• Data reporting, visualisation and analysis
• Data transfer/exchange/integration/extract/transform/load
• Data operations – backup/recovery/replication/business continuity/disaster recovery
• Data anonymisation/pseudonymisation/encryption
• Data monitoring/performance/capacity planning
• Master data management platform
• Reference data platform
• Data catalog/semantic layer
• Document management
• Data analysis
− Standards
• Data security
• Data quality
• Metadata
• Data discovery
• Data modelling
• Data management – classification/retention/archive/deletion
January 9, 2023 58
Organisation Data Architecture And Solution Data
Architecture
January 9, 2023 59
Organisation Data
Architecture
Common Data
Standards
Common Data
Infrastructure
Tools and
Facilities
Common Data
Operations
Individual
Solutions
Individual
Solutions
Individual
Solutions
Common Data
Design and
Implementation
Approaches
Common Data
Model
Organisation Data Architecture And Solution Data
Architecture
• Common data infrastructure tools allows reuse, reduces
decision-making overhead and delays, reduces cost,
accelerates individual deployment and achieves
standardisation
• Toolset will need to change in response to changing
business needs and technology landscape
January 9, 2023 60
Common Data Plumbing Infrastructure
• Data architecture should take on the projects required to deliver the
common data plumbing infrastructure
• These are foundational components
• Individual solution delivery activities should not have to be responsible for
their implementation
January 9, 2023 61
Common Data Plumbing Infrastructure
January 9, 2023 62
Common ETL
Common Data
Transfer/
Exchange
Common API
Layer
Common API
Layer
Common Data
Infrastructure/
Platform
Common Data
Warehouse
Common Data
Reporting
Common Data
Analytics
Common Data
Backup and
Recovery
Common
Business
Continuity and
Disaster Recovery
Common
Document
Management
Common Data
Analytics
Common
Performance
Monitoring and
Capacity Planning
Common Audit
Logging Data
Management
Common Data
Catalog
Common
Reference and
Master Data
Individual
Solutions
Common Data Plumbing Infrastructure
• This ideal is regularly not fully in place
• Individual solutions often have to implement some of
these capabilities that are not available centrally
• The leads to sub-optimal solutions with point resolutions
to specific requirements
January 9, 2023 63
Data Design And Modelling For Solutions
• The objective of data design for solutions is the same as
that for overall solution design:
− To capture sufficient information to enable the solution design to
be implemented
− To unambiguously define the data requirements of the solution
and to confirm and agree those requirements with the target
solution consumers
− To ensure that the implemented solution meets the requirements
of the solution consumers and that no deviations have taken place
during the solution implementation journey
January 9, 2023 64
Why Pay Attention To Solution Data Architecture?
• Solution data architecture avoids problems with solution
operation and use:
− Poor and inconsistent data quality
− Poor performance, throughput, response times and scalability
• Poorly designed data structures can lead to long data update times leading to
long response times, affecting solution usability, loss of productivity and
transaction abandonment
− Poor reporting and analysis
− Poor data integration
− Poor solution serviceability and maintainability
− Manual workarounds for data integration, data extract for reporting and
analysis
• Data-design-related solution problems frequently become
evident and manifest themselves only after the solution goes
live
• The benefits of solution data architecture are not always
evident initially
January 9, 2023 65
New Technology And Impact on Solution Data
Architecture
• New solution deployment and operating models affects
solution data architecture
− New solution design, deployment and operating models
− Greater use of platform-based solution implementation and
deployment
− Wider range of complex data technology options, especially in
terms of data analysis
− Distributed solution components, distributed solution consumer
base, distributed access with many interfaces, integration points
and data flows
− Complexity with multiple data integrations
January 9, 2023 66
Solution Design – From …
January 9, 2023 67
Solution
Central Data
Store
Solution
Central
Application
Component
Solution API
Solution
Central
Infrastructure
Solution
Hosted
Infrastructure
Solution
Internal
Consumers
Solution
External
Private
Consumers
Solution
Hosted Data
Store
Solution
Hosted
Application
Component
Solution
Hosted
Analytics
Access and
Security
Infrastructure
Central To
Hosting
Facility
Connectivity
Solution
External Public
Consumers
Solution
Mobile App
To …
• Increasing solution landscape complexity and diversity gives rise to greater data
design complexity
January 9, 2023 68
Solution
Central Data
Store
Solution
Central
Application
Component
Solution API
Solution
Central
Infrastructure
Solution
Hosted
Infrastructure
Solution
Internal
Consumers
Solution
External
Private
Consumers
Solution
Hosted Data
Store
Solution
Hosted
Application
Component
Solution
Hosted
Analytics
Access and
Security
Infrastructure
Central To
Hosting
Facility
Connectivity
Solution
External Public
Consumers
Solution
Mobile App
Solution Entity Model
January 9, 2023 69
Solution
Component
Types
Solution
Components
Solution
Solution Zones
Solution
Zone Types
Solution
Topology
Solution Consists Of
Multiple Components
Each Solution
Component
Has A Type
Solution Exists
Within A
Topology Of
Many Solutions
Solution Components
Are Located In Solution
Zones
Each Solution
Zone Has A Type
Solution
Operational
Entity
Solution
Operational
Entity Type
Deployed
Solution
Consists Of
Multiple
Operational
Entities
Each Solution
Operational
Entity Has A Type
Solution Operational Entities
Are Located In Solution Zones
Some Solution
Components
Become
Deployed
Operational
Entities
Solution Zone Types and Zones
January 9, 2023 70
Solution
Component
Types
Solution
Components
Solution
Solution Zones
Solution
Zone Types
Solution
Topology
Solution Consists Of
Multiple Components
Each Solution
Component
Has A Type
Solution Exists
Within A
Topology Of
Many Solutions
Solution Components
Are Located In Solution
Zones
Each Solution
Zone Has A Type
Solution
Operational
Entity
Solution
Operational
Entity Type
Deployed
Solution
Consists Of
Multiple
Operational
Entities
Each Solution
Operational
Entity Has A Type
Solution Operational Entities
Are Located In Solution Zones
Some Solution
Components
Become
Deployed
Operational
Entities
Solution Zones
• Solution zones are locations where groups of closely related solution
components reside
• They represent containers for solution components
• Zones are located within the wider physical solution landscape
• Each zone and the components it holds have different security
requirements
• Not all solutions will have components in all zone and not all
organisations will have all the zone types
• The solution and its constituent components can span multiple
different zones of the same type
• The zone approach is useful way of representing the entirety of a
solution, its constituent components, their connectivity, linkages and
interactions, especially data storage, processing and interactions
• You will have different levels of control over different solution zones
(including no control) – this impacts data design considerations
January 9, 2023 71
Sample Solution Zone Types
January 9, 2023 72
Sample Solution Zone Types
January 9, 2023 73
Sample Solution Zone Types
Zone Description
Insecure External Organisation
Presentation And Access
Where publicly accessible or accessing entities reside. These entities are regarded
as insecure and/or untrusted.
Secure External Organisation
Participation and Collaboration
Outside the physical organisation boundary where entities that are provided by or
to trusted external parties reside
Secure External Organisation Access Contain entities that enable secure access or are securely accessible from outside
the organisation
Organisation Contain the entities within the organisation boundary and contains all the
locations, business units and functions within it
Central Solutions and Access Contains the solution entities and their data
Solution Zone Contains the solution entities
Data Zone Zone within the organisation where data is segregated for security
Remote Business Unit Solutions and
Access
Remotely located organisation business unit or location and the entities it
contains
Workstation Zone Zone within the organisation where users accessing data and solutions are
segregated for security
Outsourced Service Provider Solutions
and Access
Contains solutions provided by and located in facilities provided by outsourced
partners
Cloud Service Provider Solutions and
Access
Contains solutions - platform, infrastructure and service - provided by and located
in cloud service providers
Co-Located Solutions and Access Contains solutions the organisation has located in facilities provided by co-
location providers
January 9, 2023 74
Solution Consumers
• Any solution with have different sets of consumers of different
types:
− Controlled Consumers – typically organisation personnel over whom the
solution owner has substantial control
− Partially Controlled Consumers – typically external business partners
and other interacting parties over whom the solution owner has some
control and influence
− Uncontrolled Consumers – typically members of the public at whom the
solution is targeted and whose needs must be inferred through groups
of proxy consumers
• Each consumer type will have different data-related needs and
expectations
• Classifying and understanding the target solution consumers
will contribute to solution data architecture design
January 9, 2023 75
Data Design And Modelling For Solutions Activities
Data
Modelling
Data Journeys
Design
Data
Processing
Design
January 9, 2023 76
Logical
Data Model
Physical Data
Model
Conceptual
Data Model
Data Design And Modelling For Solutions Activities
• This includes the following activities:
− Data Modelling – define the data entities, structures, attributes
and contents
• Conceptual Data Model (CDM) – create an initial high-level view of solution
• Logical Data Model (LDM) – expand the CDM with detailed data
requirements
• Physical Data Model (PDM) – translate the LDM into implementation-
specific details
− Data Journeys Design – create an inventory of data journeys and
identify the steps within the journeys and the data entities
involved
− Data Processing Design – define the detail of the processing
performed on data entities
January 9, 2023 77
Data Design And Modelling For Solutions Activities
• The data analysis and design activities are not linear or
sequential
• As the analysis progresses, earlier work may need to be
revisited to be elaborated and expanded on
January 9, 2023 78
Conceptual Data
Model
Logical Data
Model
Physical Data
Model
Data Journeys
Design
Data Processing
Design
Data Modelling
Packaged Solutions And Platforms
• Packaged solution components and platforms on which
solutions components are implemented and deployed will
have pre-defined data models with a greater or lesser
degree of configuration and customisation
• The data design activities should still be performed for
these solution components
• The inherent data limitations and restrictions of the
packages and platforms should be clearly defined and
understood
January 9, 2023 79
Solutions And Shared/Private Data
January 9, 2023 80
• Solutions and
their
components
within the
organisation
solution
landscape will
have both local
data and data
that is shared
with or between
other solutions,
either for
upstream/
downstream
processing or as
shared data
repositories
Solutions And Shared/Private Data
• Data design activities are different for shared and private
solution data
• Shared solution data includes reference and master data
− Reference data consists of common code, structural and identifier
values
• Their purpose is to ensure consistency across data values
• Reference data is static or slowly changing
− Master data relates to common transaction identifiers such interacting
parties (customer, partner, etc,) details
• Having a single version of master data ensures single view of all interactions with
the party across the organisation can be identified
• Solution data modelling activities should identify the
occurrences of reference and master data to maximise reuse
and contribute to maintaining a single version of the truth
January 9, 2023 81
Shared/Common Data Issues
• Because shared data is shared, the main concerns and
issues relate to:
− Ownership – who owns and is responsible
− Maintenance – who maintains it and keeps it current, who is
responsible for allowing updates, what is the process for applying
updates and changes
− Quality – who is responsible for maintaining data quality
• Individual solutions should not have to solve these
problems, but commonly and unfortunately have to
January 9, 2023 82
Common Data Formats
• Master data will have a common format
• Individual solutions need to adhere to the common master
data format
January 9, 2023 83
Data Modelling – Conceptual Data Model
• The Conceptual Data Model (CDM) represents concepts,
entities and their relationships within the scope of the
solution
• It is used to create a common understanding among all the
solution stakeholders
• The CDM defines the scope of the solution
• The functional requirements of the solution provide an
input to the CDM and its constituent entities
January 9, 2023 84
Data Modelling – Logical Data Model
• The Logical Data Model (LDM) expands on the agreed CDM
• Detailed data requirements for the specified data entities
are defined, including solution zones
January 9, 2023 85
Data Modelling – Physical Data Model
• The Physical Data Model (PDM) translates the LDM into
technology-specific implementation details and a
technology structure across the solution zones
• The LDM may be updated to reflect and accommodate
technology and platform specific features, limitations,
capabilities and restrictions
January 9, 2023 86
Data Journeys Design
• Solutions have journeys as consumers use the solution to
achieve results
• Solution journeys reflect solution consumer experiences
• Data journeys represent the data exchanges and transfers
that occur to support the solution journeys
• Solution data architecture should first create an inventory
of data journeys
• The processing data journeys can then be expanded to
reflect the lifecycle for the data type(s) associated with the
data journeys
January 9, 2023 87
Data Processing Design
• Data processing design describes the detailed processing
that is performed on data
• It defines the business rules that are applied to data within
the scope of the solution
January 9, 2023 88
Data Processing Design
Identify the
data
processing
performed on
data entities
and objects
by each
solution
component
for each
solution
journeys
January 9, 2023 89
Data
Entity/Object
Solution
Component
Data
Entity/Object
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Solution
Component
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
Data
Entity/Object
D1
D2
D3
D4 D5
Summary
• The data architecture of solutions is frequently not given the attention it deserves or needs
• Frequently, too little attention is paid to designing and specifying the data architecture within
individual solutions and their constituent components
• This is due to the behaviours of both solution architects ad data architects
• Solution architecture tends to concern itself with functional, technology and software
components of the solution
• Data architecture tends not to get involved with the data aspects of technology solutions,
leaving a data architecture gap
• Combined with the gap where data architecture tends not to get involved with the data aspects
of technology solutions, there is also frequently a solution architecture data gap
• Solution architecture also frequently omits the detail of data aspects of solutions leading to a
solution data architecture gap
• These gaps result in a data blind spot for the organisation
• Data architecture tends to concern itself with post-individual solutions
• Data architecture needs to shift left into the domain of solutions and their data and more
actively engage with the data dimensions of individual solutions
• Data architecture can provide the lead in sealing these data gaps through a shift-left of its scope
and activities as well providing standards and common data tooling for solution data
architecture
January 9, 2023 90
More Information
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
https://www.researchgate.net/profile/Alan-Mcsweeney
https://www.amazon.com/dp/1797567616
9 January 2023 91

More Related Content

What's hot

Overcoming the Challenges of your Master Data Management Journey
Overcoming the Challenges of your Master Data Management JourneyOvercoming the Challenges of your Master Data Management Journey
Overcoming the Challenges of your Master Data Management JourneyJean-Michel Franco
 
Data platform architecture
Data platform architectureData platform architecture
Data platform architectureSudheer Kondla
 
Data Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data StrategyData Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data StrategyAlan McSweeney
 
Review of Data Management Maturity Models
Review of Data Management Maturity ModelsReview of Data Management Maturity Models
Review of Data Management Maturity ModelsAlan McSweeney
 
Data Profiling, Data Catalogs and Metadata Harmonisation
Data Profiling, Data Catalogs and Metadata HarmonisationData Profiling, Data Catalogs and Metadata Harmonisation
Data Profiling, Data Catalogs and Metadata HarmonisationAlan McSweeney
 
Reference master data management
Reference master data managementReference master data management
Reference master data managementDr. Hamdan Al-Sabri
 
Capability Model_Data Governance
Capability Model_Data GovernanceCapability Model_Data Governance
Capability Model_Data GovernanceSteve Novak
 
Selecting Data Management Tools - A practical approach
Selecting Data Management Tools - A practical approachSelecting Data Management Tools - A practical approach
Selecting Data Management Tools - A practical approachChristopher Bradley
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data MeshLibbySchulze
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureDATAVERSITY
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?DATAVERSITY
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsDATAVERSITY
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
 
Business Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected ApproachBusiness Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected ApproachDATAVERSITY
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data GovernanceDATAVERSITY
 
How to Strengthen Enterprise Data Governance with Data Quality
How to Strengthen Enterprise Data Governance with Data QualityHow to Strengthen Enterprise Data Governance with Data Quality
How to Strengthen Enterprise Data Governance with Data QualityDATAVERSITY
 
Data Modeling, Data Governance, & Data Quality
Data Modeling, Data Governance, & Data QualityData Modeling, Data Governance, & Data Quality
Data Modeling, Data Governance, & Data QualityDATAVERSITY
 

What's hot (20)

Overcoming the Challenges of your Master Data Management Journey
Overcoming the Challenges of your Master Data Management JourneyOvercoming the Challenges of your Master Data Management Journey
Overcoming the Challenges of your Master Data Management Journey
 
Data platform architecture
Data platform architectureData platform architecture
Data platform architecture
 
Data Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data StrategyData Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data Strategy
 
Review of Data Management Maturity Models
Review of Data Management Maturity ModelsReview of Data Management Maturity Models
Review of Data Management Maturity Models
 
Data Profiling, Data Catalogs and Metadata Harmonisation
Data Profiling, Data Catalogs and Metadata HarmonisationData Profiling, Data Catalogs and Metadata Harmonisation
Data Profiling, Data Catalogs and Metadata Harmonisation
 
Reference master data management
Reference master data managementReference master data management
Reference master data management
 
Capability Model_Data Governance
Capability Model_Data GovernanceCapability Model_Data Governance
Capability Model_Data Governance
 
Selecting Data Management Tools - A practical approach
Selecting Data Management Tools - A practical approachSelecting Data Management Tools - A practical approach
Selecting Data Management Tools - A practical approach
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
 
Data modelling 101
Data modelling 101Data modelling 101
Data modelling 101
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business Goals
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
Business Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected ApproachBusiness Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected Approach
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
How to Strengthen Enterprise Data Governance with Data Quality
How to Strengthen Enterprise Data Governance with Data QualityHow to Strengthen Enterprise Data Governance with Data Quality
How to Strengthen Enterprise Data Governance with Data Quality
 
Data Modeling, Data Governance, & Data Quality
Data Modeling, Data Governance, & Data QualityData Modeling, Data Governance, & Data Quality
Data Modeling, Data Governance, & Data Quality
 

Similar to Data Architecture for Solutions.pdf

Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Nathan Bijnens
 
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Alan McSweeney
 
Introduction to Business and Data Analysis Undergraduate.pdf
Introduction to Business and Data Analysis Undergraduate.pdfIntroduction to Business and Data Analysis Undergraduate.pdf
Introduction to Business and Data Analysis Undergraduate.pdfAbdulrahimShaibuIssa
 
DI&A Slides: Data-Centric Development
DI&A Slides: Data-Centric DevelopmentDI&A Slides: Data-Centric Development
DI&A Slides: Data-Centric DevelopmentDATAVERSITY
 
Decoding the Role of a Data Engineer.pdf
Decoding the Role of a Data Engineer.pdfDecoding the Role of a Data Engineer.pdf
Decoding the Role of a Data Engineer.pdfDatavalley.ai
 
Data Mesh using Microsoft Fabric
Data Mesh using Microsoft FabricData Mesh using Microsoft Fabric
Data Mesh using Microsoft FabricNathan Bijnens
 
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?DATAVERSITY
 
Data-Ed Webinar: Data Modeling Fundamentals
Data-Ed Webinar: Data Modeling FundamentalsData-Ed Webinar: Data Modeling Fundamentals
Data-Ed Webinar: Data Modeling FundamentalsDATAVERSITY
 
Data Con LA 2022 - Self-Service Success and Data Products
Data Con LA 2022 - Self-Service Success and Data ProductsData Con LA 2022 - Self-Service Success and Data Products
Data Con LA 2022 - Self-Service Success and Data ProductsData Con LA
 
Five Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceFive Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceDATAVERSITY
 
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?Albert Hoitingh
 
Big Data Analytics Architecture Powerpoint Presentation Slides
Big Data Analytics Architecture Powerpoint Presentation SlidesBig Data Analytics Architecture Powerpoint Presentation Slides
Big Data Analytics Architecture Powerpoint Presentation SlidesSlideTeam
 
Conceptual vs. Logical vs. Physical Data Modeling
Conceptual vs. Logical vs. Physical Data ModelingConceptual vs. Logical vs. Physical Data Modeling
Conceptual vs. Logical vs. Physical Data ModelingDATAVERSITY
 
SG Data Mgt - Findings and Recommendations.pptx
SG Data Mgt - Findings and Recommendations.pptxSG Data Mgt - Findings and Recommendations.pptx
SG Data Mgt - Findings and Recommendations.pptxssuser57f752
 
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...DATAVERSITY
 
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016Caserta
 
Building the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseBuilding the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseDatabricks
 
MDM & BI Strategy For Large Enterprises
MDM & BI Strategy For Large EnterprisesMDM & BI Strategy For Large Enterprises
MDM & BI Strategy For Large EnterprisesMark Schoeppel
 
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...Denodo
 

Similar to Data Architecture for Solutions.pdf (20)

Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
 
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
 
Introduction to Business and Data Analysis Undergraduate.pdf
Introduction to Business and Data Analysis Undergraduate.pdfIntroduction to Business and Data Analysis Undergraduate.pdf
Introduction to Business and Data Analysis Undergraduate.pdf
 
DI&A Slides: Data-Centric Development
DI&A Slides: Data-Centric DevelopmentDI&A Slides: Data-Centric Development
DI&A Slides: Data-Centric Development
 
Decoding the Role of a Data Engineer.pdf
Decoding the Role of a Data Engineer.pdfDecoding the Role of a Data Engineer.pdf
Decoding the Role of a Data Engineer.pdf
 
Data Mesh using Microsoft Fabric
Data Mesh using Microsoft FabricData Mesh using Microsoft Fabric
Data Mesh using Microsoft Fabric
 
Data Science
Data ScienceData Science
Data Science
 
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?
DAS Slides: Emerging Trends in Data Architecture — What’s the Next Big Thing?
 
Data-Ed Webinar: Data Modeling Fundamentals
Data-Ed Webinar: Data Modeling FundamentalsData-Ed Webinar: Data Modeling Fundamentals
Data-Ed Webinar: Data Modeling Fundamentals
 
Data Con LA 2022 - Self-Service Success and Data Products
Data Con LA 2022 - Self-Service Success and Data ProductsData Con LA 2022 - Self-Service Success and Data Products
Data Con LA 2022 - Self-Service Success and Data Products
 
Five Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceFive Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data Governance
 
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
 
Big Data Analytics Architecture Powerpoint Presentation Slides
Big Data Analytics Architecture Powerpoint Presentation SlidesBig Data Analytics Architecture Powerpoint Presentation Slides
Big Data Analytics Architecture Powerpoint Presentation Slides
 
Conceptual vs. Logical vs. Physical Data Modeling
Conceptual vs. Logical vs. Physical Data ModelingConceptual vs. Logical vs. Physical Data Modeling
Conceptual vs. Logical vs. Physical Data Modeling
 
SG Data Mgt - Findings and Recommendations.pptx
SG Data Mgt - Findings and Recommendations.pptxSG Data Mgt - Findings and Recommendations.pptx
SG Data Mgt - Findings and Recommendations.pptx
 
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...
Data-Ed Slides: Data Modeling Strategies - Getting Your Data Ready for the Ca...
 
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
 
Building the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseBuilding the Artificially Intelligent Enterprise
Building the Artificially Intelligent Enterprise
 
MDM & BI Strategy For Large Enterprises
MDM & BI Strategy For Large EnterprisesMDM & BI Strategy For Large Enterprises
MDM & BI Strategy For Large Enterprises
 
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...
Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...
 

More from Alan McSweeney

Solution Architecture and Solution Estimation.pdf
Solution Architecture and Solution Estimation.pdfSolution Architecture and Solution Estimation.pdf
Solution Architecture and Solution Estimation.pdfAlan McSweeney
 
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...Alan McSweeney
 
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...Alan McSweeney
 
IT Architecture’s Role In Solving Technical Debt.pdf
IT Architecture’s Role In Solving Technical Debt.pdfIT Architecture’s Role In Solving Technical Debt.pdf
IT Architecture’s Role In Solving Technical Debt.pdfAlan McSweeney
 
Solution Architecture And Solution Security
Solution Architecture And Solution SecuritySolution Architecture And Solution Security
Solution Architecture And Solution SecurityAlan McSweeney
 
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Alan McSweeney
 
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Alan McSweeney
 
Solution Security Architecture
Solution Security ArchitectureSolution Security Architecture
Solution Security ArchitectureAlan McSweeney
 
Solution Architecture And (Robotic) Process Automation Solutions
Solution Architecture And (Robotic) Process Automation SolutionsSolution Architecture And (Robotic) Process Automation Solutions
Solution Architecture And (Robotic) Process Automation SolutionsAlan McSweeney
 
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...Alan McSweeney
 
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...Alan McSweeney
 
Operational Risk Management Data Validation Architecture
Operational Risk Management Data Validation ArchitectureOperational Risk Management Data Validation Architecture
Operational Risk Management Data Validation ArchitectureAlan McSweeney
 
Ireland 2019 and 2020 Compared - Individual Charts
Ireland   2019 and 2020 Compared - Individual ChartsIreland   2019 and 2020 Compared - Individual Charts
Ireland 2019 and 2020 Compared - Individual ChartsAlan McSweeney
 
Analysis of Irish Mortality Using Public Data Sources 2014-2020
Analysis of Irish Mortality Using Public Data Sources 2014-2020Analysis of Irish Mortality Using Public Data Sources 2014-2020
Analysis of Irish Mortality Using Public Data Sources 2014-2020Alan McSweeney
 
Ireland – 2019 And 2020 Compared In Data
Ireland – 2019 And 2020 Compared In DataIreland – 2019 And 2020 Compared In Data
Ireland – 2019 And 2020 Compared In DataAlan McSweeney
 
Review of Information Technology Function Critical Capability Models
Review of Information Technology Function Critical Capability ModelsReview of Information Technology Function Critical Capability Models
Review of Information Technology Function Critical Capability ModelsAlan McSweeney
 
Critical Review of Open Group IT4IT Reference Architecture
Critical Review of Open Group IT4IT Reference ArchitectureCritical Review of Open Group IT4IT Reference Architecture
Critical Review of Open Group IT4IT Reference ArchitectureAlan McSweeney
 
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020Alan McSweeney
 
Agile Solution Architecture and Design
Agile Solution Architecture and DesignAgile Solution Architecture and Design
Agile Solution Architecture and DesignAlan McSweeney
 
Solution Architecture and Solution Acquisition
Solution Architecture and Solution AcquisitionSolution Architecture and Solution Acquisition
Solution Architecture and Solution AcquisitionAlan McSweeney
 

More from Alan McSweeney (20)

Solution Architecture and Solution Estimation.pdf
Solution Architecture and Solution Estimation.pdfSolution Architecture and Solution Estimation.pdf
Solution Architecture and Solution Estimation.pdf
 
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...
Validating COVID-19 Mortality Data and Deaths for Ireland March 2020 – March ...
 
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...
Analysis of the Numbers of Catholic Clergy and Members of Religious in Irelan...
 
IT Architecture’s Role In Solving Technical Debt.pdf
IT Architecture’s Role In Solving Technical Debt.pdfIT Architecture’s Role In Solving Technical Debt.pdf
IT Architecture’s Role In Solving Technical Debt.pdf
 
Solution Architecture And Solution Security
Solution Architecture And Solution SecuritySolution Architecture And Solution Security
Solution Architecture And Solution Security
 
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
 
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...
 
Solution Security Architecture
Solution Security ArchitectureSolution Security Architecture
Solution Security Architecture
 
Solution Architecture And (Robotic) Process Automation Solutions
Solution Architecture And (Robotic) Process Automation SolutionsSolution Architecture And (Robotic) Process Automation Solutions
Solution Architecture And (Robotic) Process Automation Solutions
 
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...
Comparison of COVID-19 Mortality Data and Deaths for Ireland March 2020 – Mar...
 
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...
Analysis of Decentralised, Distributed Decision-Making For Optimising Domesti...
 
Operational Risk Management Data Validation Architecture
Operational Risk Management Data Validation ArchitectureOperational Risk Management Data Validation Architecture
Operational Risk Management Data Validation Architecture
 
Ireland 2019 and 2020 Compared - Individual Charts
Ireland   2019 and 2020 Compared - Individual ChartsIreland   2019 and 2020 Compared - Individual Charts
Ireland 2019 and 2020 Compared - Individual Charts
 
Analysis of Irish Mortality Using Public Data Sources 2014-2020
Analysis of Irish Mortality Using Public Data Sources 2014-2020Analysis of Irish Mortality Using Public Data Sources 2014-2020
Analysis of Irish Mortality Using Public Data Sources 2014-2020
 
Ireland – 2019 And 2020 Compared In Data
Ireland – 2019 And 2020 Compared In DataIreland – 2019 And 2020 Compared In Data
Ireland – 2019 And 2020 Compared In Data
 
Review of Information Technology Function Critical Capability Models
Review of Information Technology Function Critical Capability ModelsReview of Information Technology Function Critical Capability Models
Review of Information Technology Function Critical Capability Models
 
Critical Review of Open Group IT4IT Reference Architecture
Critical Review of Open Group IT4IT Reference ArchitectureCritical Review of Open Group IT4IT Reference Architecture
Critical Review of Open Group IT4IT Reference Architecture
 
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020
Analysis of Possible Excess COVID-19 Deaths in Ireland From Jan 2020 to Jun 2020
 
Agile Solution Architecture and Design
Agile Solution Architecture and DesignAgile Solution Architecture and Design
Agile Solution Architecture and Design
 
Solution Architecture and Solution Acquisition
Solution Architecture and Solution AcquisitionSolution Architecture and Solution Acquisition
Solution Architecture and Solution Acquisition
 

Recently uploaded

Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoTAnalytics
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoUXDXConf
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKUXDXConf
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfFIDO Alliance
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 

Recently uploaded (20)

Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 

Data Architecture for Solutions.pdf

  • 1. Data Architecture For Solutions Alan McSweeney http://ie.linkedin.com/in/alanmcsweeney https://www.researchgate.net/profile/Alan-Mcsweeney https://www.amazon.com/dp/1797567616
  • 2. Introduction • These notes discuss how overall organisation data architecture can positively impact solution design and how solution data architecture competence within solution architecture can contribute to solution design success January 9, 2023 2 Data Architecture Solution Architecture Solution Data Architecture Can Contribute to Common Data Infrastructure Tools and Data Standards Can Contribute to Overall Organisation Data Quality Can Ensure that the Data Aspects of Solution Design Are Covered in Solution Designs Can Ensure that Solution Data Concerns Are Addressed in Solution Designs
  • 3. Topics • Data Architecture For Solutions • Traditional Scope Of Data Architecture • Solution Data Architecture • What Do We Mean By Data Architecture? • Data Architecture And Common Data Tooling And Standards • Data Design And Modelling For Solutions January 9, 2023 3
  • 4. Data Architecture For Solutions • Data breathes life into solutions • Solutions get data, use data, share data, process data and create data • There will be many different types of data used by a solution − Master data − Reference data − Input data − Interim data − Generated data − Solution activity and usage data • Any solution will consist of many different components of different types • Solution components and their data will be deployed and operated across a solution landscape that can span multiple zones and platforms • Within the solution, each data type will have a different lifecycle • The solutions within the organisation solution landscape will have both shared and private data − Shared - common data (master or reference) or upstream data from other solutions or data sent downstream − Private – data held locally within the solution January 9, 2023 4
  • 5. Solution And Data • All IT solutions support, implement and operate business processes that take data inputs, process data, generate result and create primary and supporting data output − Direct data outputs – what the process in intended to create − Indirect data outputs – logs, audit trails, reports, analyses • Data outputs are then used in different ways − Generated results − As a record that the work was performed − As inputs into other processes and solutions − To report on the operation of the process or as an audit log • Data breathes life into and activates the static components of a solution • The data architecture of solutions is frequently not given the attention it deserves or needs January 9, 2023 5
  • 6. Data Architecture For Solutions • Frequently, too little attention is paid to designing and specifying the data architecture within individual solutions and their constituent components • This is due to the behaviours of both solution architects ad data architects • Solution architecture tends to concern itself with functional, technology and software components of the solution • Data architecture tends to concern itself with post- individual solutions January 9, 2023 6
  • 7. Traditional Scope Of Data Architecture January 9, 2023 7 Data Ingestion and Integration Data Validation and Error Handling Data Encryption, Anonymisation, Pseudonymisation Security and Access Control Data Processing Workflow Data Model and Data Store API Interface Data Interrogation and Analysis Data Visualisation Data Extract Management and Administration Data Publication and Sharing Existing and New Reports Data Storage Platform/ Infrastructure Usage and Performance Monitoring Semantic Layer Data Sources (Internal, External) Extract, Transform, Load Data Platform Access and Usage Merge, Aggregate, Transform Data Sources (from Solutions)
  • 8. Traditional Scope Of Data Architecture January 9, 2023 8 Data Ingestion and Integration Data Validation and Error Handling Data Encryption, Anonymisation, Pseudonymisation Security and Access Control Data Processing Workflow Data Model and Data Store API Interface Data Interrogation and Analysis Data Visualisation Data Extract Management and Administration Data Publication and Sharing Existing and New Reports Data Storage Platform/ Infrastructure Usage and Performance Monitoring Semantic Layer Merge, Aggregate, Transform Scope Of Data Architecture Data Sources (from Solutions)
  • 9. Traditional Scope Of Data Architecture • Traditional approaches to data architecture effectively appends or layers newer technologies on top on existing solutions and data sources and their data structures • Data architecture largely ignores data architectures within individual solutions • Data architecture needs to shift left into the domain of solutions and their data and more actively engage with the data dimensions of individual solutions January 9, 2023 9
  • 10. Traditional Scope Of Data Architecture January 9, 2023 10 Data Ingestion and Integration Data Validation and Error Handling Data Encryption, Anonymisation, Pseudonymisation Security and Access Control Data Processing Workflow Data Model and Data Store API Interface Data Interrogation and Analysis Data Visualisation Data Extract Management and Administration Data Publication and Sharing Existing and New Reports Data Storage Platform/ Infrastructure Usage and Performance Monitoring Semantic Layer Merge, Aggregate, Transform This Is Not A Modern Data Architecture Data Sources (from Solutions)
  • 11. Not A Modern Data Architecture • You are fooling yourself if you believe that enveloping existing data solutions, sources and structures with a skin of modernity comprises a data architecture • If you put lipstick on a pig, it is still a pig January 9, 2023 11
  • 12. Common and Shared Data Processes and Standards Data Architecture Operation, Measurement Data Architecture Review, Improvement, Update Common and Shared Data Infrastructural Components Data Solutions, Sources and Structures This Is A Data Architecture January 9, 2023 12 Data Architecture Overview Data Management, Governance, Supporting Processes Data Infrastructure, Storage and Operations Software, Hardware and Processes Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Content, Unstructured Data, Records and Document Management Master and Reference Data Management Data Warehouse, Data Marts, Data Lakes Data Reporting and Analytics, Visualisation Tools and Facilities Data Discovery, Analysis, Design and Modelling External Data Sources and Interacting Parties Data Transfer/Exchange/Integration/Publication Metadata Data Management Data Quality Data Solution Design
  • 13. Traditional Scope Of Data Architecture And Data Architecture Gap Data architecture tends not to get involved with the data aspects of technology solutions, leaving a data architecture gap January 9, 2023 13 Data Ingestion and Integration Data Validation and Error Handling Data Processing Workflow Data Sources (from Solutions) Merge, Aggregate, Transform New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models … … Data Architecture Gap Data Architecture
  • 14. Solution Data Aspects Across The Solution Landscape January 9, 2023 14 New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Span Of Organisation Solution Landscape Changes to Existing Systems and Their Data Models
  • 15. Solution Components And Their Types January 9, 2023 15 Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Changes to Existing Systems New Custom Developed Applications Acquired and Customised Software Products System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Sets of Installation and Implementation Services Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Central, Distributed and Communications Infrastructure Cutover/ Transfer to Production And Support Operational Functions and Processes Parallel Runs Enhanced Support/ Hypercare Sets of Maintenance, Service Management and Support Services Application Hosting and Management Services Changes to Existing Business Processes New Business Processes Organisational Changes, Knowledge Management Training and Documentation Component Type Solution Components
  • 16. Reduced Scope Of Traditional Solution Architecture Scope • The solution is the sum of the components needed to deliver and operate it • Solution architecture tends not to concern itself with some key aspects of the complete solution, including some of those related to data • Solution architecture tends to focus on technology aspects of a solution, omitting business and data facets • The data dimensions of other solution components also tends to be omitted partially or completely by solution architecture January 9, 2023 16 Complete Solution
  • 17. Data Related Solution Components And Their Types January 9, 2023 17 Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Component Changes to Existing Systems New Custom Developed Applications Acquired and Customised Software Products System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Sets of Installation and Implementation Services Information Storage Facilities Existing Data Conversions/ Migrations New Data Loads Central, Distributed and Communications Infrastructure Cutover/ Transfer to Production And Support Operational Functions and Processes Parallel Runs Enhanced Support/ Hypercare Sets of Maintenance, Service Management and Support Services Application Hosting and Management Services Changes to Existing Business Processes New Business Processes Organisational Changes, Knowledge Management Training and Documentation Component Type Solution Components
  • 18. Data Dimensions Of Solution Component Types January 9, 2023 18 Changes to Existing Systems Possible new data stores and their design and data models Data tools Data processes Changes to existing data stores and their design and data models Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data Metadata Data quality New Custom Developed Applications New data stores and their design and data models Data tools Data processes Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data design Metadata design Data quality design Acquired and Customised Software Products New data stores and their design and data models Data tools Data processes Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data Metadata Data quality System Integrations/ Data Transfers/ Exchanges Source and target data stores and their design and data models Data transformations and aggregations Data tools Data processes Changes to existing data stores and their design and data models Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data Metadata Data quality Reporting and Analysis Facilities Reporting data stores and their design and data models Reporting tools Reporting processes Data security, encryption and access control Master and reference data Metadata Data quality Information Storage Facilities Performance, capacity and throughput Data management and governance Existing Data Conversions/ Migrations Source and target data stores and their design and data models Data transformations and aggregations Data tools Data processes Changes to existing data stores and their design and data models Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data Metadata Data quality New Data Loads Target data stores and their design and data models Data tools Data processes Changes to existing data stores and their design and data models Performance, capacity and thoughput Data security, encryption and access control Data management and governance Master and reference data Metadata Data quality
  • 19. Data Dimensions Of Solution Component Types • The following solution component types involve data architecture and design activities: − Changes to Existing Systems − New Custom Developed Applications − Acquired and Customised Software Products − System Integrations/ Data Transfers/ Exchanges − Reporting and Analysis Facilities − Information Storage Facilities − Existing Data Conversions/ Migrations − New Data Loads • The components of each of these types will potentially involve data and therefore data design work across a range of areas that will need to be included in the solution design and subsequent solution implementation activities • Solution architecture can fail t include some of the data dimensions of the solution components of these types January 9, 2023 19
  • 20. Solution Components And Their Types • Any technology solution will consist of a potentially large number of components, each of a give type • Each solution component type belongs to one of three classes 1. Time-Bounded Delivery Entity Types • Time-bounded solution component types required to get the solution fully operational 2. Enduring Functional and Operational Technology Entity Types • Operational instrumentation and functional component types required for the solution to operate and be usable by its target consumers 3. Enduring Organisational, Process, Procedure and Structural Entity Types • Organisation and process changes and other supporting activities and sets of effort required to use the solution optimally January 9, 2023 20
  • 21. Solution Components Classes And Types January 9, 2023 21 Solution Component Classes and Types Time-Bounded Delivery Entity Types Sets of Installation and Implementation Services Existing Data Conversions/ Migrations New Data Loads Parallel Runs Enhanced Support/ Hypercare Enduring Functional and Operational Technology Entity Types Changes to Existing Systems New Custom Developed Applications Acquired and Customised Software Products System Integrations/ Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Central, Distributed and Communications Infrastructure Application Hosting and Management Services Enduring Organisational, Process, Procedure and Structural Entity Types Cutover/ Transfer to Production And Support Operational Functions and Processes Sets of Maintenance, Service Management and Support Services Changes to Existing Business Processes New Business Processes Organisational Changes, Knowledge Management Training and Documentation
  • 22. Solution Architecture Data Gap • Combined with the gap where data architecture tends not to get involved with the data aspects of technology solutions, there is also frequently a solution architecture data gap • Solution architecture also frequently omits the detail of data aspects of solutions across the various components of the types: − Changes to Existing Systems − New Custom Developed Applications − Acquired and Customised Software Products − System Integrations/ Data Transfers/ Exchanges − Reporting and Analysis Facilities − Information Storage Facilities − Existing Data Conversions/ Migrations − New Data Loads January 9, 2023 22
  • 23. Solution Architecture Data Gap • These gaps result in a data blind spot for the organisation January 9, 2023 23 Central, Distributed and Communications Infrastructure Changes to Existing Systems New Custom Developed Applications Information Storage Facilities Acquired and Customised Software Products System Integrations/ Data Transfers/ Exchanges Changes to Existing Business Processes New Business Processes Organisational Changes, Knowledge Management Reporting and Analysis Facilities Existing Data Conversions/ Migrations New Data Loads Training and Documentation Sets of Installation and Implementation Services Operational Functions and Processes Parallel Runs Cutover/ Transfer to Production Sets of Maintenance, Service Management and Support Services Application Hosting and Management Services Enhanced Support/ Hypercare Data Architecture Solution Gap Data Architecture Data Ingestion and Integration Data Validation and Error Handling Data Processing Workflow Merge, Aggregate, Transform … Solution Architecture Data Gap Solution Architecture
  • 24. Solution Architecture Data Gap • Data architecture can provide the lead in sealing these data gaps through a shift-left of its scope and activities as well providing standards and common data tooling for solution data architecture January 9, 2023 24
  • 25. Shift Left Of The Scope Of Data Architecture January 9, 2023 25 Source Systems Extract, Transform, Load Data Platform Access and Usage Changes to Existing Systems and Their Data Models New Custom Developed Applications and Their Data Models Acquired and Customised Software Products and Their Data Models System Integrations / Data Transfers/ Exchanges Reporting and Analysis Facilities Information Storage Facilities Existing Data Conversions / Migrations New Data Loads • Shifting data architecture to the left means getting involved in the data aspects of solution design, specification, selection and implementation at the earliest opportunity • This then needs to be repeated for each solution within the organisation solution landscape • The data aspects of solutions should be closely integrated within the organisation’s data architecture Shift Left of Scope of Data Architecture
  • 26. Generalised Data Lifecycle • Each data type within a solution will have a lifecycle from design and creation to ultimate archival and possible deletion January 9, 2023 26 Enter, Create, Acquire, Derive, Update, Integrate, Capture Secure, Store, Replicate and Distribute Preserve, Protect and Recover Archive and Recall Delete/Remove Implement Underlying Technology Architect, Budget, Plan, Design and Specify Present, Report, Analyse, Model • A set of data lifecycle view for solutions can assist in solution data architecture
  • 27. Generalised Data Lifecycle Stages • Architect, Budget, Plan, Design and Specify - This relates to the design and specification of the data storage and management and their supporting processes − This establishes the data management framework • Implement Underlying Technology - This is concerned with implementing the data-related hardware and software technology components − This relates to database components, data storage hardware, backup and recovery software, monitoring and control software and other items • Enter, Create, Acquire, Derive, Update, Integrate, Capture - This stage is where data originated, such as data entry or data capture and acquired from other systems or sources • Secure, Store, Replicate and Distribute - In this stage, data is stored with appropriate security and access controls including data access and update audit − It may be replicated to other applications and distributed • Present, Report, Analyse, Model - This stage is concerned with the presentation of information, the generation of reports and analysis and the created of derived information • Preserve, Protect and Recover - This stage relates to the management of data in terms of availability, backup, recovery and retention/preservation • Archive and Recall - This stage is where information that is no longer active but still required in archived to secondary data storage platforms and from which the information can be recovered if required • Delete/Remove - The stage is concerned with the deletion of data that cannot or does not need to be retained any longer − Data has to be able to be disposed of in a managed, systematic and auditable way • Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer, Standards, Governance, Fund - This is not a single stage but a set of processes and procedures that cross all stages and is concerned with ensuring that the processes associated with each of the lifestyle stages are operated correctly and that data assurance, quality and governance procedures exist and are operated January 9, 2023 27
  • 28. Solution Data Types And Lifecycles • Every solution will have one or more types of data it reads, processes or creates • Each data type will have a separate lifecycle that reflects how it is processed and how its attributes need to be reflected in its governance and management January 9, 2023 28 Solution Data Type 1 Data Type 2 … Data Type N
  • 29. Data Lifecycle Stages And Solution Component Types January 9, 2023 29 Central, Distributed and Communications Infrastructure Changes to Existing Systems New Custom Developed Applications System Integrations/ Data Transfers/ Exchanges Changes to Existing Business Processes Organisational Changes, Knowledge Management Training and Documentation Sets of Installation and Implementation Services Parallel Runs Enhanced Support/ Hypercare Information Storage Facilities Acquired and Customised Software Products New Business Processes Reporting and Analysis Facilities Existing Data Conversions/ Migrations New Data Loads Operational Functions and Processes Cutover/ Transfer to Production Sets of Maintenance, Service Management and Support Services Application Hosting and Management Services Secure, Store, Replicate and Distribute Archive and Recall Delete/Remove Implement Underlying Technology Architect, Budget, Plan, Design and Specify Present, Report, Analyse, Model Enter, Create, Acquire, Derive, Update, Integrate, Capture Preserve, Protect and Recover
  • 30. Data Lifecycle Stages And Solution Components • Each stage within the lifecycle of a solution data type will be realised by a solution component • Mapping the stages within the lifecycle of solution data types and identifying the impact on solution component types can contribute to effective solution data architecture design − This provides traceability to ensure the data is being handled correctly • For example, the Preserve, Protect and Recover data stage involving solution activities such as backup and recovery, replication, business continuity and disaster recovery may require solution components of the types: − Information Storage Facilities − Sets of Maintenance, Service Management and Support Services − Operational Functions and Processes January 9, 2023 30 Operational Functions and Processes Sets of Maintenance, Service Management and Support Services Information Storage Facilities Preserve, Protect and Recover
  • 31. So, What Do We Mean By Data Architecture? • If data architecture can contribute to solution architecture then the scope of data architecture should be defined and agreed to ensure this is possible January 9, 2023 31
  • 32. Data Architecture And Data Strategy • Data architecture defines the target data structures, operations, principles, standards, organisation, tools, management, governance that the organisation is aiming to define, implement and operate − The data architecture is designed to be implemented and operated • Data strategy defines how the organisation intends to use data to deliver on its business strategy − Data strategy precedes and feeds into the data architecture January 9, 2023 32
  • 33. Data Strategy And Data Architecture In A Wider Business And Technology Context January 9, 2023 33 Business Objectives Business Architecture Enterprise Architecture Solution Implementation and Delivery Support, Management and Operations Business Processes Required Operational Business Solutions Business Strategy Business Solution Analysis and Design/ Selection Business IT Strategy IT Function Strategy Required Operational Processes Required Supporting and Enabling Business Solutions Support Solution Analysis and Design/ Selection Required Structure, Capabilities and Resources Digital Strategy Digital IT Architecture Solution Portfolio Design And Specification Solution Portfolio Management Solution Change and Evolution Business Structure and Operational Model Data Strategy Data Architecture
  • 34. Data Strategy And Data Architecture In A Wider Business And Technology Context • Data strategy follows from business strategy and business objectives • Data architecture translates the conceptual nature of the data strategy into a more implementation- specific and – oriented view January 9, 2023 34 Business Architecture Enterprise Architecture Required Operational Business Solutions Business Solution Analysis and Design/ Selection Business IT Strategy IT Function Strategy Digital Strategy Digital IT Architecture Data Strategy Data Architecture Business Objectives
  • 35. Data Architecture • A Data Architecture exists to support the objectives and the operations of the organisation • This includes enabling individual functional solutions to be designed and implemented in accordance with the wider organisation data architecture January 9, 2023 35 Organisation Data Architecture Data Infrastructure Tools and Facilities Functional Solutions Data Standards
  • 36. Data Architecture Structure • For each set of subject arears within the data architecture design and specification process, create an activity breakdown based on the phases − Research, Design, Define, Plan − Implement, Operate − Administer, Manage, Monitor, Improve • Data architecture cannot be separated from its implementation, operation and subsequent measurement and improvement • Architecture without execution and employment is incomplete January 9, 2023 36
  • 37. Data Architecture Evolution And Development • The data architecture is not static – it must be responsive to and accommodating of change • It needs to evolve and develop in response to: − Changing organisation needs and direction, driven by internal and/or external demands − Changing organisation business strategy − New technologies and capabilities that the organisation can usefully avail of − Experience from implementation and operation • The architecture should embed within itself explicitly the ability to assess its implementation and operation and to grow, change, improve in response to these factors January 9, 2023 37 Research, Design, Define, Plan Implement, Operate Administer, Manage, Monitor, Improve Define Measurement Framework, Results and Performance Indicators Review Delivery and Operation of Architecture Experience and Lessons from Implementation and Operation Changing Organisation Needs and Direction Changes to Organisation Business Strategy New Data Technologies and Capabilities Data Architecture Changes
  • 38. Data Architecture Subject Areas Data Architecture Data Architecture Overall data architecture and data technology standards and design and implement data infrastructural technology solutions Data Management, Governance, Supporting Processes Standards, processes and their enforcement, planning, supervision, control and usage of data resources and the design and implementation of data management processes, data ownership Data Infrastructure, Storage and Operations Software, Hardware and Processes Infrastructure hardware and software required to store and provide access to data, either on-premises or hosted and facilities and processes required to operate and support the infrastructure, approach to analysis, design, implementation, testing, deployment, maintenance and data storage structures Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Approach to ensuring data security and protection, designing and implementing data security model covering data, tools and infrastructure, ensuring compliance with regulatory standards, controlling access to data, designing and implementing data authorisation model Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Data resource integration, extraction, transformation, movement, delivery, replication, transfer, sharing, federation, virtualisation and operational support and approach to implementing a common approach and providing a common set of tools Content, Unstructured Data, Records and Document Management Approach to the implementation and management of acquisition, storage, indexing of and access to unstructured data resources such as files and digitised paper records and the integration of these resources with structured data resources Master and Reference Data Management Approach to the implementation and management of master versions of shared data resources to reduce redundancy and maintain data quality through standardised data definitions and use of common data lookup values including data dictionaries Data Warehouse, Data Marts, Data Lakes Facilities for storing data extracted from operational systems for long-term storage and to enable access for reporting and analysis Data Reporting and Analytics, Visualisation Tools and Facilities Approach to providing a common approach and providing a common set of tools, facilities and supporting technologies and standards for data reporting, decision support, analysis and visualisation Data Discovery, Analysis, Design and Modelling Approach to the implementation and management of data description standards and the collection, categorisation, maintenance, integration, application, use and management of data descriptions including data catalogs External Data Sources and Interacting Parties Data Transfer/Exchange/ Integration/Publication Management of data sources and targets outside the organisation and the parties that provide that data or to whim the data is made available including contracts and agreement, service levels, access approaches Metadata Data Management Approach to the implementation and management of data description standards and the collection, categorisation, maintenance, integration, application, use and management of data descriptions including data catalogs Data Quality Designing, implementing and operating approach, processes and standards to ensure and maintain data quality Data Solution Design Defining and implementing standards relating to the use of data within solutions January 9, 2023 38
  • 39. Data Architecture Subject Areas • This is intended to represent a comprehensive view of data architecture January 9, 2023 39
  • 40. Data Architecture Subject Areas • The proposed subject areas do not exist in isolation • They are interrelated areas on which to focus analysis, planning and design effort and attention while maintaining a higher level and more complete and integrated view • The individual topics allow each subject area to be analysed and specified in detail that is appropriate for the organisation • The topics are designed to be independent of any specific hardware, software or platform technology January 9, 2023 40
  • 41. Relationships Between Data Architecture Topics January 9, 2023 41 Data Solution Design Data Quality External Data Sources and Interacting Parties Data Transfer/Exchange/ Integration/Publication Data Discovery, Analysis, Design and Modelling Data Infrastructure, Storage and Operations Software, Hardware and Processes Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Content, Unstructured Data, Records and Document Management Master and Reference Data Management Data Architecture Data Management, Governance, Supporting Processes Data Reporting and Analytics, Visualisation Tools and Facilities Data Warehouse, Data Marts, Data Lakes Metadata Data Management Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract
  • 42. Data Architecture Topic Scope Data Architecture Research, Design, Define, Plan Data Architecture Strategy and Scope Definition Data Architecture Capability Establishment Define Current Data Architecture Baseline, Inventory, Gaps, Issues, Concerns Define Architecture Supporting Tools and Processes Definition Data Architecture Scope and Activities Data Architecture Strategy and Scope Definition Data Architecture Capability Establishment Define Current Data Architecture Baseline, Inventory, Gaps, Issues, Concerns Define Architecture Supporting Tools and Processes Definition Data Architecture Scope and Activities Data Architecture Implementation Planning Implement, Operate Data Architecture Supporting Tools and Processes Implementation and Operation Data Architecture Team Formation Data Architecture Implementation Data Architecture Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Architecture Review and Improvement Data Architecture Management Data Architecture Operation Assessment January 9, 2023 42
  • 43. Data Management, Governance, Supporting Processes Topic Scope Data Management, Governance, Supporting Processes Research, Design, Define, Plan Data Governance Capability Establishment Define Governance Strategy Define Current Data Governance Baseline Define Governance Supporting Tools and Processes Definition Data Governance Scope and Activities Define Governance Policies Define Governance Standards Define Governance Compliance, Monitoring and Reporting Data Persistence Standards Data Lifecycle Definition and Management Create Data Asset Inventory Create Business Glossary Perform Data Value Assessment Data Governance Implementation Planning Data Governance Process Definition Implement, Operate Data Governance Supporting Tools and Processes Implementation and Operation Data Governance Team Formation Data Governance Implementation Data Governance Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Governance Review and Improvement Data Governance Management Data Governance Operation Assessment Data Governance Implementation and Operation Reporting January 9, 2023 43
  • 44. Data Infrastructure, Storage and Operations Software, Hardware and Processes Topic Scope Data Infrastructure, Storage and Operations Software, Hardware and Processes Research, Design, Define, Plan Data Infrastructure, Storage and Operations Capability Establishment Data Infrastructure and Storage Hardware, Software and Platform Inventory Data Operations and Process Inventory Data Infrastructure, Storage and Operations Existing Processes and Standards Inventory and Review Data Infrastructure, Storage and Operations Supporting Tools and Processes Definition Data Infrastructure, Storage and Operations Software and Hardware Scope and Activities Data Storage Hardware Technology Target Definition Data Storage Software Technology Target Definition Data Storage Platform Technology Target Definition Data Product, Platform and Vendor Selection and Management Data Backup and Recovery Data Infrastructure Performance Monitoring Tools Data Archival and Purge Tools Data Infrastructure, Storage and Operations Availability, Business Continuity, Disaster Recovery and Replication Definition Data Performance Testing and Validation Approach Data Infrastructure, Storage and Operations Standards Definition Data Infrastructure, Storage and Operations Performance and Capacity Planning Standards and Data Collection and Analysis Data Infrastructure, Storage and Operations Implementation Planning Implement, Operate Data Infrastructure, Storage and Operations Supporting Tools and Processes Supporting Tools and Processes Implementation and Operation Data Infrastructure, Storage and Operations Supporting Tools and Processes Team Formation Data Infrastructure, Storage and Operations Supporting Tools and Processes Implementation Data Infrastructure, Storage and Operations Supporting Tools and Processes Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Infrastructure, Storage and Operations Review and Improvement Data Infrastructure, Storage and Operations Management Data Infrastructure, Storage and Operations Operation Assessment January 9, 2023 44
  • 45. Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Topic Scope Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Research, Design, Define, Plan Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Capability Establishment Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Existing Approach Inventory and Baseline Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Supporting Tools and Processes Definition Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Scope and Activities Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Architecture Definition Compliance, Regulatory and Data Protection Requirements Across All Data Types Security Information, Event and Alert Logging and Auditing Data Loss Prevention Data Security Product, Platform and Vendor Selection and Management Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Standards Definition Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Monitoring, Data Collection and Analysis Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Implementation Planning Implement, Operate Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Supporting Tools and Processes Implementation and Operation Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Team Formation Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Implementation Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Review and Improvement Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Management Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Operation Assessment January 9, 2023 45
  • 46. Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Topic Scope Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Research, Design, Define, Plan Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Capability Establishment Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Existing Approach Inventory and Baseline Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Supporting Tools and Processes Definition Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Scope and Activities Data Integration Security, Authentication, Authorisation Data Integration Product, Platform and Vendor Selection and Management Data Integration Scheduler and Rules Engine Internal and External Data Sources, Targets and Channels Definition Data Integration Development, Testing and Deployment Data Integration Operations Management, Administration Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Standards Definition Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Monitoring, Data Collection and Analysis Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Implementation Planning Implement, Operate Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Supporting Tools and Processes Implementation and Operation Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Team Formation Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Implementation Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Review and Improvement Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Management Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Operation Assessment January 9, 2023 46
  • 47. Content, Unstructured Data, Records and Document Management Topic Scope Content, Unstructured Data, Records and Document Management Research, Design, Define, Plan Content, Unstructured Data, Records and Document Management Capability Establishment Content, Unstructured Data, Records and Document Management Existing Approach Inventory and Baseline Content, Unstructured Data, Records and Document Management Supporting Tools and Processes Definition Content, Unstructured Data, Records and Document Management Scope and Activities Content, Unstructured Data, Records and Document Management Security, Authentication, Authorisation Data Integration Product, Platform and Vendor Selection and Management Records Management Strategy Metadata Management Content, Unstructured Data, Records and Document Lifecycle Management Content, Unstructured Data, Records and Document Management Standards Definition Content, Unstructured Data, Records and Document Management Monitoring, Data Collection and Analysis Content, Unstructured Data, Records and Document Management Implementation Planning Implement, Operate Content, Unstructured Data, Records and Document Management Supporting Tools and Processes Implementation and Operation Content, Unstructured Data, Records and Document Management Team Formation Content, Unstructured Data, Records and Document Management Implementation Content, Unstructured Data, Records and Document Management Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Content, Unstructured Data, Records and Document Management Review and Improvement Content, Unstructured Data, Records and Document Management Management Content, Unstructured Data, Records and Document Management Operation Assessment January 9, 2023 47
  • 48. Master and Reference Data Management Topic Scope Master and Reference Data Management Research, Design, Define, Plan Master and Reference Data Management Capability Establishment Master and Reference Data Management Existing Approach Inventory and Baseline Master and Reference Data Management Supporting Tools and Processes Definition Master and Reference Data Management Scope and Activities Industry Data Standards Data Glossaries and Taxonomies Business Rules Analysis and Definition Master and Reference Data Management Product, Platform and Vendor Selection and Management Master Data Stores Reference Data Stores Master and Reference Data Management Standards Definition Master and Reference Data Management Monitoring, Data Collection and Analysis Master and Reference Data Management Implementation Planning Implement, Operate Master and Reference Data Management Supporting Tools and Processes Implementation and Operation Master and Reference Data Management Team Formation Master and Reference Data Management Implementation Master and Reference Data Management Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Master and Reference Data Management Review and Improvement Master and Reference Data Management Management Master and Reference Data Management Operation Assessment January 9, 2023 48
  • 49. Data Warehouse, Data Marts, Data Lakes Topic Scope Data Warehouse, Data Marts, Data Lakes Research, Design, Define, Plan Data Warehouse, Data Marts, Data Lakes Capability Establishment Data Warehouse, Data Marts, Data Lakes Existing Approach Inventory and Baseline Data Warehouse, Data Marts, Data Lakes Supporting Tools and Processes Definition Data Warehouse, Data Marts, Data Lakes Scope and Activities Data Models Creation Long-term Data Storage Architecture Data Integration and Population Data Warehouse, Data Marts, Data Lakes Product, Platform and Vendor Selection and Management Data Access Metadata Management Data Virtualisation Data Warehouse, Data Marts, Data Lakes Standards Definition Data Warehouse, Data Marts, Data Lakes Monitoring, Data Collection and Analysis Data Warehouse, Data Marts, Data Lakes Performance and Capacity Planning Standards and Data Collection and Analysis Data Warehouse, Data Marts, Data Lakes Implementation Planning Implement, Operate Data Warehouse, Data Marts, Data Lakes Supporting Tools and Processes Implementation and Operation Data Warehouse, Data Marts, Data Lakes Team Formation Data Warehouse, Data Marts, Data Lakes Implementation Data Warehouse, Data Marts, Data Lakes Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Warehouse, Data Marts, Data Lakes Review and Improvement Data Warehouse, Data Marts, Data Lakes Management Data Warehouse, Data Marts, Data Lakes Operation Assessment January 9, 2023 49
  • 50. Data Reporting and Analytics, Visualisation Tools and Facilities Topic Scope Data Reporting and Analytics, Visualisation Tools and Facilities Research, Design, Define, Plan Data Reporting and Analytics, Visualisation Tools and Facilities Capability Establishment Data Reporting and Analytics, Visualisation Tools and Facilities Existing Approach Inventory and Baseline Data Reporting and Analytics, Visualisation Tools and Facilities Supporting Tools and Processes Definition Data Reporting and Analytics, Visualisation Tools and Facilities Scope and Activities Reporting and Visualisation Architecture and Approach Analytics Architecture and Approach Data Integration, Access and Security Data Reporting and Analytics, Visualisation Facility Access and Security Data Reporting and Analytics, Visualisation Product, Platform and Vendor Selection and Management Data Reporting and Analytics, Visualisation Development, Testing and Deployment Data Reporting and Analytics, Visualisation Distribution and Security Data Reporting and Analytics, Visualisation Tools and Facilities Standards Definition Data Reporting and Analytics, Visualisation Tools and Facilities Monitoring, Data Collection and Analysis Data Reporting and Analytics, Visualisation Tools and Facilities Performance and Capacity Planning Standards and Data Collection and Analysis Data Reporting and Analytics, Visualisation Tools and Facilities Implementation Planning Implement, Operate Data Reporting and Analytics, Visualisation Tools and Facilities Supporting Tools and Processes Implementation and Operation Data Reporting and Analytics, Visualisation Tools and Facilities Team Formation Data Reporting and Analytics, Visualisation Tools and Facilities Implementation Data Reporting and Analytics, Visualisation Tools and Facilities Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Reporting and Analytics, Visualisation Tools and Facilities Review and Improvement Data Reporting and Analytics, Visualisation Tools and Facilities Management Data Reporting and Analytics, Visualisation Tools and Facilities Operation Assessment January 9, 2023 50
  • 51. Data Discovery, Analysis, Design and Modelling Topic Scope Data Discovery, Analysis, Design and Modelling Research, Design, Define, Plan Data Discovery, Analysis, Design and Modelling Capability Establishment Data Discovery, Analysis, Design and Modelling Existing Approach Inventory and Baseline Data Discovery, Analysis, Design and Modelling Supporting Tools and Processes Definition Data Discovery, Analysis, Design and Modelling Scope and Activities Data Modelling Definition Data Profiling Approach Definition Data Discovery and Profiling Tool Selection and Implementation Data Lineage Definition Including Tool Selection Data Catalog Definition Including Tool Selection Data Dictionary Definition Including Tool Selection Semantic Layer Definition Including Tool/Platform Selection Data Discovery, Analysis, Design and Modelling Standards Definition Data Discovery, Analysis, Design and Modelling Monitoring, Data Collection and Analysis Data Discovery, Analysis, Design and Modelling Performance and Capacity Planning Standards and Data Collection and Analysis Data Discovery, Analysis, Design and Modelling Implementation Planning Implement, Operate Data Discovery, Analysis, Design and Modelling Supporting Tools and Processes Implementation and Operation Data Discovery, Analysis, Design and Modelling Team Formation Data Discovery, Analysis, Design and Modelling Implementation Data Discovery, Analysis, Design and Modelling Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Discovery, Analysis, Design and Modelling Review and Improvement Data Discovery, Analysis, Design and Modelling Management Data Discovery, Analysis, Design and Modelling Operation Assessment January 9, 2023 51
  • 52. External Data Sources and Interacting Parties Data Transfer/ Exchange/ Integration/ Publication Topic Scope External Data Sources and Interacting Parties Data Transfer/Exchange/Integration/Publication Research, Design, Define, Plan External Data Sources and Interacting Parties Capability Establishment External Data Sources and Interacting Parties Existing Approach Inventory and Baseline External Data Sources and Interacting Parties Supporting Tools and Processes Definition External Data Sources and Interacting Parties Scope and Activities Data Transfer/Exchange/Integration/Publication Request Review and Approval Process Data Transfer/Exchange/Integration/Publication Implementation Process Definition Data Transfer/Exchange/Integration/Publication Toolset Definition and Acquisition Data Transfer/Exchange/Integration/Publication Activity Monitoring and Review Data Transfer/Exchange/Integration/Publication Access Standards Data Transfer/Exchange/Integration/Publication Open Data Approach Data Transfer/Exchange/Integration/Publication Security Definition External Data Sources and Interacting Parties Standards Definition External Data Sources and Interacting Parties Monitoring, Data Collection and Analysis External Data Sources and Interacting Parties Implementation Implement, Operate External Data Sources and Interacting Parties Supporting Tools and Processes Implementation and Operation External Data Sources and Interacting Parties Team Formation External Data Sources and Interacting Parties Implementation External Data Sources and Interacting Parties Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve External Data Sources and Interacting Parties Review and Improvement External Data Sources and Interacting Parties Management External Data Sources and Interacting Parties Operation Assessment January 9, 2023 52
  • 53. Metadata Data Management Topic Scope Metadata Data Management Research, Design, Define, Plan Metadata Data Management Capability Establishment Metadata Data Management Existing Approach Scope Definition, Inventory and Baseline Metadata Data Management Supporting Tools and Processes Definition Metadata Data Management Scope and Activities Define Metadata Architecture and Approach Metadata Standard Review Metadata Data Management Standards Definition Define Metadata Management Tools Metadata Creation and Maintenance Approach Metadata Repositories Definition Metadata Integration and Usage Approach Metadata Data Management Monitoring, Data Collection and Analysis Metadata Data Management Performance and Capacity Planning Standards and Data Collection and Analysis Metadata Data Management Implementation Planning Implement, Operate Metadata Data Management Supporting Tools and Processes Implementation and Operation Metadata Data Management Team Formation Metadata Data Management Implementation Metadata Data Management Performance and Results Indicators and Measurement Framework Definition Baseline Metadata Creation Administer, Manage, Monitor, Improve Metadata Data Management Review and Improvement Metadata Data Management Management Metadata Data Management Operation Assessment January 9, 2023 53
  • 54. Data Quality Topic Scope Data Quality Research, Design, Define, Plan Data Quality Capability Establishment Data Quality Existing Approach Inventory, Profile and Baseline Data Quality Supporting Tools and Processes Definition Data Quality Scope and Activities Data Quality Requirements Data Quality Rules Approach Data Quality Service Level Management Approach Data Quality Analysis and Reporting Data Quality Standards Definition Data Quality Monitoring, Data Collection and Analysis Data Quality Implementation Planning Implement, Operate Data Quality Supporting Tools and Processes Implementation and Operation Data Quality Team Formation Data Quality Implementation Data Quality Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Quality Review and Improvement Data Quality Management Data Quality Operation Assessment January 9, 2023 54
  • 55. Data Solution Design Topic Scope Data Solution Design Research, Design, Define, Plan Data Solution Design Advisory Capability Establishment Data Solution Design Existing Approach Review, Inventory and Baseline Data Solution Design Scope and Activities Data Management and Governance Standards Data Modelling and Design Standards Operational and Archival Data Data Storage and Persistence Standards Data Infrastructure Standards Data Transfer/Exchange/Integration Standards Data Reporting and Analysis Standards Data Performance and Throughput Standards Data Security Standards Data Solution Design Monitoring, Data Collection and Analysis Data Solution Design Implementation Planning Implement, Operate Data Solution Design Supporting Tools and Processes Implementation and Operation Data Solution Design Team Formation Data Solution Design Implementation Data Solution Design Performance and Results Indicators and Measurement Framework Definition Administer, Manage, Monitor, Improve Data Solution Design Review and Improvement Data Solution Design Management Data Infrastructure, Storage and Operations Operation Assessment January 9, 2023 55
  • 56. Data Architecture Coverage Of Solution Component Types January 9, 2023 56 Solution Component Types Data Architecture Subject Areas Data Architecture Data Management, Governance, Supporting Processes Data Infrastructure, Storage and Operations Software, Hardware and Processes Data Security, Protection, Compliance, Access Control, Authentication, Authorisation Data Integration, Access, Flow, Exchange, Transfer, Transformation, Load And Extract Content, Unstructured Data, Records and Document Management Master and Reference Data Management Data Warehouse, Data Marts, Data Lakes Data Reporting and Analytics, Visualisation Tools and Facilities Data Discovery, Analysis, Design and Modelling External Data Sources and Interacting Parties Data Transfer/ Exchange/ Integration/ Publication Metadata Data Management Data Quality Data Solution Design Changes to Existing Systems X X X X X X X X X X X X New Custom Developed Applications X X X X X X X X X X X X Acquired and Customised Software Products X X X X X X X X X X X X System Integrations/ Data Transfers/ Exchanges X X X Reporting and Analysis Facilities X X X X X X X X Sets of Installation and Implementation Services Information Storage Facilities X Existing Data Conversions/ Migrations X X X X X X New Data Loads X X X X X X Central, Distributed and Communications Infrastructure X X Cutover/ Transfer to Production And Support Operational Functions and Processes Parallel Runs Enhanced Support/ Hypercare Sets of Maintenance, Service Management and Support Services Application Hosting and Management Services Changes to Existing Business Processes New Business Processes Organisational Changes, Knowledge Management Training and Documentation
  • 57. Data Architecture Coverage Of Solution Component Types • The data architecture subject areas impact the data aspects of many solution component types within solutions • An effective data architecture can contribute to effective solution architecture and solution data architecture January 9, 2023 57
  • 58. Data Architecture And Common Data Tooling And Standards • Data architecture needs to provide common infrastructural data tools and common data standards for solutions − Tools • Data storage infrastructure – hardware, software and platforms • Data warehouse platform • Data reporting, visualisation and analysis • Data transfer/exchange/integration/extract/transform/load • Data operations – backup/recovery/replication/business continuity/disaster recovery • Data anonymisation/pseudonymisation/encryption • Data monitoring/performance/capacity planning • Master data management platform • Reference data platform • Data catalog/semantic layer • Document management • Data analysis − Standards • Data security • Data quality • Metadata • Data discovery • Data modelling • Data management – classification/retention/archive/deletion January 9, 2023 58
  • 59. Organisation Data Architecture And Solution Data Architecture January 9, 2023 59 Organisation Data Architecture Common Data Standards Common Data Infrastructure Tools and Facilities Common Data Operations Individual Solutions Individual Solutions Individual Solutions Common Data Design and Implementation Approaches Common Data Model
  • 60. Organisation Data Architecture And Solution Data Architecture • Common data infrastructure tools allows reuse, reduces decision-making overhead and delays, reduces cost, accelerates individual deployment and achieves standardisation • Toolset will need to change in response to changing business needs and technology landscape January 9, 2023 60
  • 61. Common Data Plumbing Infrastructure • Data architecture should take on the projects required to deliver the common data plumbing infrastructure • These are foundational components • Individual solution delivery activities should not have to be responsible for their implementation January 9, 2023 61
  • 62. Common Data Plumbing Infrastructure January 9, 2023 62 Common ETL Common Data Transfer/ Exchange Common API Layer Common API Layer Common Data Infrastructure/ Platform Common Data Warehouse Common Data Reporting Common Data Analytics Common Data Backup and Recovery Common Business Continuity and Disaster Recovery Common Document Management Common Data Analytics Common Performance Monitoring and Capacity Planning Common Audit Logging Data Management Common Data Catalog Common Reference and Master Data Individual Solutions
  • 63. Common Data Plumbing Infrastructure • This ideal is regularly not fully in place • Individual solutions often have to implement some of these capabilities that are not available centrally • The leads to sub-optimal solutions with point resolutions to specific requirements January 9, 2023 63
  • 64. Data Design And Modelling For Solutions • The objective of data design for solutions is the same as that for overall solution design: − To capture sufficient information to enable the solution design to be implemented − To unambiguously define the data requirements of the solution and to confirm and agree those requirements with the target solution consumers − To ensure that the implemented solution meets the requirements of the solution consumers and that no deviations have taken place during the solution implementation journey January 9, 2023 64
  • 65. Why Pay Attention To Solution Data Architecture? • Solution data architecture avoids problems with solution operation and use: − Poor and inconsistent data quality − Poor performance, throughput, response times and scalability • Poorly designed data structures can lead to long data update times leading to long response times, affecting solution usability, loss of productivity and transaction abandonment − Poor reporting and analysis − Poor data integration − Poor solution serviceability and maintainability − Manual workarounds for data integration, data extract for reporting and analysis • Data-design-related solution problems frequently become evident and manifest themselves only after the solution goes live • The benefits of solution data architecture are not always evident initially January 9, 2023 65
  • 66. New Technology And Impact on Solution Data Architecture • New solution deployment and operating models affects solution data architecture − New solution design, deployment and operating models − Greater use of platform-based solution implementation and deployment − Wider range of complex data technology options, especially in terms of data analysis − Distributed solution components, distributed solution consumer base, distributed access with many interfaces, integration points and data flows − Complexity with multiple data integrations January 9, 2023 66
  • 67. Solution Design – From … January 9, 2023 67 Solution Central Data Store Solution Central Application Component Solution API Solution Central Infrastructure Solution Hosted Infrastructure Solution Internal Consumers Solution External Private Consumers Solution Hosted Data Store Solution Hosted Application Component Solution Hosted Analytics Access and Security Infrastructure Central To Hosting Facility Connectivity Solution External Public Consumers Solution Mobile App
  • 68. To … • Increasing solution landscape complexity and diversity gives rise to greater data design complexity January 9, 2023 68 Solution Central Data Store Solution Central Application Component Solution API Solution Central Infrastructure Solution Hosted Infrastructure Solution Internal Consumers Solution External Private Consumers Solution Hosted Data Store Solution Hosted Application Component Solution Hosted Analytics Access and Security Infrastructure Central To Hosting Facility Connectivity Solution External Public Consumers Solution Mobile App
  • 69. Solution Entity Model January 9, 2023 69 Solution Component Types Solution Components Solution Solution Zones Solution Zone Types Solution Topology Solution Consists Of Multiple Components Each Solution Component Has A Type Solution Exists Within A Topology Of Many Solutions Solution Components Are Located In Solution Zones Each Solution Zone Has A Type Solution Operational Entity Solution Operational Entity Type Deployed Solution Consists Of Multiple Operational Entities Each Solution Operational Entity Has A Type Solution Operational Entities Are Located In Solution Zones Some Solution Components Become Deployed Operational Entities
  • 70. Solution Zone Types and Zones January 9, 2023 70 Solution Component Types Solution Components Solution Solution Zones Solution Zone Types Solution Topology Solution Consists Of Multiple Components Each Solution Component Has A Type Solution Exists Within A Topology Of Many Solutions Solution Components Are Located In Solution Zones Each Solution Zone Has A Type Solution Operational Entity Solution Operational Entity Type Deployed Solution Consists Of Multiple Operational Entities Each Solution Operational Entity Has A Type Solution Operational Entities Are Located In Solution Zones Some Solution Components Become Deployed Operational Entities
  • 71. Solution Zones • Solution zones are locations where groups of closely related solution components reside • They represent containers for solution components • Zones are located within the wider physical solution landscape • Each zone and the components it holds have different security requirements • Not all solutions will have components in all zone and not all organisations will have all the zone types • The solution and its constituent components can span multiple different zones of the same type • The zone approach is useful way of representing the entirety of a solution, its constituent components, their connectivity, linkages and interactions, especially data storage, processing and interactions • You will have different levels of control over different solution zones (including no control) – this impacts data design considerations January 9, 2023 71
  • 72. Sample Solution Zone Types January 9, 2023 72
  • 73. Sample Solution Zone Types January 9, 2023 73
  • 74. Sample Solution Zone Types Zone Description Insecure External Organisation Presentation And Access Where publicly accessible or accessing entities reside. These entities are regarded as insecure and/or untrusted. Secure External Organisation Participation and Collaboration Outside the physical organisation boundary where entities that are provided by or to trusted external parties reside Secure External Organisation Access Contain entities that enable secure access or are securely accessible from outside the organisation Organisation Contain the entities within the organisation boundary and contains all the locations, business units and functions within it Central Solutions and Access Contains the solution entities and their data Solution Zone Contains the solution entities Data Zone Zone within the organisation where data is segregated for security Remote Business Unit Solutions and Access Remotely located organisation business unit or location and the entities it contains Workstation Zone Zone within the organisation where users accessing data and solutions are segregated for security Outsourced Service Provider Solutions and Access Contains solutions provided by and located in facilities provided by outsourced partners Cloud Service Provider Solutions and Access Contains solutions - platform, infrastructure and service - provided by and located in cloud service providers Co-Located Solutions and Access Contains solutions the organisation has located in facilities provided by co- location providers January 9, 2023 74
  • 75. Solution Consumers • Any solution with have different sets of consumers of different types: − Controlled Consumers – typically organisation personnel over whom the solution owner has substantial control − Partially Controlled Consumers – typically external business partners and other interacting parties over whom the solution owner has some control and influence − Uncontrolled Consumers – typically members of the public at whom the solution is targeted and whose needs must be inferred through groups of proxy consumers • Each consumer type will have different data-related needs and expectations • Classifying and understanding the target solution consumers will contribute to solution data architecture design January 9, 2023 75
  • 76. Data Design And Modelling For Solutions Activities Data Modelling Data Journeys Design Data Processing Design January 9, 2023 76 Logical Data Model Physical Data Model Conceptual Data Model
  • 77. Data Design And Modelling For Solutions Activities • This includes the following activities: − Data Modelling – define the data entities, structures, attributes and contents • Conceptual Data Model (CDM) – create an initial high-level view of solution • Logical Data Model (LDM) – expand the CDM with detailed data requirements • Physical Data Model (PDM) – translate the LDM into implementation- specific details − Data Journeys Design – create an inventory of data journeys and identify the steps within the journeys and the data entities involved − Data Processing Design – define the detail of the processing performed on data entities January 9, 2023 77
  • 78. Data Design And Modelling For Solutions Activities • The data analysis and design activities are not linear or sequential • As the analysis progresses, earlier work may need to be revisited to be elaborated and expanded on January 9, 2023 78 Conceptual Data Model Logical Data Model Physical Data Model Data Journeys Design Data Processing Design Data Modelling
  • 79. Packaged Solutions And Platforms • Packaged solution components and platforms on which solutions components are implemented and deployed will have pre-defined data models with a greater or lesser degree of configuration and customisation • The data design activities should still be performed for these solution components • The inherent data limitations and restrictions of the packages and platforms should be clearly defined and understood January 9, 2023 79
  • 80. Solutions And Shared/Private Data January 9, 2023 80 • Solutions and their components within the organisation solution landscape will have both local data and data that is shared with or between other solutions, either for upstream/ downstream processing or as shared data repositories
  • 81. Solutions And Shared/Private Data • Data design activities are different for shared and private solution data • Shared solution data includes reference and master data − Reference data consists of common code, structural and identifier values • Their purpose is to ensure consistency across data values • Reference data is static or slowly changing − Master data relates to common transaction identifiers such interacting parties (customer, partner, etc,) details • Having a single version of master data ensures single view of all interactions with the party across the organisation can be identified • Solution data modelling activities should identify the occurrences of reference and master data to maximise reuse and contribute to maintaining a single version of the truth January 9, 2023 81
  • 82. Shared/Common Data Issues • Because shared data is shared, the main concerns and issues relate to: − Ownership – who owns and is responsible − Maintenance – who maintains it and keeps it current, who is responsible for allowing updates, what is the process for applying updates and changes − Quality – who is responsible for maintaining data quality • Individual solutions should not have to solve these problems, but commonly and unfortunately have to January 9, 2023 82
  • 83. Common Data Formats • Master data will have a common format • Individual solutions need to adhere to the common master data format January 9, 2023 83
  • 84. Data Modelling – Conceptual Data Model • The Conceptual Data Model (CDM) represents concepts, entities and their relationships within the scope of the solution • It is used to create a common understanding among all the solution stakeholders • The CDM defines the scope of the solution • The functional requirements of the solution provide an input to the CDM and its constituent entities January 9, 2023 84
  • 85. Data Modelling – Logical Data Model • The Logical Data Model (LDM) expands on the agreed CDM • Detailed data requirements for the specified data entities are defined, including solution zones January 9, 2023 85
  • 86. Data Modelling – Physical Data Model • The Physical Data Model (PDM) translates the LDM into technology-specific implementation details and a technology structure across the solution zones • The LDM may be updated to reflect and accommodate technology and platform specific features, limitations, capabilities and restrictions January 9, 2023 86
  • 87. Data Journeys Design • Solutions have journeys as consumers use the solution to achieve results • Solution journeys reflect solution consumer experiences • Data journeys represent the data exchanges and transfers that occur to support the solution journeys • Solution data architecture should first create an inventory of data journeys • The processing data journeys can then be expanded to reflect the lifecycle for the data type(s) associated with the data journeys January 9, 2023 87
  • 88. Data Processing Design • Data processing design describes the detailed processing that is performed on data • It defines the business rules that are applied to data within the scope of the solution January 9, 2023 88
  • 89. Data Processing Design Identify the data processing performed on data entities and objects by each solution component for each solution journeys January 9, 2023 89 Data Entity/Object Solution Component Data Entity/Object Solution Component Solution Component Solution Component Solution Component Solution Component Solution Component Solution Component Data Entity/Object Data Entity/Object Data Entity/Object Data Entity/Object Data Entity/Object Data Entity/Object D1 D2 D3 D4 D5
  • 90. Summary • The data architecture of solutions is frequently not given the attention it deserves or needs • Frequently, too little attention is paid to designing and specifying the data architecture within individual solutions and their constituent components • This is due to the behaviours of both solution architects ad data architects • Solution architecture tends to concern itself with functional, technology and software components of the solution • Data architecture tends not to get involved with the data aspects of technology solutions, leaving a data architecture gap • Combined with the gap where data architecture tends not to get involved with the data aspects of technology solutions, there is also frequently a solution architecture data gap • Solution architecture also frequently omits the detail of data aspects of solutions leading to a solution data architecture gap • These gaps result in a data blind spot for the organisation • Data architecture tends to concern itself with post-individual solutions • Data architecture needs to shift left into the domain of solutions and their data and more actively engage with the data dimensions of individual solutions • Data architecture can provide the lead in sealing these data gaps through a shift-left of its scope and activities as well providing standards and common data tooling for solution data architecture January 9, 2023 90