Eric Nelson of Synaptitude Consulting led a discussion on Agile BI and Data Chaos at the January 2014 meeting of the DC Business Intelligentsia held at Excella Consulting's headquarters in Arlington, VA.
Business Intelligence and The Enterprise
Reporting is useful at the department level.
Business Information is useful only across the Whole Business.
Enterprise BI Architecture
data, relate to the
increase in benefits.
What is essential is to
evaluate where the
cut-off point is for
optimum benefits at a
But to provide actionable intelligence across the enterprise don’t you need to
capture all the data in the enterprise?
Traditional Approach to Building
the Factory – The Data Warehouse
Traditional Approach to Building
Traditional DW/BI methodology has a heavy emphasis on
requirements gathering, because you want to precalculate/pre-model all the answers. You need to know all
the questions that might get asked. But if you are working
with new information, new data, new business dynamics –
how do you know what you don’t know?
Understanding the quality of the data, and the effectiveness
of controls, is critical to maintaining the underlying DW
But I need information now!
Building data warehouses takes years and $millions
But I have to deliver ROI on all this new gear and
systems soon than that, and I need information to do
so…..Can I just get some reports?
And maybe a dashboard or 2?
And maybe then some analytics/forecasting/predictive…
A simple 5-step process to avoiding
data chaos while building the factory
Find your data
Current systems, other capabilities (available data that is not
currently collected, or is being collected but not utilized)
Evaluate your data
Understand how your data is actually behaving vs. how you
expected it to behave
Assess data quality across stacks/domains
3. Start with some departmental initiatives
4. Transform your data into information
Turn data into information by mapping to the enterprise model
(the data warehouse thing…)
Your company doesn’t stand still, and neither does your data…
So, what to do – Agile BI
Deliver actionable results in weeks vs. years
Discover how your data is behaving before you design
your information factory
Because as we know, Agile is the cure for everything IT
BI Characteristics and Goals (Kenochan 2year study)
Data entry — accuracy
Data consolidation — consistency
Data aggregation — scope
Information targeting — fit
Information delivery — timeliness
Information analysis — analyzability
Common BI Issues (Kenochan)
20% of data has errors in it (accuracy)
50% of data is inconsistent (consistency)
It typically takes 7 days to get data to the end user
It isn't possible to do a cross-database query on 70% of
company data (scope)
65% of the time, executives don't receive the data they need
60% of the time, users can't do immediate online analysis
of data they receive (analyzability)
75% of new key information sources that surface on the
Web are not passed on to users within the year (agility)
Agile addresses BI Issues
Data Entry - Processes need to re-evaluated and corrected to minimize data
Data Consolidation - Often companies have multiple data stores and data is
scattered across multiple data stores. "Agility theory emphasizes autodiscovery of each new data source, and automated upgrade of metadata
repositories to automatically accommodate the new information".
Data Aggregation - Is a process in which information from many data stores is
pulled and displayed in a summary report. OLAP is a simple type of data
aggregation tools which is commonly used.
Information Delivery - One of the key principal of Agile BI is to deliver the
right data at the right time to the right individual. Historical data should also
be maintained for comparing the current performance with the past.
Information Analysis - One of the largest benefits of Agile BI is in improving
the decision-making of its users. Real Agile BI should focus on analysis tools
that make an operational process or new product development better.[
Agile BI Key Principals
Agile Best Practices also address
the Data Issues
A program charter should be created, which will set the stakeholder
expectations on how Agile BI system will work.
Start with the business information needs to provide context for scope.
Iterations should be time-boxed.
Stress on data discovery through the requirements and design phase.
Use the Agile process of incremental and iterative development and
Validate the BI Architecture and get approval on the proof of concept.
Data validation and verification should be completed for each development
Use flow charts or diagram to explain the BI process along with some
Any change that will be deployed to production should be thoroughly tested in
Have a formal change control; this will minimize the risk as all changes have to
be approved before it goes into production.
So, if Agile solves the BI Data Issues, why do we
continually fail to deliver Quality, Actionable Data
ACROSS THE ENTERPRISE?
Enterprise BI vs. Departmental BI
Since Agile cannot deliver across the
Enterprise, Agile BI cannot deliver across
Enabling Enterprise Action vs. Departmental
Intelligence – Some Observations for
Agile Development is fairly straightforward to execute
at a departmental level
Therefore, Agile BI is fairly straightforward to execute at
a department level
Generally, Agile Development falls apart at the
Scaled Agile Framework
Generally, Agile BI falls apart at the Enterprise Level