Life in Hell: The Experience of Successful BI Managers - retitled from the boring original. Growing business intelligence from a departmental deployment or pilot project to an enterprise scale is not as simple as buying more licenses. Increasing the scope of BI means taking into account a broader set of information needs in the organization, increasing the size and variety of data needed in a warehouse or mart. Broader use implies more varied delivery needs, as well as scaling up the user count. Increasing the scope of use also raises the importance of managing day to day operations and reliability. This webcast will discuss factors to consider when increasing the scope of a BI program, and what requirements that puts on components of the technology infrastructure. Different axes of enterprise scale How information delivery requirements change as you expand BI The impacts of growth on operations and administration Slides from a webcast for TDWI. You can listen to the full webcast and see the Jaspersoft/Talend presentations here: http://ow.ly/3lkHj

1. What's Required to Grow an
Enterprise BI Deployment?
December 7, 2010
Mark Madsen
www.ThirdNature.net
1

2. Maturation of BI
A healthy BI program gets more
complex and harder to manage
over time.
Growth from the initial
installation to broader
organizational use has many
aspects:
▪ Number of users
▪ Multiplying uses
▪ More information
Growth requires adapting
processes and technology.

3. Maturation: Initial Build, the Early Stage
Consultants leave here
ROI Minimum ROI hurdle for
initial project to be built,
infrastructure created
Future work planned for
Done Projects
Copyright Third Nature, Inc.

4. The Origin of BI Backlog: Next Phase
Fewer resources, so work
takes slightly longer to
complete, but not so long as
initial build
ROI
Minimum ROI hurdle is
lower for subsequent work
Done Projects
Copyright Third Nature, Inc.

5. The Long Tail of BI
This is what happens to
successful data warehouses
ROI
“Oh crap”
Done Projects
Copyright Third Nature, Inc.

6. Prioritizing the Long Tail of BI
Financial priorities, business
priorities, steering committees,
budget limits, time-boxing…
ROI (Guess which things get done)
Projects
Copyright Third Nature, Inc.

7. Prioritizing the Long Tail of BI
Financial priorities, business
priorities, steering committees,
budget limits, time-boxing…
ROI (Guess which things get done)
Executive pet projects
Projects
Copyright Third Nature, Inc.

8. The Long Tail of BI: Why We Have Spreadmarts
Mismanage this process and
you have a legacy system
everyone complains about
ROI
*sigh*
Low hanging fruit Projects The Kingdom of Excel
Copyright Third Nature, Inc.

9. Development Process Designed to Minimize Later Change
Requirements, This process is fine
data sources
for the initial build.
ETL / DI
Three months later,
not so much.
Warehouse /
Mart
BI / Analytics
server
Metadata stored here,
almost but not quite Clients
the same each time
The common MD repo was supposed to fix this

10. Two Things People Don’t Want
Data integration and BI projects that
take months to deliver for business
needs that may be one‐time or done
in weeks.
Least‐common denominator
financial and transaction data with
contextual information and details
stripped away in the name of speed.
Slide 10

11. The Process is Not a Waterfall, It’s an Ongoing Cycle
Requirements,
data sources
Clients ETL / DI
BI / Analytics Warehouse /
server Mart
The BI layer is the starting point for users, not the end point.
As people adopt new information, needs alter, driving change.
Your processes switch from “build” to “keep it running”.Slide 11
Copyright Third Nature, Inc.

12. Growth: Increased Use
If you’re successful, users
become more proficient
and BI use increases.
Effects are:
▪ Performance problems
▪ Capacity problems
▪ Shrinking data load
windows and response
time requirements
This presages more user
growth.

13. Aspects of Growth: More Users
Raises problems of scaling not only performance,
but also managing accounts, data security, licenses.

14. Growth: Increased Variety of Uses
Once people master the basics, diversifying demands
require new tools, more complex analysis and models.

15. Warning about software vendors:
The Swiss knew when to stop. Vendors often don’t.
BI and DI vendor response has been to add features to
the tools to meet all the different use cases.
Different uses can drive conflicting tool requirements.

16. Growth: Data
Data is the item everyone
focuses on when talking
about growth and BI.
The primary impact is on
the database, both
getting data out and
getting data in.
Data volume is the
easiest problem to
address (in most common
BI / DW situations).

17. Other aspects of data growth are harder to address
Variety:
1. More sources
▪ More system types
▪ APIs and other oddities
2. More types
▪ The usual suspects
▪ And more
3. Uses that require more
complex data models or
transformations.
▪ Like data mining
▪ And sandboxes

18. Growth often drives the need for lower latency
12x5 moves to 24x7 operation, driving SLAs, capacity
planning, failover and disaster recovery.
Methodology, organization and technology are related.
Speeding up one won’t always speed up the others.
18

19. New Data Integration Methods and Tools Required
The initially designed ETL may not handle the varied
data latencies, or deliver the performance to meet
smaller batch windows.
Single batch
Frequent batch
Mini‐batch
Continuous load
Streaming
Daily+ Copyright Third Nature, Inc. Immediate

20. Development, Maintenance
& Operations
Real time decisions on low
latency data mean data quality
plays a larger role, and it’s
harder to address.
Warehouse availability becomes
much more important to the
business, and it isn’t just the
database – it’s everything.
Performance and meeting strict
BI SLAs will rise in importance
since you are now tied in to
business operations.
Slide 20

21. Administration
As BI grows in importance
within the organization this
becomes a focal point.
▪ Any problem is magnified
due to the broader scope.
▪ There are more products.
▪ There are more inter‐tool
dependencies and problems
are more distributed.
▪ The least‐emphasized set of
features in most BI tools.
▪ And most DI tools.

22. Overall Effect of Growth
Complexity, which leads to
• Performance problems
• Reliability problems
• Maintenance problems
• Difficulty adapting to change
Usually we forego features
useful in the long term during
our product evaluations in
favor of features important for
initial delivery.

23. About the Presenters
Mark Madsen is president of Third
Nature, a technology research and
consulting firm focused on analytics,
business intelligence and data
management. Mark is an award‐
winning author, architect and CTO
whose work has been featured in
numerous industry publications. He is
an international speaker, a contributing
editor at Intelligent Enterprise, and
manages the open source channel at
the Business Intelligence Network. For
more information or to contact Mark,
visit http://ThirdNature.net.