Life in Hell: The Experience of Successful BI Managers


Published on

Life in Hell: The Experience of Successful BI Managers - retitled from the boring original.

Growing business intelligence from a departmental deployment or pilot project to an enterprise scale is not as simple as buying more licenses. Increasing the scope of BI means taking into account a broader set of information needs in the organization, increasing the size and variety of data needed in a warehouse or mart. Broader use implies more varied delivery needs, as well as scaling up the user count. Increasing the scope of use also raises the importance of managing day to day operations and reliability.

This webcast will discuss factors to consider when increasing the scope of a BI program, and what requirements that puts on components of the technology infrastructure.
Different axes of enterprise scale
How information delivery requirements change as you expand BI
The impacts of growth on operations and administration

Slides from a webcast for TDWI. You can listen to the full webcast and see the Jaspersoft/Talend presentations here:

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Life in Hell: The Experience of Successful BI Managers

  1. 1. Whats Required to Grow an Enterprise BI Deployment?December 7, 2010Mark 1
  2. 2. Maturation of BIA healthy BI program gets more complex and harder to manage over time.Growth from the initial installation to broader organizational use has many aspects: ▪ Number of users ▪ Multiplying uses ▪ More informationGrowth requires adapting processes and technology.
  3. 3. Maturation: Initial Build, the Early Stage Consultants leave here ROI Minimum ROI hurdle for  initial project to be built,  infrastructure created Future work planned for Done ProjectsCopyright Third Nature, Inc.
  4. 4. The Origin of BI Backlog: Next Phase Fewer resources, so work  takes slightly longer to  complete, but not so long as  initial build ROI Minimum ROI hurdle is  lower for subsequent work Done ProjectsCopyright Third Nature, Inc.
  5. 5. The Long Tail of BI This is what happens to  successful data warehouses ROI “Oh crap” Done ProjectsCopyright Third Nature, Inc.
  6. 6. Prioritizing the Long Tail of BI Financial priorities, business  priorities, steering committees,  budget limits, time-boxing… ROI (Guess which things get done) ProjectsCopyright Third Nature, Inc.
  7. 7. Prioritizing the Long Tail of BI Financial priorities, business  priorities, steering committees,  budget limits, time-boxing… ROI (Guess which things get done) Executive pet projects ProjectsCopyright Third Nature, Inc.
  8. 8. The Long Tail of BI: Why We Have Spreadmarts Mismanage this process and  you have a legacy system  everyone complains about ROI *sigh* Low hanging fruit Projects The Kingdom of ExcelCopyright Third Nature, Inc.
  9. 9. Development Process Designed to Minimize Later Change Requirements,  This process is fine data sources for the initial build. ETL / DI Three months later, not so much. Warehouse /  Mart BI / Analytics  server Metadata stored here, almost but not quite Clients the same each time The common MD repo was supposed to fix this
  10. 10. Two Things People Don’t Want Data integration and BI projects that  take months to deliver for business  needs that may be one‐time or done  in weeks.Least‐common denominator financial and transaction data with contextual information and details stripped away in the name of speed. Slide 10
  11. 11. The Process is Not a Waterfall, It’s an Ongoing Cycle Requirements,  data sources Clients ETL / DI BI / Analytics  Warehouse /  server Mart The BI layer is the starting point for users, not the end point. As people adopt new information, needs alter, driving change. Your processes switch from “build” to “keep it running”.Slide 11Copyright Third Nature, Inc.
  12. 12. Growth: Increased UseIf you’re successful, users become more proficient and BI use increases.Effects are: ▪ Performance problems ▪ Capacity problems ▪ Shrinking data load  windows and response  time requirementsThis presages more user growth.
  13. 13. Aspects of Growth: More UsersRaises problems of scaling not only performance, but also managing accounts, data security, licenses.
  14. 14. Growth: Increased Variety of UsesOnce people master the basics, diversifying demands require new tools, more complex analysis and models.
  15. 15. Warning about software vendors:The Swiss knew when to stop. Vendors often don’t.BI and DI vendor response has been to add features to the tools to meet all the different use cases.Different uses can drive conflicting tool requirements.
  16. 16. Growth:  DataData is the item everyone focuses on when talking about growth and BI.The primary impact is on the database, both getting data out and getting data in.Data volume is the easiest problem to address (in most common BI / DW situations).
  17. 17. Other aspects of data growth are harder to addressVariety:1. More sources ▪ More system types ▪ APIs and other oddities2. More types ▪ The usual suspects ▪ And more3. Uses that require more  complex data models or  transformations. ▪ Like data mining ▪ And sandboxes
  18. 18. Growth often drives the need for lower latency12x5 moves to 24x7 operation, driving SLAs, capacity planning, failover and disaster recovery.Methodology, organization and technology are related. Speeding up one won’t always speed up the others. 18
  19. 19. New Data Integration Methods and Tools RequiredThe initially designed ETL may not handle the varied data latencies, or deliver the performance to meet smaller batch windows.Single batch Frequent batch Mini‐batch Continuous load Streaming Daily+ Copyright Third Nature, Inc. Immediate
  20. 20. Development, Maintenance  & OperationsReal time decisions on low latency data mean data quality plays a larger role, and it’s harder to address.Warehouse availability becomes much more important to the business, and it isn’t just the database – it’s everything.Performance and meeting strict BI SLAs will rise in importance since you are now tied in to business operations. Slide 20
  21. 21. AdministrationAs BI grows in importance within the organization this becomes a focal point. ▪ Any problem is magnified  due to the broader scope. ▪ There are more products. ▪ There are more inter‐tool  dependencies and problems  are more distributed. ▪ The least‐emphasized set of  features in most BI tools. ▪ And most DI tools.
  22. 22. Overall Effect of GrowthComplexity, which leads to• Performance problems• Reliability problems• Maintenance problems• Difficulty adapting to changeUsually we forego features useful in the long term during our product evaluations in favor of features important for initial delivery.
  23. 23. About the Presenters Mark Madsen is president of Third  Nature, a technology research and  consulting firm focused on analytics,  business intelligence and data  management. Mark is an award‐ winning author, architect and CTO  whose work has been featured in  numerous industry publications. He is  an international speaker, a contributing  editor at Intelligent Enterprise, and  manages the open source channel at  the Business Intelligence Network. For  more information or to contact Mark,  visit