ETIS11 - Agile Business Intelligence - Presentation


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

ETIS11 - Agile Business Intelligence - Presentation

  2. 2. AGILE MANIFESTO CYPRUS ‘11 We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:Individuals and interactions over processes and toolsWorking software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Utah, Feb 2001 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   2  
  3. 3. CYPRUS ‘11TWELVE PRINCIPLES OF AGILE SOFTWARE •  Our highest priority is to satisfy the •  Working software is the primary customer through early and measure of progress. continuous delivery of valuable software. •  Welcome changing requirements, •  Agile processes promote sustainable even late in development. Agile development. The sponsors, processes harness change for the developers, and users should be able customers competitive advantage. to maintain a constant pace indefinitely. •  Deliver working software frequently, •  Continuous attention to technical from a couple of weeks to a couple of excellence and good design enhances months, with a preference to the agility. shorter timescale. •  Business people and developers must •  Simplicity--the art of maximizing the work together daily throughout the amount of work not done--is essential. project. •  Build projects around motivated •  The best architectures, requirements, individuals. Give them the and designs emerge from self- environment and support they need, organizing teams. and trust them to get the job done. •  The most efficient and effective •  At regular intervals, the team reflects method of conveying information to on how to become more effective, and within a development team is then tunes and adjusts its behavior face-to-face conversation. accordingly.14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   3  
  4. 4. LARGE ORGANIZATIONS FORGET CYPRUS ‘11 •  Most companies start with an innovative and entrepreneurial people and processes that get things done – they are inherently agile •  As organizations grow they put structures in place that standardize the organization but these limit creativity, increase timescales and reduce risk at the cost of reducing benefits and reward 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   4  
  5. 5. THE PROCESS IMPROVEMENT PARADOX CYPRUS ‘11 •  The paradox (…) is that process improvement is good, but process improvement programs arent, or at least they often arent. •  Organizations become more and more averse to risk as they "mature”. An organization under the gun to demonstrate increased CMM level is not going to go looking for real challenge. •  This (…) entices the play-it-safe behavior of low-risk, and therefore low-benefit, projects. Peopleware; Tom De Marco & Timothy Lister; 1987 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   5  
  6. 6. SUCCESSFUL LARGE ORGANIZATIONS CYPRUS ‘11RECOGNIZE AND REACT •  SkunkWorks –  Lockheed Martin Advance Development Projects –  A group within an organization given a high degree of autonomy and unhampered by bureaucracy, tasked with working on advanced projects. –  Responsible for U-2 & Blackbird Spy Planes, Stealth Fighter & Bomber –  Founded in June 1943 specifically to overcome the hurdles of government and large corporations working together –  SkunkWorks has 14 rules and practices •  Very Similar to the 12 Agile Principles Ben Rich & Leo Janos; SkunkWorks; 1996 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   6  
  7. 7. OTHER EXAMPLES CYPRUS ‘11 •  In the late 80s/early 90s Telcos led the way in Agile BI development –  Vodafone: Mast Placement Project –  Swisscom: Geo-located SMS Marketing •  But they have been overtaken by the search and social networks websites that rely on rapidly creating and consuming BI data to survive –  Examples: Facebook, Google & LinkedIn –  These companies are also creating and using the next generation of BI tools and BI engines 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   7  
  8. 8. AGILE BI IS NOW A 2-SPEED HIGHWAY CYPRUS ‘11 •  Truly Agile BI Organizations –  Small, strong, highly skilled teams with a strong, trusting, focused user relationship –  Project team having delegated authority and responsibility for delivery with long term funding –  The ability to operate outside standard corporate procedures (e.g. procurement) –  Regular delivery of incremental improvement •  Those that describe themselves as “wanting to be more agile” –  Willing to adopt some of the methods but unable to break free from the corporate chains 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   8  
  9. 9. CYPRUS ‘11 TECHNIQUES BECOMING MORE AGILE14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   9  
  10. 10. HOW TO BE MORE AGILE CYPRUS ‘11 •  Team Structures •  Wiki-fy Everything •  Use A Dedicated Platform •  Build Literal Staging Areas •  Throw out your ETL Tool? •  More Dynamic Reporting Tools •  Develop reports with your users •  Embrace (not so) New Technologies 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   10  
  11. 11. TEAM STRUCTURES CYPRUS ‘11 •  Agile teams are generally smaller but with broader and deeper skill sets –  These resources are more expensive individually but cheaper collectively –  Smaller teams significantly reduce the management and communication overhead –  Close communications and broad skills are more likely to generate innovative solutions 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   11  
  12. 12. WIKI-FY EVERYTHING CYPRUS ‘11 •  Projects need a consistent, persistent, versioned knowledge store •  Use a Wiki and train your business users how to use it •  Optimizes the ‘documentation’ set collected in a single structured way •  Abandon office products and SharePoint for critical documents •  Don’t store your requirements in e-mails, IM chats and documents 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   12  
  13. 13. USE A DEDICATED PLATFORM CYPRUS ‘11 •  Marketplace has a choice of BI specific, low maintenance, low TCO appliances –  IBM Netezza –  Teradata 14 (AsterData) –  Sybase IQ •  And some high power workhorses –  Oracle Exadata –  Teradata •  There are plenty of emerging technologies –  Curt Monash: 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   13  
  14. 14. BUILD LITERAL STAGING AREAS CYPRUS ‘11 •  These are 1:1 copies of your source systems (hence the ‘Literal’) •  Do this whilst others are collecting requirements, building data models, etc. •  Do some test reporting off the LSA with the users – it starts the user engagement, helps evolve the business requirements, and develops communication •  They will be an essential source for the data warehouse as it evolves and remove load from the operational systems 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   14  
  15. 15. THROW OUT YOUR ETL TOOL ? CYPRUS ‘11 •  ETL tools require product specific expertise –  Value add features are under-utilized –  Developers and DBAs use them as glorified scheduling tools •  Good Source Code Control and Management Scripting can compensate for much of the lost benefits •  Significantly reduces the project cost and increases the available skilled resources •  InsureTheBox/Netezza; YapiKredi/SybaseIQ; NonDisclosure/Greenplum have major data warehouses with no ETL product 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   15  
  16. 16. MORE DYNAMIC REPORTING TOOLS CYPRUS ‘11 •  Reports on your hand held device –  Qlikview, RoamBI, PushBI, Tableau •  Use Dynamic Web Deployed Tools –  Panopticon, Tableau •  RSS Feeds –  Usable by any RSS Reader on any platform •  Requires dedicated, reactive reporting experts that work enthusiastically with the business users and are willing to go the extra mile 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   16  
  17. 17. DEVELOP REPORTS WITH YOUR USERS CYPRUS ‘11 •  De La Rue / Tableau Reporting •  First 50 reports deployed over 28 days •  Initially deployed to users from the LSA and then migrated to the data marts •  Near real-time (fifteen minutes maximum delay) data •  Averages –  Day 5: 17 changes / 5 users –  Day 10: 1 change / 42 users –  Day 15: 0 changes / 50 users 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   17  
  18. 18. EMBRACE (NOT SO) NEW TECHNOLOGY - CYPRUS ‘11MAPREDUCE/HADOOP/NOSQL •  Software frameworks that supports data-intensive distributed applications and enables them to work with thousands of nodes and petabytes of data. •  Hadoop is a filesystem (HDFS) and distributed programming framework (MapReduce) whilst a NoSQL database consists of key-value pairs, no joins, data is sharded and replicated, no single point of failure. •  Notable BI Use (but not the only one): Rapid loading and processing of volume data to create and validate data sets for onward inclusion in the data warehouse whilst doing large scale, near real-time ‘dirty’ analysis •  Notable Users: Facebook, Twitter, LinkedIn, Yahoo!, eBay, Amazon, Google •  Technologies in use since at least 2004 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   18  
  19. 19. EMBRACE (NOT SO) NEW TECHNOLOGY – CYPRUS ‘11COMPLEX EVENT PROCESSING (CEP) •  Complex event processing (CEP) consists of processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time. •  Already used for fraud and network management but has applications in customer service and market to fine tune customer interactions in real time •  Notable BI Use: real-time information alongside existing BI – rather than trying to create an entire real-time solution •  Technologies in use since at least 1999 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   19  
  20. 20. BUT REMEMBER … CYPRUS ‘11 •  Agile is all about approach and people: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan •  Your management need to be willing to create: A group within an organization given a high degree of autonomy and unhampered by bureaucracy, tasked with working on advanced projects. 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   20  
  21. 21. REFERENCE MATERIAL CYPRUS ‘11 •  Websites –  Agile Manifesto •  12 Principles of Agile –  SkunkWorks •  14 Rules & Practices –  Hadoop and NoSQL Myth-busting •  Books –  Peopleware: Productive Projects & Teams –  SkunkWorks: A Personal Memoir –  Mythical Man Month: Essays on Software Engineering 14-­‐Oct-­‐2011   ©  2011  Data  Management  &  Warehousing   21