1. DATA MANAGEMENT & WAREHOUSING CYPRUS ‘11
DAVID M WALKER
AGILE
BUSINESS
INTELLIGENCE

2. AGILE MANIFESTO
CYPRUS ‘11
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
http://www.agilemanifesto.org/ Utah, Feb 2001
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
2

3. CYPRUS ‘11
TWELVE PRINCIPLES OF AGILE SOFTWARE
• Our highest priority is to satisfy the • Working software is the primary
customer through early and measure of progress.
continuous delivery of valuable
software.
• Welcome changing requirements, • Agile processes promote sustainable
even late in development. Agile development. The sponsors,
processes harness change for the developers, and users should be able
customer's competitive advantage. to maintain a constant pace
indefinitely.
• Deliver working software frequently, • Continuous attention to technical
from a couple of weeks to a couple of excellence and good design enhances
months, with a preference to the agility.
shorter timescale.
• Business people and developers must • Simplicity--the art of maximizing the
work together daily throughout the amount of work not done--is essential.
project.
• Build projects around motivated • The best architectures, requirements,
individuals. Give them the and designs emerge from self-
environment and support they need, organizing teams.
and trust them to get the job done.
• The most efficient and effective • At regular intervals, the team reflects
method of conveying information to on how to become more effective,
and within a development team is then tunes and adjusts its behavior
face-to-face conversation. accordingly.
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
3

4. LARGE ORGANIZATIONS FORGET
CYPRUS ‘11
• Most companies start with an
innovative and entrepreneurial
people and processes that get things
done – they are inherently agile
• As organizations grow they put
structures in place that standardize
the organization but these limit
creativity, increase timescales and
reduce risk at the cost of reducing
benefits and reward
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
4

5. THE PROCESS IMPROVEMENT PARADOX
CYPRUS ‘11
• The paradox (…) is that process
improvement is good, but process
improvement programs aren't, or at least
they often aren't.
• Organizations become more and more
averse to risk as they "mature”. An
organization under the gun to demonstrate
increased CMM level is not going to go
looking for real challenge.
• This (…) entices the play-it-safe behavior of
low-risk, and therefore low-benefit, projects.
Peopleware; Tom De Marco & Timothy Lister; 1987
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
5

6. SUCCESSFUL LARGE ORGANIZATIONS CYPRUS ‘11
RECOGNIZE AND REACT
• SkunkWorks
– Lockheed Martin Advance Development Projects
– A group within an organization given a high degree
of autonomy and unhampered by bureaucracy,
tasked with working on advanced projects.
– Responsible for U-2 & Blackbird Spy Planes, Stealth
Fighter & Bomber
– Founded in June 1943 specifically to overcome the
hurdles of government and large corporations
working together
– SkunkWorks has 14 rules and practices
• Very Similar to the 12 Agile Principles
Ben Rich & Leo Janos; SkunkWorks; 1996
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
6

7. OTHER EXAMPLES
CYPRUS ‘11
• In the late 80s/early 90s Telcos led the
way in Agile BI development
– Vodafone: Mast Placement Project
– Swisscom: Geo-located SMS Marketing
• But they have been overtaken by the
search and social networks websites
that rely on rapidly creating and
consuming BI data to survive
– Examples: Facebook, Google & LinkedIn
– These companies are also creating and
using the next generation of BI tools and BI
engines
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
7

8. AGILE BI IS NOW A 2-SPEED HIGHWAY
CYPRUS ‘11
• Truly Agile BI Organizations
– Small, strong, highly skilled teams with a
strong, trusting, focused user relationship
– Project team having delegated authority and
responsibility for delivery with long term
funding
– The ability to operate outside standard
corporate procedures (e.g. procurement)
– Regular delivery of incremental improvement
• Those that describe themselves as “wanting
to be more agile”
– Willing to adopt some of the methods but
unable to break free from the corporate chains
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
8

9. CYPRUS ‘11
TECHNIQUES
BECOMING MORE AGILE
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
9

10. HOW TO BE MORE AGILE
CYPRUS ‘11
• Team Structures
• Wiki-fy Everything
• Use A Dedicated Platform
• Build Literal Staging Areas
• Throw out your ETL Tool?
• More Dynamic Reporting Tools
• Develop reports with your users
• Embrace (not so) New Technologies
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
10

11. TEAM STRUCTURES
CYPRUS ‘11
• Agile teams are generally smaller but
with broader and deeper skill sets
– These resources are more expensive
individually but cheaper collectively
– Smaller teams significantly reduce the
management and communication
overhead
– Close communications and broad skills
are more likely to generate innovative
solutions
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
11

12. WIKI-FY EVERYTHING
CYPRUS ‘11
• Projects need a consistent, persistent,
versioned knowledge store
• Use a Wiki and train your business
users how to use it
• Optimizes the ‘documentation’ set
collected in a single structured way
• Abandon office products and
SharePoint for critical documents
• Don’t store your requirements in
e-mails, IM chats and documents
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
12

13. USE A DEDICATED PLATFORM
CYPRUS ‘11
• Marketplace has a choice of BI specific,
low maintenance, low TCO appliances
– IBM Netezza
– Teradata 14 (AsterData)
– Sybase IQ
• And some high power workhorses
– Oracle Exadata
– Teradata
• There are plenty of emerging
technologies
– Curt Monash: http://www.dbms2.com/
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
13

14. BUILD LITERAL STAGING AREAS
CYPRUS ‘11
• These are 1:1 copies of your source systems
(hence the ‘Literal’)
• Do this whilst others are collecting
requirements, building data models, etc.
• Do some test reporting off the LSA with the
users – it starts the user engagement, helps
evolve the business requirements, and
develops communication
• They will be an essential source for the data
warehouse as it evolves and remove load
from the operational systems
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
14

15. THROW OUT YOUR ETL TOOL ?
CYPRUS ‘11
• ETL tools require product specific expertise
– Value add features are under-utilized
– Developers and DBAs use them as glorified
scheduling tools
• Good Source Code Control and Management
Scripting can compensate for much of the
lost benefits
• Significantly reduces the project cost and
increases the available skilled resources
• InsureTheBox/Netezza; YapiKredi/SybaseIQ;
NonDisclosure/Greenplum have major data
warehouses with no ETL product
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
15

16. MORE DYNAMIC REPORTING TOOLS
CYPRUS ‘11
• Reports on your hand held device
– Qlikview, RoamBI, PushBI, Tableau
• Use Dynamic Web Deployed Tools
– Panopticon, Tableau
• RSS Feeds
– Usable by any RSS Reader on any platform
• Requires dedicated, reactive reporting
experts that work enthusiastically with
the business users and are willing to go
the extra mile
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
16

17. DEVELOP REPORTS WITH YOUR USERS
CYPRUS ‘11
• De La Rue / Tableau
Reporting
• First 50 reports deployed
over 28 days
• Initially deployed to users
from the LSA and then
migrated to the data
marts
• Near real-time (fifteen
minutes maximum delay)
data
• Averages
– Day 5: 17 changes / 5 users
– Day 10: 1 change / 42 users
– Day 15: 0 changes / 50 users
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
17

18. EMBRACE (NOT SO) NEW TECHNOLOGY - CYPRUS ‘11
MAPREDUCE/HADOOP/NOSQL
• Software frameworks that supports data-intensive
distributed applications and enables them to work with
thousands of nodes and petabytes of data.
• Hadoop is a filesystem (HDFS) and distributed
programming framework (MapReduce) whilst a NoSQL
database consists of key-value pairs, no joins, data is
sharded and replicated, no single point of failure.
• Notable BI Use (but not the only one): Rapid loading
and processing of volume data to create and validate
data sets for onward inclusion in the data warehouse
whilst doing large scale, near real-time ‘dirty’ analysis
• Notable Users: Facebook, Twitter, LinkedIn, Yahoo!,
eBay, Amazon, Google
• Technologies in use since at least 2004
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
18

19. EMBRACE (NOT SO) NEW TECHNOLOGY – CYPRUS ‘11
COMPLEX EVENT PROCESSING (CEP)
• Complex event processing (CEP) consists of
processing many events happening across all
the layers of an organization, identifying the
most meaningful events within the event cloud,
analyzing their impact, and taking subsequent
action in real time.
• Already used for fraud and network
management but has applications in customer
service and market to fine tune customer
interactions in real time
• Notable BI Use: real-time information alongside
existing BI – rather than trying to create an
entire real-time solution
• Technologies in use since at least 1999
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
19

20. BUT REMEMBER …
CYPRUS ‘11
• Agile is all about approach and people:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
• Your management need to be willing to create:
A group within an organization given a high degree of
autonomy and unhampered by bureaucracy, tasked with
working on advanced projects.
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
20

21. REFERENCE MATERIAL
CYPRUS ‘11
• Websites
– Agile Manifesto
• 12 Principles of Agile
– SkunkWorks
• 14 Rules & Practices
– Hadoop and NoSQL Myth-busting
• Books
– Peopleware: Productive Projects & Teams
– SkunkWorks: A Personal Memoir
– Mythical Man Month: Essays on Software
Engineering
14-­‐Oct-­‐2011
©
2011
Data
Management
&
Warehousing
21

22. THANK YOU CYPRUS ‘11
AGILE
BUSINESS
INTELLIGENCE