Data Vault automation conference - all presentations

Content accountability

Ronald Damhof

Tom Breur

Organization accountability

Simone Molenaar

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

Prudenza !

To Push or To Pull, That is the question

Ronald Damhof

woensdag 21 september 11 1

!"#$%&
!"#$%&
Taken from ‘Out of the Crisis’, Dr. W.Edwards Deming

'!&()*+%',)-

!"#$%!&'(%)*+!,+(-#./('!,+(.*$$


Push characteristics

! Mass production

! Known specifications, operational definitions, standards

! Repeatable, predictable, & even better; uniform process

! Part of the system that needs statistical control

! Inventory allowed/necessary

! Supply driven

! Reliability over flexibility

Pull characteristics

! Just in time

! Demand driven

! Build to order

! Preferably no inventory

! Flexibility over Reliability


D&%4-4&*/

?@'A&*&$)/&'F*1"$()+"*-$"8#%/2
0'1(+2&/('!3*456*+7!,+(.*$$

7*89#2&$':;"%)<=
.)/)'0'1#*%+"*'2&$34%&

? ? ? ? ?
>@'7*$4%B')*8'%<&)*2&'8)/)

> > > > >
5@'D&,42/&$'0'!/)*8)$84E&
5 5 5 5 5

6@'A&/'/B&'$)C':#*%#/='8)/) 6 6 6 6 6
8*'*+5.!90!,+(.*$!:;*'%+&4<

.)/)'2"#$%&2

0'1(+2&/('!3*456*+7!=+(.*$


E5*BF!;(2,('*'%!65*B

6 5 > ?
;(2,&'7!AAA!-&%&!B&+*)(#$*!C!9#$5'*$$!0'%*445D*'.*!!3(2&5'

?

>
HF')--2
?*,(+%$

5
!"#$%&'2/"$&

6PJ'5P

H#24*&22'O4&CJ'
!"#$%&2

.)/)'1&&82
6
HF'I--2
>'&47$5$

7*/&$-$42&'
.)/)'G)$&B"#2&

HF'I--2
>-@)(.

.)/)J'KGB)/L M#*%+"*J'KN"CL KGB&$&LJ'KGB"(L
7Q/&$*)<
'2"#$%&2

!"#$%&'/"' !"#$%&2/"$&' !"#$%&2/"$&'
7.G':.O=
-$"8#%/ /"'-$"8#%/ /"'HO

>-&,%&H4*

G#$%&5'&H4*

;(2,45&'%

3*.(#,4*-

JK*./6*

G%&'-&+-5I*-

;*'%+&45I*-


E5*BF!;(2,('*'%!65*B

6 5 > ?
;(2,&'7!AAA!-&%&!B&+*)(#$*!C!9#$5'*$$!0'%*445D*'.*!!3(2&5'

?

>
HF')--2
?*,(+%$

5
!"#$%&'2/"$&

6PJ'5P

H#24*&22'O4&CJ'
!"#$%&2

.)/)'1&&82
6
HF'I--2
>'&47$5$

7*/&$-$42&'
.)/)'G)$&B"#2&

HF'I--2
>-@)(.

.)/)J'KGB)/L M#*%+"*J'KN"CL KGB&$&LJ'KGB"(L
7Q/&$*)<
'2"#$%&2


I8(4*42/$)+3&'-$"%&22 F*1"$()+"*'.&<43&$S'R$"%&22 .&%424"*9'0'%"*/$"<

8*'*+&%*C! .)/)'0'F*1"$()+"*'$&%4-4&*/2
35$%+5H#%*
J'+5.)
?*D5$%*+!C!
G%&'-&+-5I*
>L&5'
R$"%&2

R.UI

U"(-<4)*%&'$&-"$+*,
F*1"$()+"*'
!"#$ -$"8#%/2
D42V'W)*),&(&*/
!"#$
!S2/&(2 .O'T)2&8 !"%%
:4*/&$*)<'0 .)/)' R&$1"$()*%&'
&Q/&$*)<= G)$&B"#2& W)*),&(&*/
H#24*&22 !#--<S'%B)4*'
!/),4*, $#<&2 "-+(4E)+"*
.)/)'-$"8#%/2
M$)#8'8&/&%+"*

W)$V&/'T)2V&/'
)*)<S242

U"*/$"<'X'W&/)8)/)

Remember the Push characteristics
! Mass production Data Vault

! Known specifications, operational definitions, standards Data Vault

! Repeatable, predictable, & even better; uniform process Data Vault

! Part of the system that needs statistical control Data Vault

! Inventory allowed/necessary Data Vault

! Mainly supply driven Data Vault

! Reliability over flexibility Data Vault

Automation of a Data Vault production system is just common sense


WS'R"O'
IT"#/':.)/)'O)#</=')#/"()+"*'Y""<4*,
! A&*&$)+"*'42')*')48J'*"/')',")<'4*'4/2&<1
– '."'*"/')%%"("8)/&'/B&'-$4*%4-<&2'/"'Z/'/B&'/""<@@@@
– ';""V'1"$'8&%"#-<4*,
! Y$#<S'#*8&$2/)*8'/B&'(&%B)*4%2'9'B)*8%$)['4/'Z$2/
– 'F*3&2/'4*'-$"-&$'&8#%)+"*')*8'<&)$*4*,
– 'F*3&2/'4*'K,&]*,'$&)8SL'+(&
– 'F*3"<3&'S"#$'K%#2/"(&$2L'1$"('/B&'2/)$/
! R"UJ'R"UJ'R"U
! .&<43&$J'.&<43&$J'.&<43&$


YS-&'6'9'U<)224%'.)/)'O)#</
H#24*&22'
Y$)*2)%+"*'
!S2/&('
!/),4*,'
.)/)'O)#</ .)/)2&/2
b#/
H#24*&22'
Y$)*2)%+"*' 8*'*+5.!9#$5'*$$!?#4*$
!S2/&('
D#<&'O)#</

!/$#%/#$&'/$)*21"$()+"* H#24*&22'$#<&'&Q&%#+"*
N#T'^'T#24*&22'V&S2 !/$#%/#$&')*8'3)<#&'/$)*21"$()+"*

I8)-/)T<& !#2/)4*)T<& U"(-<4)*/ .&%"#-<&8 7_&%+3&*&22 !/)*8)$84E&8 U&*/$)<4E&8

` `

YS-&'5'9'!"#$%&'.)/)'O)#</
H#24*&22'
Y$)*2)%+"*' !/),4*,'O)#</
!S2/&('
H#24*&22' .)/)'W)$/2
.)/)'O)#</
H#24*&22'
Y$)*2)%+"*' !/),4*,'O)#</
!S2/&('

!/$#%/#$&'/$)*21"$()+"* H#24*&22'$#<&'&Q&%#+"* !/$#%/#$&'/$)*21"$()+"*
a"'4*/&,$)+"*J'N#T^2#$$",)/&'V&S2 F*/&,$)+"*
R&$242+*,'2/),4*,'4*'.O'1"$()/ .O'("8&<<&8'

I8)-/)T<& !#2/)4*)T<& U"(-<4)*/ .&%"#-<&8 7_&%+3&*&22 !/)*8)$84E&8 U&*/$)<4E&8

` ` `

!"#$%&

!"#$%&

'6ccd'!&()*+%',)-

!"#$%& !/),4*,'.O
H#24*&22'.O
!"#$%& !/),4*,'.O

6ccd'!&()*+%',)-

!+<<'/B&'2"#$%&

F*/&,$)+"*J'%<&)*24*,J'%"*2"<48)+"*
H#24*&22'$#<&'&Q&%#+"*'#-2/$&)('``
.O'("8&<<&8'


!"#$%&

!"#$%&

'6ccd'!&()*+%',)-

!"#$%& !"#$%& !/),4*,'.O
H#24*&
.)/)'
G)$&B"#2&
!"#$%& !"#$%& !/),4*,'.O 22'.O

6ccd'!&()*+%',)-

!+<<'/B&'2"#$%&

F*/&,$)+"*J'%<&)*24*,J'%"*2"<48)+"*
H#24*&22'$#<&'&Q&%#+"*'#-2/$&)('``
.O'("8&<<&8'


W&/)("8&<'8$43&*')#/"()+"*
9 W"8&<2':-$"%&22J'$#<&2')*8'8)/)='8&/&$(4*&'/B&'(&/)8)/)J'/B&'(&/)8)/)'8&/&$(4*&2'/B&')#/"()+"*')$+1)%/2
9 I4('42'/"'T&'6ccd'8&%<)$)+3&
9 F/'%)*'*"/'T&',&*&$)/&8')<<J'2-&%4Z%'/)4<"$&8'(&/)8)/)'C4<<'$&()4*'*&%&22)$S

W&/)8)/)'8$43&*')#/"()+"*
9'F*-#/2e'!"#$%&'("8&<:2=J'/)$,&/'("8&<J'Y&(-<)/&'.&24,*J'a)(4*,'%"*3&*+"*2
9'I83)*%&8'4*-#/2e'a"$()<4E)+"*'-$&1&$&*%&2J'b*/"<",4&2

Y)V&*'1$"('.)*';4*2/&8/L2'T<",'-"2/e'Bf-eXX8)*<4*2/&8/@%"(X8)/)3)#</%)/X%"8&9,&*&$)+"*91"$98)/)93)#</9*"/9)29&)2S9)29S"#9/B4*VX

.)/)'O)#</'
4(-<&(&*/)+"*2 Y&(-<)/&'8$43&*')#/"()+"*
9 F*'/B&'("2/'T)24%'1"$(2g'8"%#(&*/)+"*''9'8&2%$4T4*,')'-)f&$*
9 W"$&')83)*%&8g',&*&$)+*,'hW;'%"8&'1"$'5*8',&*@'7Y;'/""<4*,
9 OT'9'Bf-eXXCCC@,$#*82)/E<4%B94/@*<XT49/""<29/&(-<)/"$@B/(<


I#/"()+"*'/S-"<",S
• YB"2&'/B)/'2#--"$/'2-&%4&2'i6':T#4<84*,')'!"#$%&'O)#</=
– Y&(-<)/&'8$43&*'"$'W&/)8)/)'8$43&*
– b[&*',&*&$)/&2'/B&'("8&<')*8'/B&'<",42+%2
• YB"2&'/B)/'2#--"$/'2-&%4&2'i5':T#4<84*,')'U<)224%'O)#</=
– Y&(-<)/&'8$43&*'"$'W&/)8)/)'8$43&*
– A&*&$)/&':(&/)8)/)'"1='/B&'<",42+%2
– W"8&<4*,'$&()4*2')'%$)['j'F.7aYFMk'YN7'Hl!Fa7!!'m7k!
• YB"2&'/B)/',"'T&S"*8'
– W&/)("8&<'8$43&*
– H)2&8'"*'/B&'T#24*&22'-$"%&22J'/B&'$#<&2')*8'/B&'8)/)
– YB&'8)/)("8&<':.OJ'IWJ'@@='42')'%"*2&n#&*%&'"1'/B&'-$"%&22
– !#--"$/'1"$'I;W'%B)$)%/&$42+%2


YB)*V'k"#
&'#()*+,-%.)&()&-/$+0

H<", Bf-eXX-$#8&*E)@/S-&-)8@%"(X
Bf-eXXCCC@T9&S&9*&/C"$V@%"(XT<",2X8)(B"1X'
;4*V&84* Bf-eXX*<@<4*V&84*@%"(X4*X$"*)<88)(B"1

7()4< $"*)<8@8)(B"1o-$#8&*E)@*<

YC4f&$ D"*)<8.)(B"1

!VS-& D"*)<8@.)(B"1

W"T4<& p>6:c=q'5qr'qs'6t?

b/B&$2 F*1"$()+"*'u#)<4/S'U&$+Z&8'R$"1&224"*)<':FuUR=
.)/)'O)#</'U&$+Z&8'A$)*8'W)2/&$
U&$+Z&8'!%$#('W)2/&$
W&(T&$'"1'/B&'H"#<8&$'HF'H$)4*'Y$#2/':iHHHY=

*+,-%.)&-/$+0)42')*'4*8&-&*8&*/'-$)%++"*&$'4*'/B&'Z&<8'"1'8)/)'()*),&(&*/')*8'8&%424"*'2#--"$/@'A$)8#)/&8'4*'6rrv'4*'
/B&'2/#8S'"1'7%"*"(4%2@'!4*%&'6rrv'B&'C"$V&8')2')'-$)%++"*&$'4*/"'/B&'Z&<8'"1'F*1"$()+"*'W)*),&(&*/'C4/B')'1"%#2'"*'
8&%424"*'2#--"$/')*8'8)/)'()*),&(&*/J'/$S4*,'B)$8'/"'&*B)*%&'/B&'$4,"$')*8'$&<&3)*%&'4*'/B&2&'Z&<82'TS'%"(T4*4*,'2%4&*+Z%'
$&2&)$%B'C4/B'/B&'&3&$S8)S'%B)<<&*,&2'"1'/B&'-$)%++"*&$@'D"*)<8'42'()4*<S'B4$&8'TS'%#2/"(&$2'4*'/B&'$"<&'"1'T#24*&22XFY'
)$%B4/&%/J')#84/"$J'%")%B'0'/$)4*&$@'N&'T<",2'"*'H97S&9a&/C"$V@%"(')2'C&<<')2'B42'"C*'T<",J'42')'(&(T&$'"1'/B&'-$&2+,4"#2'
HHHYJ'C$"/&'2&3&$)<')$+%<&2'$&,)$84*,'8&%424"*'2#--"$/')$%B4/&%/#$&2')*8'42')'$&2&)$%B&$'4*'/B&'Z&<8'"1'F*1"$()+"*'
W)*),&(&*/@'

I</B"#,B'D"*)<8'<4V&2'/"'C"$V'C4/B'/B&"$&+%)<',$"#*8&8'$&2&)$%B')*8'-$"3&*'-$)%+%&2J'D"*)<8'42'*"/')'wCB4/&'-)-&$w'
)$%B4/&%/g'-#/'S"#$'("*&S'CB&$&'S"#$'("#/B'42J'42'B42'("f"@'N&'<4V&2'/"'2&&')$%B4/&%/#$&2'w<43&w'4*'&*/&$-$42&2J'*"/'x#2/'C$4/&'
)T"#/'4/@'F*'("2/'"$,)*4E)+"*2'B42'$"<&'&Q/&*82')$%B4/&%/#$&'"[&*@'F*'/$#&<S'),4<&'2-4$4/'/B&'$"<&2'B&'-<)S2'8&-&*8'"*'/B&'
%"*/&Q/'"1'/B&'%<4&*/g'B&'%)*'T&')'(4224"*)$S':2&<<4*,'/B&'3)<#&=J')'-$"x&%/'()*),&$':,&]*,'4/'8"*&=J')'2%$#('()2/&$':$&("34*,'
4(-&84(&*/2=J'2-&%4)<42/':&8#%)+*,'B)$8C)$&'-&&-2J'8)/)')$%B4/&%/2J'8)/)'<",42+%2'&/%@='"$')'<&)8&$@


!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

Qosqo !

Introducing QUIPU

October 2011

Jeroen Klep
QOSQO

+31 6 2953 2342
Jeroen.Klep@QOSQO.nl

open source data warehousing

Agenda

New
Background Architecture
developments

What is a quipu?

AD 1300 - 1600

•  BI strategy development •  Maintenance & support
•  Information analysis •  Data vault technology
•  (E)DW architecture •  Quipu development
•  Project management
•  Adapttm training

QUIPU: Open Source DW generation
•  Open Source Data Warehouse Generation System, based on
Data Vault principles
•  First public release July 1st 2010
•  QOSQO takes a leading role in continuous development and
support

Fast implementation
of DV based EDWH

Removal of
repetitive tasks

Reduction of risk
of modeling errors

Source:

QUIPU - Key IT benefits
•  Automated data warehouse data model design
and implementation
•  Fully repository based metadata driven data
model and load code generation
•  Supports most common database platforms using
ANSI-SQL over JDBC
–  Template based platform support
•  Integration with ETL and scheduling tools
•  Lower total cost of ownership using open source
licensing model

Characteristics

Design time

Run time
Source(s) Target DW

Quipu basic architecture: ‘classic’

Business model
•  Development of new functionality
–  Paid customer assignments
–  QOSQO roadmap priority
•  Support
–  Quick start consultancy
–  Proof of Concepts
–  Flexible support model
•  On site
•  Remote
•  Training
•  Quipu Model Manager
–  Paid software
–  Hosting

Quipu products
Community Model
Edition Manager Powered by

New DWH Management &
developments Maintenance

-  Open source -  Closed source -  Embedded in BI
-  Generate models -  Manage models solutions
-  Single user -  Delta changes
-  Continuous -  Multi-user -  CaseWise Modeler
developments and solution
improvements

-  New: Data mart -  New
New product -  New solutions
DM product
generation roadmap
generation roadmap
assistance

Data Mart assistance
•  In cooperation with BinckBank
•  Logical layer on top of DataVault
•  Basic Starschema or snowflake generation

Quipu Model Manager
•  Version control of data models
•  Multiple users, projects, versions
•  Quipu Community Edition as client
•  Check in / Check out
•  Migration of run time DW data
•  Central repository of models and code
Quipu
CE
Quipu
Quipu MM
CE

Quipu
CE

•  Download and evaluate Quipu (it’s free!)

•  Share your experience and feature wishes

•  Hire us

More info
•  www.datawarehousemanagement.org
•  @OS_Quipu
•  Demo Youtube channel: ‘osquipu’
•  Sourceforge: https://sourceforge.net/projects/quipu/

•  www.QOSQO.nl

QOSQO, the DataVault Our sister company
Karel Doormanlaan 1b
specialist, is the leading Nippur assists in
5688 BP OIRSCHOT company behind Quipu executing business
The Netherlands intelligence projects
E: info@QOSQO.nl
T: +31 ( 0499 ) 577 562 www.QOSQO.nl www.nippur.nl
F: +31 ( 0499 ) 577 059

open source data warehousing

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
Infosupport

26-09-2011

!"#$%&'#()'$(#%

*"#$+,-./0+)110#'230*%%

45(.6"%
•  7./0%&)110#'%
•  8.6("90)#%:7%
•  4#3;-'(3')#(%
•  !('"%6"'"%6#-9(.%6(9(<01*(.'%
•  =(<-9(#"><(+%
•  ?0.3<)+-0.%

1

26-09-2011

!"#$%&'(($)*%
•  +!%,$-(.*."/.%,."*.)%
•  0.*1$2$3$45.6%5"/3'25"4%75-8933:%!"-$":%;9*9%
<9'3*%
•  =)95"5"4%,."*.)%
•  +!%;.>.3$(-."*%?39@$)-A%B"2.9>$')%+!%

!"#$%&'(($)*%+!%;.>.3$(-."*%?39@$)-%
•  B"2.9>$')%+!%

2

26-09-2011

!"#$%&'(#)*+&%)&,#%'

9/:.'-,77.#&'5%&"'0"&"'6%7.4+&.#8'

6%7.#&4'
0"&"' 3,4+/%44' 0"&"'
-.,#)%' -&"$+/$'
1",2&' 1",2&' 5"#&'
(/"284+4'
Source Back End Front End Reporting &
Systems Systems Systems Analysis

9/:.'-,77.#&';./&#.2'</=+#./>%/&'

!"#$%&'(#)*+&%)&,#%'

9/:.'-,77.#&'5%&"'0"&"'6%7.4+&.#8'

6%7.#&4'
0"&"' 3,4+/%44' 0"&"'
-.,#)%' -&"$+/$'
1",2&' 1",2&' 5"#&'
(/"284+4'

9/:.'-,77.#&';./&#.2'</=+#./>%/&'

3

26-09-2011

!"#$%&$#$%&'()"*%&")"+,-."*#%

!"#$%&$#$% &"+()"'$8+"%
/,0'1"%!,2"+%
4"-,5(#,'6%

3.-,'#% !$*$7"% 9"*"'$#"%

:'"$;*7%$%&$#$%<$0+#%!,2"+%
=  /,0'1"%!,2"+% =  >+7,'(#?.%

3*7'"2("*#5% 4"1(-"%

=  &$#$%<$0+#% =  :,*A70'$;,*5%
!,2"+%
9"*"'$#,'%

@,,+5% &"1,'$;,*%

4

26-09-2011

!"#"$%"&'#$()*+'$,+-+."/)-$

•  Configurations

Generator

•  Source Model •  Staging Model
•  Data Vault Model
•  Mappings
(+#"$!"#"$
0+1)23#).4$

!+'35+."6'+2$
•  789$7+.5+.$!"#"6"2+$()*+':2;$
–  <"6'+2$:=&6>$93-?>$7"#+''3#+;$
–  %3+@2$:"62#."A/)-$'"4+.;$
–  B)-2#."3-#2$
–  C-*+D+2$
•  E<9$
–  77C7$1"A?"F+2$
•  G"2+*$)-$6+2#$1."A/A+2$"-*$F&3*+'3-+2$
–  H&*3#$#."3'$

5

26-09-2011

!"#$%"&'(#)*%"#+,#$'-.*&/*#0'-&"12'
•  344&*156"#'7'-5$18'9#*$'
•  :$5#/5%/*;,/'&"00*#0'
•  :$5#/5%/*;,/',<1,46"#85#/&*#0'
•  !"#=0.%56"#>'
•  ?#1%,+,#$>'7'@"A1".#$>'B"%'5./*$5C*&*$D'
•  ?#1&./,>'%,4"%$>'B"%'+5*#$,#5#1,'

E%"#$'(#/':D>$,+'
?#B"':.44"%$'G,$5'F5$5'@,4">*$"%D'

@,4"%$>'
F5$5' -.>*#,>>' F5$5'
:".%1,' :$50*#0'
H5.&$' H5.&$' G5%$'
3#5&D>*>'

?#B"':.44"%$'!"#$%"&'(#)*%"#+,#$'

•  F5$5'G5%$'7'-.>*#,>>'H5.&$'
–  !5#'C,',*$8,%')*%$.5&'"%'48D>*15&'
•  !.C,>'I'@,4"%$>'
–  !%,5$,/'+5#.5&&D'

6

26-09-2011

!"#$%&'("#)
•  *#+,-./+,0)'"%&1"#)
–  2/+/3/',)
–  456)
–  !"#+."%)4#7(."#8,#+)
•  !"'+),9,$17,)
•  :,+/)0/+/)0.(7,#)
•  ;.,0($+/3%,)<&/%(+=)
•  :".,)18,)+")>"$&')"#)3&'(#,'').,<&(.,8,#+')

7

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
Centennium

!"#$

!%&'%&&()*$"+'+,+-%./)0%$#%'./1/2/34$
5'-)6')-%$ $#/1%2$ $7%&%-+'%$ $8&/,2%13%$9+-'&%-0.(:$

!
"#$%!&#'()*(!
+(,-$(*!.,*/0'!
12,-3*#!45!6788!

1

;3%&1+$
9*(,*(($:0!;<!*=>*#,$)*?:$)!
9@A!
!"#$%"$#&! ! B*C*#*(2*!+#2?$,*2,:#*!
'()&*! ! ! @','!D':/,!
+&,&#-"&! ! E*0>/',*!;')*F!@*G*/->0*(,!
.,(/*&)0&11
2-#",&#3456! ! H(-I/*FJ*!,#'()C*#5!2-'2?$(J!
9@A!>#-2*))!
!
!
2

!"#$"##%&'()*("+,"-$%."/&%.(
!"#$%$"#$"&''
($'%)*+,#$'*-)'.-/&*0$)/'1,&2'&2$'),32&'/4,55/6'7&'&2$'
),32&'&,0$'7"#'&2$'),32&'178'
9*"/-5&7".86'7#:,"&$),0'/-%%*)&6'%)*;$.&',0%5$0$"&7&,*"'
7"#'&)7,","3'/$)+,.$/6'<-&'%),07),58'7.&/'7/'7'=!>?(@'
4"*15$#3$'%7)&"$)'A*)',&/'.5,$"&/'
($'&74$'*)'/27)$')$/%*"/,<,5,&8'A*)'&2$'$B$.-&,*"'7"#'
07"73$0$"&'*A'=!'7"#'?(@'%)*;$.&/'7"#'%)*+,#$'/$.*"#'
*)'&2,)#'5,"$'/-%%*)&'A*)'$B,/&,"3',0%5$0$"&7&,*"/'
($'*AA$)'7"'$B&$"/,+$'%*)&A*5,*'*A'.*-)/$/'7"#'&)7,","3'
/$)+,.$/''
($'%)*+,#$'*-)'.-/&*0$)/'1,&2'&2$'4"*15$#3$'7"#'
%)7.&,.75',"/,32&/')$C-,)$#'&*'<$D.*0$E'/$5A:/-AA,.,$"&',"'
07,"&7,","3'7"#'$B%7"#,"3'&2$,)'=!:$"+,)*"0$"&/'
'
111F.$"&$"",-0F"5'
3 '

!"#$"##%&'(01$121-"/3&."(4"$/3536378(
G'0$&2*#*5*386',".5-#,"3'?(@:&**5/'
=-,5#'*)'0,3)7&$'#7&717)$2*-/$/'A7/&6'1,&2'2,32'C-75,&8'7"#'
5*1'.*/&'
G-&*07&,.'3$"$)7&,*"'*A'#7&717)$2*-/$'<7/$#'*"'#$/.),%&,+$'
0$&7#7&7'
9?H',".5-#$/I'
J$0%57&$'=7/$#'?$+$5*%0$"&'
=$/&'%)7.&,.$/'
K-75,&8'.*"&)*5'0$.27",/0'
L"*15$#3$'%7)&"$)/2,%'
=$/&'M)7.&,.$/I'?7&7'N7-5&6'L,0<7556'O$A$)$".$'G).2,&$.&-)$'
K-75,&8'.*"&)*5'0$.27",/0I'$B&$"/,+$'.2$.45,/&/'7"#'
#*.-0$"&7&,*"'
L"*15$#3$'&)7"/A$)'<8'&)7,","36'.$)&,A,.7&,*"'7"#'5$7)","3'*":
&2$:;*<'
'
4 '

!"#$"##%&'()*$*+*,"-.&/"(0"$-.1.2.34(
Knowledge Partnership

5
Structuring Modelling Generating

5#.+2"13"(6*,$#",/-%6(
!"#$%&&'($)*+,--"./0123&456*#7#.(&8,+/"9(.+&
:.#01012&"-/0"1+& &
;<&#1=&5>?&),1=#9(1/#$+&
456&=(+021(.&@&:;5&=(A($"-(.&/.#01012& !
5#/#&B#,$/&),1=#9(1/#$+&& !
5#/#&B#,$/&8(./0)08#/0"1&C!(1(+((&D8#=(9EF& !
509(1+0"1#$&9"=($$012&
"#$%!&'(#!#'!)))*+&,#-.%&/&'0%'*'.!
G1*/H(*I"J&8"#8H0123&$(#.1012&JE&="012!
4(1/(110,9&+,--"./+& &8,+/"9(.+&JE&&
C9#1#2(9(1/F&8"1+,$/#18E3&#++(++9(1/+3&-."I(8/+3&
/.#01012&#1=&+",.8012&
6

!"#"$"%&"'($&)*+"&+,$"-''
.+$,&+,$*%/'

7

0"1234+"'546"7'8"9"3:21"%+-'
;"%"$4+*%/'
'
!"#$%&'&()*&+$),,$-!.$)'/$012&3*+$40($
5&%6+*()*60'$,)7&($
8(&+&'*)*60'$,)7&($
5&90+6*0(7$)'/$+3(69*+$)(&$4(&&$04$3:)(%&$
;(&)*6'%$*:&$+*)%6'%$,)7&($6+$'0*$9)(*$04$!"#$1<*$
3)'$1&$)<*0=)*&/$9&($3<+*0=&($
$

8

!"#$%&'&(&)*)$

<*=1)7'1->$3<?94$

5'&67,6$ +*,'-&.$#&'&$ 92(.7:&'71,$
35!84$ /&-*012)*$3+#/4$ 39;"4$

9

!"#$-*=1)7'1->$

<*=1)7'1->$3<?94$

5'&67,6$ +*,'-&.$#&'&$ 92(.7:&'71,$
35!84$ /&-*012)*$3+#/4$ 39;"4$

10

-3D,

J4K, GHI, DEF,

!"#$%&'#()&%*+,-.+/01'2&%'/*+
)'3+%(4)02+567+
!"#$%&'#()&%*+,,
-./&0%1(%&'#,'2,()),345,61'$.00.0,
7#0&/8%,&#,9(%(,)'/&0%&$0,
:&0%'1&$,61'2&).,'2,)'(9&#/,61'$.00.0,
,
;'()+,
41($.(<&)&%*,,
=(&#%.#(#$.+,)'$()&0(%&'#,(#9,&9.#%&2&$(%&'#,'2,()),
9(%(>%1(#0($%&'#0,?@1'#/,(#9,$'11.$%A,
5&#B(/.,%',0'"1$.,0*0%.C0,

11

-3D,

J4K, GHI, DEF,

!"#$%&'#()&%*+,-.+/01'2&%'/*+
%081)(%0+%(4)02+
!"#$%&'#()&%*+,,
G.#%1(),0%'1(/.,'2,%.C6)(%.,0$1&6%0,2'1,/.#.1(%&#/,
%(<).0,(#9,345,61'$.9"1.0,
3(0*,61'C'%&'#,%','%8.1,.#L&1'#C.#%0,?M4KDA,
41(#06(1(#%,
,
;'()+,,
N.10&'#,C(#(/.C.#%,'2,%.C6)(%.0,
M<O.$%,?1.PA$1.(%&'#,(#9,C(&#%.#(#$.,

12

<8@,

A9H, 6-G, @EF,

!"#$%&'#()&%*+,-.+/01'2&%'/*+
30%(+4(%(+%(5)02+
!"#$%&'#()&%*+,,
-./&#.0,1.2&0%1(%&'#,(#3,4"5)&$(%&'#,)(*.1,
6'#%(&#0,'57.$%,(#3,89:,3./&#&%&'#0,
6'#%(&#0,3(%(,)'2&0%&$0,
,
;'()+,
;.#.1(%&#2+,
<.2&0%1(%&'#,)(*.1,=-(%(,>(")%,%(5).0?,
@"5)&$(%&'#,)(*.1,=A%(1,0$B.C.0?,,
89:,41'$.00.0, 0%'1.3, ,
D57.$%,$1.(%&'#,(#3,C(&#%.#(#$.,
13

<8@,

A9H, 6-G, @EF,

!"#$%&'#()&%*+,-.+/01'2&%'/*+
30%(+4(%(+%(5)02+
9(5)., <.4'0I9(5). ,
6'#%(&#0,'57.$%,#(C.0,/'1,,
A%(2&#2,,
<.2&0%1(%&'#,
@"5)&$(%&'#,
9(5).,&0,/&)).3,5*,(#,(44)&$(%&'#,'1,8J$.),0B..%,
,
, <.4'0IK(44&#2 ,
6'#%(&#0,C(44&#2,'/,0%(2&#2L,1.2&0%1(%&'#,(#3,
41.0.#%(%&'#,
9(5).,&0,/&)).3,5*,(#,(44)&$(%&'#,'1,8J$.),0B..%,
14

!"#$%&'(&$)*+,(-"'+

!"=)%$&)'-+3!>84+

5&(#$*#+ ."*&'(,+/(&(+ 829,$:(&$)*+
35674+ 0('"1)2%"+3./04+ 38;<4+

15

G7B!

@EF! ./0! BCD!

!"#$%&'(&$)*+,(-"'+
!
"#$#%&'#(!&))!*+,!'&,)#(!-$!'*#!./01!&(!2#3-$#2!-$!
%#45(-'5%6!
78#%6!*+,!95$'&-$(!'*#!95)+:$(;!!
!"#$%&!'(&)#*+),-#.(/*0&1!2345*+)-#'+1(56(("5"17-#
'+1(58&02#+2#'+1(5(9"!15!"#
<&%-&,)#(!3%5:!%#45(-'5%6!&%#!&44)-#2!
=$9#!'*#!*+,(!&%#!>#$#%&'#21!'*#!%#45(-'5%6!-(!
+42&'#2!
?+,(!>#$#%&'#2!&995%2-$>!'5!/&'&!<&+)'!@'&$2&%2(A!

16

G7B!

@EF! ./0! BCD!

!"#$%&'(&$)*+,(-"'+
!
"#$#%&'#(!&))!*+,!'&,)#(!-$!'*#!./01!&(!2#3-$#2!-$!
%#45(-'5%6!
78#%6!*+,!95$'&-$(!'*#!95)+:$(;!!
!"#$%&!'(&)#*+),-#.(/*0&1!2345*+)-#'+1(56(("5"17-#
'+1(58&02#+2#'+1(5(9"!15!"#
<&%-&,)#(!3%5:!%#45(-'5%6!&%#!&44)-#2!
=$9#!'*#!*+,(!&%#!>#$#%&'#21!'*#!%#45(-'5%6!-(!
+42&'#2!
?+,(!>#$#%&'#2!&995%2-$>!'5!/&'&!<&+)'!@'&$2&%2(A!

17

G7B!

@EF! ./0! BCD!

!"#$%&'(&$)*+,(-"'+
@9%-4'! '&,#)! !
"#$#%&'#(!&))!(&'#))-'#!'&,)#(!-$!'*#!./01!&(!2#3-$#2!
-$!%#45(-'5%6!
78#%6!(&'#))-'#!95$'&-$(!'*#!'*#!95)+:$(;!!
:5!"#$/0&+!32#*+),-#'+1(56(("5"17-#
'+1(56(("5+!2"5"17-#'+1(58&02#+2#'+1(5(9"!15!"#
<&%-&,)#(!3%5:!%#45(-'5%6!&%#!&44)-#2!
=$9#!'*#!*+,(!&%#!>#$#%&'#21!'*#!%#45(-'5%6!-(!
+42&'#2!
@&'(!>#$#%&'#2!&995%2-$>!'5!/&'&!<&+)'!@'&$2&%2(A!

18

F8A'

!DE' 012' ABC'

!"#$%&'(&$)*+,(-"'+
!"#$%&' &()*+' '
,*-*#(&*.'(++'.(&*++$&*'&()+*.'$-'&/*'0123'(.'4*5$-*4'
$-'#*%6.$&6#7'
89*#7'.(&*++$&*'"6-&($-.'&/*'&/*'"6+:;-.<''
!"#$%&'()*#+,%-*./0%1*23"433$"$250%
1*23"433$"*#,$"$250%1*23"6)(,%*,%1*23"37$#2"#$%
=(#$()+*.'5#6;'#*%6.$&6#7'(#*'(%%+$*4'
>-"*'&/*'/:).'(#*'?*-*#(&*43'&/*'#*%6.$&6#7'$.'
:%4(&*4'
!(&.'?*-*#(&*4'(""6#4$-?'&6'1(&('=(:+&'!&(-4(#4.@'

19

F8A'

!DE' 012' ABC'

!"#$%&'(&$)*+,(-"'+
!"#$%&' &()+*'+$-G '
,*-*#(&*.'(++'+$-G'&()+*.'$-'&/*'0123'(.'4*5$-*4'$-'
#*%6.$&6#7'
89*#7'+$-G'$.'"6--*"&*4'&6'&H6'6#';6#*'/:).%
=(#$()+*.'5#6;'#*%6.$&6#7'(#*'(%%+$*4'
>-"*'&/*'+$-G.'(#*'?*-*#(&*43'&/*'#*%6.$&6#7'$.'
:%4(&*4'
I$-G.'?*-*#(&*4'(""6#4$-?'&6'1(&('=(:+&'!&(-4(#4.@'
'

20

H9C'

!FG' 123' CDE'

!"#$%&'(&$)*+,(-"'+
!"#$%&' &()*+'*$,- '
.+,+#(&+/'(**'*$,-'&()*+/'$,'&0+'1234'(/'5+6$,+5'$,'
#+%7/$&7#8'
9:+#8'*$,-'$/'"7,,+"&+5'&7'&;7'7#'<7#+'0=)/!
>(#$()*+/'6#7<'#+%7/$&7#8'(#+'(%%*$+5'
?,"+'&0+'*$,-/'(#+'@+,+#(&+54'&0+'#+%7/$&7#8'$/'
=%5(&+5'
A$,-/'@+,+#(&+5'(""7#5$,@'&7'2(&('>(=*&'!&(,5(#5/B'
'

21

H9C'

!FG' 123' CDE'

!"#$%&'(&$)*+,(-"'+
!"#$%&' '
.+,+#(&+/'(**'*$,-'/(&+**$&+'&()*+/'$,'&0+'1234'(/'
5+6$,+5'$,'#+%7/$&7#8'
>(#$()*+/'6#7<'#+%7/$&7#8'(#+'(%%*$+5'
?,"+'&0+'*$,-'/(&+**$&+/'(#+'@+,+#(&+54'&0+'#+%7/$&7#8'
$/'=%5(&+5'
A$,-'/(&+**$&+/'(#+'@+,+#(&+5'(""7#5$,@'&7'2(&('>(=*&'
!&(,5(#5/B'

22

EF@'

!CD' 123' @AB'

!"#$%&'(&$)*+,(-"'+
!"#$%&' '
()*)#+&),'+--'-$*.',+&)--$&)'&+/-),'$*'&0)'1234'+,'
5)6$*)5'$*'#)%7,$&7#8'
9+#$+/-),'6#7:'#)%7,$&7#8'+#)'+%%-$)5'
;*")'&0)'-$*.',+&)--$&),'+#)'<)*)#+&)54'&0)'#)%7,$&7#8'
$,'=%5+&)5'
>$*.',+&)--$&),'+#)'<)*)#+&)5'+""7#5$*<'&7'2+&+'9+=-&'
!&+*5+#5,?'

23

./0,$1(&$)*+,(-"'+

!"=)%$&)'-+6!>.7+

8&(#$*#+ 2"*&'(,+3(&(+ ./0,$1(&$)*+
689:7+ 4('"5)/%"+62347+ 6.;<7+

24

FGC$

?>E$ 012$ CD6$

!"#$%&'(%)*+$',-.+
/%0-*1%)*1+'*/+2'&(1+
!"#$%&'($)&#$#*+",-'($.+%/$012$
0%&.%+/3$-%$ $3-)+$3,4"/"$3-)&#)+#3$
5+$)&($%-4"+$.%+/)-$$
673*&"33$+7'"3$,)&$8"$)99'*"#$
07++"&-'($73*&:$;*"<3$
673*&"33$+7'"$"#*-%+$*&$&"=-$+"'")3"$
>(9"$?01$@A$@@A$"-,B$
0%&.%+/"#$#*/"&3*%&3$<4"&$&""#"#$
$
$
25

3456+7.)&-11+
@&,+"/"&-)'$)99+%),4$
>*/"8%="3$%.$HIJ$<""K3$
$
$
$

26

!"#$%&'()*+,%)-*./0/-&%
90% Centennium 70% Customer 100% Customer
100% Centennium 30% Customer 40% Centennium 10% Centennium

!"#$%&'& !"#$%&(& !"#$%&)& !"#$%&*&

+,-.%/%,0&

1-8-2011 31-12-2011
!"#1 "+&+%3+4,&%
7.+)-)-2%+-8%!6+*9)-2%6-1&9/1:6;% <4((6.&)-2%*4=&60/.%
&.+)-)-2% !/.&)5)*+&)6-%

Typical increment ranges from 2 to 6 months
Centennium role changes from LEAD to FOLLOW
Customer is fully CDM-aware at the end of the increment
Centennium continues supporting customers through
knowledge partnership

>?@A7%!BC7BCCDA#%

28

!"#$"##%&'()*("+,"-$%."/&%.(/0&.".(122($/"("+,"-$.(&#3"-(0#"(-0045(
!"#"$%&'(("#)*+&,--&.*'/-"0+"&,*0&"12"#3)4"&3'&,00#"44&3!"&5'62-"1&
$74)*"44&)*3"--)+"*5"&)447"4&(,5)*+&'7#&5-)"*34&3'0,%&

;18$.(1#3(4%<&-".:( 6"-7%8".(07"-7%"9:(
L"%#4/4M+NOOP+ !"#$%&'(#)*+
QRS+6%$1#/$$+1#'/&&12/#)/+)"#$%&'(#'$+ ,-"./)'$++
+ 0/$"%-)1#2++
!0-"(712&".:( 34%)('1"#+
T%H(#+!(?1'(&+ +
K#+)&"$/+)"&&(6"-('1"#+ 60'"(04(0&-(82%"#$.:(5""#6-"#7+8&6-"#7++
<6./)'1A/+(#4+K#4/?/#4/#'+ 9:(7+!8;7+<=07+>"?(@7+$/A/-(&+B%')C++
( D%#1)1?(&1'1/$7+8/2"#7+9%'-/)"7+E9<7+
=+,"-$%.":( F/#G*H/7+E('(+I'//&7+;,97+B3JE87++K;387+
8))/&&7+E"HE"H7+;8I+=89;7+J/($/,&(#7++
=%$1#/$$+1#'/&&12/#)/+
=-(6(#'+5('/-+
I'-('/21)7+'()'1)+(#4+"?/-('1"#(&+
+

!"-$%4%81$%0#(1#3(*#3&.$->(?"80<#%$%0#(
8&&+!/#'/##1%H+)"#$%&'(#'$+(-/+!/-'1U1/4+=K+,-"U/$$1"#(&$+V!=K,W+(X(-4/4+6*+
EB5KY++
+
!=K,7+6()@/4+6*+EB5KZ$+-/?%'('1"#+($+'C/+&/(4/-+1#+=K+/4%)('1"#+(#4+-/$/(-)C7+
H(@/$+(+$'-"#2+$'('/H/#'+'C('+X/+(-/+(H"#2+'C/+&/(4/-$+1#+'C/+1#4%$'-* %?+'"+
4('/+X1'C+=K+'/)C#"&"21/$7+@#"X&/42/(6&/+(6"%'+6/$'+?-()'1)/$+(#4+$'('/["U['C/[
(-'+$"&%'1"#$7+(#4+1#U"-H/4+"U+/H/-21#2+'-/#4$+

8&&+!/#'/##1%H+B('(+>(%&'+$?/)1(&1$'$+(#4+(-)C1'/)'$+(-/+)/-'1U1/4+"#+B('(+
>(%&'+H"4/&1#2+6*+B(#+J1#$'/4'7+B('(+5(-/C"%$/+1#4%$'-*+'C"%2C'+&/(4/-+
(#4+U"%#4/-+"U+'C/+B('(+>(%&'+H"4/&1#2+$'(#4(-4.

!/#'/##1%H+1$+'C/+6%$1#/$$+?(-'#/-+"U+F/#/$//+8)(4/H*+1#+"-2(#1G1#2+'C/+B('(+
>(%&'+D"4/&1#2++!/-'1U1)('1"#+ +3%-"?/+!"%-$/Y++
5/+(-/+())-/41'/4+'"+?/-U"-H+'C/+F/#/$//+8)(4/H*+B('(+>(%&'+'-(1#1#2$Y+

+)%$'"H/-+9%'-/)"+
B('(+D"4/&+1#+]^^O+U"-+'C/1-+)"-?"-('/+U1#(#)1(&+(#4+?-")%-/H/#'+4('(+X(-/C"%$/+
H"4/&Y+

31

!"#$"##%&'()*("+,"-$%."/&%.(
01#2"(344-/4&$(56(
7895(:!(;.<=-1>"#/12"((
?"@"A44#( BCB(69(7B(6CB(
D1+(( BCB(69(7B(6C9(
EF0(( GGGHI"#$"##%&'H#@((
( ( ( GGGHJ%<4,@"%K%#2"#H#@(
!
! ! !

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
Logica

29 september 2011

Metadata driven Data Integration
Hype or reality ?

Datavault Conference - Automation
Bertram Hof & Tom van Gessel
6-10-2011

Generating or still Programming

!  Do you use Data Integration tools ?

!  Do you use Metadata Exchange ?

!  Do you use design patterns / reusable components

!  Do you spend much time testing

!  Do you have metadata management in place ?

© Logica 2011. All rights reserved Datavault Conference – Automation 6 oct 2011 No. 2

Title of Presentation 1

29 september 2011

Agenda

!  Logica and our BI Practice
!  Framework approach
!  Best practices
!  Demo Mapping Builder
!  One step beyond, Business Metadata driven
!  Recap
!  Q & A


Logica, our presence - Europe

" Our widespread presence
means we have the capability 9,600
to sell and deliver where our UK Nordics
clients work and live
5,400
" Speaking the same language 1,900
Germany
gives us strong client and
5,500
cultural intimacy Benelux

200
" Combining these skills with
blended delivery is a platform 8,900 CEE
France
to deliver services in the most
efficient way to our clients 900
Portugal



29 september 2011

Logica, our BI workforce world wide

> 3000 consultants work on BI every day,
on site, remote, near- & offshore


in
Roll out thought leader in
Logica, plan
ce
en
Business Intellig w
Europe, launches its ne
book to share its vision
How to Transform Information
Into a Competitive Asset
Discover the BI Framework

Investing in Business Intelligence to aid
competitiveness is, for the fourth year in a row,
top priority for CIOs, say analysts. BI is even
more important when times are tough: it can
help find bottlenecks and inefficiencies or
expose areas that are profitable.

Knowing that most organisations already have some
BI solutions in place, this publication focuses on cost
effective management of BI and provides with a clear
roadmap on how to lower the total cost of
ownership of the current landscape.

Discover a structured approach to manage the
BI life cycle in a cost effective and efficient
manner.



29 september 2011

Agenda

!  Best practices
!  Recap
!  Q & A


Logica - BI Framework
Business Focus
ICT Focus

Operation Focus Change Focus



29 september 2011

Logica - BI Referentie architectuur

Operational Actionable
Data Information

Client
Operations
Services
Reporting PDA

Product X
Sales
Services RSS
Enterprise Analytics
Data
Product Y Warehouse Finance Mail
Services

Mining
Product Z Web
Marketing
Services
Extract Access Publish
Source Integrate Storage Subject Area Utilities Personalise
Present

Data Warehouse (back-end) Business Intelligence (front-end)
Sequential Development Iterative Development


Logica - Engineering Framework
BI Engineering Framework subject models
Deliverable
Data Function Network Timing People Motivation
Mission & Vision statement
Services
Business Business Business Organisational Goals &
&
Terms Locations Events Entities Strategy
Business Products
Context Semantic Business
Logistic Master Organisational Objectives
data process
System Plan Structure & Policies
model model
Enterprise Architecture criteria, topologies and standards
BI
BI BI
System semantic BI infra BI event BI user task
essential semantic
Context data context model model
context rule model
model
BI
BI architecture criteria, topologies and standards
Architecture
Logical Logical Logical
System Logical Logical user Logical
data process control
Concept Infra. Model interface mdl. rule model
model model model
Physical Physical Physical
System Physical Physical user Physical
d ata process control
Specification Infra. Model interface mdl. rule model
model model model
Busines
Repository Database Process Infrastructure Procesflow User interface
rule
data & Code code code Environments code code
code
Business
BI Solution Database Process Infrastructure Procesflow User interface
rule
Configuration objects objects Environments objects objects
objects



29 september 2011

Agenda

!  Best practices
!  Recap
!  Q & A


DWH, layer reference model

" 3nf " Per source " Per source " Per source " Integrated " Integrated " Subject
" incomplete " Source " Source " Storage " Target " Subject oriented
history model model model " Delta oriented " Business
" Detail " Delta " Complete " Complete " Truncate/ " Dimensional Language
" OLTP " Truncate/ History History insert Model
insert " Merge " Merge " Merge

Source IMP STG/ODS DVT
1 1 1 1 D D D D

Source IMP STG/ODS DVT F F
2 2 2 2

D D D D
Bron n
Source IMP STG/ODS DVT
n n n n

IMP STG/ODS DVT INT/BVT STO/DMT

Source Knowledge
system Reference data
Worker

Meta data

Processflow



29 september 2011

Productivity Boosters

!  Import flat files – Import Tabel + Import Mapping

!  Staging / ODS – Staging/ODS tabel + Merge / SC Mapping

!  Storage / Datamart – Dimension / Fact + Mapping

!  Processflows

!  Quantitative Measures

!  Seedfile driven generation – flatfiles / imp /ods

!  Seedfile driven generation – XML delivery / interfaces

!  Seedfile driven generation – dimension/fact loading

!  Datavault Experts – Hup, Link en Satellite generation


Example of Productivity Booster Datavault Link



29 september 2011

Some Practical results

!  Seedfile driven approach ODS => 15-30% of budget

!  Productivity Boosters during development => 10-20% of budget

!  Quality improvement => 40%

!  Test reduction => 70%

!  Exploitation reduction

!  Time to market

!  Impact Analysis

! …


Agenda

"  Logica and our BI Practice
"  Framework approach
"  Best practices
"  Demo Mapping Builder
"  One step beyond, Business Metadata driven
"  Recap
"  Q & A



29 september 2011

Demo, mapgen the functionality

Mapping generation with informatica powercenter

Parameters

Repository
Mapgen Informatica

Visio
Template


Demo, mapgen the templates

Target ODS

Target Satellite (DataVault)



29 september 2011

Demo Mapping Builder

!  Ferarri case


Agenda

!  Best practices
!  Recap
!  Q & A



29 september 2011

ETL Development Process


ETL Framework a different perspective



29 september 2011

Logica - ETL Framework, components


ETL Generator



29 september 2011

Some lab implementations

Microsoft
SSIS

IBM Cognos/
Infosphere


Lab results

•  BI- Platform indepedant ETL methode
•  Generic ETL model/design

•  Cost reduction of 8% with ETL Framework

•  Cost reduction of 17% with ETL Generator

•  Combination of ETL Framework and ETL Generator will
result in cost reduction > 26%



29 september 2011

Agenda

!  Best practices
!  Recap
! Q & A


Recap

Generation of Dataware House not a hype but reality

"  Main parts of the datawarehouse can be generated

"  Requirement Capture needs further maturity
"  Framework approach provides the structures needed to generate
"  Mature enough to use within projects and organisations
"  Quality results obvious /Testtime reduction
"  Faster implementation, time to market / Reduced TCO



29 september 2011

Agenda

!  Best practices
!  Recap
! Q & A


Q&A



29 september 2011

Thank you

BI brilliant together

Ing. Bertram Hof Tom van Gessel
Principal Consultant BI Software architect


BI brilliant together

© Logica 2011. All rights reserved


!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

XLNTconsulting !

Agile BI:
Accounting for progress

Tom Breur
Data Vault Automation
Utrecht, 6 Oktober 2011

“Our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software”

Agile Manifesto, 2001
Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham,
Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern,
Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

www.xlntconsulting.com 2

Counter intuitive Agile practices
! People are more productive if nobody tells
them what to do
! Pair programming leads to more (effective)
production code
! Business partners must be full-time
engaged (co-located) with the
development team


Counter intuitive Agile practices
! Only the business has the right to choose
what gets done
! An efficient team must have “slack”, must
have people sitting idle, with nothing
productive to do, on a regular basis
! Etc.


Software ‘inventory’

“Work-in-Progress is a liability
– not an asset”
Tom Breur, 2011


Simplified development

Error Reports

Idea Develop Test Working Code


(More) realistic development

Idea Analysis Design Code

Error Error Error

Working Acceptance System Unit
Code Test Test Test


Agile manufacturing

Theory Focus
J-i-T Inventory
TQM/QA Quality & Conformance
T-o-C Bottlenecks
Lean Inventory, Quality &
Conformance
Six Sigma Quality & Variance


Throughput Accounting metrics
THROUGHPUT INVENTORY

Rate of cash* generated through Quantity of ideas for client-valued
delivery of working code into functionality queing for input to, in-
production, not merely code process through, or waiting for
complete output, from the system

*Assuming a constant level of Investment

INVESTMENT OPERATIONAL EXPENSE

The sum of money invested in the The sum of money spent in the
system of software production plus system to produce working code from
the sum spent to obtain the ideas for ideas for client-valued functionality
client-valued functionality input to the (marginal expense to create
system (gathering requirements) production code)


ROI in Throughput Accounting

Unknown (T) – Pretty hard to guess (OE)
ROI =
Didn’t bother to measure (I)


NP in Throughput Accounting

(more) Net Profit (NP) = T – (less) OE



Throughput (T) – Operating Expense (OE)
(more) ROI =
(less) Investment in Inventory



(more) Net Profit (NP) = (more) T - OE

(more) Throughput (T) – Operating Expense (OE)
(more) ROI =
Investment


Focus on Throughput
! Focus on T, I, or OE?
! Throughput is unlimited, it can grow
forever
! Focusing on cost has a logical (yet
unattainable) lower bound – namely zero
! Throughput focuses on the customer –
externally
! Cost focuses on the team – internally

Investment
! Minimizing Investment (I) drives ROI up
! Minimizing Investment also reduces OE,
by reducing carrying cost of capital
! And, most importantly
! Lower I means lower inventory,
which leads to reduced Lead Times,
hence earlier delivery of value
(Agile Manifesto principle #1)


Cost vs Throughput Accounting
Cost Accounting Throughput Accounting
!  Inventory is an asset !  Inventory is a liability
!  Efficiency = function/ !  Efficiency = function/
dollar (hours) " labor is direct costs (idle or not)
a “variable” cost " labor is a “fixed” cost
!  People sitting idle are !  People sitting idle are a
discarded! part of the system!


Cost vs Throughput Accounting

Cost Accounting

Operating
Inventory Production

Least Focus
Expense
Most Focus

Throughput Operating
Inventory
(Production) Expense

Throughput Accounting


Agile & Data Vault
! (very) few other architectures allow
incremental build at such low marginal
cost
! Deliver early – in (very) small increments
! (very) few other architectures allow
‘mistakes’ in your model, that you can
recover from inexpensively
! Deliver early – (long) before you have settled
on “the” final business model

Conclusion
! By providing appropriate metrics
(=Throughput Accounting), complex
adaptive systems (Agile projects) will
display the desired emergent properties
! Agile BI is not about delivering faster (or
cheaper) – efficiency
! Agile BI is about delivering in arbitrarily
smaller increments to end-users – hence
gathering feedback about effectiveness

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

Genesee Academy !

25568 Genesee Trail Rd
Golden, Colorado 80401
(303) 526-0340

Data Vault Modeling and Approach DW2.0 and Unstructured Data Master Data Management and Metadata

Data Vault
DW Automation
Classification Matrix

Data Vault Automation Conference 2011

©2011 Genesee Academy, LLC

25568 Genesee Trail Rd
Golden, Colorado 80401
Hans Hultgren

© 2011 Genesee Academy, LLC

Welcome

Overview of Data Warehouse Automation
Scope of the Classification Matrix
Classification Criteria
Automation Categories
The Automation Matrix
Applying the Matrix



Operational Applications support business processes.
Typically this implies the support and partial
automation of components of a particular business
process.
In addition to software, business processes are also
supported by methodologies, frameworks, specialized
techniques, and also forms, templates and checklists.
Together these form a pool of tools and techniques that
support certain aspects of these business processes.



With Data Warehousing and Business Intelligence
another pool of tools and techniques exists to support
particular aspects of these programs.
In fact these tools and techniques are vast and varied
each addressing some combination of DWBI activities.
To limit these tools to some degree, this presentation
will focus mainly on Enterprise Data Warehousing and
in particular those that utilize data vault modeling.


Scope of the Classification Matrix

As mentioned, the focus is on Enterprise Data
Warehousing:
* Integrated * Non-‐Volatile * Time-‐Variant
* Subject Oriented * Auditable * Adaptable

* Atomic Level * All Data * Traceable
* Business Key Based * Business Aligned * Hub/Link/Sat
Plus all forms of automation tools and techniques:
* Software Tools * Methodologies * Frameworks
* Code Generators * Templates * Shells
* Common Models * Documentation * Metadata



To begin working with the automation matrix we must
consider and understand the various classification
criteria.
Effectively (in simple terms) this means that we look at
different ways of thinking about these tools and
techniques.
As you will find, when a certain classification criteria is
presented, and you begin to think about that criteria in
context, the meaning becomes clear.



For example, consider the following classification
criteria:
Templates for ETL
Support for Data Modeling
Generation of Mappings
Automation of Testing
For each one, consider them in the context of some of
the tools and techniques presented earlier today.
This process of contemplating criteria in the context of
particular tools and techniques is the purpose of the
automation matrix


Notice in the prior examples there are two parts to

Templates for ETL
Support for Data Modeling
Generation of Mappings
Automation of Testing

The LEFT side items are tool or technique Features

The RIGHT side items are DWBI Functions


Automation Categories

Combinations of these Classification Criteria help us to
form sets of Automation Categories
While there are some obvious ones to consider (ETL
code generators, DWBI program methodologies, Model
and Integration Templates, etc.) we are also able to
assemble a custom set of criteria for our own
automation category.



Header Section
Main Matrix
Profile



Header Section

Capture name, note and date
Categorize based on
Tool / Application -‐ Software tool, application, template, shell, etc.
Methodology -‐ PM, program, management, governance, etc.
Framework -‐ Overall comprehensive end-‐to-‐end components



Main
Matrix

Sets of Features and Functions the classification criteria


Features

Manage Assists in the management of this function
Support Directly supports the function itself
Structure Provides structure and structural components
Organize Helps to organize the function
Automate Automates components of the function
Generate Actual generation of artifacts related to the function
Templates Templates to provide consistency & to expedite
Patterns Design, architectural, and software patterns
Document Creates or provides documentation related to function
Test Helps with testing related to this function


Functions

Scoping Mapping
Requirements Integration
Analysis Transform Rules/Logic
Design Profiling, Data Quality
Visualization Build ETL/ELT
Information Modeling Testing
Data Modeling Metadata
Creating Databases Documentation
Semantic Alignment



Profile

Considers the scope of what the tools and techniques support, the
primary value proposition and uses, what type of DWBI program is
supported, and the overall approach for the DWBI program.

Questions?

www.GeneseeAcademy.com

Data Vault Certification CDVDM

Register now for November 17-‐18
Centennium.nl

© 2011 Genesee Academy, LLC Hans@GeneseeAcademy.com
25568 Genesee Trail Rd USA +1 303.526.0340
Golden, Colorado 80401 Sverige 070 250 2102

© 2011 Genesee Academy, LLC 17

Architectural Layers


!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

IN2BI !

!"#$"#!$%%&

DATA WAREHOUSE AUTOMATION WITH..

DWH DECK AND BIML

MARCO SCHREUDER

Jumpers are closing ..
the gap between data and information.

%&

!"#$"#!$%%&

Alternatively!!"""!
They build a data warehouse

Rigid modular approach of Data Vault ...
Means easy automation

!&

!"#$"#!$%%&

Data warehousing ..
Available to more companies

Focus of DWH Deck on
Microsoft SQL Server

'&

!"#$"#!$%%&

Source Driven
Rich meta data and profile information helps with choices.

Model window allows ..
Generating, publishing and running the model

'&

!"#$"#!$%%&

SSIS PACAKGES WITH BIML

•  Alternative for ETL stored procedures
•  Business Intelligence Markup Language
•  To automate the creation of SSIS Packages
•  Invention of Varigence (varigence.com)
•  Donated to the (open source)BIDS helper project
bidshelper.codeplex.com

ROADMAP WITH FOCUS ON DATA MART AREA

•  Support reference or mapping tables to cleanse certain
column attributes.
•  Support combining multiple hubs and their satellites into
one dimension.
•  Allow defining of hierarchies.
•  Allow the creation of snapshot fact tables.
•  Support building Analysis Services (SSAS) cubes.

'&

!"#$"#!$%%&

GOAL: -80% TO 20%

But .. There is only so much one person can do
So ... I need partners

Interested?
E-mail: marco@in2bi.nl
Tel: +31 (0)6 26479075
Internet: http://www.dwhdeck.com
http://www.in2bi.nl

'&

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
BI Team

Meta & Master Data Management
Data Vault Generation

Data Vault Automation – 6 oktober 2011, Utrecht

!"#$%&

1. Inleiding

2. Meta- & Master Data Management
RDF - Resource Description Framework
Structured (Semantic) Wiki

2. Data Vault - ETL Code Generator

BI-Team
The Joy of Accomplishment
HLKH!MHN5H <M(MAN

• !"#$%&"' ()*+,#-,%#$ ./01"2,+ #, 3#10/ 0/4#-%&#,%0-+
! 503./"6"-+%7" 80/9#/' :00;%-4 <%+%0- =96%,".#."/+>
! 5"/,%?%"' @#,# <#)$, (."2%#$%+,
• A7"/ BC D"#/+ 0? "E."/%"-2" 9%,6 F(!"#$%&'##( )*++#)(#% ,-./G =HE2"$ I A:JK +"/7"/>

H++"-2"
A/#-4"O"-P
"#$%&'() *#)+&,-,.(#/ 0%/#1
Q",# R Q#+,"/ @#,# Q#-#4"3"-,

2&,3-#1.# 0%/#1
HS: 50'" O"-"/#,0/

455-()%'(,& 6#&#7%',7

Actionable Knowledge

!"#$%&%'' (&)*
+#', -)%'.#/ 0 123,% #4 #2,56&% 732 ,"5' .#/ 6#&&%. 8!"#$%&' (% )*+ ,-)- .-"/)9:
04 ;<= >#*%' ,%?, 5, >2%'%4,' '3$% 7)4.#$%4,#& ,"3)*",' 34 ,"% @#,# A#)&,< #4. #' ')6"
')$$%25B%' '3$% C#'56 ,"3)*",' C%"54. D2#4*%E%4:

-)%'.#/ %F%454* 0 .5. 2%6%5F% ,"% 73&&3154* 2%#6,534' 34 ,"% >2%F5%1 0 .5. '%4. 3),G
!"# $%#&'()'*
! "#$%%& %'(# )*# $")'+%# )*', ', $-#,./#0 1
! *$2# 3. +.//#3) .3 ')4 .)*#" )*$35 #6+#%%#3) 7.819
:.; $"# $ )";# <'.3##" '3 )*', $"#$9
+,- ./(0/*
!( ($3 =$3>, "#$+)'# $%%##3 /$$" .3?#",+*"'72#30 =') ', '3?#"?$$? 8$$38"#(#3? -#"(9
1,#"2) !"-3,4*
@A#/$3)'+ ,<$+#@ ', B#-..3 8"'%7$3) 1
).#2$% -'% ?$) /'73 )-##?# ,%'?# .< ?# (#&3.)# ?$$" /## ,)$") C3'#) *#%#/$$% <$,,#3?4 /$$" ?#
B#?$+*)# ', B#%'7(D9 1
E.B/$$%, #"B .3?#" ?# '3?";( %5 6(/) 7/,2%85 7"# )%' "/'%5(29

!546% 5, 1#' 125,,%4 &%'' ,"#4 HI "3)2' #*3< 5, 5' 43, 54 ,"% >236%%.54*'< '3
*%, 5, "3, 37 ,"% >2%'' #, ,"% J0 -%#$ ',#4.:

012*$)+2)"1+
0 2(//+2)$(% (3 4+5 6+2$'$(%'

!"%(&% 84"(&) "@ 4'& A(*+&%)56 ,- B%3'*4&342%&

• !"#$%&'&()*+& ,-./0 12(34*"(56*47
• !"#$%&' !()* +,- ./)/
• !"#$%&' 0/12. 34 +,- (45367/)(34
• !%*4*356 8233&)) 1534"%) 9 :;)4536&)
•8#"9%:- 1;--: "< (=;>-=-&+?+%"& @A 5B&C+%"&%&'D
•*?&:>%&' "< 6-EB%#-=-&+F
• /%"<&34 =&56*47
• 7B>+%;>- (&%+%?+%9-FG H"&+%&B"BF H,?&'-
• 4"& -I%F+-&+ "# >%=%+-: 7-+? A 7?F+-# .?+? 7?&?'-=-&+
• >"( 12(34*"(56 =&?2*%&#&(4)
• H"=;>%?&C-
• /B:%+?J%>%+K

,- C&5# A(*+&%)56 ,-./0 B%3'*4&342%&

© 2009-2011 bi-team

!"#" $"%&'()*&+ !"#" ,")-#
Fractal like structure, basic elements that are repeated
•Easy to generate
•Atomic statements – efficient, robust - research
•Can be stored in SQL and No-SQL databases

Semantic Transformation (& auditable)
Data domain to Business domain (& structured)

.&/"0#12 30"-4*1* 5 67 89:

Risk Management
Complex Interactions

!"#$%&

1. Inleiding



!"#$%&'(&) *&+, - *,.+&/ 0,+, *,",(&1&"+

The Whole Picture

*&+, - *,.+&/ 0,+,
*&+, !"#"
• $%& '"(#) #%"# #*+&#%&, -"./# " (*0-1&#& -.(#2,& *' " (2)#*0&,3 -,*42(#3 &#(5
• !&'./&) %*6 ./'*,0"#.*/ .) ("-#2,&4 "/4 )#*,&4
• 789&(#3 $ 8*:3 !"#" ;*4&13 <8)#,"(#.*/3 0,+, 23 0,+,

*,.+&/ !"#"
• $%& 1.)# *' ,&"1 &:.)#./+ -,*42(#)3 (2)#*0&,)3 &#(5
• $%& =+*14&/> )#"/4",4 *, ,&'&,&/(& 1.)#
• ?/)#"/(&3 < 8*:3 @2)./&)) A&B)3 C&"1.)"#.*/3 4&5&/&"6& 0,+,

;&#" "/4 ;")#&, !"#" 4.''&, ./
• 78& 6#".&9:&"6&. #5 68,"(&. D;&#" E ?$3 ;")#&, E F.&,",(%B3 &#(5G
• ;.,(& ,"' </=1,/> :.&/. D?$ "/4 @2)./&))G

*&+, ,"' *,.+&/ 0,+, /&9:=/& .8,/&' *,",(&1&"+

7#%&, (*/(&-#H @2)./&)) I E $&(%/.("1

!"#$%&'(& )*"*(&+&",
The Meaning (Semantics) of Data/Information is:

• The main ingredient in realizing BI/PM applications
• About 60% to 80 % of the effort in realizing BI/PM
applications is spend on Data Access
• At the heart of information supply
• Important for the whole information chain
• From input, through processing to interpretation
• Important for all activities of the whole organization
• Essential for cooperation

Need for: - Encyclopedia of the Organization
- Pragmatic input and management
Solution: - Wikipedia of the Organisation
- PiggyBack on BI/PM initiative

.&2#638& /&28317,1#" 03*+&$#34 ./0
R D317%& @,#3&
• !"# $% & '()*+ '$+, ',- .(/%()0$12 34567 %0&/+&)+8
-./0 12 * 03*+&$#34 5#3 2677#3,1"( .&2#638& /&28317,1#"9
#3 )&,* /*,* :/*,* *;#6, /*,*<=

• -./0 12 62&' ,# 8*7,63& 27&81518 2,*,&+&",2 *;#6, * 3&2#638&9
2,*,&+&",2 ,>*, >&%7 5#3+ * +#3& 8#+7%&,& 718,63& #5 ,>& 3&2#638&?=
3 9)&.0$.&* !"#: ;<,**,= 9(',)%7

• ><, +&0&?()2&0 $% ./0 ,317%&: '<$.< .(/%$%0% (?8
– @6;A&8,9 B3&'18*,&9 C;A&8,
– @$A, & -&%$. B/C*$%< %,/0,/.,: & 2,&/$/C?1* %0&0,2,/0
• #() ,D&2E*,8 !"#$%&'(" $% 0<, )(*+$(, (? 0<, -%$.%&,(/'#
– FD31*"(6%*,1#" #5 !"#$%&'(&G

• D317%& @,#3&
– /*,*E*2& )*"*(&+&", @F2,&+ 5#3 D317%&2
– C7&" GHI @,*"'*3'2J
• ./0 K '*,* 5#3+*,
• @BL.MN O @BL.PN K Q6&3F %*"(6*(&
• CGN 318> '&28317,1#"9 3&*2#"1"(

EF8"+ G BC@
Subject, Predicate, Object: A predicate modifies a subject.
A predicate must contain a verb and other sentence elements like an object to complete the predicate

.&"5)8#*" 9$ '#()*#+,-
!"#$%&" '#()*#+,- ./(%+#*)/0 :#$ 1&"#
:#$ ./(%+#*)/0 1&"#

;%<="8*
')*2 ,<="8*
3#+%"
14$*"&5#4 !"#$"%&'()* +,+-./0 120344
6"&+)0 5"%6'(7 4-//1-,+8 .023.8
7/05/0 5%"'# 9%:#':( +-88,-0;; 2-8++34;
;(&"#5$>""*$? @+#* @)+"$ A B"+#*)/0#+ C#*#<#$"$
#&" !"#$%"& BC@ $/%&8"$D

<6=>%#'(# :* ')$"%"(?" #> @'A B#'()'%)*
C"&& D(>E( #'A *#'()'%)* '%"F GHIG3 JKL&:( M>%"3 "#?-

;*&%8*%&"5 O ;"4#0*)8 H)I)
H)I) .#J" )$K.&/F2L /- /<="8*$? )5"#$? $24</+$? "*8M
– NO<3 N(:P>%6 O"*>K%?" <)"(#:P:"% Q NOR3 N(:P>%6 O"*>K%?" R>?'#>% S$##=FTT---U
H)I) (#J"$ #&" # 0#*%&#+ NB9ONB7 */ )5"0*)-2 B"$/%&8"$-

BC@ *#'#"6"(# :( ;"4#0*)8 !"5)#H)I)F
H( ' (#J" E:#$ :(P>%6'#:>( >( ' &"$/%&8" S' $%<="8*P
' #'A :* ?>(#':(") >P #$" P>%6F VV(&"5)8#*"QQ/<="8*RR

@/& "F#4(+" /0 # (#J" S)*> )0-/&4#*)/0 /0 14$*"&5#4Q
VV*"6'(#:?W*#'#"6"(#FFX'&K"YY
T2("5 7)0I UU9$ '#()*#+ /-QQ*>" V"*>"&+#05$RR
1**&)<%*" UU>#$ ./(%+#*)/0QQWXYMYYYRR

:%4#0$ A '/4(%*"&$ 8#0 '&"#*"? B"#5? N(5#*" A C"+"*" *>" C#*#
;"4#0*)8 H)I) G BC@ E5)*/& O @&/0* E05 */ BC@ T&)(+" ;*/&"

;%<%
Three A’s: • !"#$%&'( )* +, -"+( ./,-)0+1
Anybody can say – !" #$"% &' %($)$"*
Anything about
Anything • 2,#1(#1)1 3"04%#(
• +,-. /-%,0%1 23#&,"
If you are known (logged in) – 4$1&311$'" 5,*%
• 5(*,1%+,/6
Mediawiki software – 2%-1$'"$"*

scalable • 7+/)0+)/( 8,'',91 :"+"
• 6$## 7"8'" 9 "'" 1)-3&)3-%( (,),
capable
– :),"(,-(1; <4= > :?@<AB
tested
• 7("/04
proven
– =-%% C%D) :%,-&E > :%8,")$& =$"(

=$(#-"

1. Inleiding



!"#$%&'($ )* !+,-.$/$-

/$-$+,&$ 0)1$
!" #$%&' ()' &"(' !*+
2,&, 3,+$4)56$ ()' 2,&, 7,+&6
",6$1 )- &4$ 8-)9:$1.$ '- &4$ !+.,-';,&')-

<4'*& ", =&&$-&')- ,-". >$?5'+$@$-&6 !" 8-)9:$1.$

•0,&$+ *)+ <A$$1 )* B@A:$@$-&,&')-C04,-.$
•B-*+,6&+5%&5+$ *)+ <$:* <$+('%$ DB

2,&, 3,+$4)56$ E 2,&, 7,+&6 /$-$+,&)+

>$?5'+$@$-&6
Knowledge
Source Semantic
B-*)+@,&')- WIKI
8-)9:$1.$ K
Triple
<&)+$ F(,:5,&')-
G>5:$6H $&%IJ

RDF to DDL & ETL /00&%1(!%")

2,&,
Building & Loading
L,5:&

'"()*+, -.& /*0* 1 2"%#3&+-*+, -.& !"#$%&

67,#&8-7 9(-.:
1-<%&%"%+&=
4++)
11;

!"#$%&
?%5% "-@" A B1' ".%>)-=

'()) *+&,-." /
0!' 12 3%$.#"%+& !"#"%="%8=
4++)5%"
!"#$%&$
4.%>)-
!"+.-

F:I.%7 1#"# E#()" 6>>.+#8C
Structured Wiki

RDF
4-8C&D/ FG36H/
B6? 2(=%&-==
1DED RULES 1DED

RDF
“structured” One Environment “unstructured”

!+(.8- 1.%,-& 3-"# 1#"# 1.%,-&

,-&./(0(.
1 2 4
RULE Engine

7484 !"#$ 9&/(: !"#$ !"#$ 9&/(:
7484 "%&'(
3
Triple )*+*)*+* 9&/(:
7484 )*+* 9&/(: 7484 "%&'(
"62-(

))< = 12.34/5-&642. )2';
"6&64:64':

+&-/(6
"25-'( #+< )&6& >&5?6
! "#$# %#&$'

BBB;,-&./(0(.;.(6

$!<# #./4.(

7484 !"#$ 9&/(: !"#$ !"#$ 9&/(:
7484 @&A(:%&'(

7484 )*+* 9&/(: Triple )*+*)*+* 9&/(:
7484 @&A(:%&'(
"62-(

))< = 12.34/5-&642. )2';
"6&6:;

+&-/(6
"25-'( #+< )> = )C

7,F")
*+,-."
(-+/"-01") 81-)0 @))"))A"20 !"# $%&)
!"#$%&' ()")$*)$+*' ,)+-
+2 ":1)012&
$%&) +2 3141 3141
5%0% (%&")
67589 7;<=* 5%0% (%&")

$%-&"0 @))"))A"20
.*,/ 0%12) ' 3"4$5"),5 6"&*

=!>?!= !"# $%&&"'
;)"- ?2/,0
$%-&"0 5"B12101+2 (%&")
3"4$5"),5 6"&*' 789 6"/&,) 8,:$%$)$;%

C *0"/)D 81-)0 @))"))A"20 E $%-&"0 @))"))A"20 E $%-&"0 5"B12101+2

0%12+#11 3%)#1 !+/2+#
• !"#$%&## "#&'# %&&( )* +& ,+-& )* "%(&'#),%( ,%( .,-$(,)& )/& '"-&#
• !"#$%&'()# *+,-)#./# 01234 5&+#&'.$6&7 8,., -$+','9:
– !"#$%&## ;%*5-&(<&
– =>?@ ($A&%#$*%#7 &)6B
• C& ,'& %*) ,+-& ,%9 )$A& #**% )* 5'$)& '"-&# )/,) ,'& DEEF '$</) DEEF *G
)/& )$A&
• C& %&&( )* +& ,+-& )* )5&,H )/& *")6*A&
• I/& '"-& &%<$%& 5'$)&# ),<# )* )/& J,<&7 )/,) 6,% +& ,-)&'&( ,%( &%/,%6&(
A,%",--9
• K'**-#
– 8,.,LM'**69
– NJ'&,(/&&) (&6$#$*% ),+-&
– K'**-# -,%<",<& 0,-#* $% OP>:
– K*A,$% NJ&6$G$6 >,%<",<&
– C$(&-9 1#,+-&
– Q%)'9 R,-$(,)$*%7 S*AJ-&T Q.&%) @'*6&##$%<
– @-,%%$%<7 U*'&6,#)$%<

DRools Guvnor
•Web Based Rules Management System
•Rules Editor
•Rules Engine Webservice
•Rules Repository (Webservice)

Logic Can be defined with:
•Guided Rules Editor
•Decision Tables .XLS input
•Domain Specific Language
•DRools language (XML)
•Java/Groovy

!"#$%& '()
'*"+,-% ./-%0 '() 1#&2.%3*4

!"#$%&'&$ (#$%)*+& ,-$./)0$1
• 2)3) 4&/ 56
– !&%/5% 6&#,%&5+%-7 (%8,"/5%-7 !/5%9#&+%- : !#35%35 6/9%-
– ;<3 =/+35%3/3*% >*&+,5-
• 70#8)$% /9& 8#/#
– !&%/5% 1+2+ 6/9%- ?+54 (/99%0 @3A#&8/5+#3 ./-%0 #3 B/5/ B%A+3+5+#3
)/39</9% /30 B/5/./-% *#35%35 >5/5+-5+*-
– !&%/5% (/99%0 =/-5%& B/5/ C!#35%35D 6/9%- .E )#/0+39 B/5/
• :&#10$)$%
– F00 5/9- #& *4/39% $/"<%- ./-%0 #3 *#3*"<-+#3- C*&%/5% : %0+5 ;BGD
– !&%/5% 1+2+ ,/9%- ./-%0 #3 *#3*"<-+#3- C0%-*&+.% 5/&9%5 +3 ;BGD
– !&%/5% *#33%*5+#3 #.H%*5- A#& &<"%- %39+3%
• I/$/ .%/3- ./-%0 #3 ,&#,%&5+%- #3 -/8,"% ,/9%- /30 *#3A+9 -%55+39-
• ;0$.+-8)$%
– !&%/5% J=) A+"% 0%-*&+.+39 5/&9%5 #.H%*5 C;BGK>6F;L) M J=)D
– !&%/5% BB) /30 '() *#0%

!""#$%&'$() *+)+,&'(,
• !"#$%&'($ )*+(&, (# $&& $'"&&+$
– --.- *%%/0'*(0#+
– -*$(&".*(* 1 0+2*3#4$ /0$($

• 5.6 ,*(* 3#,&/ 7 84$0+&$$ 54/&$ 9
6/&:0;/& <+=0"#+3&+(

• <:'&/ ,&20+0(0#+ ;*$&, *%%/0'*(0#+ >&+&"*(#"

!"#$%&'&$
• (&)# * (#+)&" ,#)# (#$#%&-&$)
• ,#)# .#/0) '&$&"#)1"
• 233045#)41$ '&$&"#)1"
• 6"#%-#)457 899&5)4:& * ;0&<4=0&
• 6"1:&$ >&5?$101%@
• A1B C$:&+)-&$) * D/$$4$% E1+)+
• !"#$%&' ())"*+&#$), )- .&, / .&(0$,*
• 1,0&,(* ())"*+&#$), 2$#0$, #0* )+3&,$4&#$),
• >&5?$45#0F D,; ,#)# (1G&0 H I/+4$&++ D/0&+

!"#$% .(,/+-'$/ 0
&'( )'* &'(* #++,$+-'$

BI-Team
The Joy of Accomplishment

Musings on the Data Vault
In the following I like to express some thoughts on the Data Vault that are not of an immediate technical nature.
First let me ponder on the Data Vault as fractal, thereafter I will draw attention to the semantic transformation
and integration involved.

Data Vault as Fractal

Illustration 1 - Iterated Function.

When looking at the Data Vault its fractal structure is something that immediately draws attention. e
question is however, whether this notion does help us understand the Data Vault better, or makes it easier to
communicate the concept. e idea here is that as a metaphor it may help us to convey an essential aspect of the
Data Vault, however as with any analogy it should not be expanded to its extremities or taken too seriously.

Illustration 2 - Cantor Fractal.

Fractals come in sorts and types. One of the ways of distinguishing fractals is how they are generated. It is the
class of so called iterated function systems, or IFS fractals, that show a remarkable resemblance to the Data Vault.
It is the class of fractals that is most used for analysis, and IFS is also used for encoding images. e ideas behind
this type of fractal can be traced back to Gottfried Leibniz, who started to think about recursion in the 17’th
century. In 1883 Georg Cantor published about set theory mentioning what is now called the Cantor Fractal,
actually discovered in 1875 by Henry John Stephen Smith. rough consideration of it, Cantor and others helped
lay the foundations of modern general topology.

In the use of Iterated Function Systems for coding and analysing images and diverse signals we nd our analogy.
By repeating and transforming some simple elements over and over a complete image can be coded. is analogy
shows that the power of the Data Vault to describe a complete data-landscape can be linked to the repeated use
of just a few standard elements. It is the repeated use of just a few standard elements, that is one of the main
aspects of the Data Vault. By just using a few standard architectural elements, these elements can be studied
and understood well. In addition also using just a few atomic statements in the database management system
over and over, makes that the implementation of a Data Vault is robust and independent of a speci c database
management system. Also it can be envisioned that implementing a Data Vault in a DBMS that is not based on
the relational paradigm, may open up a whole new level of performance and capability. e fractal nature of the
Data Vault will help to implement it on a lot of new platforms that are out there now or about to come. e
work of the Systems Group at the ETH Zürich and the Avalanche project show how joins may be speeded up
dramatically by just adding eld programmable gate arrays (FPGA) based co-processing to a traditional RDBMS.
A Data Vault will immediately run better because of the work on the join statement.

Netezza which has recently been bought by IBM, could prove to be an ideal platform for Data Vault
implementations, as many new appliances and NO-SQL databases that are out there. Dan Linstedt has recently
published on this. ese options to implement a Data Vault on diverse and new platforms and the promise
it holds to boost the performance and capability of the Data Vault are there because of what intuitively can
be perceived as the fractal nature of the Data Vault. e power of the use of just a few basic architectural and
technical elements is not only of importance now, for the future it makes Data Vault even more applicable. e
Data Vault paradigm is even relevant when implementing a system based on an RDF Triple Store, where tables
or columns do not exist at all anymore on the logical level. I will re ect on that in the second part of this article.

Another aspect of the use of just a few basic architectural and technical elements is that because of this, the
generation of the code to build and load a Data Vault and querying it can very well be automated. Again this is
a very powerful aspect of the Data Vault that can be associated with its perceived fractal nature. So it seems that
the image of a fractal may very well express some aspects of the power of the Data Vault when talking about it,
for instance when introducing the concept.

Sat Sat Sat
Sat
a Sat Sat
S
Sat Sat
t S
Sat Sat Sat Sat
Hub Link Hub Sat Sat Sat
Sat Sat
t Sat Hub Link Hub Hub
u Link Hub Hub Link Hub
Sat Sat Sat
Sat Sat Sat
Hub Link
k Hub
Sat Sat Sat Sat Sat Sat Hub Link
k Hub
Hub Link Hub

Hub Link Hub
Sat Sat Sat
Hub Link Hub Hub Link Hub
Link
Sat
a Sat
a Sat Link
n Hub Link Hub

Hub
u Link
n Hub

Illustration 3 - Data Vault as Fractal.

Data Mining, Statistics, Spreadsheets Standard Reporting, Semantic Document Knowledge
Visual Analysis Life Read/Write connected to OLAP Dashboards, Briefing Books, etc. Analysis Management Management

SQL MDX & SQL & Native MDX & SQL & Native SPARQL

Semantic
Integration
Data Marts
Spreadsheet connected OLAP + Relational + RDF Requirements; Truths
Semantic Wiki
Triple Store

Transform & Load Data Mart - Builder & Loader

Sat Sat Sat
Sat Sat Sat
S
Sat S
Sat S
Sat Sat Sat Sat
Sat Sat Sat
Hub Link Hub Sat Sat Sat
a
Hub Link Hub Hub Link Hub Hub Link Hub
Sat Sat Sat
Sat Sat Sat
Hub Link Hub
Sat Sat Sat
t Sat Sat Sat Hub Link Hub
u
Hub Link Hub

Hub Link Hub
Sat Sat Sat
Hub Link Hub
b Hub Link Hub
Link
Sat Sat Sat
Sa Link Hub Link Hub

Hub Link Hub
Hu Data Warehouse
Data Vault Knowledge; Facts

Extract & Load ETL Tools

Complete Architecture for Source Source
Source
Pragmatic Implementation
v1.1 - hscholten@bi-team.com

Illustration 4 -

Semantic Transformation and Integration
Now let me share another observation on Data Vault. It is common sense to say that the data will not be
transformed when loading the Data Vault. And indeed by technically transforming at most in a reversible way,
the important aspect of auditability is guaranteed. However in my opinion the data is subject to an important
transformation and integration process when it is stored in a Data Vault. A Semantic Transformation and
Integration is applied to the data, that is of great importance. is transformation and integration is one of the
powers behind the Data Vault. And I think that it is important to be consciously aware of this.

What do I mean by Semantic Transformation? By Semantic Transformation I mean that the data is transformed
from the technical context to the business context when it is being loaded from the source into the Data Vault.
e data in the vault is structured and integrated according to its business (organizational) meaning. e main
architectural feature of a Data Vault is the hub with its associated business key. is is not a technical key, and
for a good reason it is called the Business Key. e integration on the basis of the business (organizational) keys
is one of the strengths of the Data Vault paradigm and is the reason why new data can be added and data can be
separated so easily. Because the data is transformed from its technical context to the business context, it cannot
be generated only on the basis of the technical information on the source data. e data is going to be re-ordered
and integrated based on the business keys, based on the knowledge the business (organization) has on the
information. e fact that the data is semantically transformed and integrated is the reason why a Data Vault is a
good source for Data Marts, especially OLAP cubes where usually the dimensions are associated with a business
key. e combination of the semantic transformation with the rule that the data will not be transformed in the
sense of changing the numbers or facts, makes that the Data Vault is such an excellent paradigm.

As an example of the value of transformations we can look at image manipulation. When we talk about printers
and monitors we talk in RGB (or CYMK) space, because this is how printers and monitors and the rst level of
our vision system work. Red-Green-Blue color space is tied to the physical level. When an image has to be edited
for color we transform it to Lab color space. at is because this is how our eyes/brains process the information
which is received by the rods and cones in our eye. Some image manipulations are near impossible and most are
very hard to perform in RGB space, but easy to do in Lab space. Professionals and image manipulation software
mostly work in Lab space when correcting or manipulating color and other technical image issues. Further on in
the image chain we mainly talk about Hue Saturation Lightness and use that for instance in Photoshop, because
this is how we think and feel about complete images. Hue-Saturation-Lightness is also how people talk about
images in general. Consumers never see Lab space and almost no-one is aware of it, but behind the scenes the
software uses it. By space we mean a set of information consisting of dimensions, reference points and scales.

Likewise when data is stored in a system of record, we think in relational (transactional) space. When storing
data in a Data Vault it is transformed to Semantic Space, and for reporting it will again be transformed.

A Data Vault is all about expressing the information in the context of the organizational / business meta- and
master-data and translating it from the context of the technical systems to the organizational basic categories
and their properties and relations (the business or organizational ‘ontology’). Because a real and important
transformation is applied, it is logical that some e ort and judgement is involved. is is why building a Data
Vault is not just straight forward, a Data Vault is not just another data storage layer. Because the data is organized
according to the business meaning (ontology), it is so easy to add new data to a Data Vault.

e notion that a Semantic Transformation and Integration is applied, makes it a natural choice to look at
semantic technologies when de ning and building a Data Vault. I will not get into that here, the BI-Team
white paper about OrangeGen addresses that. But we do not have to stop at applying semantic technologies to
building the Data Vault. e data in the vault is because of its structure waiting, or actually it may be screaming!,
to be published as RDF data. For instance the D2R open source software is available to add a server layer on
top of the Data Vault so that the data in the vault can e ciently be published as RDF triples. is opens up the
data for a whole new class of analysis, which can be called Semantic Analysis. Network analysis on behalf of risk
management is only one of the many possible applications. e data in every Data Vault can immediately and
optimally be opened up for this type of analyses, because it already is semantically transformed and integrated.

Next we do not have to stop at applying the Data Vault paradigm to an RDBMS, we can apply the Data Vault
paradigm even when letting go in the data model of tables and records altogether. e Amdocs-Franz AIDA
project uses the Allegrograph RDF triple store as back-end for a CRM system, read about it on the web. is
project shows that RDF triples are a viable data model, and as well that the technology is available to do
something useful with it today. e relevant issue here is that an application still needs a data-model on the
logical level, even if we do not have to think about that on the physical level. It seems clear that the Data Vault as
paradigm is just as viable when applied as logical model in the semantic space, as it is when applied to relational
databases.
I do hope that by drawing attention to the Semantic Transformation and Integration that takes place and
naming it as such, the awareness of this important aspect of the Data Vault may grow and that as a result this
transformation and integration may be better understood by discussing and studying this aspect consciously. At
least the notion that we are designing an ontology when de ning a Data Vault, may point to technologies, tools
and methodologies that are used in the eld of knowledge engineering and ontology building that can be of use
when de ning a Data Vault.
Henk Scholten
BI-Team B.V.
www.bi-team.com
Text version 1.1 / d.d. 20111010 BI-Team uses the Cantor Fractal as its logo

For more information on this subject read the Bi-Team white paper:
OrangeGen, Meta- & Master Data management

Data Mining & Classic BI Spreadsheet Friendly OLAP Knowledge / Semantics
Based Applications

Ontology Editor

Relational: SQL OLAP: MDX + native Knowledge / RDF: SPARQL

Knowledge RDF

Data Marts
OLAP + Relational + RDF triple store RDF TRIPLE STORE
Meta- & Master Data

Transform & Load RDF ETL

Sat Sat Sat Rule
Sat Sat Sat
Sat
a Sat Sat
Hub Link Hub
b Engine
Hub Link
i Hub

Hub
u Link Hub
Sat Sat Sat Transformations
Data Warehouse Hub Link Hub
b
Data Vault RDF

Extract & Load RDF ETL

Source Source RDF

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
Conspect

06 October 2011

Data Vault Automation
Wisdom never fails

So Conspect?
Wisdom never fails

! Dutch ICT & Consulting organization
! Headquarters @ Almere
! 120 Highly Skilled Professionals
! Conspect is a company that excels
in Attitude and Behavior
! Within the ICT domain we provide
professional services in Application
Integration, Business Intelligence,
Custom Development and CRM on
Demand
! Conspect = Open Culture - Social
Engaged – Sustainability &
Corporate Social Responsibility

Why Conspect….
Wisdom never fails

‘Think Big, Start small, Scale fast’

Conspect and BI, our history
Wisdom never fails
2001 2009 2011

Developed
the idea of
Beginning automating
of our Data made our
Conspect Vaults ideas real

First Data Brought BI
Vault Team
together
with
Conspect

2007 2011

Who is this guy?
Wisdom never fails
Name: Justin Eelzak
Date of birth: January 13th 1981
Employer: Conspect
Occupation: Business Intelligence Consultant
Active in BI since: 2008

BI Specialty: Reporting and as of today speaking to large groups about Data Vault

Next Generation of Data Warehouses
Wisdom never fails

An Actual Slide Slide
Wisdom never fails

Value in automation Difficulty of automation
• Complex • Non standard
• Dynamic • Complex
• Great deal of initial work

! “I was so hoping it wouldn’t be a guy reading the bullet points from his slides.
! He lasted ten slides before failing…”

DWH to Staging Out
Wisdom never fails

Value in Great deal
Complex Dynamic of initial
automation work

Difficulty of Non Very
automation standard Complex

Source to Staging
Wisdom never fails

Value in Reasonably Static once Not a lot of
automation simple in place initial work

Difficulty of Metadata
Non
rules are
automation standard
complex

Staging to Data Vault
Wisdom never fails

Moderate
Value in Moderate
initial
maintenance
automation development
costs
time

Difficulty of Simple
Highly
Metadata
automation standardized
rules

What did we end up doing?
Wisdom never fails

! CDC (hardly a confusing
acronym)
! Fully automated the Staging
to Data Vault E(T)L
! Fully automated the
generation of the Data Vault
Database
! Left a human being in charge
of the Data Vault design.

100% Reusable ETL
Wisdom never fails

Generated Data Vault
Wisdom never fails

!"#"$%"&'#$
("#")"*+,-&.+

Custom Component
Wisdom never fails

Results
Wisdom never fails

Current record:
5 hours of ETL
in 5 minutes. Build ETL get your
data vault for free

What will we add later?
Wisdom never fails

You wouldn’t know us from our work in Business Intelligence, however today is
about to change that.

While to me today isn’t about me it is all about telling you about what we
at Conspect are doing regarding Data Vault automation, but before we can get
to that point you’ll need to know a little about Conspect, the background our
BI/Data Vault team has and how we feel about this specific area of automation.

I’ll start off with a little bit about the company I work for, we’ll stop ever so
quickly at who I am and then we’ll go have a look at what we’ve automated
and why, that last bit in reverse order the why always goes before the what. And
as a finishing touch a glimpse of the future.

2

We’ve developed a broad skill base as far as tools are concerned, tools are not
sacred, however we have a specialism working with MS tooling

3

2001 started doing business as an ICT consultancy firm, developing Quality
Management and Business Consultancy.
2007 first data vault by our BI Architect
2011 brought the BI team and Conspect together
2011 developed long desired automation tooling for our data vaults under
Conspect R&D

4

I’m not that special and today most certainly isn’t about me. I’m just a BI
Consultant who speaks English reasonably well. However since it would be rude
to talk to you for half an hour without even so much as introducing myself I will
do that. So while the slide take care of this I’ll stand here and be ornamental.

*perform 6 clicks*

5

Next Gen DWH, next in relation to what is currently the vast majority of the
implemented datawarehouses on the market. When we design or build a modern
day BI solution including a datawarehouse we combine, the best of both worlds.
Data Vault is rather exclusively our DWH-ing design method of choice while the
delivery to the business uses Kimball’s methods.

At the heart of our typical Next Generation DWH solution lies the Data Vault.

The reason we prefer using Data Vault over other datawarehouse design methods
boils down to reducing costs for our clients and increasing the predictability of
our BI implementation projects. Reducing TCO by making future customer needs
easier to address is the main selling point. Using automation to reduce TCO
further and to increase predictability is the next logical step for a BI Service
Provider. We don’t datawarehouse unless it’s meant to last.

Since we only have half an hour and this day is all about Data Vault
automation we’ll look at just the part of this picture related to that topic, if
you ever feel like discussing the entire architecture you’re more than
welcome to engage me or a colleague of mine in in-depth discussion. For
now let’s zoom in.

6

Zooming in simplifies the picture, what we’ve got here are the core
components of our Data Vault and the most vital connections to it. These
are the most likely candidates for automation.

I’m grouping together the ETL automation and the automation of the
database it writes to since once you automate the generation of ETL
automating the generation of the database is halfway done already.

Source to staging and the staging database
Staging to data vault and the data vault itself
Data Vault to Business Vault/Data Mart or Staging Out and the Business
Vault/Data Mart or Staging Out Database.

Since Business Vault/Data Mart/Staging Out is quite a mouthful and the
architectural decision between the three is not really our focus for today
we’ll go with Staging Out from this point on. If you have another
preference feel free to mentally substitute Staging Out with your personal
favorite whenever I say it.

7

Now those of you I haven’t lost completely will have noticed one database
obviously missing from this picture. Were I a betting man I’d ask for some
audience participation...

8

Allow for recovering from the shock of being constructively heckled.

The metadata database is rather important in the scheme of automating our
Data Vault environment however generating your metadata database leaves
you metaphorically standing between two automation mirrors. Since we
have to store our rules and directive information for our automation process
somewhere we’ll just do it in one database by hand.

!"#$#%&'%()&*#%+%,&*%-.%/!0%"#$#1%*"-)2"%3-*%+%4-*%-.%!%"+55#3&32%&3%*"#%4#.*%
*6-%5$-7#''#'1%4#*8'%2-%+"#+9%+3%!"#!$"#!%&%!'&()* 2-&32%-3%"#$#:

9

!"#$%&#'()(#*%#+,-#+.#/01#(23()*#+4%&*#')5*5.6#/01#7(#'588#*(88#$%&#')5*5.6#
!"#$%&$&%'()* 4&*#')5*5.6#6%%9#/01#5,#7+)9:#;(#'%&89#3)%4+48$#*(88#$%&#*7+*#
4(<+&,(#'(#+88#7+=(#+#9(,5)(#*%#"((8#6%%9#+4%&*#*7(#'%)-#'(#9%#+.9#3+)*8$#
>&,*#4(<+&,(#5*#!?#*)&(:#;%'(=()#'(@)(#.%*#6%5.6#*%#+&*%A+*(#*7(,(#
4)567*8$#75678567*(9#35(<(,#%"#/01#4(<+&,(#')5*5.6#6%%9#/01#"%)#*7(A#5,#
7+)9B#+&*%A+*5.6#,%A(*75.6#3)%3()8$#"%)#*7(#"5),*#*5A(#5,#+8'+$,#7+)9()#
*7+.#>&,*#9%5.6#*7(#'%)-:

C%B#+,-$+*$./01'.0*$%&$&%'()* 5*#5,#+4%&*#,+=5.6#A%.($#%)#+4%&*#A+-5.6#
A%.($#4$#saving you money:#07)%&67#D&5<-()#9(38%$A(.*B#A%)(#)(85+48(#
)(,&8*,B#)(9&<*5%.#%"#A+5.*(.+.<(#<%,*,#%)#+88#%"#*7(#+4%=(#'%)-:

!"#$%&#'()(#*%#+,-#*7+*#,+A(#/01#(23()*#+4%&*#'7()(#*7(#A+>%)5*$#%"#*7(#
<%,*,#)(8+*(9#*%#/01#85(#7(#'%&89#*(88#$%&#+,#7(#*%89#A(#*7+*#5.#*7(#8%.6#)&.#
*7(#A+>%)5*$#%"#*7(#<%,*,#)(8+*(9#*%#/01#<%A(#")%A#2*+123$.45$2*5*&%64:

0;E0#5,#'7()(#'(#,7%&89#"%<&,#'7(.#+&*%A+*5.6#/01B#*%#A+-(#)('%)-#+.9#
)(9(,56.#(+,5()#+.9F%)#8(,,#3)(=+8(.*:

10

!"#$%&'(()&*#&#+"&,-..-/0'#1&*2,&3*-2%&#(&4"&5*,"&+"6"7
8$5&*&6"9(6#-23&5*2&*#&+"*6#&%(&'"#$%&%""&-.&:"&/*2&";96"%%&#+"&<*'0"&*2,&
"..(6#&-2&#+"&9-/#06"&:"$<"&3(#7&=*'0"&";96"%%",&-2&%->"&*2,&"..(6#&
";96"%%",&-2&/('(67&?+"&5(6"&6",&-#&-%&#+"&5(6"&"..(6#&1(0$6"&3(-23&#(&4"&
";9"2,-23&#(&5*)"&-#&+*99"27

11

A traditional Slide, I was trying to steer clear of these and just go with a dynamic
report style presentation. Didn’t quite work out that way.

I was going to go down the spectrum from red to green but here I stand before a
mostly male audience picking red and green as my reporting colors, let’s try that
again.

12

There we go.

Going down the spectrum from black through orange to blue, we’ll start with the
ETL between our Data Vault and Staging Out and the generation of the resulting
database.

Why is the triangle representing this ETL so big and why is it deep black? It is so
large because there is a great deal of value in automating the application of
business rules to our datawarehouse. And it’s black because it is difficult to
automate and statistically speaking too many people in this room cannot
distinguish green from red.

Let me start by explaining what value I see in automating this part of our data
warehouse architecture?

Well.

For starters business rules change, business needs change, you will need
more information from your datawarehouse as the years go by and business
users discover new questions they want answered.

13

Additionally this is the place where business rules are applied in your ETL. Probably the
most complex ETL you’ll encounter in a data vault DWH as all the transformations
will take place here. Business rules, from how EBIT is calculated to result seeking
algorithms to confidence intervals, you name it and it should, in my professional opinion
be done here.

And lastly, this is the first ROI you’d receive by automating this section and depending
on the business at hand there is potentially a huge load of work to be done here initially as
all their calculations need to be implemented here. There is no customer that hates early
ROI.

If we’re able to automate all of this we’re saving our customers a lot of money and unless
we’re working for charity some of that money will end up with us.

Now to rain on my own parade. The big triangle was rather black, the blackest of black
Visio had to offer without it looking hideous. While going from a hub with its orbiting
satellites to a dimension is a reasonably uniform action going from information from
several different hubs, links and satellites to a fact table is not. (almost) Every business
rule needs to be implemented individually, writing software that interprets a formula, goes
into your datawarehouse and returns it perfectly and in an optimized fashion is a LOT of
work and rather complex work at that. Add to this that you still need to assemble the
formula in the new tool you’ve built to handle it and you’ve basically added a layer of
separation while you still need to perform most of the work.

We’d love to do this but as of today the value a good consultant can add in this regard
over what a piece of software could easily be made to achieve is just too great. As an aside
next year in SQL Denali the Rule Management component for SSIS should be able to
greatly reduce the required effort for automation. Perhaps in a years time you’ll see me
standing here to present our new automation solution for Data to Business Vault.

13

On to the small orange bit of ETL, why is it small and why is it orange?

It is small because once you’ve written the SQL to get your source tables into
your staging that specific bit of EL is not going to change. You might add more
EL for new systems and you might shut down some EL for retired systems and
while this happens on a nearly regular basis there should not be a great deal of
work involved in adding these relatively flat data transfers to your existing
packages.

Now why is it orange? None of those source systems is the same as any of the
other ones (at least usually), an automated process that needs to interpret what
data is in a specific table and how to bring it into staging needs to be smart or it
needs to be fed with inordinate amounts of metadata, as per our large black
triangle defeating the purpose of automating in the first place.

The database itself is reasonably stable as well and while you could generate the
tables based on metadata extracted from your EL there isn’t a great deal of value
in doing so.

Now this is probably the time to mention the raw data vault and futuristic
staging automation possibilities.

14

I’m not going to spend a lot of time talking about Raw data vaulting, in our
architecture there really is no place for something like a raw data vault, a historical
source system takes an inordinate amount of space on your servers whilst not
reducing the need for a true data vault. So while automating a raw data vault is
relatively straightforward we won’t be spending our time trying to create a tool to
facilitate this. Without integration across multiple sources there is no BI value to
making a Raw Data Vault.

The second version of staging automation is the product of nearly artificially
intelligent programming where analysis of the source systems is executed
programmatically and the database is generated from large amounts of metadata
information. It has the same stumbling blocks as automating the EL from the source
to this database. There is some value in performing this automation step, in our
opinion however the value does not weigh up to the required investment.

14

On to our reasonably mid sized blue bit of ETL and the related database our data
vault, why are they blue and why are they the size they are?

This is easily the most worthwhile part of ETL related to our Data Vault, at least
where automation is concerned, changes and additions happen during
implementations as well as during the remaining lifecycle of the datawarehouse.
What makes it the most worthwhile however is the fact that every link, every hub
and every satellite is mechanically identical to every other link, hub and satellite.
This makes loading and generating your data vault possible using just a tiny
amount of metadata.

15

We as a group of Business Intelligence consultants/developers balance between
ambition and pragmatism. We’ve had to make choices as we made our own
implementation of Data Vault automation. I think I’ve done a reasonable job
detailing where we’ve seen the potential profits in automation as well as the
expected investment costs, we have gone forward with this information and
built a tool to save the most effort for the most reasonable investment.

The result of this is our Conspect Datavault Component. I DO have a quote ready
by Dan saying CDC is a critical tool for next generation datawarehousing. I know
it’s Change Data Capture but hey.

And because I feel this is a very important point to get across I’ll include
something we didn’t do on the list of things we did, because we did decide not to
do it. We didn’t take out the Data Vault modeler.

If you were to perform some data mining on the data vault certification
course as it was given to me by Hans. Basket analysis would give you an
interesting correlation to the word ‘it’. The most used word used by Hans
in combination with ‘it’ is the word ‘depends’. ‘It depends on the business.’
was very popular. So creating a data vault is dependent on the wants and
needs of the business. Building a good data vault, one that delivers on the
selling points of the method our client relies on us to deliver, requires

16

critical thought. Even with some of the recent advancements in the field of AI, it
will be a while before determining the needs of the business is something we can
usefully automate.

16

We configure our package once for a data vault. After that just add metadata.

17

Generates the entire data vault database from minimal metadata, tables,
relationships, indices, data types.

18

Entirely built using the most current Microsoft technology.

19

As if an enormous euro sign couldn’t become any more commercial, and oddly I
am not a very commercial minded man, I have some anecdotal results to share
with you.

20

There is a pretty bright future for DV automation, we’re currently looking
towards the partial automation of the application of business rules.

21

Data Vault automation conference - all presentations

More Related Content

What's hot

Viewers also liked

Similar to Data Vault automation conference - all presentations

More from Prudenza B.V

Recently uploaded

Data Vault automation conference - all presentations