Conducting Experiments in Software Industry

398 views

Published on

What use are experiments for industry?
1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions
2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development

Published in: Engineering
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
398
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Conducting Experiments in Software Industry

  1. 1. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid
  2. 2. FiDiProFinland Distinguished Professor Program Project start, January 2013 Experimental Software Engineering Industrial Laboratory (ESEIL)
  3. 3. MISSION !"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)% +/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15% % 6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$+&&% "7*.)%)(+%)(%*)$&%((%*8%+/0+1#2+$)&%% 9.&(%8*14"1-%)(+%%+,%-./%&0#$1,#-#2.'/1 "00:#+-%)*%3&45$%2'%1'%&%-#64&1"7*.)% &*34"1+%-+;+:*02+$)%
  4. 4. !"#$%#&'$()*$'+$,-.$*+/0&#12$ 3&$'+$1+&#$*+/0&#12$3-$'"#$ &#0435#$+2$+'"#0& 6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$ :#-#0)'#$"3:"$;/)13'*$$ 8-+(1#.:#$ <+-./59-:$#=>#037#-'&$%03-:&$'"#$ &53#-9,5$7#'"+.$'+$?+@()0#$ A-:3-##03-:$
  5. 5. !"#$%&'$()*+&*+),$+-(./+01/++ )-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+ 8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+ ;$,&(9+),$+;$,12&-%+0$+-;*$%2$+ <9$(7=/+!"#$"%&'(+#.1/&(:+1+%-.$++ &(+*->01%$+9$2$.-#'$()+1(9+),$++ )*++'),*+(+;$)0$$(+),$'+
  6. 6. !"#$"%%&$"#'($)*$+,$"%)' !"#$"%&'($)(*+(),-.%/$0&"s and market-speak to 1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$ Fact-based management -*$%"./*0,,1'20,$('/"($"#)' )345,('3%,+'64'$(%".71' 5))4&+"0&4&6#$+-.'&,-.%/$7("8.(%%(%$".'$%6*(.96:%$ -2$%-;7"*($'(<(4-)3(.6$6(+:.-4-9&(%$ $ =10>(+,<($-)&.&-.%$+".$4("'$6-$3&%6"8(.$ )(*+(),-.%$-2$*("4&6#$
  7. 7. What Use Are Experiments for Industry? !"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+ )$56(-1-4&$*+ 7  !"#$%&'$()*+#%-2&8$+,(+-9:$532$+5-'#,%&*-(+%,)6$%+)6,(+*09:$532$+ -#&(&-(*+ ;6$+%$*01)*+-/+$"#$%&'$()*+*6-018+,11-.+&(80*)%<+)-+ 7  =,*$+8$5&*&-(*+-(+-9:$532$+4%-0(8*+ >  ?$*01)*+/%-'+,(+$"#$%&'$()+,%$+2,1&8+,)+5-'#,(<+1$2$1+ 7  @(5%$,*$+5-()%-1+-(+*-A.,%$+8$2$1-#'$()+ >  @8$(3/<+%$1$2,()+2,%&,91$*+-/+*-A.,%$+)$56(-1-4&$*+ 1. 2.
  8. 8. What an Experiment Is Not !"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$ 1*%"$-%"#&$,*'*/2&$%#&',$ $ 3+#4,"-%$#()#&*#+5#$ 6  78$-+5#$1#+%$92//--+*+:;$ 6  <2++-%$9#$:#+#&2/*=#0$%-$-%"#&$02>,?$-%"#&$)#-)/#$ @.9A#5BC#$&#)#2%#0$#()#&*#+5#$ 6  7D2./$"2,$2$/-%$-E$#()#&*#+5#$'2+2:*+:$)&-A#5%,;$ 6  <2+$:#+#&2%#$&./#,4-E4%".'9,?$*,$5-+F+#0$%-$D2./?$*,$,.9A#5BC#$ <-//#5%#0$02%2$E&-'$)&-A#5%,$ 6  73.&$0#)2&%'#+%$,%-&#,$02%2$E&-'$2//$)&-A#5%,;$ 6  8+$&#2/$)&-A#5%,?$C2&*29/#,$52++-%$9#$5-+%&-//#0$2,$+#5#,,2&>$%-$.+0#&,%2+0$1"2%$%"#$ 52.,#,$2+0$#G#5%,$2&#$$
  9. 9. A SE Experiment Task i A B M1 M1 Task i M2 M2 Software process Software process Task i Task i Task i Task i
  10. 10. What is Needed to Conduct an Experiment? !  "#$%&%'()%*+# ,  -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+# !  6&%*#780#29#)(++24'%# !  "#$%&%'()%*+#5:$%*#1.%#+3/%#;(:$2<(:+# ,  =(:>:%$#2:#3#*((/0#5+2:?#+3/%#2:9*3+1*5;15*%0#@# !  "#$%&%'()%*+#$(2:?#1.%#+3/%#13+A#5:$%*#+15$B# ,  C#13+A#1B)2;3''B#13A%+#31#'%3+1#1D(#.(5*+#1(#;(/)'%1%# !  E%)%:$2:?#(:#1.%#%F)%*2/%:13'#$%+2?:#D%#;3:#+)'21#)3*<;2)3:1+#(*#%&%*B4($B# $(2:?#21#1D2;%#
  11. 11. Train people in exchange Researchers provide free training Companies provide participants for the experiment
  12. 12. Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Reminder of Unit Testing concepts 3 Exercises to warm up CONTROL TASK (Test-last) DAY 2 Test-driven Development 3 TDD Exercises DAY 3 2 TDD Exercises TREATMENT TASK (TDD)
  13. 13. Quality @ F-Secure TDD Experiments
  14. 14. •  The mean for ITL is 54% of correctly implemented user stories •  The mean for TDD is 76% •  Subjects applying TDD get a 17% improvement in quality compared to subjects applying ITL Quality @ F-Secure TDD Experiments •  The observed difference between ITL and TDD is not significant EFFECT SIZE HYPOTHESIS TESTING
  15. 15. Productivity @ F-Secure TDD Experiments
  16. 16. •  The mean for ITL is 23% •  The mean for TDD is greater at 48% •  Subjects applying TDD show a 24% improvement in productivity compared to subjects applying ITL •  The observed differences between ITL and TDD is significant Productivity @ F-Secure TDD Experiments
  17. 17. Other Data Analysis
  18. 18. Dissecting TDD: Post-hoc analysis !  "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%, /)8'*#+,*37&96%$,:8)2*(;, <  =)#9,/;/2%', <  0%'(,>&'(?,(.%,/9$%, <  @%A)/(9&*#+,)B%&,&)(.%&,(.)#,$%'*+#,>&'(, !  =)#9C$%6%2973%#(,.)',39'(,4%)&*#+,9#,*37&96%$,:8)2*(;,,, <  0*#;,DC3*#8(%,$%6%2973%#(,/;/2%', <  E,8#*(,*#/&%)'%,*#,:8)2*(;,A9&,%6%&;,FGDH,&%$8/59#,*#,/;/2%,'*I%, !  =%%$,(9,4%,/9#>&3%$,-*(.,)#,%J7%&*3%#(K,
  19. 19. Schedule: Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Testing tool Concepts 2 Tool Exercises 1 Motivational exercise (ITL) BASELINE TASK (Your Way) DAY 2 Slicing 2 Slicing Exercises CONTROL TASK (ITL) DAY 3 Test-driven Development 3 TDD Exercises TREATMENT TASK (TDD)
  20. 20. Replications of Our TDD Experiment !"#$%&'$()*+,-$.&/(,01,#%$2#3.),, 4  5*6*,7,58(&),.$9(/, :  ;2<$=>%$,?@,.&)$.A,2B,CD,#*%E=&#*()., :  8FG,?<#*&(A,2B,@0,#*%E=&#*()., 4  HII,7,JK$.),.$9(/, :  !+$L)%3M&),2B,N,#*%E=&#*()., !"#$%&'$()*+,-$.&/(,C1,=%3..,7,.+&=&(/,)%*&(&(/, 4  5*6*,7,58(&),.$9(/, :  H3'#*(O,P,?!.)3(&*A, :  8FQ,?<#*&(A, :  !<F!,?!=>*R3%A, !"#$%&'$()*+,-$.&/(,@1,!-C,7,S3>%T*O,=3()%3+,)%$*)'$(),, 4  HII,7,U(O,.$9(/, :  !%&=..3(, 4  FO)V3(,7,WW, :  ;2<$=>%$,
  21. 21. Quality: TL vs TDD F-Secure Elektrobit Univ. Politecnica Valencia TL < TDD (42%) Java 31 Students No need more subjects TL = TDD (32%) C++ 9 Professionals Need more subjects TL = TDD (17%) Java 24 Professionals No need more subjects
  22. 22. Productivity: ITL vs TDD ITL < TDD (41%) Java 31 Students No need of more subjects ITL < TDD (21%) C++ 9 Professionals No need of more subjects ITL < TDD (24%) Java 24 Professionals No need of more subjects F-Secure Elektrobit Univ. Politecnica Valencia
  23. 23. !"#$$%$&'#$%()*+,+-'+.% /)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<% =>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(% 6,&(4'@5)28% A&-B.)0-)C'#-2%()*+,+-'+.%9&,%?4#$)28% !)0-)C'#-2%()*+,+-'+.%9&,%6,&(4'@5)28% % !2#3$+%,+.4$2.%9&,%()*+,+-2%)-9,#.2,4'24,+.% D#5#%E%FGG% DH-)2%E%I2+.2% %
  24. 24. We Are Now Recruiting ! !  "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2#&#12#3$*+$&#.#*4*+,$(5&$.(5&1#$ (+$677$ 8  "#$9*''$&5+$2:#$#;0#&*/#+2$%1$0%&2$(-$2:#$2&%*+*+,$ <&##$=$3%>1$2&%*+*+,$(+$677$ ?(5&$'(.%'$#;0#&*/#+2$&#15'21$ @*3#$50$2($<%.2AB%1#3$C%+%,#/#+2$ natalia.juristo@oulu.fi
  25. 25. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid Email: natalia@fi.upm.es

×