Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Conducting Experiments in Software Industry

477 views

Published on

What use are experiments for industry?
1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions
2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Conducting Experiments in Software Industry

  1. 1. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid
  2. 2. FiDiProFinland Distinguished Professor Program Project start, January 2013 Experimental Software Engineering Industrial Laboratory (ESEIL)
  3. 3. MISSION !"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)% +/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15% % 6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$+&&% "7*.)%)(+%)(%*)$&%((%*8%+/0+1#2+$)&%% 9.&(%8*14"1-%)(+%%+,%-./%&0#$1,#-#2.'/1 "00:#+-%)*%3&45$%2'%1'%&%-#64&1"7*.)% &*34"1+%-+;+:*02+$)%
  4. 4. !"#$%#&'$()*$'+$,-.$*+/0&#12$ 3&$'+$1+&#$*+/0&#12$3-$'"#$ &#0435#$+2$+'"#0& 6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$ :#-#0)'#$"3:"$;/)13'*$$ 8-+(1#.:#$ <+-./59-:$#=>#037#-'&$%03-:&$'"#$ &53#-9,5$7#'"+.$'+$?+@()0#$ A-:3-##03-:$
  5. 5. !"#$%&'$()*+&*+),$+-(./+01/++ )-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+ 8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+ ;$,&(9+),$+;$,12&-%+0$+-;*$%2$+ <9$(7=/+!"#$"%&'(+#.1/&(:+1+%-.$++ &(+*->01%$+9$2$.-#'$()+1(9+),$++ )*++'),*+(+;$)0$$(+),$'+
  6. 6. !"#$"%%&$"#'($)*$+,$"%)' !"#$"%&'($)(*+(),-.%/$0&"s and market-speak to 1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$ Fact-based management -*$%"./*0,,1'20,$('/"($"#)' )345,('3%,+'64'$(%".71' 5))4&+"0&4&6#$+-.'&,-.%/$7("8.(%%(%$".'$%6*(.96:%$ -2$%-;7"*($'(<(4-)3(.6$6(+:.-4-9&(%$ $ =10>(+,<($-)&.&-.%$+".$4("'$6-$3&%6"8(.$ )(*+(),-.%$-2$*("4&6#$
  7. 7. What Use Are Experiments for Industry? !"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+ )$56(-1-4&$*+ 7  !"#$%&'$()*+#%-2&8$+,(+-9:$532$+5-'#,%&*-(+%,)6$%+)6,(+*09:$532$+ -#&(&-(*+ ;6$+%$*01)*+-/+$"#$%&'$()*+*6-018+,11-.+&(80*)%<+)-+ 7  =,*$+8$5&*&-(*+-(+-9:$532$+4%-0(8*+ >  ?$*01)*+/%-'+,(+$"#$%&'$()+,%$+2,1&8+,)+5-'#,(<+1$2$1+ 7  @(5%$,*$+5-()%-1+-(+*-A.,%$+8$2$1-#'$()+ >  @8$(3/<+%$1$2,()+2,%&,91$*+-/+*-A.,%$+)$56(-1-4&$*+ 1. 2.
  8. 8. What an Experiment Is Not !"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$ 1*%"$-%"#&$,*'*/2&$%#&',$ $ 3+#4,"-%$#()#&*#+5#$ 6  78$-+5#$1#+%$92//--+*+:;$ 6  <2++-%$9#$:#+#&2/*=#0$%-$-%"#&$02>,?$-%"#&$)#-)/#$ @.9A#5BC#$&#)#2%#0$#()#&*#+5#$ 6  7D2./$"2,$2$/-%$-E$#()#&*#+5#$'2+2:*+:$)&-A#5%,;$ 6  <2+$:#+#&2%#$&./#,4-E4%".'9,?$*,$5-+F+#0$%-$D2./?$*,$,.9A#5BC#$ <-//#5%#0$02%2$E&-'$)&-A#5%,$ 6  73.&$0#)2&%'#+%$,%-&#,$02%2$E&-'$2//$)&-A#5%,;$ 6  8+$&#2/$)&-A#5%,?$C2&*29/#,$52++-%$9#$5-+%&-//#0$2,$+#5#,,2&>$%-$.+0#&,%2+0$1"2%$%"#$ 52.,#,$2+0$#G#5%,$2&#$$
  9. 9. A SE Experiment Task i A B M1 M1 Task i M2 M2 Software process Software process Task i Task i Task i Task i
  10. 10. What is Needed to Conduct an Experiment? !  "#$%&%'()%*+# ,  -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+# !  6&%*#780#29#)(++24'%# !  "#$%&%'()%*+#5:$%*#1.%#+3/%#;(:$2<(:+# ,  =(:>:%$#2:#3#*((/0#5+2:?#+3/%#2:9*3+1*5;15*%0#@# !  "#$%&%'()%*+#$(2:?#1.%#+3/%#13+A#5:$%*#+15$B# ,  C#13+A#1B)2;3''B#13A%+#31#'%3+1#1D(#.(5*+#1(#;(/)'%1%# !  E%)%:$2:?#(:#1.%#%F)%*2/%:13'#$%+2?:#D%#;3:#+)'21#)3*<;2)3:1+#(*#%&%*B4($B# $(2:?#21#1D2;%#
  11. 11. Train people in exchange Researchers provide free training Companies provide participants for the experiment
  12. 12. Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Reminder of Unit Testing concepts 3 Exercises to warm up CONTROL TASK (Test-last) DAY 2 Test-driven Development 3 TDD Exercises DAY 3 2 TDD Exercises TREATMENT TASK (TDD)
  13. 13. Quality @ F-Secure TDD Experiments
  14. 14. •  The mean for ITL is 54% of correctly implemented user stories •  The mean for TDD is 76% •  Subjects applying TDD get a 17% improvement in quality compared to subjects applying ITL Quality @ F-Secure TDD Experiments •  The observed difference between ITL and TDD is not significant EFFECT SIZE HYPOTHESIS TESTING
  15. 15. Productivity @ F-Secure TDD Experiments
  16. 16. •  The mean for ITL is 23% •  The mean for TDD is greater at 48% •  Subjects applying TDD show a 24% improvement in productivity compared to subjects applying ITL •  The observed differences between ITL and TDD is significant Productivity @ F-Secure TDD Experiments
  17. 17. Other Data Analysis
  18. 18. Dissecting TDD: Post-hoc analysis !  "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%, /)8'*#+,*37&96%$,:8)2*(;, <  =)#9,/;/2%', <  0%'(,>&'(?,(.%,/9$%, <  @%A)/(9&*#+,)B%&,&)(.%&,(.)#,$%'*+#,>&'(, !  =)#9C$%6%2973%#(,.)',39'(,4%)&*#+,9#,*37&96%$,:8)2*(;,,, <  0*#;,DC3*#8(%,$%6%2973%#(,/;/2%', <  E,8#*(,*#/&%)'%,*#,:8)2*(;,A9&,%6%&;,FGDH,&%$8/59#,*#,/;/2%,'*I%, !  =%%$,(9,4%,/9#>&3%$,-*(.,)#,%J7%&*3%#(K,
  19. 19. Schedule: Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Testing tool Concepts 2 Tool Exercises 1 Motivational exercise (ITL) BASELINE TASK (Your Way) DAY 2 Slicing 2 Slicing Exercises CONTROL TASK (ITL) DAY 3 Test-driven Development 3 TDD Exercises TREATMENT TASK (TDD)
  20. 20. Replications of Our TDD Experiment !"#$%&'$()*+,-$.&/(,01,#%$2#3.),, 4  5*6*,7,58(&),.$9(/, :  ;2<$=>%$,?@,.&)$.A,2B,CD,#*%E=&#*()., :  8FG,?<#*&(A,2B,@0,#*%E=&#*()., 4  HII,7,JK$.),.$9(/, :  !+$L)%3M&),2B,N,#*%E=&#*()., !"#$%&'$()*+,-$.&/(,C1,=%3..,7,.+&=&(/,)%*&(&(/, 4  5*6*,7,58(&),.$9(/, :  H3'#*(O,P,?!.)3(&*A, :  8FQ,?<#*&(A, :  !<F!,?!=>*R3%A, !"#$%&'$()*+,-$.&/(,@1,!-C,7,S3>%T*O,=3()%3+,)%$*)'$(),, 4  HII,7,U(O,.$9(/, :  !%&=..3(, 4  FO)V3(,7,WW, :  ;2<$=>%$,
  21. 21. Quality: TL vs TDD F-Secure Elektrobit Univ. Politecnica Valencia TL < TDD (42%) Java 31 Students No need more subjects TL = TDD (32%) C++ 9 Professionals Need more subjects TL = TDD (17%) Java 24 Professionals No need more subjects
  22. 22. Productivity: ITL vs TDD ITL < TDD (41%) Java 31 Students No need of more subjects ITL < TDD (21%) C++ 9 Professionals No need of more subjects ITL < TDD (24%) Java 24 Professionals No need of more subjects F-Secure Elektrobit Univ. Politecnica Valencia
  23. 23. !"#$$%$&'#$%()*+,+-'+.% /)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<% =>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(% 6,&(4'@5)28% A&-B.)0-)C'#-2%()*+,+-'+.%9&,%?4#$)28% !)0-)C'#-2%()*+,+-'+.%9&,%6,&(4'@5)28% % !2#3$+%,+.4$2.%9&,%()*+,+-2%)-9,#.2,4'24,+.% D#5#%E%FGG% DH-)2%E%I2+.2% %
  24. 24. We Are Now Recruiting ! !  "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2#&#12#3$*+$&#.#*4*+,$(5&$.(5&1#$ (+$677$ 8  "#$9*''$&5+$2:#$#;0#&*/#+2$%1$0%&2$(-$2:#$2&%*+*+,$ <&##$=$3%>1$2&%*+*+,$(+$677$ ?(5&$'(.%'$#;0#&*/#+2$&#15'21$ @*3#$50$2($<%.2AB%1#3$C%+%,#/#+2$ natalia.juristo@oulu.fi
  25. 25. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid Email: natalia@fi.upm.es

×