Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Where User Experience And Software Engineering Meet

593 views

Published on

My talk at CHOOSE 2009 in Bern, Switzerland.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Where User Experience And Software Engineering Meet

  1. 1. where  user  experience  and software  engineering  meet Andrew  J.  Ko Monday, November 30, 2009
  2. 2. in  method,  I’m  an  HCI  researcher I  study   problems I  evaluate   I  design   solutions solutions 2 Monday, November 30, 2009
  3. 3. in  method,  I’m  an  HCI  researcher I  study   problems I  evaluate   I  design   solutions solutions 2 Monday, November 30, 2009
  4. 4. in  topic,  I’m  an  SE  researcher I  want  to  make  it  easy  to   create  useful,  usable  software that  empowers   and  enriches   users’  lives 3 Monday, November 30, 2009
  5. 5. human-­‐computer   software   interaction engineering me 4 Monday, November 30, 2009
  6. 6. human-­‐computer   software   interaction engineering how  do  we  get  the   how  do  get  the   right  design? design  right ? (the  lives  of  users) (the  lives  of  developers) 5 Monday, November 30, 2009
  7. 7. user-­‐centered   software  evolution how  do  we  get  the right  design right? design   6 Monday, November 30, 2009
  8. 8. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? (#$!!!" (!$!!!" debugging  with  the  Whyline !"#$%&'(, '$!!!" 9/829?/9B" how  can  users  help  software  evolution? &$!!!" !"#$%&'(+ !"#$%&'(- 5B/9B" %$!!!" !"#$%&'()*+ *>C3/" #$!!!" !" >29/" recent  work  in  leveraging  the  crowd )*+",-'" )*+",!(" ./0",--" ./0",!#" 489",!%" 489",!A" =>?",!<" =>?",!'" :5@",!&" 123",--" 123",!#" :*;",!<" :*;",!'" 456",!!" 456",!7" 7 Monday, November 30, 2009
  9. 9. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? debugging  with  the  Whyline !"#$%&'()*+ !"#$% how  can  users  help  software  evolution? recent  work  in  leveraging  the  crowd )*+",!(" ./0",!#" 489",!%" 123",!#" 456",!!" 456",!7" 8 Monday, November 30, 2009
  10. 10. information  needs  at  Microsoft with  the  Human  Interactions  in  Programming  group  at  Microsoft  Research observed  25  hours  of  coding  and  bug   fixing,  in  the  role  of  “new  hires” 357  pages  of  handwritten  notes 4,231  events  in   an  spreadsheet 9 Monday, November 30, 2009
  11. 11. 17  developers   hard  at  work   across  25  hours Visual  Studio Windows  Vista service  packs mobile  devices discussion  boards educational  tools SQL  server MS  Office Encryption DRM ... 10 Monday, November 30, 2009
  12. 12. 8  activities writing  code submitting  code triaging  bugs reproducing  a  failure understanding  behavior reasoning  about  design maintaining  awareness non-­‐work  activity 11 Monday, November 30, 2009
  13. 13. 8  activities writing  code submitting  code triaging  bugs reproducing  a  failure understanding  behavior reasoning  about  design maintaining  awareness non-­‐work  activity 11 Monday, November 30, 2009
  14. 14. 9  reasons  for   switching  tasks face  to  face  conversation phone  call instant  message e-­‐mail  alerts bug  report  change  alerts task  avoidance getting  blocked meetings task  completion 12 Monday, November 30, 2009
  15. 15. 9  reasons  for   switching  tasks face  to  face  conversation phone  call instant  message e-­‐mail  alerts bug  report  change  alerts task  avoidance getting  blocked meetings task  completion 12 Monday, November 30, 2009
  16. 16. software   interrupted  every   development   ~5-­‐10  minutes work  is  highly   fragmented blocked  every   ~10  minutes 13 Monday, November 30, 2009
  17. 17. next,  we  looked   for  information   that  developers   sought  to  get   their  work   done  ... 14 Monday, November 30, 2009
  18. 18. next,  we  looked   for  information   that  developers   sought  to  get   their  work   done  ... 14 Monday, November 30, 2009
  19. 19. 21  information  needs  observed  (by  frequency) what  have  my  coworkers  been  doing?   what  code  could  have  caused  this  behavior?   have  resources  I  depend  on  changed?   what  code  caused  this  program  state? how  do  I  use  this  data  structure  or  function? did  I  make  any  mistakes? what  is  the  program  supposed  to  do? in  what  situations  does  this  failure  occur? is  this  problem  worth  fixing? why  was  this  code  implemented  this  way? what’s  statically  related  to  this  code? what  are  the  implications  of  this  change? 15 Monday, Novemberdoes  the  failure  look  like? what   30, 2009
  20. 20. 5  information  needs  least  often  satisfied what  have  my  coworkers  been  doing?   %  unsatisfied what  code  could  have  caused  this  behavior?   36% have  resources  I  depend  on  changed?   what  code  caused  this  program  state? 61% how  do  I  use  this  data  structure  or  function? did  I  make  any  mistakes? what  is  the  program  supposed  to  do? 15% in  what  situations  does  this  failure  occur? 41% is  this  problem  worth  fixing? why  was  this  code  implemented  this  way? 44% what’s  statically  related  to  this  code? what  are  the  implications  of  this  change? 16 Monday, Novemberdoes  the  failure  look  like? what   30, 2009
  21. 21. 3  were  debugging  related  ... what  have  my  coworkers  been  doing?   what  code  could  have  caused  this  behavior? reproducing,   have  resources  I  depend  on  changed?   diagnosing,   what  code  caused  this  program  state? and  scoping   how  do  I  use  this  data  structure  or  function? failures  were   did  I  make  any  mistakes? the  most  time-­‐ what  is  the  program  supposed  to  do? consuming   activities in  what  situations  does  this  failure  occur? is  this  problem  worth  fixing? why  was  this  code  implemented  this  way? what’s  statically  related  to  this  code? what  are  the  implications  of  this  change? 17 Monday, Novemberdoes  the  failure  look  like? what   30, 2009
  22. 22. 2  were  design  related what  have  my  coworkers  been  doing?   what  code  could  have  caused  this  behavior? have  resources  I  depend  on  changed?   what  code  caused  this  program  state?   how  do  I  use  this  data  structure  or  function? did  I  make  any  mistakes? what  is  the  program  supposed  to  do? knowing  what   in  what  situations  does  this  failure  occur? software   is  this  problem  worth  fixing? should  do  was   rarely  known why  was  this  code  implemented  this  way? what’s  statically  related  to  this  code? what  are  the  implications  of  this  change? 18 Monday, Novemberdoes  the  failure  look  like? what   30, 2009
  23. 23. software  development  is  tacit plans  and  specifications  are  unwritten developers  have  to   communicate  to  make  progress 19 Monday, November 30, 2009
  24. 24. software  quality  depends  highly  on  the  quality  of   human  communication  and  cognition these  human  activities  are  faulty  and  unreliable 20 Monday, November 30, 2009
  25. 25. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? debugging  with  the  Whyline !"#$%&'()*+ !"#$% how  can  users  help  software  evolution? recent  work  in  leveraging  the  crowd )*+",!(" ./0",!#" 489",!%" 123",!#" 456",!!" 456",!7" 21 Monday, November 30, 2009
  26. 26. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? debugging  with  the  Whyline !"#$%&'()*+ !"#$% how  can  users  help  software  evolution? recent  work  in  leveraging  the  crowd )*+",!(" ./0",!#" 489",!%" 123",!#" 456",!!" 456",!7" 21 Monday, November 30, 2009
  27. 27. why  is  debugging  such  a  challenge? wrong! observe   guess  the   check  the   solved! failure cause guess 22 Monday, November 30, 2009
  28. 28. Whyline a  Workspace  for  Helping  You  Link  Instructions,  Numbers,  and  Events observe   click  the   follow   solved! failure faulty  output dependencies 23 Monday, November 30, 2009
  29. 29. an  example  ... why  was  the  line  black? 24 Monday, November 30, 2009
  30. 30. an  example  ... why  was  the  line  black? 24 Monday, November 30, 2009
  31. 31. record  the  problem 25 Monday, November 30, 2009
  32. 32. record  the  problem 25 Monday, November 30, 2009
  33. 33. load  the  recording 26 Monday, November 30, 2009
  34. 34. why  was  the  line  color  black? 27 Monday, November 30, 2009
  35. 35. why  was  the  line  color  black? 27 Monday, November 30, 2009
  36. 36. why  was  the  line  color  black? 28 Monday, November 30, 2009
  37. 37. code executions  of  code (execution  events) why  was  the  line  color  black? 28 Monday, November 30, 2009
  38. 38. why  was  the  line  color  black? 29 Monday, November 30, 2009
  39. 39. followup   questions  about   selected  event why  was  the  line  color  black? 29 Monday, November 30, 2009
  40. 40. followup   questions  about   selected  event why  was  the  line  color  black? 29 Monday, November 30, 2009
  41. 41. selected   dependency   highlighted   in  source why  was  the  line  color  black? 29 Monday, November 30, 2009
  42. 42. why  was  the  line  color  black? 30 Monday, November 30, 2009
  43. 43. because  gSlider   was  used  twice,   ignoring  bSlider why  was  the  line  color  black? 30 Monday, November 30, 2009
  44. 44. a  comparison  study vs Whyline   control group   group both  groups  had  modern  IDE  features    show  declaration,  show  callers,  show  references,  etc. 31 Monday, November 30, 2009
  45. 45. subject  program ArgoUML,  an  open  source   software  design  tool ~150,000  lines  of  code 22  external  libraries chose  two  bug  reports   from  version  18.1 ■ one  w/  simple  fix ■ one  w/  complex  fix 32 Monday, November 30, 2009
  46. 46. bug  1  results whyline control #  successful minutes 10 30 8 24 6 18 4 12 2 6 0 0 p<.05 p<.05 more  successful  in  half  the  time 33 Monday, November 30, 2009
  47. 47. bug  2  results whyline control #  successful minutes 10 10 8 4  of  10 8 6 gave  up 6 4 4 2 2 0 0 p<.05 more  successful  in  the  same  time 34 Monday, November 30, 2009
  48. 48. unsolicited  quotes  from  users “This  is  great,  when  can  I  get  this  for  C?” “My  god,  this  is  so  cool.” “It's  so  nice  and  straight  and  simple...” 35 Monday, November 30, 2009
  49. 49. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? debugging  with  the  Whyline !"#$%&'()*+ !"#$% how  can  users  help  software  evolution? recent  work  in  leveraging  the  crowd )*+",!(" ./0",!#" 489",!%" 123",!#" 456",!!" 456",!7" 36 Monday, November 30, 2009
  50. 50. talk  outline why  is  software  evolution  difficult? a  study  of  information  needs  at  Microsoft how  can  tools  help  software  evolution? (#$!!!" (!$!!!" debugging  with  the  Whyline !"#$%&'(, '$!!!" 9/829?/9B" how  can  users  help  software  evolution? &$!!!" !"#$%&'(+ !"#$%&'(- 5B/9B" %$!!!" !"#$%&'()*+ *>C3/" #$!!!" !" >29/" recent  work  in  leveraging  the  crowd )*+",-'" )*+",!(" ./0",--" ./0",!#" 489",!%" 489",!A" =>?",!<" =>?",!'" :5@",!&" 123",--" 123",!#" :*;",!<" :*;",!'" 456",!!" 456",!7" 36 Monday, November 30, 2009
  51. 51. does  open  bug  reporting  work? with  my  PhD  student,  Parmit  Chilana comprehensive  analysis  of  ~500,000   reports  from  the  Mozilla  community ■ quantitative  analysis  of  to  characterize  bug   report  resolution  trends ■ qualitative  analysis  to  explain  bug  report   resolution  trends 37 Monday, November 30, 2009
  52. 52. four  types  of  contributors 1% CORE  devs  drivers,  super  reviewers,  module  owners,  peers 1% ACTIVE  devs  assigned  bug  reports 80% REPORTERS 18% USERS reported  and   only  commented   commented  on   on  bug  reports bug  reports 38 Monday, November 30, 2009
  53. 53. #  of  comment  contributors  over  time (#$!!!" !"#$%&'(, (!$!!!" '$!!!" 9/829?/9B" &$!!!" !"#$%&'(+ !"#$%&'(- 5B/9B" %$!!!" !"#$%&'()*+ *>C3/" #$!!!" !" >29/" )*+",-'" )*+",!(" ./0",--" ./0",!#" 489",!%" 489",!A" =>?",!<" =>?",!'" :5@",!&" 123",--" 123",!#" :*;",!<" :*;",!'" 456",!!" 456",!7" REPORTERs  are  the  most  active  commenters 39 Monday, November 30, 2009
  54. 54. #  of  comment  contributors  over  time (#$!!!" !"#$%&'(, (!$!!!" '$!!!" 9/829?/9B" &$!!!" !"#$%&'(+ !"#$%&'(- 5B/9B" %$!!!" !"#$%&'()*+ *>C3/" #$!!!" !" >29/" )*+",-'" )*+",!(" ./0",--" ./0",!#" 489",!%" 489",!A" =>?",!<" =>?",!'" :5@",!&" 123",--" 123",!#" :*;",!<" :*;",!'" 456",!!" 456",!7" REPORTERs  are  the  most  active  commenters 39 Monday, November 30, 2009
  55. 55. resolution  by  reporter  type $!!"# ,!"# +!"# 67/89321.# *!"# )!"# 9:30;/8.1.# (!"# <0:=9># '!"# <0-?@A0-;.# &!"# %!"# 9:52896# $!"# B>.6# !"# -./0-1.-# 2345.# 30-.# most  REPORTER  reports  are  not  FIXED 40 Monday, November 30, 2009
  56. 56. resolution  by  reporter  type $!!"# ,!"# +!"# 67/89321.# *!"# )!"# 9:30;/8.1.# (!"# <0:=9># '!"# <0-?@A0-;.# &!"# %!"# 9:52896# $!"# B>.6# !"# -./0-1.-# 2345.# 30-.# most  REPORTER  reports  are  not  FIXED 40 Monday, November 30, 2009
  57. 57. %  reports  FIXED  by  each  type (!!"# !"#$%&'()*+ !"#$%&'(, !"#$%&'(- '!"# &!"# !"#$%&'(+ 07+/# %!"# $!"# *0A8/# !"# +/27+B/+# )*+#,-'# ./0#,-'# =/>#,!%# =/>#,!@# 1/2#,--# )*+#,!(# 94:#,!?# 94:#,!'# 678#,!(# 678#,!@# 678#,!%# 94:#,!$# 345#,!!# )*;#,!<# REPORTERs  have  dropped  in  effectiveness )*;#,!&# 41 Monday, November 30, 2009
  58. 58. %  reports  FIXED  by  each  type (!!"# !"#$%&'()*+ !"#$%&'(, !"#$%&'(- '!"# &!"# !"#$%&'(+ 07+/# %!"# $!"# *0A8/# !"# +/27+B/+# )*+#,-'# ./0#,-'# =/>#,!%# =/>#,!@# 1/2#,--# )*+#,!(# 94:#,!?# 94:#,!'# 678#,!(# 678#,!@# 678#,!%# 94:#,!$# 345#,!!# )*;#,!<# REPORTERs  have  dropped  in  effectiveness )*;#,!&# 41 Monday, November 30, 2009
  59. 59. why  are  REPORTERs  ineffective? sampled  and  categorized  100  reports  of  each   resolution  type... most  REPORTER  reports  =  technical  support  for   power  users’  tinkering  and  using  old  builds rarely  provided  static  and  dynamic  context   adequate  to  reproduce  problems reported  problems,  resolved  shortly  after   42 Monday, November 30, 2009
  60. 60. is  open  bug  reporting  useful? yes,  but  ... ■ significant  overhead  to  process  bad  reports ■ only  a  skewed  subset  of  users  report  bugs ■ users  who  report  bugs  are  bad  at  providing   the  static  and  dynamic  context  of  problems ■ text  is  a  terribly  imprecise  medium  for   expressing  this  context 43 Monday, November 30, 2009
  61. 61. ongoing  work enabling  every  user  to  submit precise structured aggregatable bug  reports with  zero  training? 44 Monday, November 30, 2009
  62. 62. three  takeaway  points software  quality  depends  highly  on  the  quality   of  human  communication  and  cognition human  communication  and  cognition  are   faulty  and  unreliable carefully  designed  interactive  tools  can   compensate  for  these  limitations questions? 45 Monday, November 30, 2009

×