Evaluating and Improving Software Usability


Published on

Presented at Software Test Professionals, New Orleans 2012;
Today’s web-based applications (WebApps) containing complex business logic and which are sometimes critical to operating the business, now must have an increased focus on usability as well as the newer and broader term, user experience. Especially with SaaS based business models where users can switch applications at a heartbeat and pay by the month (or even a free trial), both usability and user experience become paramount as there is no up-front investment forcing a user to stay. The ISO 25010 standard describes a model for general usage in specifying and evaluating software quality requirements, one of which is usability, but there are no models or formal definitions for user experience, nor its relation to usability. For usability, ISO 25010 is intended as a general guideline to be adapted based on a specific context and lacks implementation specifics. UX, on the other hand, does not have any formal standard definition, although some models have been developed regarding its elements. Models and research have been used mostly for the purpose of understanding, rather than evaluating improvements. In this session, we draw relationships between usability and user experience and explore measurement and evaluation methods that can be used as the first step toward improvement.

Published in: Technology

Evaluating and Improving Software Usability

  1. 1. Evaluating and Improving Software Usability 902 : Thursday, 9:30am - 10:45am Philip Lew www.xbosoft.com
  2. 2. Understand,  Evaluate  and   Improve   2  
  3. 3. Agenda  •  Introduc7on   –  Importance  of  usability   –  What  is  usability?   –  What  is  UX?  •  Usability  Modeling  and  Measurements  •  Case  studies  on  measuring  and  improving  •  Summary    
  4. 4. What  is  Quality?  Pertaining  to  soEware  J   Requirements…   4  
  5. 5. Importance  of  Usability  
  6. 6. Why  is  Usability  Important?  •  Usability  is  important  especially  for  web   applica7ons   –  And  now  especially  mobile  applica7ons  •  Saas,  mobile  are  in  alignment  with  and  soon   becoming  equivalent  to  cloud   •  Most  prevalent  implementa7on  method  in  the  next  5  years  •  Without  good  usability:   – Users  will  leave  the  applica7ons   – For  mobile,  if  they  can’t  learn  in  30  seconds,   they  won’t  come  back  
  7. 7. Web  and  Mobile  have  Changed  the   Terrain  •  Business  models   have  changed   –  Instead  of  paying   upfront  and   ‘owning’  the   soEware   –  Pay  as  you  go,  pay   by  subscrip7on  •  Behavior  and   expecta7ons  have   changed  
  8. 8. What  is  Usability   In  rela7on  to  quality  In  rela7on  to  user  experience  
  9. 9. When  you  think  Usability…  •  Naviga7on  •  Efficiency  •  Responsiveness-­‐performance  •  Learnability  •  AZrac7veness   Anything   else  come  to  •  Understandability   mind?  
  10. 10. Current  Research  –  Usability   User   Quality Experience   Satisfaction Usability ISO 9241-11 ISO 9126-1 Usability Design ISO Experts Product 25010 Owners
  11. 11. Usability  -­‐  Key  Characteris7c  of   Product  Quality   Source: ISO 25010
  12. 12. Quality  in  Use   ISO 25010
  13. 13. Notes  on  Sa7sfac7on  and  Usability   don’t have what I want I’m unsatisfied •  Sa7sfac7on  is  a   subjec7ve  feeling   Highly usable dependent  on  many   software things  other  than   usability:   My password doesn’t work –  A  user  can  be  highly   sa7sfied  but  the   applica7on  with  low  Nice weather I’m usability.   today satisfied! –  An  applica7on  can  be   ! highly  usable  (high  low usability software usability)  but  the  user  is   not  sa7sfied!   Finishedmy work today
  14. 14. Usability-­‐Major  Component  of  Quality   Quality  and  Usability  in  the  so1ware  development  lifecycle   Usability   Usability  •  Can  be  measured  from  the  design  point  of   view  or  of  the  product  •  Can  be  measured  ‘in-­‐use’  with  real  users  
  15. 15. Evolu7on  of  SoEware  quality   ISO 9000 ISO 9126 ISO 9241 ISO 25010 Software Software Software Type of Process Quality Quality Quality quality quality (internal) (external) (in use)What ismeasured Software processes Code Product ? CMMIHowmeasured? assessment model white box testing black box testing ? CMMIWho Assessment Programmer Tester End Usermeasures? Company
  16. 16. What  is  Usability-­‐Effect  of  the   SoEware  Product  Degree to which specifiedusers can achieve specifiedgoals with effectiveness,efficiency and satisfaction in aspecified context of use. Source: ISO 25010
  17. 17. Usability  in  Actual  Usage  •  User  role   specified  users  •  Objec7ve   specified  goals  •  Task  •  Environment   specified  context  of  use  •  Domain  •  …   What  else  can  you  think  of?  
  18. 18. Usability and  User  ExperienceUX, Usability and Quality in Use•  Difficult to understand their relationships•  Lack of consensus in meaning in literature•  Not totally clear how related to quality•  Standard definition for user experience is still not available
  19. 19. Some  UX  Defini7ons   UX DESCRIPTIOND1 a continuous process of user engagement with the product [10]D2 entire set of affects that results in user-product interaction [11]D3 the evolution of usability [12]D4 elaboration of the satisfaction component of usability [13]D5 a categorization of “do-goals” (pragmatics) and “be-goals”(hedonics) [3], [14]D6 infinite small experiences relating to people, products and contexts [6]D7 consequence of user’s and product’s characteristics when interacted in a specific environment [1]D8 degree to which specified users can achieve actual usability, safety, and satisfaction in use in a specified context of use [15]D9 A persons perceptions and responses that result from the use or anticipated use of a product, system or service [16]D10 the degree of positive or negative emotions that can be experienced by a specific user in a specific context during and after product use and that motivates for further usage [17]
  20. 20. User Experience (UX) •  Pragmatic (do-goals) : refers to the products perceived ability to support the task achievement and focuses on the product’s utility and usability in completing tasks, the ‘do-goals’ of the user. •  Hedonic (be-goals) : products ability to support the user’s achievement of ‘be-goals’, such as being satisfied, pleasure. •  Real goal of the end user is fulfilling be-goals such as being autonomous, competent, related to others, stimulated, etc.” •  Be-goals are the driver of UXHassenzahl
  21. 21. Rela7ng  Usability  and  User  Experience   Quality   508   User  Interface   Func7onality   Learnability   characteris7c   compliancy   Usability   Product   Text/Sound   Appropriate   Easy  to  use   Easy  to  learn   aZributes   aZributes   Func7ons   UX  Pragma7c   Effec7ve  and  Efficient   Do  Goals   Usability   Effec7veness  in  use   Efficiency  in  use   Learnability  in  use   Measurement   UX  Hedonic   Sa7sfac7on  (pleasure,  sense  of  community…)   Be  Goals   Sa7sfac7on   Sense  of   Pleasure   Comfort   Trust   Measurement   Community   Context   USER  EXPERIENCE   21  
  22. 22. Now  that  we  understand  it  ALL…   We  Need  to  Define  Usability  For  Our   Own  Purposes   22  
  23. 23. Let’s  Define  Usability/UX   From  the  Product  Viewpoint   Usability  Characteris7c  1   Naviga7on   Understandable     Characteris7c  2 Characteris7c  n  Subcharacteris7c  1   Control  Stability   Subcharacteris7c  2   Previous-­‐Next   Subcharacteris7c    n   AZribute  1  Home  Loca7on   AZribute  2   Save  Loca7on   Ease  of  finding   AZribute  n  
  24. 24. Defining  Usability  from  an     Effect-­‐Real  usage  Point  of  View   Usability Quality  Effec7veness   Efficiency   Sa7sfac7on   Characteris7c  n   Accuracy   Completeness   Subcharacteris7c    n   Errors   AZribute  2   AZribute  3   AZribute  1  
  25. 25. Some  Poten7al  AZributes   Measurements  for  ‘in  use’  Usability  •  Effec9veness   –  Comple7on  rates   Can  you  think   –  Error  rate   of  any  others,   –  Help  usage   par9cular  to  •  Efficiency   your   applica9on?   –  Task  7me   –  Backtracking  •  Learnability   –  Learning  rate   –  Task  7me  devia7on  
  26. 26. Adding  in  UX  Sa9sfac9on  in  use  •  Degree  to  which  users  are  sa7sfied  in  a  specified   context  of  use.  Sa7sfac7on  is  the  user’s  response  to   interac7on  with  the  product,  including  amtudes   towards  use  of  the  product.  Subdivided  into  sub-­‐ characteris7cs:   –  Likability  (cogni7ve  sa7sfac7on)   UX Component –  Pleasure  (emo7onal  sa7sfac7on)   –  Comfort  (physical  sa7sfac7on)   –  Trust   26  
  27. 27.  Example  of  Product  Quality  Model   with  AZributes   (Operability in ISO 25010)    External  Quality  Requirements  (for   Shopping  Cart  Entity)   1 Usability   1.1 Understandability   1.1.1 Icon/label  ease  to  be  recognized     1.1.2 Information  grouping  cohesiveness   1.2 Learnability   1.2.1 ………………………………………………………..     1.3 Ease  of  Use   1.3.1 Control  permanence     1.3.2 Helpfulness      
  28. 28. Learnability  •  Degree  to  which  the  soEware  product  enables  users  to   learn  its  applica7on  -­‐  7  principles  (ISO  9241-­‐10)   –  Suitability  for  the  task  -­‐  should  be  suitable  for  the  user’s   task  and  skill  level     –  Self-­‐descrip7veness-­‐  should  be  clear  what  the  user   should  do  next   –  Controllability    -­‐  the  user  should  be  able  to  control  the   pace  and  sequence  of  the  interac7on     –  Conformity  with  user  expecta7ons  -­‐  should  be   consistent   –  Error  tolerant  -­‐  forgiving   –  Suitability  for  individualiza7on  -­‐  should  be  able  to  be   customized  to  suit  the  user     –  Suitability  for  learning  -­‐  should  support  learning   28  
  29. 29. Ease  of  use  •  Degree  to  which  the  soEware  product   makes  it  easy  for  users  to  operate  and   control  it.   – controllability   – error  tolerance  (by  operator)     – conformity  with  user  expecta7ons   29  
  30. 30. Ease  of  Use  -­‐  Helpfulness  •  Degree  to  which  the  soEware  product   provides  help  when  users  need   assistance  including  help  that  is:   – easy  to  find     – comprehensive   – effec7ve   30  
  31. 31. Example:  Usability  Measurement  ACribute   Scale How Calcula9on Goal CurrentHelp   Percent    of   Compare   % 90% 40%completeness Menu  items   menus  and   with     help  items helpEase  of   Keystrokes   Sample  50   average 3.5 12access to   items find/use  a   feature/ func7on/ informa7onConsistency Number   Examine   integer 1 5 loca7ons  for   menus  and   same  buZon doc. What types of usability measures are these? 31  
  32. 32. Using  a  measurement  model   Evalua7on   Usability  composed of Indicator   Metrics   Evalua7on   Evalua7on   Characteris7c   Measurement  composed of Func7on   Evalua7on   Evalua7on   Measurements   Sub-­‐characteris7cs   32  
  33. 33. Usability-­‐Review   usability   usability  in  use   UX  •  Can  be  measured  from  the  design  point  of   view  or  of  the  product  •  Can  be  measured  ‘in-­‐use’  with  real  users  
  34. 34. Measuring  Usability   Methods  and  models  Now  that  we  know  what  it  is,  and   what  to  measure,  then  HOW?   34  
  35. 35. Measurable  AZributes  •  Usability  characteris7c  •  Descrip7on  and  purpose   Once you have a model (what you•  How  to  measure   are going to•  What  is  measured   measure), then•  Measurement/Calcula7on   you start doing IT!•  Range  (min,  max)  •  Metric  •  Objec7ve  (goal)  versus  Current  (indicator)  
  36. 36. UX/Usability  Measurement  Methods   Focus   groups   Ques9on naires   Observa9on   Heuris9c   Labs   Evalua9on   Logging   Let’s  get  started  
  37. 37. Measurement  methods  Expert  Evalua9on  (Heuris9c  evalua9on)  •  Define  a  set  of  rules  or  criteria  with   measurements  and  evaluate  against  them  Web-­‐based  logs  •  Collect  user  ac:vity  data   – Mistakes  and  errors   – How  long  it  takes  to  complete  tasks   – Comple:on  rates  Ques9onnaires  •  Quan7ta7ve  subjec7ve  measurement  of  UX   characteris7cs   37  
  38. 38. Heuris7c  Evalua7on   Using  a  checklist  But  a  liZle  more  complicated  than   yes/no   38  
  39. 39. Heuristic Evaluation of a Pharmacy applicationEach usability attribute has a quantifying metric.•  For  example,  for  Predic9ve  textual  aCribute,  users  should  be  able  to   understand  a  buCon’s  results  prior  to  pressing  it.    •  Direct  metrics  need  to  be  designed,  i.e.:     –  (0)  No  support  at  all   –  (1)  Par9al   –  (2)  Complete    •  Need  a  mapping  from  0,  1,  2  to  something  more  understandable,  i.e.  2  =   100,  and  1  =  60  •  Need  an  indicator  to  interpret  the  level  of  sa9sfac9on  met  with  decision   criteria  with  acceptability  ranges  in  a  percentage  scale:     –  0-­‐40  (unsa9sfactory  –red)  means  changes  must  take  place  with  high   priority;   –  40-­‐70  (marginal  –yellow)  indicates  a  need  for  improvement  ac9ons;     –  70-­‐100  indicates  a  sa9sfactory  level  –green-­‐  for  the  analyzed   aCribute.     39
  40. 40. 2.2.2 Error Recovery Support" In the current state, users filling a new prescription are supported well in error recovery (2.2.2) in automatic cursor positioning 40
  41. 41. Heuristic Evaluation – Pharmacy SoftwareThe purpose of the evaluation•  Understand the external quality level of the learnability sub-characteristic for filling a new prescription Table 2 - excerpt of the whole current evaluation 41
  42. 42. Heuris7c  Evalua7on  of  Usability-­‐ Shopping  Cart  External Quality Requirements Measure EI value P/GI value Global Quality Indicator 61.97% 1 Usability 60.88% 1.1 Understandability 83% 1.1.1 Icon/label ease to be recognized 100% 1.1.2 Information grouping cohesiveness 66% 1.2 Learnability 51.97% 1.2.1 ……………………………………………… … 1.3 Operability 49.50% 1.3.1 Control permanence 100% 1.3.2 Expected behaviour 50% 2 Content Quality 63.05% 2.1 Content Suitability 63.05% 2.1.1 Basic Information Coverage 50% Line item information completeness 2 50% Product description appropriateness 50% 2.1.2 Coverage of other Contextual Information 76.89% ……………………………………………….. … Return policy information completeness 33%
  43. 43. JIRA Heuristic Usability Evaluation
  44. 44. Usability  Logging  Collec7ng  user  ac7vity  and  behavior     Tradi7onally  used  for  other  things   like…     44  
  45. 45. Usability  Logging  •  Iden7fy  users  by  using  session   ID  to  iden7fy  a  unique  user.    •  Itera7vely  insert  code  into  the   applica7on  •  Collect  data  •  Analyze  the  data  for  each   aZribute  in  different   dimensions  and  aggrega7ons  •  Determine  the  need  for  further   calcula7ons  and  what  aZributes   to  measure  further  •  Revise  the  data  we  are   collec7ng,  adding  or  decreasing   granularity  
  46. 46. Some  easy  and  some  hard   46  
  47. 47. Jira Evaluation Summary 47
  48. 48. Using  Both     Product  Perspec7ve  and  the  User  Perspec7ve  •  If  we  find  some  aZribute  with  low   performance  from  the  user  perspec7ve  •  Then  we  can  go  back  to  the  product   perspec7ve,  figure  out  why,  and  improve   48  
  49. 49. JIRA Evaluation by Attribute Excerpt 49
  50. 50. For  example:Login  BuZon  Visibility   (product  usability  aZribute)   50  
  51. 51. Using  Ques7onnaires  The  process:  1. Use  your  model  of  what  you  want  to  measure   and  improve  2. Design  the  ques7onnaire  according  to  the   model  3. Execute  the  ques7onnaire  4. Analysis  
  52. 52. One  UX  Model  
  53. 53. Ques7onnaires  Used  SUMI  (So1ware  Usability  Measurement  Inventory  )  QUIS  (Ques:onnaire  for  User  Interface  Sa:sfac:on  )  PSSUQ  (Post-­‐Study  System  Usability  Ques:onnaire  )  ASQ  (A1er-­‐Scenario  Ques:onnaire  )  PUTQ  (Purdue  Usability  Tes:ng  Ques:onnaire  )  PHUE  (Prac:cal  Heuris:cs  for  Usability  Evalua:on  )  SUS  (System  Usability  Scale  )  IUI  (Isometrics  Usability  Inventory)  
  54. 54. 54  
  55. 55. 55  
  56. 56. What  do  you  do  with  all  this?   56  
  57. 57. Itera7ve  Improvement  •  Measure  external  quality  –  usability  with   heuris7cs  •  Measure  in-­‐use  usability  with  logging  or   ques7onnaires  
  58. 58. Goal  is  to  Understand,  Evaluate  and  Improve   58  
  59. 59. Next  Steps  •  Produce  an  ac7on  plan   –  What  usability  aZributes  are  important  to  your   organiza7on?   •  Develop  a  model   –  What  data  can  you  collect/Which  technique  can   you  use   •  Maybe  some  elements  of  the  model  drop  out-­‐can’t  be   measured  that  easily  •  Start  collec7ng  and  developing  benchmarks  
  60. 60. Conclusion  •  Usability  and  UX  are  both  abstract  concepts   –  Cri7cal  component  of  quality  •  Defining  is  different  for  each  organiza7on  •  Need  a  model  for  your  organiza7on  •  The  model  is  the  founda7on  of  what  to   measure  •  Once  you  can  measure,  then  you  can  evaluate   and  improve  
  61. 61. Resources  These are all journal publications that canbe accessed via these links… 61  
  62. 62. Thanks   Ques7ons  and  Answers    Please  fill  out  an  evalua:on  form  and  drop  it  in  the   collec:on  basket  located  at  the  back  of  the  room.       Philip  Lew     @xbosoE     philip.lew@xbosoE.com   phone:  408-­‐350-­‐0508   www.xbosoE.com