Cloud  Testing  Bill  of  Rights                                                Top  10  Decision  Criteria  for  a  Web  ...
II.     Provides   an   elastic,   globally   distributed   test   environment   on   demand   and                        ...
There  is  rarely  a  need  to  generate  load  from  actual  client  machines.  Generating  exter-­                      ...
Clearly,   this   right   is   not   exclusive   to   cloud-­based   performance   testing   technologies.                ...
mance  issues  and  under-­tuned  sites  and  applications  than  it  would  cost  to  switch  to  a                      ...
IX.   Test  platform  that  allows  functional  and  load  testing  from  lab  to                                 producti...
Upcoming SlideShare
Loading in …5
×

Bill of-rights-white-paper-final-012312

1,971 views

Published on

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
1,971
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
29
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Bill of-rights-white-paper-final-012312

  1. 1. Cloud  Testing  Bill  of  Rights Top  10  Decision  Criteria  for  a  Web  &                                                                       Mobile  Application  Testing  Platform By  Scott  Barber,  Chief  Technologist,  President  &  CEO,  PerfTestPlus,  Inc. Why  a  Cloud  Testing  Bill  of  Rights? My   collaboration   with   When  the  governmental  structure  of  the  United  States  was  initially  put  into  place,   SOASTA  began  in  the  sum-­ it  did  not  include  the  Bill  of  Rights.  While  many  of  the  Founding  Fathers  were  far   mer   of   2011   when   I   was   from  pleased  about  the  absence  of  explicit  citizens’  rights  in  the  Constitution,  they   asked  to  preview  a  beta  ver-­ knew  that  if  they  didn’t  get  the  basic  governmental  structure  in  place  quickly,  the   sion   of   CloudTest   Lite.   The   new   country   wouldn’t   last   long   enough   for   citizens’   rights   to   matter.  This   sounds   positive  experience  I’ve  had   very  much  like  how  development  and  IT  groups  were  formed,  with  quick  and  dirty   with   both   the   product   and   organizational   and   governmental   structures   that   made   sense   to   someone   in   the   the   SOASTA   team   has   en-­ moment. abled   our   work   together   to   continue.   This  begs  the  question:  In  your  development  or  IT  group,  do  you  feel  like  you  have   the  right  to  voice  dissatisfaction  and  organize  to  champion  change?  If  your  experi-­ If   you   are   familiar   with   my   work   then   you   know   that   I   what  change  we  are  championing.”   don’t   commonly   write   about   vendors   or   tools   aside   from   I  can  think  of  no  techniques,  ideas  or  practices  related  to  testing,  which  require  sup-­ deliberately   neutral   educa-­ port  from  management,  that  get  shut  down  more  quickly  than  those  that  include  the   tional  pieces.   cloud.  The  cloud  is  gaining  a  degree  of  credibility  when  it  comes  to  operations,  but   when  it  comes  to  testing,  there  are  a  disproportional  number  of  vociferous  detrac-­ SOASTA   has   proven   to   be   tors  as  compared  to  champions.  I  have  to  wonder  if  the  detractors  are  being  driven   one   of   those   rare   vendors   by  fear:  fear  of  change  and  fear  of  being  rendered  irrelevant  by  advancement. speak   freely   about   without   This   observation   leads   me   to   see   the   Cloud  Testing   Bill   of   Rights   as   a   series   of   violating   what   mom   taught   proposed  revisions  to  commonly  accepted  software  development  practices.  They   me   about   refraining   from   ensure   citizens   of   the   nation   of   software   development   the   freedom   of   expanded   saying   anything   at   all   if   you   options   and   the   right   to   pursue   betterment   of   their   current   situation.   These   revi-­ don’t   have   anything   nice   to   say. simplify  application  performance  testing.   Your  Cloud  Testing  Rights I’m  hard  pressed  to  imagine  many  individuals  who  would  object  to  having  the  right   to  pursue  these  items  appropriate  to  the  needs  and  context  of  their  project,  applica-­ tion  or  organization  without  fear  of  being  labeled  a  troublemaker.©  Soasta,  Inc.  2012 1
  2. 2. II.   Provides   an   elastic,   globally   distributed   test   environment   on   demand   and   within  minutes III.   Has  dynamic  and  fully  automated  test  grid  provisioning IV.   Has  the  ability  to  build  and  modify  complex  tests  within  hours   V.   Supports  dynamic  web  and  mobile  technologies  and  protocols VI.   Controls  test  sessions  to  start,  pause,  stop,  modify,  re-­start  and  analyze VII.   Has  a  single  integrated  view  of  performance  and  monitoring  metrics VIII.   Rapidly  isolates  performance  issues  in  real  time  at  any  scale IX.   Allows  functional  and  load  testing  from  lab  to  production X.   Is  affordable  enough  for  everyday  testing In  case  these  rights  don’t  strike  you  as  important  or  it’s  not  immediately  obvious  to   you  what  problems  they  address,  read  on.   Load  Generation A  cornerstone  of  a  business  creating  value  by  adding  application  performance  test-­ ing  is  the  ability  to  exercise  the  application  under  test  with  realistic  loads.  “Realistic”   doesn’t   only   refer   to   the   ability   to   deliver   the   correct   volume,   but   also   to   deliver   that  volume  from  the  places  where  actual  users  will  be  using  your  application.  The   before  deciding  which  tool  to  use  for  your  application  performance  testing  needs. -­ ternally  and  from  an  end-­user’s  perspective.  Each  of  these  perspectives  add  value   that  the  other  two  don’t,  meaning  that  failing  to  consider  and  account  for  each  of   these  will  ultimately  lead  to  unwanted  surprises. Internal  load  generation  is  the  most  effective  option  for  early  performance  assess-­ ments  and  for  continually  assessing  and  tuning  hardware  and  software  that  will  live   in   your   hosted   environment.   External   load   generation   is   necessary   to   determine   of  your  site  or  application  and  geographic  proximity  to  the  data  center.  Taking  mea-­ surements   from   a   true   end-­user   perspective,   or   “Last   Mile”   testing,   is   necessary   to  assess  and  tune  the  front-­end  aspects  of  the  site  or  application  and  determine   actual  user  experience. When  it  comes  to  load  generation,  in  almost  all  situations  you’ll  want  the  ability  to   generate  load  both  internally  and  externally.  If  you’re  part  of  an  organization  that   hosts  every  component  internally  and  access  to  those  sites  or  applications  will  be   available  only  to  folks  on  the  same  internal  network,  then  there  is  no  need  for  ex-­ ternal  testing.  ©  Soasta,  Inc.  2012 2
  3. 3. There  is  rarely  a  need  to  generate  load  from  actual  client  machines.  Generating  exter-­ nal  load  from  the  cloud  while  observing  a  sampling  of  end-­user  interactions  is  entirely   -­ ready  part  of  your  overall  testing  program. II.   Elastic,  globally  distributed  test  environment -­ ters  is  that  you  have  the  right  to  be  able  to  generate  realistic,  production  volume  loads   from  geographies  representative  of  your  actual  user  base.  While  generating  load  ex-­ ternally  from  across  the  street  is  better  than  only  being  able  to  generate  internal  load,   it’s  not  nearly  as  valuable  as  concurrently  generating  load  from  a  variety  of  geographic   locations.   It’s  far  more  challenging  today  to  predict  performance  from  a  particular  geography  than   -­ able  to  the  producers  of  sites  and  applications  to  improve  performance  in  a  particular   geography.   Implementing   a   cloud-­based   technology   for   load   generation   means   that   you  no  longer  have  to  wait  for  users  to  complain  before  becoming  aware  of  a  problem.   Cloud-­based  load  generation  can  enable  you  to  assess  performance  by  user  location   by  clicking  a  couple  of  extra  buttons  and  running  a  couple  of  extra  of  tests.     III.   Dynamic  and  fully  automated  test  grid  provisioning  on  demand   and  within  minutes   Very  much  related  to  the  previous  two  rights,  you  have  the  right  to  mix  and  match  the   generation  points  for  your  load  quickly  and  simply.  Having  the  ability  to  generate  load   from  anywhere  isn’t  enough,  and  isn’t  unique  to  cloud-­based  solutions.  I,  for  one,  was   doing  that  ten  years  ago.  Of  course,  it  always  seemed  to  involve  a  month  to  ship  hard-­ ware  with  the  right  software  pre-­installed  to  remote  locations,  or  to  talk  people  working   of,  coordinate  network  connectivity  and  the  other  hundred  details.  And  all  this  before   -­ gested  such  a  solution  today! Speed  and  Completeness enough  to  be  relevant  without  needing  to  learn  a  new  tool  each  time  a  new  technology   is  implemented.  Performance  engineers  have  long  been  begging  for  rapid  testing  tools   that  can  be  used  with  a  wider  range  of  technologies. Today,  traditional  testing  tools  continue  to  take  weeks  and  months  and  only  test  older   technologies.  Two  rights  speak  to  today’s  performance  testers  not  being  condemned   to  telling  those  same  war  stories  ten  years  from  now.©  Soasta,  Inc.  2012 3
  4. 4. Clearly,   this   right   is   not   exclusive   to   cloud-­based   performance   testing   technologies.   Rather,  you  have  the  right  to  demand  this  from  your  tool  or  service  provider  no  matter   don’t   have   days,   let   alone   weeks,   to   wait   for   performance   assessments.  They   have   the  right  to  results  in  enough  time  to  do  something  more  useful  than  shrug,  mumble   something  about  wishing  they’d  known  sooner  and  logging  an  action  item  for  the  next   release.  The  only  hope  we  have  of  providing  results  on  their  timeline  is  if  we  can  build   and  modify  realistic  tests,  complex  tests,  in  hours  or  less. V.   Support  for  dynamic  web  and  mobile  technologies  and   protocols One  could  say  that  we  have  the  right  to  demand  that  our  tools  provide  support  for  new   technologies  and  protocols.  At  the  time  of  this  writing,  the  most  challenging  of  those   from  a  load  generation  perspective  is  related  to  asynchronous  yet  inter-­related  use  of   both   client-­   and  server-­side  communications  within  a  single  site  or  application.  Next   year,  it  may  be  something  entirely  different.  What  will  remain  the  same,  however,  will   be  our  desire  to  not  purchase  and  learn  a  new  tool  or  service  due  to  a  technological   advance.   Of   course   there   will   always   be   bleeding-­edge   technologies   that   tools   don’t   support.   And  when  one  of  those  technologies  catches  on,  vendors  need  not  charge  us  obscene   amounts  of  money  for  updates.  The  fact  that  some  vendors  still  expect  their  loyal  cus-­ tomers  to  pay  exorbitant  fees  to  AJAX  or  SOAP  support,  protocols  that  crossed  from   have  the  right  to  be  treated  better  than  that. Control Following  the  ability  to  create  and  implement  tests  quickly,  and  building  tests  against   leading   technologies  without  purchasing  new  tools,  tool  extensions  are  the  ability  to   performance  test  effort. VI.   Control  test  sessions  to  start,  pause,  stop,  modify,  re-­start  and   analyze There  was  a  time  when  it  was  a  rare  tool  that  allowed  expert  performance  testers  to   write  custom  code  to  give  them  the  kind  of  detailed  control  they  needed  to  maximize   a  tool  so  they  can  accomplish  tasks  like  changing  load  levels  on  demand  during  test   runs.  There  are  plenty  of  tools  on  the  market  that  come  with  this  kind  of  control  built  in. This  speaks  to  performance  testers  having  the  right  to  spend  the  majority  of  their  time   doing  performance  testing  instead  of  working  around  tool  limitations.  Any  organization   disabling   their   performance   testing   program   by   forcing   their   testers   to   work   around   their  tools  today  is  spending  far  more  money  in  terms  of  wasted  time,  unfound  perfor-­©  Soasta,  Inc.  2012 4
  5. 5. mance  issues  and  under-­tuned  sites  and  applications  than  it  would  cost  to  switch  to  a   tool  or  service  that  just  works.  We  most  certainly  deserve  the  right  to  ask  our  compa-­ nies  to  stop  wasting  money. Analytics When  I  was  starting  out  as  a  performance  tester,  virtually  all  performance  analysis  was   into  the  same  spreadsheet  of  data  that  was  manually  exported  from  the  load  genera-­ tion  tool  to  manually  create  charts  and  graphs  so  the  team  could  manually  analyze  the   modify  the  next  test. VII.      Single  integrated  view  of  performance  and  monitoring  metrics Year   after   year   my   pleas   for   this   capability   fell   on   deaf   ears.  Today,   most   load   gen-­ eration  tools  come  with  built-­in  monitoring.  Many  even  come  with  real-­time,  integrat-­ ed  monitoring  on  a  single,  customizable  dashboard.  To  me,  this  is  nothing  short  of  a   dream  come  true;;  for  any  performance  tester  this  is  at  least  a  right,  if  not  an  expecta-­ tion.  Aside  from  saving  an  unbelievable  amount  of  time,  the  increase  in  value  per  test   rights,  it’s  the  right  to  have  monitoring  software  that  works  as  part  of  your  performance   testing  program  that  matters.   VIII.      Rapidly  isolate  performance  issues  in  real  time  at  any  scale -­ ect.  Today,  these  tools  are  commonplace,  affordable,  approachable,  and  even  easy  to   -­ ties.  The  ability  to  isolate  performance  issues  in  real  time  not  only  saves  time,  as  the   right’s  title  implies,  it  all  but  eliminates  the  “it’s  your  tool,  not  my  application”  response,   enabling  the  team  to  get  right  to  work  on  resolving  issues  instead  of  wasting  time  –   The   overwhelming   majority   of   my   clients’   performance   testing   efforts   are   completely   hamstrung   by   tools,   services   and   license   agreements   that   make   it   impossible   to   do   continual  performance  testing.  This  situation  leads  to  the  propagation  of  the  belief  that   performance  testing  can  be  effective  as  a  last-­minute  task.  The  reality  is  that  treating   performance   testing   as   an   isolated   afterthought   leads   to   unnecessary   heroic   efforts   and  frequently  to  epic  failures  in  production.  This  is  a  sad  and  decidedly  unhelpful  state   of  affairs.©  Soasta,  Inc.  2012 5
  6. 6. IX.   Test  platform  that  allows  functional  and  load  testing  from  lab  to   production Good  performance  testing  begins  when  a  project  is  conceived  and  ends  when  a  site   or  application  is  retired.  Sites  and  applications  that  achieve  consistent  acceptable  (or   better)  performance  integrate  performance  testing  into  every  aspect  of  design,  devel-­ opment,   deployment   and   maintenance   of   their   product.   It   used   to   be   that   doing   this   took  a  large  and  expensive  toolbox  as  well  as  a  small  army  of  tool  specialists.  That  is   no  longer  the  case. With  modern  cloud-­based  performance  testing  tools,  activities  that  used  to  be  excep-­ tionally  painful,  like  moving  a  test  from  the  lab  environment  to  a  staging  or  production   environment,  take  minutes  instead  of  days  or  weeks.  Combine  this  with  capabilities  of   individual  tools  to  perform  multiple  functions  means  that  fully  and  seamlessly  integrat-­ ing  performance  testing  is  entirely  viable.  Once  again,  we  have  the  right  to  demand  this   kind  of  simple  and  seamless  integration  of  performance  testing  into  the  entire  product   lifecycle  and  the  right  to  demand  that  vendors  support  our  efforts  in  this  regard. X.     Affordable  enough  for  everyday  testing This  right  simply  cannot  be  overstated.  For  the  majority  of  my  career  as  a  performance   tester,  only  the  richest  and  most  elite  software  development  organizations  were  able   to  justify  the  expense  of  buying  or  leasing  the  software  and  infrastructure  necessary  to   run  tests  at  will.  Shared  licenses,  temporary  environment  provisioning,  and  administra-­ tive   overhead   has   been   the   bane   of   a   performance   testers   existence   for   years.  Any   organization  that  is  still  facing  this  situation  is  doing  so  out  of  negligence. Advances  in  cloud-­based  technologies  and  pricing  models  have  all  but  eliminated  the   need   for   additional   hardware   and   infrastructure   to   be   procured   and   maintained   in-­ ternally.   This   has   reduced   the   total   cost   of   enabling   on-­demand,   daily   performance   testing,  conducted  by  any  member  of  any  team,  without  the  need  to  cross-­coordinate   hardware  or  license  availability.  Add  that  to  the  savings  that  an  organization  will  realize   by  deploying  and  maintaining  well-­performing  applications,  and  it  seems  nothing  short   of  irresponsible  to  continue  paying  so  much  more  for  so  much  less  value.   Conclusion Cloud-­based   performance   testing   technologies   have   converted   the   dreams   of   many   old-­school  performance  testers  into  the  rights  of  today’s  performance  testers.  Failure   to  take  advantage  of  these  rights  wastes  time  and  money.  It  ensures  acceptable  site   expensive,   or   result   from   sheer   dumb   luck.   In   short,   not   knowing   and   taking   advan-­ tage  of  your  rights  as  described  by  the  Cloud  Testing  Bill  of  Rights  is  nothing  short  of   irresponsible. About  the  Author: A  prominent  throught-­leader  in  the  area  of  software  system  performance  and  testing  software  systems,  Scott   Barber,  founder  and  Chief  Technologist  of  PerfTestPlus  &  Director  of  the  Computer  Measurement  Group,   makes  his  living  writing,  speaking,  consulting,  and  coaching  with  the  goal  of  advancing  the  understanding   and  practice  of  software  testing.  Scott  has  contributed  to  four  books,  composed  over  100  articles  and  papers,   the  Association  for  Software  Testing,  and  co-­founded  the  Workshop  on  Performance  and  Reliability.©  Soasta,  Inc.  2012 6

×