White Paper: Creating a World-Class Build System, and Getting It Right
Upcoming SlideShare
Loading in...5
×
 

White Paper: Creating a World-Class Build System, and Getting It Right

on

  • 769 views

White Paper for the Citrix Systems' presentation by Scott Estes

White Paper for the Citrix Systems' presentation by Scott Estes

Statistics

Views

Total Views
769
Views on SlideShare
769
Embed Views
0

Actions

Likes
0
Downloads
32
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

White Paper: Creating a World-Class Build System, and Getting It Right White Paper: Creating a World-Class Build System, and Getting It Right Document Transcript

  • Creating  a  World-­‐Class  Build  System,  and  Getting  It   Right   Scott  Estes   Citrix  Systems,  Inc.        Copyright Citrix Systems, Inc. All rights reserved.   1    
  • 1 Table  of  Contents  2   Introduction  .........................................................................................................................................  4  3   SCM  and  Build  are  Critical   ....................................................................................................................  4  4   Who  Are  the  Vision  Stakeholders?  ......................................................................................................  4  5   What  are  the  Requirements?  ..............................................................................................................  5  6   Evolution  versus  Revolution  ................................................................................................................  6  7   Build  versus  Buy?  .................................................................................................................................  7  8   Leveraging  Others  to  Accomplish  Common  Goals  ...............................................................................  7  9   SCM  and  the  Build  System  at  Citrix   ......................................................................................................  8   9.1   Engineering  Structure  ..................................................................................................................  8   9.2   XenApp/Desktop  Component  Structure  ......................................................................................  9   9.3   PLANB  Structure  ..........................................................................................................................  9  10   The  Flow  of  the  Build  .....................................................................................................................  11   10.1   Input  ..........................................................................................................................................  12   10.1.1   Automation  at  Citrix   ..........................................................................................................  12   10.2   Pre-­‐Build  Steps  ...........................................................................................................................  13   10.2.1   Automation  at  Citrix   ..........................................................................................................  13   10.3   Build  ...........................................................................................................................................  14   10.3.1   Automation  at  Citrix   ..........................................................................................................  15   10.4   Post-­‐Build  Steps  .........................................................................................................................  16   10.4.1   Automation  at  Citrix   ..........................................................................................................  17   10.5   Output   ........................................................................................................................................  18   10.5.1   Automation  at  Citrix   ..........................................................................................................  19  11   Optimizing  the  System  ...................................................................................................................  20   11.1   Keep  it  Deterministic  .................................................................................................................  20   11.2   Keep  it  fast  .................................................................................................................................  22   11.3   Keep  it  simple  ............................................................................................................................  25   11.4   Keep  to  standards  ......................................................................................................................  25  12   Scheduling  Updates  and  Maintenance  ..........................................................................................  25   12.1   How  to  do  it  ...............................................................................................................................  25   12.2   When  to  do  it  .............................................................................................................................  26  Copyright Citrix Systems, Inc. All rights reserved.   2    
  • 13   Final  Thoughts  ...............................................................................................................................  26    Copyright Citrix Systems, Inc. All rights reserved.   3    
  • 2 Introduction  Anything  worth  doing  has  to  begin  with  a  vision,  regardless  of  how  mundane  it  may  seem.    Without  a  vision,  whatever  it  is  we’re  trying  to  do  is  inevitably  doomed  to  either  go  supernova  in  the  short  term,  or  die  a  drawn  out,  laborious  death  over  a  longer  period.    In  both  scenarios,  the  collateral  damage  is  devastating  and  usually,  the  only  way  to  recover  is  to  trash  everything  and  start  over  at  the  beginning  again.    An  interesting  study  could  be  done  to  show  how  much  time,  money  and  resources  are  wasted  by  various  industries,  but  that  is  a  topic  for  another  time.    What  is  explained  and  demonstrated  in  this  paper  is  how  to  define,  implement  and  maintain  an  optimized  Build/SCM/ALM  vision,  and  avoid  common  (and  not  so  common)  pitfalls  along  the  way.  3 SCM  and  Build  are  Critical  It’s  been  said  that  the  Build  is  the  heartbeat  of  a  project.    This  is  a  fundamental  truth,  yet  it’s  amazing  how  more  often  than  not,  person-­‐months  are  invested  in  the  Develop  and  Test  visions  and  strategies  of  a  project,  and  scant  person-­‐hours,  (if  that)  are  given  to  how  the  project  code  base  will  be  managed  and  built,  until  the  very  last  possible  point  in  time.    This  “Build  Avoidance”  is  understandable  because  everyone  is  usually  focused  on  customer  requirements,  product  features,  schedules  and  a  myriad  of  other  things  that  siphon  attention  away  from  SCM  and  Build.    All  too  often,  the  Product  team  will  simply  assume  that  the  product  will  magically  coalesce,  or  that  someone  else  is  thinking  about  how  to  build  it.    This  is  the  point  where  the  savvy  Build  Visionary  can,  with  the  proper  support  and  resources,  create  order  from  the  technical  chaos,  and  provide  a  quality-­‐built  product  that  meets  everyone’s  expectations.  It  must  also  be  stressed  that  there  must  be  executive  level  support  for  SCM.    If  the  senior  management  staff,  and  those  in  charge  of  driving  the  overall  Engineering  roadmap(s)  are  not  bought  in  to  the  Build  Vision,  there  is  little  to  no  chance  of  obtaining  the  resources  required  to  create  a  successful  SCM  and  Build  infrastructure.  4 Who  Are  the  Vision  Stakeholders?  Identifying  and  knowing  all  the  stakeholders  of  the  Build  system  is  the  first  step  to  eliminating  any  “blind  spots”  in  the  Build  vision.    Understanding  the  motives  and  mindsets  of  the  stakeholders  also  goes  a  long  way  towards  ensuring  that  the  Build  system  is  optimized  and  meeting  all  the  requirements.  At  the  high  level,  the  Build  stakeholders  can  be  organized  as  follows,   • Customers   • Project  Management   • Product  Development   • Finance   • Product  Marketing/Sales   • Quality  Assurance/Test  Copyright Citrix Systems, Inc. All rights reserved.   4    
  • • Lifecycle  Maintenance/Customer  Support   • Build  These  stakeholders  will  have  varying  levels  of  input  and  requirements  to  the  Build  vision  and  some  may  not  immediately  realize  that  they  are  stakeholders.    This  is  acceptable  as  long  as  the  Build  Visionaries  remain  aware  of  all  the  stakeholders  and  are  proactive  at  meeting  their  requirements.    As  an  example,  the  customer  is  as  key  a  stakeholder  as  is  Product  Development.    That  being  said,  it  might  be  considered  a  bit  odd  to  find  a  Build  engineer  consulting  with  a  customer  on  how  the  product  should  be  built.    Here,  the  forward  thinking  Build  visionary  should  be  looking  at  project  schedules  and  planning  how  the  Build  vision  can  facilitate  delivering  a  quality  product  on  time  to  the  customer.    Conversely,  the  Build  Visionary  should  be  in  constant  communication  with  Product  Development  to  ensure  that  their  requirements  are  being  accounted  for  in  the  vision.  5 What  are  the  Requirements?  Once  the  stakeholders  have  been  identified,  the  next  step  is  to  gather  their  requirements.    The  following  table  defines,  at  the  high  level,  some  of  these  requirements.   Stakeholder     Requirements    Customers   • Quality  product   • Delivered  on  time   • Affordable   • Maintainable  Project/Release   • Deterministic  builds  Management   • On  schedule   • Of  quality  Product  Development   • Fast  builds   • Efficient  Build  system   • Fully  supported  Finance   • Adherence  to  budget   • Proper  forecasting  Product  Marketing/Sales   • Features   • Quickly  available  Quality  Assurance/Test   • Fast  builds   • Automated  delivery  and  testing   • Fully  repeatable  Lifecycle  Maintenance   • Escrow-­‐able,  (“goldenized”)  builds   • Fully  repeatable   • Patchable   • Business  Continuity  Build   • Stakeholder  requirements   • Adhere  to  SCM  standards   • Resourced  and  supported  Copyright Citrix Systems, Inc. All rights reserved.   5    
  •  There  may  be  additional  requirements  to  what  is  listed  here,  but  having  the  ones  listed  here,  well  defined,  will  ensure  a  smooth  departure  on  the  path  towards  the  world-­‐class  build  system.  There  is  one  additional  stakeholder  with  requirements  dependent  on  the  maturity  level  of  the  company,  and  that  is  Security.    The  requirements  here  range  from  simply  protecting  the  Intellectual  Property  of  the  company  to  full-­‐scale  security  audits,  threat  models  and  so  on.    These  requirements  can  be  a  stand-­‐alone  topic  by  themselves,  and  as  such,  left  for  another  time.  6 Evolution  versus  Revolution  At  this  point,  the  Build  Visionary  should  have,   a) Executive  sponsorship   b) Identified  stakeholders   c) Stakeholder  requirements  With  these,  he/she  can  then  decide  which  of  the  two  approaches,  (Evolution  or  Revolution)  best  suit  the  needs  of  the  product  and  company  as  a  whole.    Each  approach  has  associated  pros  and  cons,  and  often  a  combination  of  the  two  is  what  proves  to  be  most  effective  moving  the  vision  forward.    The  high-­‐level  pros/cons  are  listed  in  the  following  tables,   Evolution   Pros       Cons  • Usually  a  slow,  stable  rate  of  change     • Can  be  too  slow  to  affect  meaningful   change  • Training/documentation  can  keep  pace     • Impetus  can  be  lost  as  others  vie  for   resources       Revolution   Pros       Cons  • Shortcut/mitigate  barriers  to  change     • Amount  of  change  can  adversely  affect   stakeholders’  ability  to  absorb  it.  • Can  open  avenues  for  other  shift  changes.     • Training/documentation  will  usually  not   keep  pace.     • Can  cause  unintended  side-­‐effects  if  not   carefully  planned.      Focusing  exclusively  on  only  one  of  these  two  approaches  is  going  to  result  in  failure  of  the  Build  Vision  at  some  point,  either  from  stakeholder  exhaustion  or  loss  of  confidence  in  the  Vision/Visionary.    While  there  are  no  hard  and  fast  “rules”  on  how  to  combine  these  into  an  effective  vision,  experience  indicates  that  Revolution  works  best  with  small,  experienced  teams  that  can  then  act  as  “ambassadors”  to  the  larger  organization  when  the  vision  is  introduced  at  that  scale.    Evolution  can,  in  fact,  harness  these  Copyright Citrix Systems, Inc. All rights reserved.   6    
  • “mini”  revolutions  such  that  the  larger  teams  can  absorb  the  changes  as  schedules  and  resources  permit.    The  most  important  task  for  the  Build  Visionary  is  to  stay  on  top  of  the  teams  to  ensure  that  they  remain  committed  to  the  Vision.  7 Build  versus  Buy?  There  is  no  magic  answer  to  this  perennial  question;  it  can  be  different  to  varying  degrees,  for  every  application  at  every  company.    The  first  major  challenge  the  Build  Visionary  must  overcome,  is  to  adequately  evaluate  the  available  product  offerings  to  see  which  ones  could  fit  into  the  Build  Vision  compared  to  building  and  supporting  the  infrastructure  in-­‐house.    It’s  not  enough  simply  to  determine  if  the  tools  work  technically,  the  evaluation  and  resulting  recommendation  has  to  include  the  overall  return  on  investment,  (ROI)  and  ongoing  total  cost  of  ownership,  (TCO).    The  recommendation  will  be  based  on,   • The  amount  of  funds/resources  available,  and   • The  amount  of  time  allocated  for  the  evaluations.  The  second  major  challenge  for  the  Visionary  is  to  convince  the  company  to  allocate  the  amount  of  time  needed  to  properly  answer  this  question,  while  keeping  pace  with  the  ongoing  requirements  of  the  product  release  cycle(s).  8 Leveraging  Others  to  Accomplish  Common  Goals  Very  rarely,  if  ever,  will  there  be  enough  SCM/Build  resources  to  singularly  accomplish  the  Vision.    Therefore,  the  savvy  Build  Visionary  will  look  to  others  outside  the  immediate  Build  team  to  assist.    This,  aside  from  moving  the  Vision  forward,  has  two  important  side  effects,   1. People  who  help  feel  like  their  ideas  are  being  heard  and  valued.    As  people  see  the  results  of   their  input  and  work,  they  will  become  more  comfortable  with  continuing  to  present  new  ideas   and  processes.    When  channeled  properly,  this  can  become  a  self-­‐sustaining  cycle  that  further   ensures  the  relevance  of  the  Vision.   2. Those  who  help  in  turn  become  “Build  Ambassadors”  to  the  rest  of  their  respective  teams.     Generally  speaking,  the  best  customer  of  an  invention  is  the  inventor.    If  a  person  on  a   development  team  has  contributed  to  the  Build  Vision,  that  person  will  then  most  likely  take  the   vision  and  spread  it  among  the  rest  of  the  team,  thus  ensuring  that  the  Vision  is  widely  accepted   without  relying  on  the  Visionary  to  do  all  the  “selling”.  There  are  a  couple  of  “gotchas”  that  must  be  mitigated  when  leveraging  resources  from  other  teams,   1. Don’t  interfere  with  other  teams’  mandates  and  objectives.    Most  development  managers  like   to  keep  their  teams  100%  focused  on  the  team’s  particular  roadmap,  leaving  no  time  for  other   pursuits.    Most  likely,  the  help  provided  will  be  in  the  form  of  “skunk-­‐works”,  or  after-­‐hours  type   projects.    Sometimes  the  benefit  to  the  overall  group  will  be  significant  such  that  a  Development  Copyright Citrix Systems, Inc. All rights reserved.   7    
  • manager  will  assign  a  certain  percentage  of  his/her  resources  to  formally  helping  on  SCM/Build   objectives.    In  any  scenario,  the  Build  Visionary  must  take  extreme  care  to  balance  the  Build   needs/wants  with  those  of  the  supporting  team(s).   2. Make  certain  to  manage  expectations.    The  last  thing  any  of  the  Build  stakeholders  want  is  to   feel,  (whether  real  or  imagined)  that  their  ROI  is  not  realized.    Failure  to  manage  the   expectations  is  one  of  the  quickest  ways  to  destroy  the  credibility  and  success  of  the  Vision.  In  summary,  the  Build  Visionary  must  exert  all  his/her  diplomatic  abilities  when  working  with  extended  teams  and  other  Stakeholders  to  ensure  continued  support  for  the  Build  Vision.  9 SCM  and  the  Build  System  at  Citrix  Up  to  this  point,  the  discussion  has  focused  primarily  on  non-­‐technical,  logistical  and/or  philosophical  aspects  that  must  be  considered  when  creating  an  ideal  Build  Vision.    In  parallel,  the  Build  Visionary  should  be  constructing  the  building  blocks  that  will  comprise  the  system  that  will  execute  as  mandated  by  the  Vision.    The  core  of  the  ideal  Build  system  should  revolve  around  well-­‐defined  processes  and  tools  needed  to  facilitate  the  fundamentals  of  good  SCM  practices  such  as,   • “Determinism”  (  the  ability  to  determine  attributes  of  a  build)   • Branching/Merging   • Parallel  development   • Lifecycle  maintenance   • “Goldenization”/escrow   • Business  Continuity   • Some  level  of  Development  automony/self-­‐service   • Etc.  Discussions  around  some  of  these  areas  can  be  separate  topics,  and  as  such  will  not  necessarily  be  discussed  in  detail  here,  other  than  to  show  how  the  Citrix  Enterprise  and  Desktop  Applications  Build  team  have  leveraged  the  power  of  Perforce  to  facilitate  the  realization  of  our  ideal  Build  Vision.  Before  the  technical  details  of  the  build  system  are  discussed,  the  composition  of  the  Engineering  teams  responsible  for  the  product,  and  the  structure  of  the  product  itself,  will  be  explained  to  show  how  Citrix’  ideal  Build  vision  has  evolved  to  meet  the  requirements  of  the  stakeholders.  9.1 Engineering  Structure  There  are  over  500  engineers  organized  in  development  and  supporting  teams  located  in  the  following  sites,   • Bangalore,  India   • Bedford,  Massachusetts  Copyright Citrix Systems, Inc. All rights reserved.   8    
  • • Cambourne  and  Chalfont,  UK   • Fort  Lauderdale,  Florida   • Santa  Clara,  California   • Sydney,  Australia   • Tokyo,  Japan  The  global  Build  team  consists  of,   • 1  engineer  in  Bangalore   • 1  engineer  in  Bedford   • 5  engineers  in  UK   • 4  engineers  in  Fort  Lauderdale   • 1  engineer  in  Santa  Clara  The  Build  engineers  are  primarily  responsible  for  maintaining  the  Build  Vision  and  Systems  at  the  site  where  they  are  located.    Additionally,  the  team  can  leverage  a  “follow-­‐the-­‐sun”  work  schedule  to  be  able  to  maximize  Build  resources  and  coverage  without  requiring  any  one  of  the  teams  to  provide  support  outside  of  normal  working  hours  at  each  site.  9.2 XenApp/Desktop  Component  Structure  The  code  base  that  comprises  the  product  is  structured  along  core  technology  boundaries,  known  as  Technology  Components,  (TCs),  (e.g.  Client,  Server,  protocols,  etc.).    The  engineers  are  assigned  into  TC  teams,  and  focus  on  their  technology  instead  of  a  particular  release  of  the  overall  product.    Each  TC  team  has  a  junior  architect,  (or  Principal  Design  Engineer)  who  is  responsible  for  the  evolution  of  the  technology,  and  coordinating  what  features  can  be  delivered  for  a  particular  product  release  with  the  TC  Manager,  Release  and  Product  Management  teams.      The  deliverables  of  the  TCs  are  released  as  SDKs  which  can  consume/be  consumed  in  such  a  way  as  to  create  a  complete  product.    There’s  a  lot  more  that  could  be  discussed  about  the  TC  methodology,  and  its  advantages  and  disadvantages.      In  synopsis  for  this  discussion,  the  main  advantage  of  this  approach  is  that  a  huge  code  base  is  organized  into  small,  manageable  chunks  that  can  be  “snapped”  together  to  create  a  product  release.    The  disadvantage  is  that  there  is  more  responsibility  placed  on  the  architects,  managers  and  Release  Team  to  ensure  that  the  TC  teams  are  communicating,  and  any  component  dependencies  are  aligned  for  a  particular  release.  9.3 PLANB  Structure  The  build  system  at  Citrix  is  called  the  Perforce  Linked  Automated  Network  Builder,  (or  PLANB).    It  is  comprised  of  six  distinct  components  that  communicate  with  the  Perforce  servers,  and  are  responsible  for,     • Determining  when  to  build   • Creating  the  Build  environment   • Building  the  binaries  Copyright Citrix Systems, Inc. All rights reserved.   9    
  • • Creating/updating/publishing  the  TC  SDKs   • Creating  the  installers/DVD  layout   • Publishing  the  product  DVD  There  are  two  types  of  builds  that  PLANB  controls,   • SDKs   • DVD  layouts  When  building  SDKs,  there  is  an  optional,  manual  SDK  updater  component,  (using  ASP.net)  provided  to  “publish”  the  output  as  determined  by  the  TC  owner.    This  process  is  intended  to  be  used  when  the  TC  output  is  at  some  level  of  quality  as  to  be  considered  “locked”  for  the  overall  product  DVD  release.    If  a  TC  owner  chooses  to  not  publish  his/her  SDK,  then  the  latest  successful  build  of  the  output  is  used  for  the  DVD  layout.  A  Watcher  service  (using  WCF  functionality)  polls  the  Perforce  servers  for  code  submissions.    When  a  submission  is  detected,  the  service  determines  if  any  client  spec  template  changes  are  required,  and  make  any  needed  changes  before  passing  the  change  list(s)  to  be  built,  and  control  to  the  next  component.  The  Build  VM  creator  (also  using  WCF),  seeks  out  an  available  “bare-­‐metal”  machine,  starts  up  a  XenServer-­‐based  virtual  machine  template  on  it,  and  passes  control  to  the  Building  component.  The  Binaries  Builder,  (using  C#  and  MSBuild  functionality)  synchronizes  the  code  using  the  change  lists  sent  from  the  Watcher,  compiles  the  code,  creates  the  SDK(s)  and/or  other  components  into  a  “save”  area  which  can  either  be  used  as  input  to  publish  a  TC’s  component(s)  or  as  direct  input  into  the  Installer/DVD  creator.  The  Installer/DVD  creator,  (using  PowerShell  scripts),  collects  the  binaries,  (SDKs,  installers,  etc.)  from  Perforce  and  the  “save”  area,  and  creates  the  DVD  layout  that  will  be  published  to  Test.  The  Publisher,  (also  using  PowerShell  cmdlets)  then  copies  the  DVD  image  out  to  an  area  where  can  be  consumed  by  the  various  Engineering  groups,  and  sends  a  semaphore  to  the  Test  infrastructure  which  initiates  automated  Build  Verification  Tests  and  subsequent  Regression  test  suites.  When  defining  these  components,  it  was  realized  that  the  overall  system  needed  to  be  broken  into  smaller  pieces  or  projects  such  that  the  architecture  of  each  component  could  be  properly  designed  without  getting  lost  in  the  larger  scope  of  the  entire  system.    The  projects  were  aligned  as  follows,   • Cumulonimbus,  responsible  for  the  Watcher  and  Build  VM  creator  components,  as  well  as  the   code  synching  process  of  the  Binaries  builder  component.    Utilizes  a  SQL  database  to  store  build   information.   • Piston  is  a  “toolbox”  of  PowerShell  scripts  used  to  create  DVD  layouts  and  associated  artifacts.   • Vulcan  is  the  core  build  tool.    It  hooks  into  Visual  Studio’s  MSBuild  engine  to  compile  Windows  Copyright Citrix Systems, Inc. All rights reserved.   10    
  • code.  Figure  9.3.1  maps  out  the  components,  and  the  interactions  between  them.    Additional  details  of  each  component  and  the  associated  projects  will  be  discussed  in  sections  10.1-­‐10.5  below.  Figure  9.3.1:   PLANB Perforce Svr  1 Svr  2 Svr  3 ... Svr  n (Project)  Cumulonimbus (Project)  Piston Updates SDKs Notified  of  submissions  and  updates  client  views Synchs  SDKs Synchs  code (Project)  Vulcan Binaries  Builder SDK  Updater Installer/DVD   Watcher Build  VM  Creator Publisher manual  process Creator “Save”  area  10 The  Flow  of  the  Build  Regardless  of  the  complexity  of  the  product/project,  the  flow  of  the  Build  should  always  follow  the  basic  flow  of  control  in  the  following  diagram,  Figure  10.1:   Input Pre-­‐Build  Step(s) Build Post-­‐Build  Step(s) Output  This  diagram  illustrates  the  flow  of  control  at  its  most  componentized  level.    The  Build  Visionary  should  clearly  define  each  of  these  components  with  the  following  attributes,   • Owner(s)   • Stakeholder(s)   • Input(s)   • Control(s)   • Output(s)  Copyright Citrix Systems, Inc. All rights reserved.   11    
  • Keep  in  mind  that  there  may  be  multiple  builds  following  this  model  that  feed  as  inputs  into  other  builds  and/or  the  ultimate  layout  of  the  product.    Having  the  attributes  for  each  “layer”  of  build  well  defined  is  key  to  ensuring  a  deterministic,  simple  and  efficient  build  system.  10.1 Input  Input  can  be  loosely  defined  as  any  new  “thing”  that  would  cause  a  subsequent  build  to  be  different  than  the  current  build.    Narrowing  this  down  a  bit,  an  input  is  usually  a  direct  code  submission,  an  SDK  or  some  other  “black  box”  component  update.    A  “black  box”  update  is,  at  some  point,  going  to  be  the  result  of  a  direct  code  submission.    Multiple  or  tiered  Build  systems,  (which  can  be  from  where  “black  box”  updates  come  will  be  discussed  later).  Applying  the  previously  defined  attributes  to  the  Input,  defines  the  component  as  follows,  Table  10.1.1:  Attribute   Details   Comments  Owner(s)   • Project/Release   Responsible   for   ensuring   builds   are   released   as   Management   per  schedule.  Stakeholder(s)   • Project/Release   Everyone   in   this   list   is   vested   in   ensuring   that   the   Management   build  is  successful  at  all  stages  of  the  process.   • Build   • Product  Development   • Quality  Assurance/Test   • Product  Marketing/Sales  Input(s)   • Code  submission(s)   Any  new  thing  that  needed  in  the  build  as  well  as   • “black-­‐box”  update   any  historical  artifacts.   • Previous  build  reports  Control(s)   • “Pre-­‐flight”  checks   Ensures  quality  and  traceability  of  submitted   • Triggers   code.   • “Determinism”  Output(s)   • “Of  quality”  code   Code  has  passed  all  controls  and  can  be  built.    10.1.1 Automation  at  Citrix  The  watcher  service,  (part  of  project  Cumulonimbus),  monitors  the  Perforce  ports  for  submissions,  and  automatically  queues  them  for  building  in  the  build  farm.    When  a  submission  is  detected,  the  service  gathers  the  change  list(s)  that  comprises  the  submission,  and  then  checks  the  project’s  configuration  file  to  see  if  any  changes  in  it  require  updates  to  the  client  view  templates,  (e.g.  a  new  SDK  is  needed  by  the  project).    If  any  such  changes  are  detected,  the  service  will  update  the  associated  templates,  before  passing  the  change  list(s),  and  a  job  ID  associated  with  the  change  lists  to  the  Build  VM  creator.    This  interaction  is  shown  in  Figure  10.1.1.  Figure  10.1.1:  Copyright Citrix Systems, Inc. All rights reserved.   12    
  • Watcher Update  Client  spec   Yes templates Submission   Project  config   Get  Change  List(s)   Code  Submission No Call  VM  Creator notification  received change? and  job  ID Perforce Job  ID Cb  database  10.2 Pre-­‐Build  Steps  There  usually  aren’t  too  many  steps  at  this  point;  usually  consisting  of  synching  the  code,  and  any  build  machine  preparation  needed  prior  to  building  the  code.  The  attributes  for  Pre-­‐Build  steps  are;  Table  10.2.1:  Attribute   Details   Comments  Owner(s)   • Build   Responsible  for  ensuring  the  process  and   infrastructure  meets  stakeholder  requirements.  Stakeholder(s)   • Project/Release   Everyone   in   this   list   is   vested   in   ensuring   that   the   Management   build  is  successful  at  all  stages  of  the  process.   • Build   • Product  Development   • Quality  Assurance/Test   • Product  Marketing/Sales  Input(s)   • “Of  quality”  code   Code  and  infrastructure  on  which  to  build  it.   • Build  machine/infrastructure  Control(s)   • Available  infrastructure   Must  have  reliable  and  available  infrastructure.   • System  resiliency  Output(s)   • Buildable  code   Code  is  now  ready  to  be  built.   • Build  environment    10.2.1 Automation  at  Citrix  The  Build  VM  creator  component,  (also  part  of  project  Cumulonimbus),  establishes  and  maintains  a  load  balanced  build  farm  consisting  of  dynamic  and  consistent  build  machines  leveraging  Citrix  XenServer®  technology.  When  the  Watcher  service  sends  a  cue  to  build,  the  creator  determines  which  virtual  machine,  (from  pool)  to  use  based  on  the  job  ID  sent  from  the  Watcher.    It  then  determines  if  the  virtual  machine  is  Copyright Citrix Systems, Inc. All rights reserved.   13    
  • available  in  the  pool.    If  it  is  not,  (i.e.  it  is  building  another  job),  the  creator  places  the  job  in  a  queue  to  build;  otherwise,  it  then  calls  into  the  Xen  API  functions  to  connect  a  data  and  tools  drives  to  the  virtual  machine,  assigns  the  machine  to  a  host  and  starts  it.    The  final  step  in  this  part  of  the  Build  process  is  a  build  script,  (which  resides  in  the  Windows  Startup  folder)  initiates  to  begin  synching  the  code  base.    The  control  flow  is  shown  in  Figure  10.2.1.1  below.  One  additional  note,  at  this  point  the  data  and  tools  drives  are  the  storage  areas  for  the  code  base  and  build  tools  required  for  building  the  code  base,  respectively.    These  drives  reside  on  a  NAS  infrastructure  which  will  be  discussed  further  in  section  11.2.  Figure  10.2.1.1:   Build  VM  creator Cb  database Connect  Data  and   Yes Tools  drives  to  VM VM  type Is  a  VM   Start  VM/ Cue  to  build Determine  VM  type Assign  VM  to  host available? Build  script Wait  for  VM  to   No become  available    10.3 Build  The  Build  Visionary  has  to  be  careful  that  this  part  of  the  system  is  maintained  properly  to  remain  in  line  with  the  Vision.    Competing  requirements  from  the  development  teams  that  for  the  most  part,  own  the  component  build  files  (solutions,  make  files,  etc.)  are  often  introduced  as  shortcuts/hacks  to  accomplish  short-­‐term  goals,  (e.g.  deadlines/milestones)  at  the  component  level.    Failure  to  review  and  make  necessary  updates  to  the  build  files  will  result  in  a  non-­‐optimized  product  build.    The  following  table  defines  the  attributes  for  Build,  Table  10.3.1:  Attribute   Details   Comments  Owner(s)   • Build   Responsible  for  jointly  maintaining  the   • Product  Development   component  build  tasks,  solution  files,  makefiles,   etc.  Copyright Citrix Systems, Inc. All rights reserved.   14    
  • Stakeholder(s)   • Project/Release   Everyone   in   this   list   is   vested   in   ensuring   that   the   Management   build  is  successful  at  all  stages  of  the  process.   • Build     • Product  Development   Test   will   mostly   be   concerned   with   build   times   • Quality  Assurance/Test   and  availability.  Input(s)   • Buildable  code     • Build  environment  Control(s)   • Build  standards   Component  build  scripts  and  associated  files   must  conform  to  company  standards.  Output(s)   • Binaries   Binaries   ready   to   be   signed,   packaged,   etc.   in   • Installers*   preparation  for  release  to  test.     *Installers   may   be   considered   part   of   the   Post-­‐ Build  steps.    10.3.1 Automation  at  Citrix  Project  Vulcan  provides  a  build  framework  around  Microsoft’s  Visual  Studio  MSBuild  engine  into  which  Windows  component  solution  files  are  fed.    This  framework,  shown  in  Figure  10.3.1.1  is  kept  as  simple  as  possible  for  maintenance  purposes.    Only  the  Input,  Post-­‐Build  and  Output  steps  are  developed  and  maintained  in-­‐house.    The  Pre-­‐Build  and  Build  steps  are  all  MSBuild-­‐centric,  and  no  additional  maintenance  is  required  outside  of  the  development  teams  maintaining  the  solution  and  project  files.  That  being  said,  the  Build  team  has  developed  a  Citrix-­‐specific  add-­‐in,  including  project  templates,  (based  on  the  out-­‐of-­‐the-­‐box  templates  provided  with  Visual  Studio),  to  Visual  Studio  which  provides  support  for  Engineering-­‐wide  requirements  such  as,   • Enforcement  of  banned  Microsoft  functions,  (e.g.  strcpy()  versus  sstrcpy()).   • Consistent  signing  of  binaries.   • Automatic  inclusion  of  test  frameworks,  (e.g.  nUnit  and  code  coverage).   • Hooks  into  in-­‐house  developed  tools  used  in  the  build,  (e.g.  custom  graphics  generator)      Copyright Citrix Systems, Inc. All rights reserved.   15    
  • Figure  10.3.1.1:   Input Pre-­‐Build  Steps Build Post-­‐Build  Steps Output Configuration   .proj/search Yes More  to  build? No Sign,  symbol   Create  SDKs,   ID  components   Check  and   Compile   Synch  code Compile  code Yes index,  copy   installers,  reports,   to  build build  deps successful? output,  etc. etc. No Log  error  and   continue    Non-­‐Windows  builds  utilize  standard  make  files  with  a  similar  framework.    The  non-­‐windows  build  system  does  not  support  all  the  functionality  that  MSBuild  provides,  which  means  it  is  somewhat  simpler,  yet  requires  more  developer  support  to  ensure  the  make  files  are  kept  as  clean  as  possible,  (especially  in  the  case  of  dependencies),  to  ensure  fast,  accurate  builds.    Additionally,  system  has  not  been  updated  to  support  the  enhanced  dependency  checking  and  reporting  capabilities  found  in  the  Windows  system.    This  does  not  currently  present  a  burden  to  the  Development  and  Build  teams,  as  the  non-­‐Windows  code  base  consists  of  reasonably  small  client  side  applications.    The  adequate  amount  of  Determinism  is  provided  by  the  Perforce  filelog  functionality.    The  framework  is  shown  in  Figure  10.3.1.2.  Figure  10.3.1.2:   Input Build Post-­‐Build  Steps Output Yes More  to  build? No Sign,  symbol   Create  SDKs,   Search  for   Compile/Link   Compile/Link   Synch  code Yes index,  copy   installers,  reports,   makefiles code successful? output,  etc. etc. No Log  error  and  die  10.4 Post-­‐Build  Steps  At  this  point,  all  binary  components  needed  for  the  product  build  should  be  available  for  additional  post-­‐build  processing.    This  post-­‐build  processing  can  consist  of  binary  signing,  installer  creation,  (some  teams  may  choose  to  make  these  part  of  the  Build  steps),  “layout”  (also  known  as  the  DVD)  creation,  gathering  data  for  the  Build  report,  and  any  other  tasks  in  preparation  for  releasing  the  build  to  Quality  Copyright Citrix Systems, Inc. All rights reserved.   16    
  • Assurance/Test.    The  attributes  are  as  follows,  Table  10.4.1  Attribute   Details   Comments  Owner(s)   • Build   Responsible  for  jointly  maintaining  the  install   • Product  Development   tasks.    Build  responsible  for  ensuring  that  the   build  is  readied  for  release.  Stakeholder(s)   • Project/Release   Everyone   in   this   list   is   vested   in   ensuring   that   the   Management   build  is  successful  at  all  stages  of  the  process.   • Build     • Product  Development   Test   will   mostly   be   concerned   with   build   times   • Quality  Assurance/Test   and  availability.  Input(s)   • Binaries     • Installers  (if  any)  Control(s)   • Build  standards   Install  and  “layout”  scripts  must  conform  to   company  standards.  Output(s)   • Installable  product  image   Everything  needed  to  continue  with  Build   • Build  reports   Verification  and  Regression  Testing.   • Email  notification   • Semaphore  for  automated   testing    10.4.1 Automation  at  Citrix  Project  Piston  provides  a  “toolbox”  of  PowerShell  scripts  which  are  used  to  create  the  product  DVD.    The  request  to  create  a  DVD  can  either  come  from  a  schedule  semaphore,  or  be  manually  requested  from  the  Build  Portal,  (more  on  the  portal  later).      When  the  request  to  create  a  layout  is  received,  the  main  PowerShell  script  extracts  the  data  identifying  the  components  needed  for  the  product  DVD  from  the  project’s  layout  configuration  file.    This  data  is  then  passed  to  the  next  steps  of  creating  the  DVD  image.    Components  that  comprise  the  DVD  contents  are,     • Merge  modules  and  installers  from  the  various  stand-­‐alone  sub-­‐components.   • Help  files  and  supporting  documentation.   • Meta-­‐installers.   • Non-­‐Windows  clients  The  next  step  in  the  process  is  to  update  the  debug  symbols  server  with  the  symbols  generated  during  the  build.    The  scripts  then  generate  the  various  reports  needed  to  provide  the  desired  level  of  Determinism  for  the  build,   • Location  of  the  build   • What  changes  went  into  the  build  Copyright Citrix Systems, Inc. All rights reserved.   17    
  • • Build  time   • Any  major  updates   • Checksum  for  comparison  with  previous  build.   • What  binaries  changed   • Etc.  The  final  stage  of  the  process  is  to  auto-­‐generate  and  send  an  email  notifying  of  the  build’s  availability,  and  sending  a  semaphore  to  the  Test  infrastructure  to  initiate  automated  testing.    The  process  is  shown  in  Figure  10.4.1.1.  Figure  10.4.1.1:   Piston Layout   configuration   file On-­‐Demand/ Get  layout   Update  Symbol   Release  DVD  for   Create  DVD  image Create  reports per  schedule information server testing Symbol  Server  10.5 Output  The  final  stage  of  the  process,  the  build  is  now  ready  for  Quality  Assurance/Test  to  begin  executing  a  set  of  “smoke  tests”  or  Build  Verification  Test  (BVT),  to  ensure  the  build  will  perform  fundamental  activities,  (e.g.  install).    Based  on  the  successful  completion  of  the  BVT,  more  extensive  test  suites  will  be  executed  on  the  build.    Often  the  results  of  these  tests  will  serve  as  input  for  subsequent  builds.    The  attribute  table  for  Output  is  as  follows,  Table  10.5.1  Attribute   Details   Comments  Owner(s)   • Quality  Assurance/Test   Responsible  for  jointly  maintaining  the  install   tasks.    Build  responsible  for  ensuring  that  the   build  is  readied  for  release.  Copyright Citrix Systems, Inc. All rights reserved.   18    
  • Stakeholder(s)   • Project/Release   Everyone   in   this   list   is   vested   in   ensuring   that   the   Management   build  is  successful  at  all  stages  of  the  process.   • Build     • Product  Development   • Quality  Assurance/Test   • Product  Marketing/Sales  Input(s)   • Installable  product  image     • Build  reports   • Semaphore   for   automated   testing  Control(s)   • Test  standards   Test  suites  must  conform  to  company  standards.  Output(s)   • Test  results     • Bug  reports    10.5.1 Automation  at  Citrix  The  Citrix  Automation  Development  Infrastructure,  (CADI)  is  a  SDK  which  provides  a  Test  Automation  framework  for  creating  test  suites  which  can  then  be  executed  in  various  test  environments.    Skynet  is  a  NAS-­‐based  cloud  infrastructure  that  provides  a  highly  available  virtualization  infrastructure  that  will  allow  for  fast,  on-­‐demand  access  to  virtual  machines  that  can  be  acquired  by  CADI.    (Skynet  is  part  of  a  larger  quota-­‐based  storage  infrastructure  and  will  not  be  discussed  in  detail  here).  CADI  polls  the  build  release  area  for  a  semaphore  file  which  indicates  a  build  is  available.    When  it  finds  a  file,  it  provisions  a  test  infrastructure  based  on  the  product  build  requirements  extracted  from  the  Test  infrastructure  database.    CADI  then  executes  the  BVT,  which  determines  if  the  build  is  good  enough  for  more  exhaustive  testing.    If  the  BVT  reports  a  failure,  testing  stops  on  the  build,  and  it  is  marked  as  a  failed  or  bad  build.  If  the  BVT  reports  success,  the  build  is  then  submitted  for  additional  testing  to  determine,   • Regressions   • Performance   • Scalability   • Overall  stability  Data  from  these  tests  is  used  to  generate  test,  bug  and  other  useful  reports  that  indicate  the  state  of  the  product  and  project.    The  high-­‐level  flow  is  shown  in  Figure  10.5.1.1.      Copyright Citrix Systems, Inc. All rights reserved.   19    
  • Figure  10.5.1.1:   CADI Test   infrastructure Provision  Test   Execute  additional   Create  reports,   Build  is  ready Execute  BVT BVT  Pass? Yes Environment test  suites generate  CPRs,  etc. No Report  and  die Skynet  11 Optimizing  the  System  The  Build  Vision  must  ensure  the  build  system  is  optimized  for  the  following,   • Determinism   • Performance   • Self-­‐service   • Standards  Having  these  areas  optimized  will  ensure  that  the  Build  “Customers”  are  happy,  and  the  overall  Build  needs  and  expectation  of  the  organization  are  being  met  and  exceeded.  11.1 Keep  it  Deterministic  The  term  “Determinism”  is  overloaded  from  its  usual,  “fatalistic”  definition,  in  that  it  means  that  the  capability  within  the  Build  system  exists  to  know  everything  about  a  product  build.    Not  only  does  it  have  to  exist,  it  has  to  be  as  efficient  as  possible.      For  example,  the  Perforce  command,  filelog  will  accurately  report  what  changes  went  into  a  build.    Having  the  additional  capability  to  determine  which  code  submissions  caused  which  binaries  to  change  is  not  necessarily  as  simple  yet  is  critical  for  Quality  Assurance/Test  to  be  able  to  determine  quickly,  in  order  to  properly  seed  their  test  bed.  Copyright Citrix Systems, Inc. All rights reserved.   20    
  • Example:   Historically,  when  a  XenApp/XenDesktop  build  was  released  to  Test,  there  was  no   capability  to  accurately  determine  which  binaries  changed  from  build  to  build,  (checksums   and  binary  comparison  tools,  while  useful,  could  not  provide  the  levels  of  detail  required   for  targeted  testing).    As  a  result,  test  cycles  for  intermediate  builds  required  the  same   amount  of  time  at  those  for  milestone  builds,  which  slowed  the  entire  release  cycle.     An  in-­‐house  developed  tool  called  the  Binary  Sleuth  leverages  the  information  stored  in   the  debugging  symbols  created  by  the  Windows  Symbol  Indexing  functionality,  (for   Windows  builds  only),  to  accurately  determine  which  binaries  change  from  build  to  build.     This  has  enabled  Test  to  target  specific  areas  to  test  based  on  the  scope  of  the  changes  to   obtain  results  much  more  quickly  than  by  starting  at  the  beginning  of  the  test  suites  and   working  down  to  the  areas  that  need  quick  turnaround  verification.    The  format  of  the   report  is  as  follows,     Figure  11.1.1:      Being  able  to  track  and  report  Build  data  as  efficiently  as  possible  is  critical  to  the  overall  speed  of  the  product  release  cycle.  Copyright Citrix Systems, Inc. All rights reserved.   21    
  • Example:   The  XenApp/XenDesktop  product  has  evolved  from  a  monolithic  build  to  a  compilation  of   30  major  components,  (with  over  64  subcomponents).    There  are  many  different  data  sets   that  need  to  be  examined  on  a  per  build  basis,  and  having  these  automatically  generated  in   report  format  is  critical  for  the  Release  team  to  keep  development  notified  of  the  schedule.     A  list  of  the  currently  available  reports  is  as  follows,     Figure  11.1.2:      11.2 Keep  it  fast  The  ideal  goal  here  is  to  provide  builds  faster  than  they  can  be  consumed.    While  this  may  be  unrealistic,  keeping  the  build  times  as  low  as  possible  will  definitely  mean  that  the  consumers  will  remain  happy  and  able  to  perform  their  respective  duties  without  having  to  wait  for  the  build.  Copyright Citrix Systems, Inc. All rights reserved.   22    
  • Example:   At  one  point  before  the  Technology  Component  methodology  was  introduced  at  Citrix,  the   build  time  for  a  monolithic  XenApp  build  was  approximately  16  hours.    These  extremely   long  build  times  made  it  very  challenging  to  keep  to  aggressive  Test  and  Release  schedules.     Several  factors  contributed  to  the  time,     • Compiling  code  multiple  times   • Compiling  “dead”  code   • Build  infrastructure  not  optimized.   • Dependencies  not  well  defined.   • Etc.     Over  time  and  with  concerted  effort,  the  build  times  have  been  reduced  significantly  to   approximately  5.5  hours  to  build  all  the  components  in  serial.    Having  said  that,  build  times   have  been  further  reduced  as  a  result  of  SDKs  and  components  releasing  semi-­‐ autonomously,  (as  per  the  TC  model)  and  simply  being  included  as  part  of  the  DVD  layout.   Additional  development  work  to  simplify  the  component  dependencies  have  reduced  the   times  as  well  as  optimized  incremental  build  times.    Build  times  reports  have  been  added   to  the  Build  portal  and  can  now  be  reviewed  to  determine  the  cause  for  any  significant   changes.    A  sample  report  is  shown  in  Figure  11.2.1.     Figure  11.2.1:    Copyright Citrix Systems, Inc. All rights reserved.   23    
  •  Leveraging  Perforce  functionality,  like  P4Proxy,  as  well  as  isolated  network  segments,  fast  hardware,  etc.  are  starting  points  to  a  fast  build.    The  Citrix  XenApp/XenDesktop  build  farm  topology  shown  in  Figure  11.2.2  below,  utilizes  these  optimizations  as  follows,   • Uses  fast  Bare-­‐Metal  machines,  (64GB  RAM,  Quad  core  half-­‐height  blades)  paired  with  a  NetApp   storage  device.   • Maintains  multiple  and  dedicated  Perforce  proxy  servers  across  the  sites  to  increase   performance.   • Isolates  the  Build  farm  on  its  own  optimized  10GB  network  segment  to  avoid  traffic  “jams”.   • Maintains  a  well-­‐defined  Business  Continuity  process  to  ensure  that  build  jobs  can  be  quickly   moved  from  site  to  site.  Figure  11.2.2:   US UK Business  Continuity  Center Cam Chf Perforce  Svr  1 Perforce  Svr  2 Perforce  Svr1/ Perforce  Svr1/ Proxy  ports Proxy  ports Builds  NAS Builds  NAS Builds  NAS (+BC  for  Chf) (+BC  for  Cam) 1GB 1GB 1GB Build  Farm 10GB NAS Servers Servers Servers P4  Proxy Data P4  Proxy Devs Devs Devs Build  machines Build  machines  (+BC  for  Chf) (+BC  for  Cam) Build  machines  Copyright Citrix Systems, Inc. All rights reserved.   24    
  •  Additional  tweaks,  like  incremental  builds,  parallel  component  builds  will  also  further  reduce  build  times.    Some  compilers,  such  as  Microsoft’s  MSBuild  engine  capitalize  on  the  multi-­‐processor  capabilities  of  current  hardware  to  spawn  non-­‐dependent  components  across  the  available  processors  to  more  fully  optimize  build  times.  11.3 Keep  it  simple  Most  people  enjoy  a  good  puzzle  for  fun.    Most  software  engineers  don’t  have  the  same  feeling  when  they’re  trying  to  meet  feature  milestones  and  project  milestones,  and  find  themselves  lost  in  an  intricate,  cryptic  build  system.    The  cardinal  rule  of  build  systems  is  to  keep  the  user  experience  as  close  to  an  “out  of  the  box”  install  as  possible.    This  means,  don’t  create  a  set  of  elaborate  scripts/steps  that  modify  the  build  environment  needlessly.    Recall  Microsoft’s  “Razzle”  build  system?    It  worked  great  as  long  as  nothing  changed,  and  when  something  within  the  environment  did,  then  the  results  were  unknown,  and  there  was  not  an  obvious  way  to  debug.    Happily,  Microsoft  learned  their  lesson,  and  the  current  Visual  Studio/MSBuild  system  is  very  extensible  to  meet  developers’  needs.    The  ideal  Build  Vision  should  support  a  core  build  “framework”  into  which  additional  layers  or  components  can  plug  to  facilitate  the  organization’s  requirements,  (e.g.  binary  signing,  security  settings,  etc.)  At  Citrix,  for  Windows-­‐based  builds,  Visual  Studio  and  Perforce  comprise  the  core  framework,  and  as  few  MSBuild  plug-­‐ins  as  required  are  leveraged  to  create  the  flagship  XenAppXenDesktop  products.    The  details  of  the  Vulcan  Build  system  were  previously  discussed  in  section  10.3.1.  11.4 Keep  to  standards  Use  standards  where  it  makes  sense  to  do  so,  and  resist  the  dark  embrace  of  the  “cookie  cutter”  approach  because  this  is  ultimately  doomed  to  fail.    How  the  Build  Visionary  chooses  to  define  the  standards  should  be  based  on  development  feedback  and  the  problems  they  are  trying  to  solve.    For  example,  there  could  be  a  standard  way  to  build  all  C#/Managed  code  components,  or  a  way  that  all  Windows-­‐based  components  are  signed.    The  reality  is  that  the  combinations  are  many,  and  to  reiterate  should  be  based  on  the  needs  of  all  the  Build  stakeholders.  12 Scheduling  Updates  and  Maintenance  12.1 How  to  do  it  The  Build  Vision  should  account  for  system  maintenance  and  updates,  as  this  is  an  important  part  of  keeping  the  Build  system  up  to  date  and  meeting  new  requirements.    As  mentioned  earlier,  keeping  the  various  parts  of  the  system  componentized  as  plug-­‐ins  to  the  core  framework  will  facilitate  ongoing  maintenance.    Additionally,  the  following  guidelines  further  ensure  seamless  transitions  as  a  result  of  updates,   • Stage  updates  to  get  maximum  ROI.    Group  similar  updates  together  to  minimize  the  amount  of   change  to  the  system.    This  may  seem  obvious,  but  is  often  overlooked  by  the  overzealous  Build  Copyright Citrix Systems, Inc. All rights reserved.   25    
  • engineer.   • Automate  as  much  as  possible.    scripting  updates  such  that  they  require  minimal  manual   intervention  also  mitigates  any  potential  human  error   • Test  to  trust.    This  is  the  most  critical  part  of  maintenance.    Nothing  will  destroy  the  credibility   of  the  Build  team  quicker  than  a  botched  upgrade  resulting  in  downtime  for  Development   teams.    Trust  between  the  stakeholders  is  a  requirement  for  a  successful  Build  Vision,  so  test,   test  and  test  some  more  before  implementing  updates  into  production.   • Keep  the  Stakeholders  in  the  loop  on  progress.    Notification  of  key  events  during  the   maintenance,  (start  time,  end  time,  benefits,  changes  in  behavior,  etc.)  will  also  shore  up  trust  in   the  Build  team  and  Vision.    The  Stakeholders  will  feel  like  they  are  “in  on”  events,  even  if  they   don’t  need  to  be,  plus,  the  Build  visibility  doesn’t  hurt  the  reputation  of  the  team  either.  12.2 When  to  do  it  Updates  come  with  some  overhead  in  that  they  can,  (and  usually  do)  come  with  some  amount  of  change  to  the  system.    It  is  critical  that  they  are  scheduled  with  the  knowledge  and  consensus  of  the  Build  stakeholders.    Following  the  following  general  guidelines  will  ensure  that  there  will  be  minimal  confusion  as  the  result  of  Build  System  maintenance,   • Align  updates,  (as  much  as  possible)  with  project  schedules  and  milestones.    The  last  thing  the   Build  Visionary  wants  is  to  conduct  maintenance  during  a  critical  project  milestone,  (e.g.  the  last   weekend  before  the  project  achieves  final  function/code  freeze).   • Attach  updates  to  a  particular  project.    At  the  end  of  the  day,  it’s  really  all  about  getting  the   product  to  the  customer,  so  having  a  Build  Vision  in  general,  (in  addition  to  the  maintenance),   attached  to  a  project  ensures  that  the  Vision  will  be  funded  properly.    The  savvy  Build  Visionary   will  work  closely  with  the  Release  and  Development  teams  to  embed  needed  Build  System   updates  into  the  project  schedule.  13 Final  Thoughts  Like  everything  else  in  the  technical  industry,  the  Build  Vision  never  ends;  it  is  always  evolving  to  meet  new  requirements,  tools  and  processes.    As  we  move  forward  into  whatever  the  next  new  “thing”  that  will  become  the  norm  in  Software  Development,  and  companies  increasingly  focus  on  trimming  costs  and  increasing  revenues,  the  Build  Visionary  must  take  care  to  nurture  the  Vision,  keep  it  looking  forward,  and  constantly  demonstrating  the  ROI  to  everyone  who  will  listen.    This  will  help  secure  the  Build  Vision’s  position  as  a  critical,  core  team  within  the  organization.    The  Build  Visionary  will  find  himself/herself  wearing  many  hats,  from  the  technical  to  the  salesperson,  ensuring  longevity  of  the  Build  Vision.  Copyright Citrix Systems, Inc. All rights reserved.   26