Slides from the talk given at the 28 meeting of Spbalt.net

1. Distributed DevelopmentProject FreeSpace Alexey Suvorovalexey@activemesa.com Dmitri Nesterukdmitri@activemesa.com

2. This Talk Is About… Speed How to get things done faster Quality How to get feedback faster How to get more testing done Manageability Cloud monitoring & control Decentralization/fault-tolerance Not only development!

3. Joel’s Test Do you use source control? Can you make a build in one step? Do you make daily builds? Do you have a bug database? Do you fix bugs before writing new code? Do you have an up-to-date schedule? Do you have a spec? Do programmers have quiet working conditions? Do you use the best tools money can buy? Do you have testers? Do new candidates write code during their interview? Do you do hallway usability testing? Typical answer: No.

4. Best Tools Money Can Buy? Hardware Fast CPU Lots of RAM SSDs Multiple monitors Software Commercial issue tracking Paid source code hosting File sync services ReSharper

5. In large/complex projects IDE interaction is slow Code analysis is slow Compilation is slow Testing is slow (Re)deployment is slow

6. IDE interaction is slow IDEs are slow, but we cannot ditch them We have nearly few software options for optimizing IDEs E.g., VS is both disk I/O-bound (SSD a must) and CPU-bound We cannot relocate, e.g., the ReSharper cache into a distributed service IDEs can be spawned on same project in many machines Multiple screens/remote desktop windows Synchronizable with Dropbox, SugarSync, etc. But project/solution reloads in VS will kill you

7. Code analysis is slow In-depth analysis of either compiled or source code is computationally intensive NDepend, FxCop and others can all be run remotely Not just on the build server Most of these tools output report files Can send these to origin Some of these tools can be made to work on per-file/per-project rather than per-solution

8. Compilation is slow Compilation is C#/VB.NET – acceptable F# – slow C++ – atrociously slow Made worse by pre/post-build PostSharp Entity Framework Code Contracts Moles Etc. VS compilation process inefficient Will rebuild projects that haven’t changed Will not parallelize by default MSBuild is parallelizable /m:n Can spawn multiple processes

9. Testing is (painfully) slow Unit testing is badly parallelized MbUnit’s [Parallelizable] Same in NUnit 3 Can easily parallelize at different granularity Test case/method Test fixture Test assembly

10. Fear of Builds/Tests Developers loath to compile or run tests too frequently Disruption of focus leads them to Surf the web Go for coffee <insert your pastime here> Everyone loses Loss of concentration/motivation Developers never ‘in the zone’ TDD does not work

11. Who cares? Developers Employers Fixed salary Don’t care about TTM Accustomed to substandard tools/equipment View compilation as a one-time process Don’t care about frequent/continuous testing See manual deployment as normal More concerned with saving money than getting things done Uninformed about good/best practices Not concerned with quick delivery (in case of service companies that charge by the hour)

12. Problem: nobody recognizes compilation/testing as wasted time

13. How to speed things up? Optimize or buy a faster computer SSDs More RAM Faster CPUs Costly! Has to be done for every developer. Alternatively… use existing infrastructure Both physical (e.g., dev machines) and virtual Distribute workload between machines Use idle resources – no need to buy new machines.

14. Status Quo Computers get faster More cores per CPU Faster hard drives (SSD, hybrid) Software gets more demanding Windows eats more and more RAM & HDD VS is slower Everyone else follows suit The overall development experience isnot getting any better

15. Why Distributed? Resource under-utilization A typical enterprise (IT-specific or not) is unlikely to use 100% of processing resources Resource overload Bottlenecks in servers Resource costs Server-grade hardware Reliability concerns (e.g. RAID)

16. Three Pillars of Distribution Get data on everyone’s machine Cloud storage/file sync Verification necessary Get machines chatting with one another XMPPclient on each node Load balance and optimize execution plan Send commands to do work, get results Work items synchronized via cloud storage Redundancy/reliability guarantees Integrate with existing systems Easy because XMPP uses XML

17. Scale Vectors Core Machine Better support for Multi-Core Exists in some cases MSBuild MbUnit (+ NUnit 3) Could be leveraged in the general case Not easy! Needs to mind end user’s preferences Support arbitrary networks (both on- and off-site) Need to control code sync (security) Can go for full resource utilization (esp. off-hours) Speculative processing E.g., Monte-Carlo simulations Operations which are prohibitively resource-intensive E.g., mutation tests

18. Leveraging the Model Compilation Compiling on dev’s machine is counterproductive Compilation of some languages (C++, Scala) takes far too much time But the problem exists everywhere (.Net, Java) Deployment Testing Large test base cannot be run on a dev’s machine CI is not the answer Constrained to a single machine Can be distributed, but not straightforward Code analysis Very costly Coverage analysis Extremely costly

19. Challenges Load balancing File synchronization Security

20. Compilation MSBuild Builds all major types of VS projects Can parallelize locally (/m:n, n=# of processes) Builds block VS Build on the UI thread Builds often inefficient Cannot build only projects affected by changes Cannot use multiple machines

21. Distributed Compilation Dramatically speed up solution build Determine project dependencies Build different projects on separate machines Use multiple MSBuild processes per machine Depending on CPU count & power Does not distract the developer Development machine usable without interruption Quicker feedback on errors Allows to instrument a continuous build policy Build on ever file save

22. Distributed Testing Testing is slow Unit testing is largely not parallelized MbUnit [Paralellizable] Nunit will only support it in v.3 Not parallelized between several machines Testing in specific environment difficult Requires complicated (possibly manual) deployment processes Developer typically only tests on their own box (+ maybe CI server) Distributed testing ensures tests work everywhere

23. Side Effects Side effects are unwelcome on users’ machines Environment changes may have undesired consequences Builds are typically exempt from this They do not affect anything beyond solution work folders Unit tests may or may not affect host system Integration tests typically do affect hosts Require ‘clean’ set-ups

24. Isolation No side-effects Irrelevant, just take care of load Side effects Process-level virtualization (for existing machines) JauntePE App-V Virtual machines Hyper-V ESX

25. Virtualized Testing Creation of multiple physical nodes is costly Physical machine re-configuration takes too much time Can configure a virtual test environment with Hyper-V System Center Virtual Machine Manager Virtual/physical migrations Different hardware requirements Multi-CPU system Fast disks Very large amounts of RAM

26. Project FreeSpace Private Cloud Infrastructure XMPP + file sync Single-MSI deployment Plugin architecture Fully self-updating Decentralized service orchestration Self-organizing Each node has identical capability Easy to administer

27. Project FreeSpace Features Distributed Compilation Initially MSBuild Distributed Unit Testing Initially via Gallio test automation framework Distributed Integration Testing Virtual machine management Initially via Hyper-V Distributed deployment E.g., create new VM for testers with appropriate binaries etc.

28. Benefit Summary Better than Continuous Integration Better than Continuous Testing Better than local compilation Better than local testing

29. Better Than Continuous Integration Good for long-running builds/tests Happens on a single machine Can set up, e.g., multiple instances, but it’s not straightforward Not designed for distributed builds Not optimized for idle processing Assumes server is dedicated Does not give immediate feedback Typically works on commit I.e., detects source control changes

30. Better Than Continuous Testing Testing is often more costly than compilation Typically, tests run on commit Continuous testing (e.g., Mighty Moose) systems ensure that Tests run on each save Only tests affected by changes are executed Fast feedback… But not instant – you still need to recompile. Given the option, why not build/test thingsall the time?

31. Better Than Local Compilation Does not block IDE Scales across your network Much faster builds Immediate feedback

32. Better Than Local Testing Much faster recompilation Tests do not tax developer CPU Allows for immediate testing in different environments Tests happen in parallel (where possible)

33. That’s all! Questions?