A talk from the London Continuous Delivery meetup group on Infrastructure development on windows, things to think about and war stories

1. INFRASTRUCTURE DEVELOPMENT
ON WINDOWS
London CD meetup – 4th September 2014
@owainperry

2. Who am I
 Owain Perry
 Software architect at Justgiving.com
 @owainperry
@owainperry

3. Justgiving.com
 Enable people to raise money for charity
 £~400 million
 20 to 75 transactions / min avg
 Steady load
 Very spike loads (viral traffic)
@owainperry

4. What am I going to talk
about?
An windows infra dev 101 for those starting out
Things to think about
@owainperry

5. Quick poll
Who’s working in a Windows environment?
 Puppet / chef / other
@owainperry

6. Our Architecture
(as of today)
 Lots of change in our software architecture
 Existing monolithic application (chopping up)
 Micro services
 Event Driven Architecture
 CQRS
 Event sourcing
@owainperry

7. Just giving server estate
 Smaller hosting supplier based on VMware
 250 - 300+ servers mostly in AWS, growing
rapidly.
 Some servers in Azure
@owainperry

8. Why do automation?
The Friday failure -We go live on Tuesday
 Auto upgrade=true
 Chocolatey upgraded to a non recoverable
broken state
 One we realised the issue (Mean Time To
Comprehension ~ 1hr)
 Changed the setting
 We blew away the whole estate ~60 servers
while at lunch
 Job done.
@owainperry

9. Tools and tech stack
 Windows
 Linux (a little)
 Chef
 Vagrant
 Minitest (test kitchen)
 Chocolatey
 Artifactory
 Ruby
 Powershell
 Berkshelf
@owainperry

10. Where do you start?
 Configuration management
 There is a need to manage configuration
between environments
 Configuration of servers
 What software is installed.
 We are using Chef
@owainperry

11. Why Chef?
 Ok windows support
 DSL and ruby at the same time....
 Search (powerful, but you need to use it with
thought)
 ChefDK
 Great community
@owainperry

12. Doing windows infrastructure
development
 Vagrant
 Work on your servers from the comfort of
your own desktop
 Images for windows are large 6+ GB
 *hint use packer.io to build your base images
@owainperry

13. Package management
 To keep things sane , you need package management
 Chocolatey - it works
 Oneget – in powershell 5
 Download a package over http
 Download a specific version
 Easy to build
 (just, but it's not apt-get or rpm) still a bit beta , but better
than nothing
 *hint, build your own self contained packages, so you don't
depend on the internet
 *rant come on MS lets party like it’s 1999
@owainperry

14. Artifact repository
 Store your Software deliverables
 Chocolatey packages
 Other stuff? (custom gems)
 Supports HA , fast , support (becomes very
central to infrastrcuture)
 examples include Nexus, Artifactory,....
@owainperry

15. Mini test (test kitchen)
 Write tests around your infrastructure - start
doing TDD
 Just provides some sanity when changing
things
 Takes discipline
@owainperry

16. Using CI
 to upload your infrastructure code to server
 Execute tests on you infra code
 Supports TDD
 Makes change to the server repeatable.
 Version stamping
@owainperry

17. Writing windows automation
 Chef primitive providers (pretty good
coverage)
 To powershell or not to powershell?
 How do you make things idempotent with
powershell?
 Writing things in ruby? (for chef it's more
native)
@owainperry

18. Who's the boss?
 Group policy? Chef?
 What is the point of group policy in a chef
world?
 E.g. Create a local user to run a service using
chef, group policy will try and remove it.
@owainperry

19. Frustrating things about
windows
 SomeUser (domain or local?)
 ./Someuser
 hostname/someuser
 etc...
 Installing .Net 3.5 can only be done using
features
 Writing a shutdown script
 Installing .Net 4 via winrm (can’t install
update)
@owainperry

20. Yesterdays issue…
 Servers Ran out of diskspace due to local
caching
 20 mins to delete the data on each server
(High CPU load) while serving traffic.
 We flicked from the blue to green side
 Just deleted the servers and recreated (~10
mins)
 Automation is just awesome….
@owainperry

21. So…
 Choose a configuration manager
 use vagrant - it's just awesome
 sort out package management
 Try and do TDD if you can
 Get some CI in place
 Write code
 (get frustrated with Microsoft every couple of hours)
 Delete your estate and rebuild it while you have
lunch , it makes you feel great
@owainperry

22. Questions?
@owainperry