Talking through our journey to achieve congruent infrastructure at Zopa.

1. Hashicorp Meetup 26th June 2018
Building Congruent infrastructure @Zopa
Ben Coughlan

2. 2
Ben Coughlan
Senior Zopa Reliability Engineer
• Version 1 (software consultancy), SSE Airtricity, Allianz, Qualcomm.
• Docker Everywhere! TLS Everywhere!
• Joined Zopa in February 2017.
• Ben.Coughlan@zopa.com

3. What is Congruent infrastructure?
Divergence:
• Manual administration.
• Scripts, ssh, rsync, manually mounting and extending disks.
• Constantly creeping from desired state.
• Snowflake Servers.
Convergence:
• Configuration Management used to spin up infrastructure.
• Ansible, chef, puppet, salt.
• CM tools are unreliable, impossible to keep the state of all
servers the same.
• SSH allows for configuration drift.
Congruence:
• No configuration management.
• Immutable infrastructure.
• No state creep, because there is no SSH.

4. How to build congruent infrastructure.
• Build software, not servers.
• The first platform will take weeks or month’s, the second one will take minutes.
• Instead of updating servers, you completely replace the servers. Think cattle, not pets.
• Build hardened systems, with lots of monitoring.
• Everything needs to be immutable.
• Manage your state.
• Standardise your tools, languages, practices, add an order to everything, and build it
into your culture.

5. The tools for SRE’s @Zopa
Tools
• Terraform
• Vault
• Consul
• CoreOS (Ignition)
• Docker
Telemetry
• Prometheus
• Grafana
• Splunk
Languages
• Golang
Platforms
• AWS
• vSphere

6. The ethos for SRE’s @Zopa
• Try to remain vendor agnostic
• Use open source where possible
• Everything needs to be immutable
• Everything runs in a Container
• TLS everywhere
• Telemetry everywhere
• Perfect is the enemy of good

7. The Challenge – Infrastructure at Zopa.
The Monolith
• Overloaded DC’s.
• Many single points of failure.
• Many snowflake servers.
• Every transaction goes through the monolith.
• Single digit production releases per week.
• Huge range of databases sitting on three servers.
• Divergent infrastructure, very little configuration
management.
Tech Stack
• Ubuntu VM’s (some running applications in Docker).
• Windows Server running C# and .NET Services.

8. The Solution – Infrastructure at Zopa.
Microservices
• Multi Cloud.
• Well defined, and split pipelines.
• Hundreds of deployments per day.
• Distributed databases.
• Immutable infrastructure.
Tech Stack
• Kubernetes!!!!!!1 
• Kafka
• Containerize everything.
• Windows Services to move to .NET Core.
• Move as much as possible into Kubernetes.

9. Use Case: Building Kafka with congruence in
mind.
Why is it hard to automate a deployment of Kafka?
1. Kafka is far from cloud native.
2. It can be unstable, even when built manually.
3. Zopa’s use case of using Kafka for critical transactions, and using it as a persistent database,
meant that we couldn’t ever afford to lose data.
4. As it’s open source, it had a lot of inconsistent behaviours, run’s different version’s of a JVM across
products of the same version, has different jar’s on each product, etc. This took a lot of pull
requests to resolve over the past 12 months.
5. Logging to file, SSL/TLS/AAA/ACL’s or “security”, dockerised images, rebalancing a cluster
without taking it offline, online operations, etc, were all an afterthought with this product.
6. Initially built outside of Kubernetes, due to the complexities of Kafka.

10. Infrastructure Orchestration - spinning up the
machines.
TERRAFORM – Infrastructure Orchestration.
IGNITION – User Data.
Remote Storage is attached -
Managing state.
Certificates are issued to each server..
Bootstrapping takes place.
Service is up and running – telemetry.

11. Terraform
1. Pulls in secrets from Vault to access AWS. (vault provider)
2. Pulls in Values from remote state files. (for the VPC, subnets, etc)
3. Utilises reusable modules from the Zopa stack, (spot pricing, service discovery, re-attachable storage,
etc)
4. Creates launch configurations, autoscaling groups, Route53 addresses, etc.
5. Passes the Ignition user data into the launch configuration.

12. Orchestration/Config management: Rationale

13. Ansible : Configuration Management
• Agentless in push mode.
• In push mode, requires a solid SSH Connection.
• Ansible's reliance on SSH makes it difficult to use securely in an environment with a lot of churn.
• Ansible has no native protection against concurrent runs.
• Ansible in push mode is really slow.
• It’s workaround for its slow performance -- tags -- encourages config drift.

14. Ansible : Configuration Management

15. IGNITION – User Data
• Kicks in at runtime.
• Templates dynamic “drop-in” files instead of carrying out configuration management.
• No Configuration drift, you kill the server every time you want to update. (clean slate)
• Adding/changing configurations is much easier.
• Far easier to use than legacy CM tools like Ansible, Puppet, Chef, etc.
• No Race conditions, no lag, no SSH.
• Secure! Specific IAM permissions to retrieve it’s own user data.

16. IGNITION – User Data

17. Infrastructure Orchestration – Rack awareness and Persistence.
1. “Smilodon” is used to attach a remote EBS volume.
2. Once the volume is attached, a file that has been placed on the
volume by Terraform, will have the “rack” or AWS site in place,
and the node number.
3. The node number is then used to attach the correct network
interface.
4. The network interface gives the node a second IP address, which
is tied to the Route53 record.
This allows us to ensure that the correct data is used for each node,
which prevents rebalancing, and ensures we can lose any AWS
availability zone, without losing any data.

18. SSL Certificates – Issuing and renewal.
• Vault PKI endpoint generated for each platform.
• Consul configuration in place to allow for distributed lock’s, so that certs are issued
one at a time.
• “Certificated” an in-house tool, contacts Vault and Consul to issue certs to each
machine.
• Attempts to get a “lock” to issue a cert for itself.
• Gets the lock.
• Issues certs, and generates keystores and truststores.
• Mounts those cert stores into Kafka’s docker container.
• Starts/Restarts the Kafka service.
• Releases the lock.

19. Bootstrapping takes place
• Services Register with each other
independently.
• Zero manual work done to carry out the required
setup steps in Kafka.
• Based off dynamic “count’s” in ignition code.
• Dynamic, and easily scalable, use of
autoscaling function in AWS makes horizontal
scaling trivial.

20. • Prometheus – Built in Application and Server
node exporter used extensively.
• Custom checks used extensively for service
specific deliverables. (Consumes, produces to
cluster regularly, checks disk, io, network, uses
probes for services, checks JVM, etc. )
• Grafana used to visualise each cluster
• Splunk used for Log aggregation, and for
monitoring ACL’s on each interaction/operation.
Service up and running

21. The Terraform lifecycle.
1. Merge changes in Git.
2. Terraform apply.
3. Destroy Servers.
4. Beer.

22. The Challenges of going Congruent.
1. Time
It takes a lot longer to produce a hardened, stable platform.
2. Complexity
Many products don’t work well in an immutable design pattern.
3. Culture
Trying to maintain the same development pattern across a team gets complicated as a team grows.
Trying to convince Puppet/Ansible and Debian fan’s to adopt CoreOS and Ignition instead.
4. Windows
Doesn’t handle configuration management well.
.NET core and dockerised services seems to be the only answer.
5. Finding people to help! 
This is another shameless plug to remind you that we are hiring.

23. Questions?