More Related Content
Similar to Preetha chaos community day lightning talk
Similar to Preetha chaos community day lightning talk (20)
Recently uploaded
Recently uploaded (20)
Preetha chaos community day lightning talk
- 2. Software Engineer @ Hashicorp Distributed Systems @preethaa @preetapan About Me
- 3. Copyright © 2017 HashiCorp Any Infrastructure. Any Application. Anywhere.
- 4. Consul ● Service Discovery ● Health Checks ● Key Value Store
- 7. Issues with consul + containers 2013 2014 2015 2016 2017 15 5
- 10. Reasonable design decisions made two years ago in software, Start to feel like driving 1980’s car in a 85 mph highway..
- 11. Design Decision #1 - Raw IP in Configuration Use raw IP addresses in configuration: e.g “123.45.6.7:8200”
- 12. Problems with raw config values Must change config on restart Configuration split across files and environment variables
- 13. Solution - Declarative Configuration ipAddress = “{{GetInterfaceIP eth0}}” https://github.com/hashicorp/go-sockaddr
- 14. Design Decision #2 - Operator API Operators issue commands Used to add/remove servers, change configuration etc.
- 15. Problems with Operator API Line between operator and user is blurry Operators make mistakes Infrastructure moves around too fast, not feasible to wait for manual intervention
- 16. Solution - Auto Pilot Prefer self healing rather than operator intervention Monitors environment and fixes problems
- 17. Design Decision #3 - Server Identifiers Use IP+Port as a unique identifier for servers
- 18. Problems with Server Identifiers Restarted container is still the same server, but now it has a different id
- 19. Solution - Address Provider Use UUIDs Treat IP addresses as an ephemeral concept. https://github.com/hashicorp/raft/issues/237
- 20. In Conclusion.. Many more interesting failure modes.. Building a new system today? Think about whether it will ever have to run inside a container Invest in destructive testing that introduces container specific failure modes
- 21. Thank you
Editor's Notes
- At Hashicorp, we build software to provision, secure, connect and run any infrastructure, anywhere.
- I work on Consul - which is the connect piece. Consul is a open source system that provides service discovery, health checks and a strongly consistent KV store.
- This is a high level architecture diagram of consul, the important detail to grasp here is that a consul deployment consists of clients and servers, and they need to be able to identify each other on a network to send messages. Servers perform leader election.
- Quick show of hands, how many of you use orchestration/containers in production? This is a snapshot from google trends - I find this more a leading indicator of this.
- I had constructed this one looking at open github issues that mention issues using consul with docker / docker swarm and kubernetes. Granted, this is in the lower double digits, but why?
- lets look at an analogy. I am from Austin, texas, AND This is highway 130 in Austin, it has the largest posted speed limit in the US. You can’t go any faster legally.
- This is the picture of the my family’s first car in the 80s
- I am going to talk about three specific examples from consul.
- The first design decision is to use ip addresses in configuration,, like this example. Servers need to bind to a specific address and port, and we might as well get that from a config file or environment variable.
- The problem with this approach is that it makes you have scripting to keep config files or environment variables in sync when containers restart and you get a new IP.
- Our solution was to do declarative configuration. Hashicorp’s go-socketaddr library does this. As an example, this template lets you specify HOW to obtain the ip address, instead of specifying WHAT the value is
- Systems like Consul have both users and operators. Users for consul are concerned with registering services and reading values from the KV store. Operators handle configuration changes, moving things around. Its quite reasonable to provide an API for operators.
- This also breaks down in a containerized world. If your infrastructure movees around every few hours, its not reasonable to expect manual intervention
- Our solution was Auto pilot. Its a self healing mechanism that monitors the environment around it and fixes problems, so the neaed to use the operator api becomes very rare.
- Servers need a way to identify themselves, and so we used use ip+port, because in a given cluster that needs to be unique.
- This causes the consensus algorithm to break because ids are not reliable.
- We solved this inside our consensus algorithm library (raft) by introducing an address provider abstraction. We treat addresses as an ephemeral concept, and use providers to figure out the value at run time. SO when the same container it restarted and it gets a new IP, it can ask the provider for its address.
- There are many more interesting ways in which software fails when its contanizered. If you are building a new system today, think about whether it will run inside containers, because that influences design choices. We also need to really invest in destructive testing around this.