Recommended
More Related Content
What's hot
What's hot (20)
Similar to Shadowing production requests
Similar to Shadowing production requests (20)
Recently uploaded
Recently uploaded (20)
Shadowing production requests
- 2. MOST COMMON QUESTIONS BEFORE A DEPLOY • Will the new implemented functionality work? • Will it break any existing functionality? - EVERY BACKEND DEVELOPER
- 3. ESTABLISHED METHODS OF TESTING • We only test the edge cases we can think of • The number of code paths grows with project complexity • Time and effort
- 5. YOLO METHODS
- 8. YOLO DRAWBACKS • Requires active monitoring and reverting • Users are still affected by regressions
- 11. SHADOWING ADVANTAGES • Users can't be affected by regressions in the new release • We can compare production and shadow response content and performance to catch regressions!
- 12. SHADOWING ADVANTAGES • Free real world test inputs • Free assertions - know good last version
- 13. REAL TIME ZERO EFFORT RISK-FREE TESTING WITH PRODUCTION REQUESTS
- 16. PROBLEMS
- 17. DATABASE • Side effects of concurrently processing the same request on two application servers • For better response comparisons we need a shadow DB that has the same data
- 18. DATABASE SOLUTION • Dynamically mock the database layer (warning, this may be hard) • Writable production read replica in the shadow environment • Re-synced once enough changes happen • Introduces some comparison noise • Good enough™
- 19. EXTERNAL SERVICES • Sending emails, processing payments, ... • Data integrity • Rate limiting + cost
- 21. KEEPING STATE ACROSS REQUESTS • CSRF tokens • The user doesn't receive the shadow responses • The production state in invalid in the shadow environment
- 22. ACTIVE REQUEST EDITING • Find state tokens in shadow responses • Persist tokens in Redis per user IP • Replace tokens in duplicated shadow requests
- 23. RESPONSE COMPARISON NOISE • Sources of noise • timestamps • RNG • external service data • intended changes
- 25. DEVELOPER'S VIEW
- 33. CONCLUSION • Risk free for end users • New releases are subject to real world edge cases • Automatic regression detection • Increased trust in new releases
- 34. Visit infinum.co or find us on social networks: infinum.co infinumco infinumco infinum Questions?
Editor's Notes
- What are the two main questions you ask before you deploy a new release to production? - does the new functionality work? - did the changes break any existing functionality? My talk is gonna focus on helping us gain confidence to answer the second one with a strong no.
- What we usually do to gain confidence is to write a bunch of tests of all shapes and sizes. They are great, but they don't get us there the whole way. There's always that lingering feeling of "what did we miss in tests, will this really work in production?" In essence we are the limiting factor when it comes to tests. We only test the edge cases we can think of, theres a lot of them and testing all possibilities takes too much time and effort. So what can we do?
- Test code in production! This seems funny at first but is the core principle of the most well known methods of deployment.
- I call these YOLO methods. We push a release to production even though it may have regressions. We don't care about that too much because we are sure we can catch them in time and revert to the previous version
- The first of these methods is blue green deployments. It consists of a pair of production like environments, with the production one containing the current production release and the other one containing the next release. The central router routes all user requests to the new release. In case we detect some regressions/ increased error rates we just reroute everyone back to the old production release.
- Canary deployment are very similar, but differ in the fact that we only send a small portion of users to the new release.
- Both of these methods work to a degree, but they have some drawbacks. We need a monkey/developer to observe the behaviour of the releases and revert back to the old version. Whats even worse is that these methods still allow users to experience regressions. They just limit the exposure to a percentage of users or a time period. Bummer
- Production request shadowing is a method that aims at solving these problems. It's not very widely used, I only found mentions in some Amazon patents and Twitter/Cookpad talks.
- The core idea of shadowing is that user requests are fed through the shadow proxy server and duplicated to both a production environment and a shadow environment which contains the new release. Both application server process the same request and then return the answer to the shadow proxy server. The user is then sent only the production response.
- It's a simple idea, but it brings a lot to the table. Because users only see production responses they can't be affected by regressions in the new release. More importantly we can compare production and shadow content and performance to catch regressions!
- We get free real world test inputs from users in production AND free assertions. If the new release response is not the same as the production one we classify it as a regression.
- This equates to real time, zero effort, risk-free testing with production requests.
- So everything up until now was theoretical, now we'll look at the example I implemented for my thesis. Lets start with the architecture
- This looks a bit more complicated than the last picture. We have a bunch of application servers, 2 in the production environment and 1 in the shadow environment. They are using corresponding separate databases. The main 2 components that I implemented were the shadow proxy server and the command & control application. User requests flow as follows: the HTTP server forwards them to either the production environment or to the shadow proxy server. That in turn duplicates them to a production and shadow app instance with the new release. The results are then stored in Redis for comparison by the command & control application. The user then gets sent just the production response
- As with anything once you start looking closer a bunch of issues start popping up.
- The biggest one is related to the database. Concurrently processing the same user request on two application servers can lead to some unexpected behaviour. Imagine trying to process a requests that wants to delete a record. The faster of the two servers would fire an SQL query and succeed while the second one would return an error as the record wouldn't exist anymore. Same with uniqueness validations. The responses would differ too much To achieve the best response comparison we would want the underlaying production and shadow databases to be separate BIT-BY-BIT copies of each other at all times.
- The correct way to do this would be to dynamically mock the database layer in the shadow environment, but that is pretty hard. Twitter supposedly did it. The easier but less correct way I chose was to introduce a writable production read replica using bucardo. This means that the 2 databases are the same most of the time but the shadow one can deviate to a certain degree. Once it deviates too much we re-sync it back with production to improve response comparisons. Having this deviation does introduce some comparison noise but allows us to have a much simpler infrastructure, all we need is another DB.
- The next problem is external services. There are certain types of requests we don't want to duplicate - sending emails to clients, processing payment data.. We could also run into rate limiting or high costs by just duplicating external service calls
- The solution for this is quite simple - Environment flags. We need to identify and mock all the external service calls in the shadow environment.
- Another issue we would quickly run into especially if we render views is keeping state across multiple requests. This can occur for example with CSRF tokens. Because the user only receives the production response and included token, all the consequent requests that he sends to the shadow environment will be invalid.
- To solve this we need to actively edit user requests. We need to search for state tokens in the shadow responses, store them per user IP in redis and replace them in the duplicated shadow request.
- So once we solve all of those technical problems there comes the biggest usability issue. In a perfect world a request being processed by 2 application servers at the same time with the same release would always return the same response. Sadly this is not true as there are many sources of noise in the responses. A few examples of these are timestamps, values dictated by random number generators, external service data and all the intended changes we made in the new release.
- To combat this I implemented global and per-endpoint comparison rules. By default all response changes result in a negative score. Using a regex expression we can catch the part of the data that changes and set it a neutral score modifier. In this example we'r ignoring the CSRF token in the meta tag