Deploy with Confidence: Knowing Which Deployment Strategy to Apply When

As you create and update your codebase, you must release that change to the production environment with minimal impact on end users. There are different ways to do this, but each one has its pros and cons. With numerous deployment strategies available, selecting the perfect one can be a bit tricky. If you find yourself in that spot, read on.

We'll take a look a six deployment strategies and the steps involved in each. For each strategy, I've also added a short note on when you can use it. Please note that these are quick guidelines and not rules set in stone. You can (and should) modify the steps based on your requirements.

1. Recreate

This is one of the oldest strategies. In this instance, this is what you do:

1. Shut down all instances of the existing application.
2. Wait till all instances are down.
3. Deploy the second version in place of the existing application.

Obviously, this means the application will be unavailable for the duration of the shutdown and boot-up of the application.

Pros

1. The main advantage of the recreate strategy is that it is easy to implement.
2. The application remains in a single state. It is either the old version of the code or the new. Both cannot be active at the same time.

Cons

1. There is going to be downtime for every deployment.
2. If an issue arises following deployment, rollback would necessitate additional downtime.

When to Use Recreate Strategy

Because of the downtime involved, this strategy is seldom used in production environments. The exception is when you are implementing a breaking change that cannot allow any traffic to hit the old code while the deployment is happening for fear of corruption. In such cases, some downtime is inevitable. It's good to set up a downtime page that informs users about the temporary unavailability of the application.

You can also use the recreate strategy in lower environments (like development or staging) where downtime is not an issue.

2. Rolling

This strategy involves slowly rolling out a change across all instances of the application. 

This is how you go about it:

1. Take one instance of the old application out of rotation.
2. Replace it with one instance of the new application.
3. Wait for the new instance to become available/healthy.
4. Repeat the above steps for all remaining instances one by one.

Depending on the system, you can change the approach as needed. For example, you can add new instances first and then remove the old ones, thus keeping the same count of available instances. If you have a lot of instances, you can release two or more instances in parallel.

Pros

1. This is relatively easy to set up.
2. You always have a minimum number of available instances. Hence there is no downtime.

Cons

1. The rolling strategy can be relatively slow to release, especially in case of a rollback.
2. Throughout most of the deployment, you will have two versions of the application available in production. This can be troublesome, especially in case of changes that are not backward compatible.

When to Use Rolling Strategy

This is the default deployment strategy used in most scenarios nowadays. It can be implemented in almost all types of applications as long as your application is load-balanced. 

3. Blue/Green

This is a complex approach that depends on the ability to direct traffic between two sets of server instances. We generally have two similar stacks of the application. Let’s call them Blue and Green stacks. 

When using a Blue/Green deployment:

1. By default, all traffic goes to one stack (Blue stack) and none goes to the other (Green stack). 
2. We deploy the changes to the Green stack and have it verified and tested.
3. Once testing is completed and it is confirmed that everything looks good, traffic is completely switched to the Green stack.
4. This leaves the Blue stack free for the next version of the application to be deployed and tested.

The main advantage of this is if you notice issues after the deployment, you can simply switch back to the Blue stack.

Pros

1. Instant rollout/rollback.
2. There is only one version of the application available at any point in time.

Cons

1. It is costly since you will always have two full stacks of instances available.
2. Testing and monitoring the Green environment is extremely important before making it live.
3. If you introduce changes to the Green stack that break the Blue stack (like a database change), it could impact production.

When to Use Blue/Green Strategy

The Blue/Green strategy is often used where speed and reliability are paramount and cost is not an issue. You can reduce the cost to some extent by launching the Green env before deployment and keeping the Blue around for a set amount of time after deployment.

4. Canary

The name of this deployment method comes from the old practice of using canary birds to assess the safety of mines. Miners used to take a canary with them. As long as the bird was alive, the mine was considered safe. But if it dropped dead, it meant it was no longer safe to venture forth.

This is how canary deployment is done:

1. Deploy a single new instance of the new application. This will be referred to as the canary instance from here on.
2. Redirect a small amount of live traffic to the canary instance.
3. Monitor the instance for issues and also have your team do the same.
4. If no issues are noticed, release the change to all instances and remove the canary instance.
5. In case any issue is noticed, remove the canary instance.

Additionally, instead of redirecting live traffic, you can add rules to allow users with specific headers to connect directly to the canary instance. You could then have your QA team set that header and always hit the Canary instance instead of the live environment. Once they confirm it is ready, introduce a small subset of traffic to the canary environment.

The above diagram shows a rolling deployment after canary testing is complete, but you could switch it using a Blue/Green or any valid deployment strategy.

Pros

1. It is easy to monitor values such as the error rate or load times of the new code.
2. Any issues due to the deployment would only affect a small subset of actual users.
3. Rollback is fast.

Cons

1. Deployment is slow.
2. Similar to rolling deployments, you have two versions of the application available in production at the same time. This means an end user might hit either version when he hits your site. This can be problematic if the two versions are not compatible with each other.
3. Similar to Blue/Green deployments, successful canary deployment depends a lot on the testing and monitoring of the canary instance. This means a dedicated QA team will need to be involved along with custom monitoring rules to verify the canary instance.

When to Use Canary Strategy

Canary services make a lot of sense in most container orchestration tools, like ECS, Kubernetes, etc. They rely a lot on the routing solution (often a load balancer) to direct traffic based on weightage and headers, which is easier to achieve in most container orchestration tools. Using the canary service gives you a lot of the advantages of a Blue/Green while not costing as much. It also gets you live traffic to test while still having a quick rollback option.

5. A/B Testing

This is not just a deployment strategy, but rather a way to make business decisions based on actual metrics. That said, it is very much related to canary deployments. Let’s say you have two possible solutions (represented by Red and Green below) that could be implemented for the same scenario, but you are unsure which will give the best end-user response. With A/B testing, you release both and route traffic to the same based on a particular parameter. And then based on feedback from users, performance monitoring, etc., you decide and go ahead with the solution that has the better response.

This is how it goes:

1. Release an instance of both A and B along with your old version.
2. Use specific routing rules to send a small subset to both A and B.
3. Collect metrics and feedback for both A and B.
4. Based on the solution that has the most positive feedback, roll out the better solution.

Pros

1. You have full control over the traffic flow.
2. You make the decisions based on metrics.

Cons

1. A/B testing is complex to set up.
2. It is extremely hard to troubleshoot.

When to Use A/B Testing

It is used in special cases where you need metrics to decide how to proceed.  That said, you must have some form of distributed tracing to be able to troubleshoot issues.

6. Shadow

This is another version of the Blue/Green deployment. But instead of routing traffic directly to the Green setup, you set up a shadow service. This service duplicates all incoming traffic to the Blue and routes it to the Green in parallel. As a result, you get actual production load to the Green environment. You then monitor the Green stack, and once everything is stable, switch all traffic to it.

This is very complex to set up and needs a lot of special considerations, especially with respect to the outgoing traffic. For example, you don’t want traffic from the Green to trigger the same external API twice. So, you would need to set up dummy services for external APIs. But in doing so, you need to ensure you switch to the actual service before you switch real traffic to the Green stack.

Pros

1. Application performance can be tested with actual production traffic.
2. There is no impact on the end user.

Cons

1. Shadow deployment is very expensive and complex.
2. It needs a lot of special tools to implement, such as tools to duplicate and route traffic, block outgoing traffic, route egress to dummy APIs, etc.
3. It is not a true end-user test and hence can be misleading.

When to Use Shadow Strategy

To be honest, I find it an overkill. The only instance where I would consider it is when I'm changing a major component in my solution. For example, when I want to migrate from an SQL to a NoSQL database. Many teams use it when they want to load-test every major change before releasing it.

Conclusion

As you can see, there are many ways to deploy an application. Some strategies might be faster but riskier, while others are more reliable but slower. It's crucial to weigh these factors and choose the one that best aligns with your project's specific needs and priorities, including the budget. You can also take an existing strategy and tweak it to better suit your needs.