CANARY DEPLOYMENTS

A Canary Deployment is a technique that reduces the risk of deploying a new version of an application by slowly rolling out the changes to a small subset of users before rolling it out to the entire customer base.

HOW DOES IT WORK?

The concepts of blue/green and canary deployments have been around for a while and have been well-established as best-practices for reducing the risk of software deployments. In traditional applications, you slowly and incrementally update the servers in your fleet while simultaneously verifying application health. However, there is somewhat of an impedance mismatch when mapping these concepts to a serverless world. You cant incrementally deploy your software across a fleet of servers when there are no servers!

The answer is that there are a couple of services and features involved in making this possible. Let us explain:

Lambda versions and aliases

AWS Lambda allows you to publish multiple versions of the same function. Each version has its own code and associated dependencies, as well as its own function settings (like memory allocation, timeout and environment variables).

You can then refer to a given version by using a Lambda Alias. An alias is nothing but a name that can be pointed to a given version of a Lambda function.

Traffic shifting with Lambda aliases

With the introduction of alias traffic shifting, it is now possible to trivially implement canary deployments of Lambda functions. By updating additional version weights on an alias, invocation traffic is routed to the new function versions based on the weight specified. Detailed CloudWatch metrics for the alias and version can be analyzed during the deployment, or other health checks performed, to ensure that the new version is healthy before proceeding.

Traffic shifting with SAM and CodeDeploy

AWS CodeDeploy provides an intuitive turn-key implementation of this functionality integrated directly into AWS SAM.

Traffic-shifted deployments can be declared in a SAM template, and CodeDeploy manages the function rollout as part of the CloudFormation stack update. CloudWatch alarms can also be configured to trigger a stack rollback if something goes wrong.

UPDATE SAM TEMPLATE

Open the SAM template (sam-app/template.yaml) in your project and add the following lines to the HelloWorldFunction properties section.

AutoPublishAlias: liveDeploymentPreference:    Type: Canary10Percent5Minutes

Take care of indentation, YAML files ! :D

Deployment Preference Types

For this workshop, we are using the Canary10Percent5Minutes strategy, which means that traffic is shifted in two increments. In the first increment, only 10% of the traffic is shifted to the new Lambda version, and after 5 minutes, the remaining 90% is shifted.

There are other deployment strategies you can choose in CodeDeploy:

• Canary10Percent30Minutes
• Canary10Percent5Minutes
• Canary10Percent10Minutes
• Canary10Percent15Minutes
• Linear10PercentEvery10Minutes
• Linear10PercentEvery1Minute
• Linear10PercentEvery2Minutes
• Linear10PercentEvery3Minutes
• AllAtOnce

The Linear strategy means that traffic is shifted in equal increments with an equal number of time interval between each increment.

Validate the SAM template

Run the following command on your terminal:

~ cd ~/environment/sam-app~ sam validate

If the template is correct, you will see template.yaml is a valid SAM Template. If you see an error, then you likely have an indentation issue on the YAML file. Double check and make sure it matches the screenshot shown above.

Push the changes

In the terminal, run the following commands from the root directory of your sam-app project.

~ git add .~ git commit -m "Canary deployments with SAM"~ git push

MONITOR CANARY HEALTH

Canary deployments are considerably more successful if the code is being monitored during the deployment. You can configure CodeDeploy to automatically roll back the deployment if a specified CloudWatch metric has breached the alarm threshold. Common metrics to monitor are Lambda Invocation errors or Invocation Duration (latency), for example.

Define a CloudWatch Alarm

Add the following alarm definition to the template.yaml file in the Resources section after the HelloWorldFunction definition.

CanaryErrorsAlarm:  Type: AWS::CloudWatch::Alarm  Properties:    AlarmDescription: Lambda function canary errors    ComparisonOperator: GreaterThanThreshold    EvaluationPeriods: 2    MetricName: Errors    Namespace: AWS/Lambda    Period: 60    Statistic: Sum    Threshold: 0    Dimensions:      - Name: Resource        Value: !Sub "${HelloWorldFunction}:live"      - Name: FunctionName        Value: !Ref HelloWorldFunction      - Name: ExecutedVersion        Value: !GetAtt HelloWorldFunction.Version.Version

And then add the following lines to the DeploymentPreference section of the HelloWorldFunction definition.

Alarms:    - !Ref CanaryErrorsAlarm

Your template.yaml should look like this:

AWSTemplateFormatVersion: '2010-09-09'Transform: AWS::Serverless-2016-10-31Description: >  sam-app  Sample SAM Template for sam-appGlobals:  Function:    Timeout: 3Resources:  HelloWorldFunction:    Type: AWS::Serverless::Function    Properties:      CodeUri: hello-world/      Handler: app.lambdaHandler      Runtime: nodejs14.x      AutoPublishAlias: live      DeploymentPreference:        Type: Canary10Percent5Minutes        Alarms:          - !Ref CanaryErrorsAlarm      Events:        HelloWorld:          Type: Api          Properties:            Path: /hello            Method: get  CanaryErrorsAlarm:    Type: AWS::CloudWatch::Alarm    Properties:      AlarmDescription: Lambda function canary errors      ComparisonOperator: GreaterThanThreshold      EvaluationPeriods: 2      MetricName: Errors      Namespace: AWS/Lambda      Period: 60      Statistic: Sum      Threshold: 0      Dimensions:        - Name: Resource          Value: !Sub "${HelloWorldFunction}:live"        - Name: FunctionName          Value: !Ref HelloWorldFunction        - Name: ExecutedVersion          Value: !GetAtt HelloWorldFunction.Version.VersionOutputs:  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function  # Find out more about other implicit resources you can reference within SAM  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api  HelloWorldApi:    Description: "API Gateway endpoint URL for Prod stage for Hello World function"    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"  HelloWorldFunction:    Description: "Hello World Lambda Function ARN"    Value: !GetAtt HelloWorldFunction.Arn  HelloWorldFunctionIamRole:    Description: "Implicit IAM Role created for Hello World function"    Value: !GetAtt HelloWorldFunctionRole.Arn

Then you can validate again:

cd ~/environment/sam-appsam validate

If it's ok, then push again:

git add .git commit -m "Added CloudWatch alarm to monitor the canary"git push

Navigate to the AWS CodeDeploy console and after a couple of minutes, you should see a new deployment in progress. Click on the Deployment to see the details.

The deployment status shows that 10% of the traffic has been shifted to the new version (aka The Canary). CodeDeploy will hold the remaining percentage until the specified time interval has ellapsed, in this case we specified the interval to be 5 minutes.

Shortly after the 5 minutes, the remaining traffic should be shifted to the new version:

ROLLBACKS

Monitoring the health of your canary allows CodeDeploy to make a decision to whether a rollback is needed or not. If any of the CloudWatch Alarms specified gets to ALARM status, CodeDeploy rollsback the deployment automatically.

Introduce an error on purpose

Lets break the Lambda function on purpose so that the CanaryErrorsAlarm gets triggered during deployment. Update the lambda code in sam-app/hello-world/app.js to throw an error on every invocation, like this:

let response;exports.lambdaHandler = async (event, context) => {    throw new Error("This will cause a deployment rollback");    // try {    //     response = {    //         'statusCode': 200,    //         'body': JSON.stringify({    //             message: 'hello my friend with canaries',    //         })    //     }    // } catch (err) {    //     console.log(err);    //     return err;    // }    // return response};

Make sure to update the unit test, otherwise the build will fail. Comment out every line in the sam-app/hello-world/tests/unit/test-handler.js file:

// 'use strict';// const app = require('../../app.js');// const chai = require('chai');// const expect = chai.expect;// var event, context;// describe('Tests index', function () {//     it('verifies successful response', async () => {//         const result = await app.lambdaHandler(event, context)//         expect(result).to.be.an('object');//         expect(result.statusCode).to.equal(200);//         expect(result.body).to.be.an('string');//         let response = JSON.parse(result.body);//         expect(response).to.be.an('object');//         expect(response.message).to.be.equal("hello my friend with canaries");//     });// });

In the terminal, run the following commands from the root directory of your sam-app project.

git add .git commit -m "Breaking the lambda function on purpose"git push

Again, wait for your Pipeline to reach the deployment phase (ExecuteChangeSet). It should turn blue when it begins:

INVOKE THE CANARY

While the deployment is running, you need to generate traffic to the new Lambda function to make it fail and trigger the CloudWatch Alarm. In a real production environment, your users will likely generate organic traffic to the canary function, so you may not need to do this.

In your terminal, run the following command to invoke the Lambda function:

aws lambda invoke --function-name \$(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \--payload '{}'  response.json

Remember: During deployment, only 10% of the traffic will be routed to the new version. So, keep on invoking your lambda many times. 1 out of 10 invocations should trigger the new broken lambda, which is what you want to cause a rollback.

Here is a command that invokes your function 15 times in a loop. Feel free to run it in your terminal.

counter=1while [ $counter -le 15 ]do    aws lambda invoke --function-name \    $(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \    --payload '{}' \    response.json    sleep 1    ((counter++))done

Navigate to the AWS CodeDeploy Console and go into the deployment In-Progress to view its details.

After a few minutes, CodeDeploy will detect that the CanaryErrorsAlarm has triggered and it will start rolling back the deployment. The screen will look something like this:

CLEANUP

DELETE S3 BUCKETS

Empty the buckets by going to the S3 console. And then Delete them (there are 2 buckets).

DELETE CF STACKS

Now that the buckets are empty, we can delete the Cloudformation stacks:

Delete the CF stacks in following sequence and please wait for each to complete before going to next one.

~ aws cloudformation delete-stack --stack-name sam-app~ aws cloudformation delete-stack --stack-name sam-app-cicd~ aws cloudformation delete-stack --stack-name aws-sam-cli-managed-default

We can now delete the Cloud9 instance we created:

• Go to your Cloud9 Environment.
• Select the environment (i.e. MyCloud9Workspace) and click delete.

Congratulations, you finished the workshop. Thank you for reading all parts of this workshop series.