How to Add KMS etcd encryption to an AKS cluster

Azure Kubernetes Service now supports Key Management System (KMS) plugin integration which enables encryption at the rest of our Kubernetes data in etcd using Azure Key Vault. We can now store secrets in bring your own key (BYOK) encrypted etcd using KMS.

KMS Plugin for Key Vault is the recommended choice for using a third-party tool for key management. KMS plugin simplifies key rotation, with a new data encryption key (DEK) generated for each encryption, and key encryption key (KEK) rotation controlled by the user.

Features

• Use a key in Key Vault for etcd encryption.
• Bring your own keys.
• Provide encryption at rest for secrets stored in etcd.

Prerequisites

• An Azure subscription. If we don't have an Azure subscription, we can create a free account.
• Azure CLI installed.

Install the aks-preview Azure CLI

We also need the aks-preview Azure CLI extension version 0.5.58 or later. Install the aks-preview Azure CLI extension by using the az extension add command or install any available updates by using the az extension update command.

# Install the aks-preview extensionaz extension add --name aks-preview# Update the extension to make sure you have the latest version installedaz extension update --name aks-preview

Register the AzureKeyVaultKmsPreview preview feature

To use the feature, we must also enable the AzureKeyVaultKmsPreview feature flag on our subscription.

Register the AzureKeyVaultKmsPreview feature flag by using the az feature register command, as shown in the following example:

az feature register --namespace "Microsoft.ContainerService" --name "AzureKeyVaultKmsPreview"

It takes a few minutes for the status to show Registered. Verify the registration status by using the az feature list command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AzureKeyVaultKmsPreview')].{Name:name,State:properties.state}"

When ready, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command:

az provider register --namespace Microsoft.ContainerService

Limitations

The following limitations apply when we integrate KMS etcd encryption with AKS:

• Disabling of the KMS etcd encryption feature.
• Changing of key ID, including key name and key version.
• Deletion of the key, Key Vault, or the associated identity.
• KMS etcd encryption doesn't work with System-Assigned Managed Identity. The keyvault access-policy is required to be set before the feature is enabled. In addition, System-Assigned Managed Identity isn't available until cluster creation, thus there's a cycle dependency.
• Using Azure Key Vault with PrivateLink enabled.
• Using more than 2000 secrets in a cluster.
• Managed HSM Support
• Bring your own (BYO) Azure Key Vault from another tenant.

Create a KeyVault and key

Use az keyvault create to create a KeyVault.

az keyvault create --name kcdchennaikv --resource-group kcdchennairg

Use az keyvault key create to create a key.

az keyvault key create --name kcdchennaikey --vault-name kcdchennaikv

Use az keyvault key show to export the Key ID.

export KEY_ID=$(az keyvault key show --name kcdchennaikey --vault-name kcdchennaikv --query 'key.kid' -o tsv)echo $KEY_ID

The above example stores the Key ID in KEY_ID.

Create a user-assigned managed identity

Use az identity create to create a User-assigned managed identity.

az identity create --name kcdidentity --resource-group kcdchennairg

Use az identity showto get Identity Object ID.

IDENTITY_OBJECT_ID=$(az identity show --name kcdidentity --resource-group kcdchennairg --query 'principalId' -o tsv)echo $IDENTITY_OBJECT_ID

The above example stores the value of the Identity Object ID in IDENTITY_OBJECT_ID.

Use az identity show to get Identity Resource ID.

IDENTITY_RESOURCE_ID=$(az identity show --name kcdidentity --resource-group kcdchennairg --query 'id' -o tsv)echo $IDENTITY_RESOURCE_ID

The above example stores the value of the Identity Resource ID in IDENTITY_RESOURCE_ID.

Assign permissions (decrypt and encrypt) to access key vault

Use az keyvault set-policy to create an Azure KeyVault policy.

az keyvault set-policy -n kcdchennaikv --key-permissions decrypt encrypt --object-id $IDENTITY_OBJECT_ID

Create an AKS cluster with KMS etcd encryption enabled

Create an AKS cluster using the az aks create command with the --enable-azure-keyvault-kms and --azure-keyvault-kms-key-id parameters to enable KMS etcd encryption.

az aks create --name kcdchnakscluster --resource-group kcdchennairg --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID

Update an exiting AKS cluster to enable KMS etcd encryption

Use az aks update with the --enable-azure-keyvault-kms and --azure-keyvault-kms-key-id parameters to enable KMS etcd encryption on an existing cluster.

az aks update --name kcdchnakscluster --resource-group kcdchennairg --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID

Use below command to update all secrets. Otherwise, the old secrets aren't encrypted.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

For more information on using the KMS plugin, see Encrypting Secret Data at Rest.

Note

Currently, this feature is in preview. In AKS, preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.

Thanks for reading my article till end. I hope you learned something special today. If you enjoyed this article then please share to your friends and if you have suggestions or thoughts to share with me then please write in the comment box.