Windows Containers in Azure Kubernetes Services

Kubernetes, or K8s, is the most popular tools to orchestrate containers. Kubernetes is widely available, you can deploy it on-premises or on any Cloud providers or use a managed service like AKS in Azure or Google Cloud.

But what happens when you want to orchestrate Windows Containers. There are many reasons to work with Windows Containers, libraries or languages are only available on Windows, you have too many dependencies or you may not have the knowledge or the time to use Linux.

Windows containers are here. They are stable and run with dockers. You can publish your app, build your solutions with it and use it like Linux containers. But can you use them in Kubernetes?
Yes, it's possible to use them in Azure with AKS or in Google cloud GKE.

How does it work in Azure AKS?

Kubernetes core services, API Engine, DNS, still need to run on Linux. Every Kubernetes cluster, including those with Windows Containers, need at least one Linux node to run core services. You can add Windows Server to run containers, but the first node needs to be a Linux VM.

Windows Containers feature in AKS is in preview. Before starting to deploy AKS you will need to configure your workstation and your subscription.
First, be sure to use the latest version of AZURE CLI. The 2.0.76 version is required to run the Windows Container feature.

On windows, you can install the latest MSI or use Chocolatey to manage and update the installation.

On Linux is you are on Debian like distro you can manage the update by APT.

You will need also to add the AKS-Preview extension

az extension add --name aks-preview

And you should try to update to be sure to have the latest version.

az extension update --name aks-preview

If you never managed an AKS cluster, install the Kubernetes command-line tools (kubectl)

az aks install-cli

After that, you will need to update your subscription to register the Windows Container feature. This action changes the behavior of Azure for any new AKS cluster and not only those you want to use with Windows Containers. In other terms, any new cluster you deploy will be in preview mode, without any SLA from Microsoft. Do not choose a subscription with production clusters.

To perform this update.

az feature register --name WindowsPreview --namespace Microsoft.ContainerService

You will have to wait a few minutes, enough to go out and grab a coffee or a pizza. To check the result you can use this command.

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/WindowsPreview')].{Name:name,State:properties.state}"

Finally, after the registration is completed, you can register the new Microsoft.ContainerService

az provider register --namespace Microsoft.ContainerService

The AKS cluster configuration is slightly different than normal AKS cluster. The Windows Preview enables multi-node pools. You can have multiple kinds of VM in different pools. Each pool can have different VM Size and OS version. You need tp have a pool for the control plane with small VMs, a pool with bigger VMs on Linux and a pool with Windows 2019 VMs for Windows containers. The Windows Preview also enables a machine scale set for nodes.

There is another important change. If with a classic AKS cluster you have the choice between Azure CNI and KubeNet for the network plugin, the Windows preview limits the choice to Azure CNI only. Azure CNI is a little more complex than KubeNet.

Pools in the same cluster must share the same subnet and VNET and pool names have different requirements for Windows and Linux; 12 characters limit for Linux and 6 for Windows.

Knowing that we can deploy the cluster with the first Linux pool for the control plane.

WINPASS="YourRealPassWord"az aks create \    --resource-group omc-lab-akswin \    --name akswin01 \    --node-count 2 \    --kubernetes-version 1.15.7 \    --generate-ssh-keys \    --windows-admin-password $WINPASS \    --windows-admin-username adminomc \    --vm-set-type VirtualMachineScaleSets \    --load-balancer-sku standard \    --network-plugin azure \    --node-vm-size Standard_B2ms \    --node-osdisk-size 80 \    --nodepool-name coreaks \    --docker-bridge-address 172.24.0.1/16 \    --dns-service-ip 10.10.0.15 \    --service-cidr 10.10.0.0/24 \    --dns-name-prefix omc-aks-windows \      --tags 'env=lab' 'app=Aks Windows'

A password is needed for Windows, even if the first action is to deploy the control plane in the Linux pool named coreaks. The password must comply with the Windows 2019 default password policy.

The command deploys a Resource Group, a public IP and a VNET. You can also use your own VNET by providing the subnet ID with the --vnet-subnet-id parameter. Be sure to have enough available IP.

If you connect to the cluster and list pods, you will not find any Windows pods.

kubectl get nodesaks-coreaks-17338944-vmss000000   Ready    agent   18m   v1.15.7aks-coreaks-17338944-vmss000001   Ready    agent   18m   v1.15.7

You need to add them

az aks nodepool add \    --resource-group omc-lab-akswin \    --cluster-name akswin01 \    --os-type Windows \    --name win01 \    --node-vm-size Standard_B2ms \    --node-count 1 \    --kubernetes-version 1.15.7

Be careful with the pool name, the maximum number of characters is 6.
Now you can see the Windows pools

kubectl get nodesNAME                              STATUS   ROLES   AGE     VERSIONaks-coreaks-17338944-vmss000000   Ready    agent   30m     v1.15.7aks-coreaks-17338944-vmss000001   Ready    agent   30m     v1.15.7akswin01000000                    Ready    agent   2m17s   v1.15.7

From the Azure Portal, you should be able to see theses two node pools in the Setting of your Aks cluster, coreaks and win01.

Before deploying containers in the cluster, we need to create a namespace. Its not a requirement and you can deploy whatever you want without a namespace (in the default namespace in fact). But namespace helps to organize resources and projects in a Kubernetes cluster.

kubectl create namespace wintest

Deploying Windows Applications

I have a simple windows container, a web site with IIS. Here's the Dockerfile

FROM microsoft/iisCOPY /site/ /inetpub/wwwroot

To deploy the container to the AKS cluster we need to put it in a registry, public or private. Azure provides a private registry service, Azure Container Registry.
To use it, login to your subscription using AZ Cli then login to the registry.

az acr login --name $acrname

Then tag the container

docker tag webpage xxxx.azurecr.io/samples/webpage

and push it to the registry

docker push xxxx.azurecr.io/samples/webpage

But if we need to login to the registry, how to manage login from the AKS cluster? During the cluster installation, the system created a Service Principal to manage VMs and networks in the AKS resource group.
Its possible to extract the Client ID using the command line to use it to manage rights in the containers registry.

$AKSClientID=az aks show --resource-group $rgname --name $AksClusterName --query "servicePrincipalProfile.clientId" --output tsv

We also need to get the Resource ID of the container registry

$AcrID= az acr show --name $acrname --resource-group $rgname --query "id" --output tsv

You can now manage permission on the container registry, pull and read are needed.

az role assignment create --assignee $AKSClientID --role acrpull --scope $AcrIDaz role assignment create --assignee $AKSClientID --role reader --scope $AcrID

You can start to deploy applications and services in the cluster. For the webpage containers we need to create a deployment, how the container should run (number of pods, port, limit, ) and service (how can we access the application?).
To be sure to run this application on a Windows server you only need to use nodeSelector property in the Deployment specification.

apiVersion: apps/v1beta1kind: Deploymentmetadata:  name: webiis  namespace: wintest  labels:    app: webiisspec:  replicas: 1  template:    metadata:        name: webiis        labels:          app: webiis    spec:      nodeSelector:        "beta.kubernetes.io/os": windows      containers:      - name: webiis        image: xxxxxx.azurecr.io/samples/webpage:latest        ports:        - containerPort: 80        resources:          limits:            cpu: 1            memory: 800M          requests:            cpu: .1            memory: 300M  selector:    matchLabels:      app: webiis---apiVersion: v1kind: Servicemetadata:  name: webiisspec:  type: LoadBalancer  ports:  - protocol: TCP    port: 80  selector:    app: webiis

to apply

kubectl apply -n wintest -f webapp.yaml

It should take a few minutes for the pod to be ready. You can monitor the pod with the get pods command.

kubectl get pods -n wintest

To monitor the service, you can use the get services command. It will be necessary to get the load balancer IP.

kubectl get service -n wintest

As you can see, deploying Windows Application on Kubernetes is almost the same thing as deploying Linux applications. There is only one parameter to add to your deployments, the nodeSelector. Windows and Linux applications can coexist in the same cluster.