Use Azure Application Gateway private link configuration for an internal API Management

TL;DR

When operating Azure API Management in an internal virtual network, already integrated with Azure Application Gateway, an upcoming feature AllowApplicationGatewayPrivateLink allows you to connect this configuration to another virtual network using Private Link and Private Endpoint.

Motivation

In a post I made beginning of February 2022 I linked a virtual network with limited address space - like in a corporate / ExpressRoute / SD-WAN connected scenario - to a Container Apps environment:

That setup works as long as ingress only goes from resources in Hub network to Spoke network. But if Container Apps needs to call (as in my target scenario) Azure API Management - calling from Spoke network into Hub network - that solution is not sufficient.

Looking for options I had this post which Marcel.L pointed me to back then, where he forwards calls to API Management in another virtual network using a Private Endpoint/Link and Virtual Machine Scale Set combination.

However, as pointed out in my earlier post, my endgame should be to replace Azure Service Fabric IaaS container hosting with a higher level abstraction, PaaS like Container Apps, hence I did not necessarily want to add just another IaaS in the process - even one with a lower complexity.

Searching for alternatives I checked on private linking capabilities of Azure API Management itself. However this cannot be used and mixed when it is already operated in external or internal virtual network mode. Hence no option for me.

"As with container compute I need my API Management with 100% ingress and egress within a virtual network"

Researching on private IP address options on Azure Application Gateway I stumbled over a private link feature in Azure CLI which - lacking tangible documentation - I exploited here and converted to Bicep

Solution Elements

To achieve this configuration

following solution elements additionally to my earlier post are required:

• a Private Link Configuration on the Application Gateway; still to subnet within Hub virtual network
• a Private Endpoint in Spoke virtual network
• a Private DNS Zone linked to Spoke virtual network so that Container Apps resolve to Private Endpoint IP address

complete configuration for all snippets shown in this post can be found in this repo / tag; I'll state the filenames so that snippets can be found more easily

Prerequisites

• Azure CLI
• Bicep
• Linux shell / bash / ...

Stage 1 - get preview feature working (as of Feb'2022)

When first trying to deploy Application Gateway with specifying privateLinkConfiguration, I was rewarded with a nice error code SubscriptionNotRegisteredForFeature and a message like Subscription /subscriptions/... is not registered for feature PrivateLinkConfigurations required to carry out the requested operation. Finding no suitable documentation I tried to find the feature switch myself with

az feature list --namespace Microsoft.Network -o table | grep Priv

I then activated what seemed to make sense for my case:

az feature register --name AllowApplicationGatewayPrivateLink --namespace Microsoft.Networkaz feature register --name AllowAppGwPublicAndPrivateIpOnSamePort --namespace Microsoft.Networkaz provider register -n Microsoft.Network

Stage 2 - configuration Application Gateway

To add private link, an entry in privateLinkConfigurations section is required. It needs to be put in the same virtual network but a different subnet as the gateway itself (once Application Gateway is deployed to a subnet, it only allows for other Application Gateways to share this subnet, but no other resources).

That privateLinkConfiguration is then to be referenced on the frontendIPConfiguration. In my sample I share it with the public IP, but I guess, this also can be separated to allow various forwarding rules depending from where ingress is coming from.

appgw-priv.bicep

...frontendIPConfigurations: [  {    name: 'default'    properties: {      publicIPAddress: {        id: pip.id      }      privateIPAllocationMethod:'Dynamic'      privateLinkConfiguration:{        id: resourceId('Microsoft.Network/applicationGateways/privateLinkConfigurations', appGwName, 'private')      }    }  }]privateLinkConfigurations: [  {    name: 'private'    properties: {      ipConfigurations: [        {          name: 'private-ip'          properties: {            privateIPAllocationMethod:'Dynamic'            subnet: {              id: subnetJumpHubId            }          }        }      ]    }  }]...

for simplification in my sample I am using HTTP on port 8080; in production this will be replaced by a proper HTTPS, FQDNs and certificates

Based on private link configuration in Hub network now a private endpoint can be added in Spoke network:

appgw-priv.bicep

resource pep 'Microsoft.Network/privateEndpoints@2021-05-01' = {  name: 'pep-priv-gateway'  location: location  properties: {    subnet: {      id: subnetJumpSpokeId    }    privateLinkServiceConnections: [      {        properties: {          privateLinkServiceId: appgw.id          groupIds: [            'default'          ]        }        name: 'pep-priv-gateway'      }    ]  }}

Stage 3 - private DNS zone

Spoke network needs a private DNS zone so that traffic towards the Private Endpoint is resolved correctly:

appgw-priv-dns.bicep

param vnetSpokeId stringparam apiName stringparam pepIp stringvar privateDNSZoneName = 'internal-api.net'resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {  name: privateDNSZoneName  location: 'global'}resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {  parent: privateDnsZone  name: '${privateDnsZone.name}-link'  location: 'global'  properties: {    registrationEnabled: false    virtualNetwork: {      id: vnetSpokeId    }  }}resource privateDnsZoneEntry 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {  name: apiName  parent: privateDnsZone  properties: {    aRecords: [      {        ipv4Address: pepIp      }    ]    ttl: 3600  }}

again for simplification I use DNS zone internal-api.net because I am not yet sure whether it is suitable here to go with privatelink.azure-api.net as the configuration I use is not purely an API Management private link configuration

Stage 4 - deploy API Management, Application Gateway and DNS zone

I had to split deployment of API Management & Application Gateway, return and extract private IP address information from private endpoint and then continue with private DNS deployment. I was not able (or just too lazy) to wire this up within one Bicep file.

deploy-stage-2.sh

...az deployment group create --resource-group $RESOURCE_GROUP \    --template-file apim.bicep \    --parameters apimName=$APIMNAME \    appInsightsName=$APPINSIGHTNAME \    logAnalyticsWorkspaceName=$LOGANALYTICSNAME \    fapp1Fqdn=$fapp1Fqdn \    fapp2Fqdn=$fapp2FqdnVNET_SPOKE_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'spoke')].id" -o tsv`PEP_NIC_ID=`az network private-endpoint list -g $RESOURCE_GROUP --query "[?name=='pep-priv-gateway'].networkInterfaces[0].id" -o tsv`PEP_IP=`az network nic show --ids $PEP_NIC_ID --query ipConfigurations[0].privateIpAddress -o tsv`az deployment group create --resource-group $RESOURCE_GROUP \    --template-file appgw-priv-dns.bicep \    --parameters "{\"pepIp\": {\"value\": \"$PEP_IP\"},\"vnetSpokeId\": {\"value\": \"$VNET_SPOKE_ID\"},\"apiName\": {\"value\": \"$APIMNAME\"}}"

along with API Management instance I deploy an API to test calls to Function Apps hosted in Container Apps environment; appgw-priv.bicep shown above is a module of apim.bicep, hence deployed in the first block

Stage 5 - testing

To check, that I have not introduced regressions I first test access to Function Apps over public IP of Application Gateway:

test-api.sh

...APIMID=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query id -o tsv`APIMURL=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query gatewayUrl -o tsv`GWURL=`az network public-ip show -n ${APIMNAME}-priv-gateway-pip -g $RESOURCE_GROUP --query dnsSettings.fqdn -o tsv`GWPORT=`az network application-gateway show -n ${APIMNAME}-gateway -g $RESOURCE_GROUP --query frontendPorts[0].port -o tsv`SUBKEY=`az rest --method post  --uri ${APIMID}/subscriptions/test-subscription/listSecrets?api-version=2021-08-01 --query primaryKey -o tsv`curl -s ${GWURL}:${GWPORT}/test/fapp1?subscription-key=$SUBKEYprintf '
'curl -s ${GWURL}:${GWPORT}/test/fapp2?subscription-key=$SUBKEYprintf '
'

To make more thorough tests of the Functions Apps in combination with calls to API Management I have to hop on a jump VM in Spoke network:

test-fapps.sh

...IP=$(az vm list-ip-addresses -g $RESOURCE_GROUP --query "[?contains(virtualMachine.name, 'hub')].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)declare -a apps=("fapp1" "fapp2")for app in "${apps[@]}"do    echo "$app"    fqdn=$(az containerapp show -n $app -g $RESOURCE_GROUP --query configuration.ingress.fqdn -o tsv --only-show-errors)    ssh ca@$IP curl -s https://$fqdn/api/health    echo " <<-- check APIM internal status"    ssh ca@$IP curl -s https://$fqdn/api/apim-status    echo " <<-- check $app APIM health"    ssh ca@$IP curl -s https://$fqdn/api/apim-internal-status    echo " <<-- check $app APIM internal status"done

where

• curl -s https://$fqdn/api/health checks Function App only with a call to a HTTP trigger
• curl -s https://$fqdn/api/apim-status and curl -s https://$fqdn/api/apim-internal-status call HTTP triggers which themselves forward calls to API Management instance (using the internal-api.net domain)

[FunctionName("Apim-Status")]public static async Task<IActionResult> ApimStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-status")] HttpRequest req)  => new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/status-0123456789abcdef"));[FunctionName("Apim-Internal-Status")]public static async Task<IActionResult> ApimInternalStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-internal-status")] HttpRequest req)  => new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/internal-status-0123456789abcdef"));

Conclusion

With this solution a major roadblock for me is out of the way and I do not need to operate my own compute resources for network traffic forwarding. It still requires a some polishing and hardening before I can use it even close to our production resources.

A positive effect for me is that with reconfiguration of the existing Application Gateway and a very low footprint of IP addresses consumed in my corporate virtual network, I can even operate the existing environment with Service Fabric in coexistence with the new Container Apps environment, which reduces migration risks tremendously.