AKS: Persistent Volume Claim with an Azure File Storage protected with a Private Endpoint

This post will show you the steps you’ll have to take to deploy an Azure Files Storage with a Private Endpoint and use it to create volumes for an Azure Kubernetes Service cluster:

Create a bicep file to declare the Azure resources

You’ll have to declare the following resources:

  • A VNET with 2 subnets. One for the private endpoint and the other for the AKS cluster.
  • An Azure Files storage.
  • A Private Endpoint for the storage.
  • A Private DNS Zone and Private DNS Zone Group.
  • A link between the Private DNS Zone to the VNET.
  • An AKS cluster.
  • A role assignment to add the kubelet identity of the cluster as a Contributor to the Storage Account.

in a main.bicep file with the following contents:

  1param sa_name string = 'akscsisa'
  2param aks_name string = 'akscsimsft'
  3
  4// Create the VNET
  5resource vnet 'Microsoft.Network/[email protected]' = {
  6  name: 'private-network'
  7  location: 'westeurope'
  8  properties: {
  9    addressSpace: {
 10      addressPrefixes: [
 11        '10.0.0.0/8'
 12      ]
 13    }
 14    subnets: [
 15      {
 16        name: 'endpoint'
 17        properties: {
 18          addressPrefix: '10.241.0.0/16'
 19          serviceEndpoints: []
 20          delegations: []
 21          privateEndpointNetworkPolicies: 'Disabled'
 22          privateLinkServiceNetworkPolicies: 'Enabled'
 23        }
 24      }
 25      {
 26        name: 'aks'
 27        properties: {
 28          addressPrefix: '10.240.0.0/16'
 29          serviceEndpoints: []
 30          delegations: []
 31          privateEndpointNetworkPolicies: 'Enabled'
 32          privateLinkServiceNetworkPolicies: 'Enabled'
 33        }
 34      }
 35    ]
 36    enableDdosProtection: false
 37  }
 38}
 39
 40// Create the File Storage Account
 41resource sa 'Microsoft.Storage/[email protected]' = {
 42  name: sa_name
 43  location: 'westeurope'
 44  sku: {
 45    name: 'Premium_LRS'
 46    tier: 'Premium'
 47  }
 48  kind: 'FileStorage'
 49  properties: {
 50    minimumTlsVersion: 'TLS1_0'
 51    allowBlobPublicAccess: false
 52    isHnsEnabled: false
 53    networkAcls: {
 54      bypass: 'AzureServices'
 55      virtualNetworkRules: []
 56      ipRules: []
 57      defaultAction: 'Deny'
 58    }
 59    supportsHttpsTrafficOnly: true
 60    accessTier: 'Hot'
 61  }
 62}
 63
 64// Create the Private Enpoint
 65resource private_endpoint 'Microsoft.Network/[email protected]' = {
 66  name: 'sa-endpoint'
 67  location: 'westeurope'
 68  properties: {
 69    privateLinkServiceConnections: [
 70      {
 71        name: 'sa-privateserviceconnection'
 72        properties: {
 73          privateLinkServiceId: sa.id
 74          groupIds: [
 75            'file'
 76          ]
 77        }
 78      }
 79    ]
 80    subnet: {
 81      id: '${vnet.id}/subnets/endpoint'
 82    }
 83  }
 84}
 85
 86// Create the Private DNS Zone
 87resource dns 'Microsoft.Network/[email protected]' = {
 88  name: 'privatelink.file.core.windows.net'
 89  location: 'global'
 90}
 91
 92// Link the Private DNS Zone with the VNET
 93resource vnet_dns_link 'Microsoft.Network/privateDnsZones/[email protected]' = {
 94  name: '${dns.name}/test'
 95  location: 'global'
 96  properties: {
 97    registrationEnabled: false
 98    virtualNetwork: {
 99      id: vnet.id
100    }
101  }
102}
103
104// Create Private DNS Zone Group 
105resource dns_group 'Microsoft.Network/privateEndpoints/[email protected]' = {
106  name: '${private_endpoint.name}/default'
107  properties: {
108    privateDnsZoneConfigs: [
109      {
110        name: 'privatelink-file-core-windows-net'
111        properties: {
112          privateDnsZoneId: dns.id
113        }
114      }
115    ]
116  }
117}
118
119// Create AKS cluster
120resource aks 'Microsoft.ContainerService/[email protected]' = {
121  name: aks_name
122  location: 'westeurope'
123  identity: {
124    type: 'SystemAssigned'
125  }
126  properties: {
127    kubernetesVersion: '1.19.9'
128    dnsPrefix: aks_name
129    agentPoolProfiles: [
130      {
131        name: 'default'
132        count: 1
133        vmSize: 'Standard_D2s_v3'
134        osDiskSizeGB: 30
135        osDiskType: 'Ephemeral'
136        vnetSubnetID: '${vnet.id}/subnets/aks'
137        type: 'VirtualMachineScaleSets'
138        orchestratorVersion: '1.19.9'
139        osType: 'Linux'
140        mode: 'System'
141      }
142    ]
143    servicePrincipalProfile: {
144      clientId: 'msi'
145    }
146    addonProfiles: {
147      kubeDashboard: {
148        enabled: false
149      }
150    }
151    enableRBAC: true
152    networkProfile: {
153      networkPlugin: 'kubenet'
154      networkPolicy: 'calico'
155      loadBalancerSku: 'standard'
156      podCidr: '10.244.0.0/16'
157      serviceCidr: '10.0.0.0/16'
158      dnsServiceIP: '10.0.0.10'
159      dockerBridgeCidr: '172.17.0.1/16'
160    }
161    apiServerAccessProfile: {
162      enablePrivateCluster: false
163    }
164  }
165}
166
167// Built-in Role Definition IDs
168// https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles
169var contributor = '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c'
170
171// Set AKS kubelet Identity as SA Contributor
172resource aks_kubelet_sa_contributor 'Microsoft.Authorization/[email protected]' = {
173  name: guid('${aks_name}_kubelet_sa_contributor')
174  scope: sa
175  properties: {
176    principalId: reference(aks.id, '2021-02-01', 'Full').properties.identityProfile['kubeletidentity'].objectId
177    roleDefinitionId: contributor
178  }
179}

Deploy the Azure resources

Run the following commands to deploy the Azure resources:

1az group create -n private-pvc-test -l westeurope
2
3az deployment group create -f ./main.bicep -g private-pvc-test

After 10 minutes or so you’ll have all resources up and running.

Install Azure CSI Driver

The Azure Files Container Storage Interface (CSI) driver can be installed on the cluster so Azure Kubernetes Service (AKS) can manage the lifecycle of Azure Files shares.

To install the driver run:

1az aks get-credentials -n akscsimsft -g private-pvc-test --overwrite-existing
2
3curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/v1.5.0/deploy/install-driver.sh | bash -s v1.5.0 --

and check the pods status:

1kubectl -n kube-system get pod -o wide --watch -l app=csi-azurefile-controller
2kubectl -n kube-system get pod -o wide --watch -l app=csi-azurefile-node

You should find instances of csi-azurefile-controller and csi-azurefile-node running without issues.

Create a Storage Class

Create a file named: storageclass-azurefile-csi.yaml with the following yaml:

 1apiVersion: storage.k8s.io/v1
 2kind: StorageClass
 3metadata:
 4  name: azurefile-csi
 5provisioner: file.csi.azure.com
 6allowVolumeExpansion: true
 7parameters:
 8  resourceGroup: <resourceGroup>  # optional, only set this when storage account is not in the same resource group as agent node
 9  storageAccount: <storageAccountName>
10  # Check driver parameters here:
11  # https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md
12  server: <storageAccountName>.privatelink.file.core.windows.net 
13reclaimPolicy: Delete
14volumeBindingMode: Immediate
15mountOptions:
16  - dir_mode=0777
17  - file_mode=0777
18  - uid=0
19  - gid=0
20  - mfsymlinks
21  - cache=strict  # https://linux.die.net/man/8/mount.cifs
22  - nosharesock  # reduce probability of reconnect race
23  - actimeo=30  # reduce latency for metadata-heavy workload

Remember to replace the values of <resourceGroup> and <storageAccountName> with the ones used in the previous deployment. (i.e. private-pvc-test and akscsisa)

Now deploy the Storage Class to the cluster:

1kubectl apply -f storageclass-azurefile-csi.yaml

Create a Private Volume Claim

Create a Private Volume Claim that uses the storage class. Create pvc.yaml with the following contents:

 1apiVersion: v1
 2kind: PersistentVolumeClaim
 3metadata:
 4  name: my-azurefile
 5spec:
 6  accessModes:
 7    - ReadWriteMany
 8  storageClassName: azurefile-csi
 9  resources:
10    requests:
11      storage: 100Gi

Deploy the Private Volume Claim to the cluster and check that everything is ok:

1kubectl apply -f pvc.yaml
2k get pvc

Now feel free to try and mount a volume using the Private Volume Claim: my-azurefile

Hope it helps!!!

Please find a bicep based sample here or if you prefer terraform here

References:


Plan IP addressing for AKS configured with Azure CNI Networking
AKS: Windows node pool with spot virtual machines and ephemeral disks
comments powered by Disqus