Today I’ll show you how to use Container Insights and Azure Monitor to check your AKS cluster for pods without requests and limits.
You’ll need to use the following tables and fields:
- KubePodInventory: Table that stores kubernetes cluster’s Pod & container information
- ClusterName: ID of the kubernetes cluster from which the event was sourced
- Computer: Computer/node name in the cluster that has this pod/container.
- Namespace: Kubernetes Namespace for the pod/container
- ContainerName:This is in poduid/containername format.
- Perf: Performance counters from Windows and Linux agents that provide insight into the performance of hardware components operating systems and applications.
- ObjectName: Name of the performance object.
- CounterName: Name of the performance counter.
- CounterValue: The value of the counter
And take a close look at the following Objects and Counters:
| ObjectName | Counter | Description |
|---|---|---|
| K8SContainer | cpuLimitNanoCores | Container’s cpu limit in nanocore/nanocpu unit. If container resource limits are not specified, node’s capacity will be rolled-up as container’s limit. |
| K8SContainer | cpuRequestNanoCores | Container’s cpu request in nanocore/nanocpu unit. If container cpu resource requests are not specified, this metric will not be collected. |
| K8SContainer | memoryLimitBytes | Container’s memory limit in bytes. If container resource limits are not specified, node’s capacity will be rolled-up as container’s limit. |
| K8SContainer | memoryRequestBytes | Container’s memory request in bytes. If container memory resource requests are not specified, this metric will not be collected. |
| K8SNode | cpuAllocatableNanoCores | Amount of cpu that is allocatable by Kubernetes to run pods, expressed in nanocores/nanocpu. |
| K8SNode | cpuCapacityNanoCores | Total CPU capacity of the node in nanocore/nanocpu unit. |
| K8SNode | memoryAllocatableBytes | Amount of memory in bytes that is allocatable by kubernetes to run pods. |
| K8SNode | memoryCapacityBytes | Total memory capacity of the node in bytes. |
Now let’s cut the chace, run the following KQL query on Azure Monitor and check the results:
1let podCounters = Perf
2 | where ObjectName == 'K8SContainer' and (CounterName == 'cpuLimitNanoCores' or CounterName == 'cpuRequestNanoCores' or CounterName == 'memoryLimitBytes' or CounterName == 'memoryRequestBytes')
3 | summarize d = make_bag(pack(CounterName, CounterValue)) by InstanceName
4 | evaluate bag_unpack(d);
5let podRequestsAndLimits = podCounters
6 | extend InstanceNameParts = split(InstanceName, "/")
7 | extend PodUI = tostring(InstanceNameParts[(array_length(InstanceNameParts)-2)])
8 | extend PodName = tostring(InstanceNameParts[(array_length(InstanceNameParts)-1)])
9 | project PodUI, PodName, cpuLimitNanoCores, cpuRequestNanoCores, memoryLimitBytes, memoryRequestBytes;
10let nodeCounters = Perf
11 | where ObjectName == "K8SNode" and (CounterName == 'cpuAllocatableNanoCores' or CounterName == 'cpuCapacityNanoCores' or CounterName == 'memoryAllocatableBytes' or CounterName == 'memoryCapacityBytes')
12 | summarize d = make_bag(pack(CounterName, CounterValue)) by InstanceName
13 | evaluate bag_unpack(d);
14let nodeCapacity = nodeCounters
15 | extend InstanceNameParts = split(InstanceName, "/")
16 | extend Computer = tostring(InstanceNameParts[(array_length(InstanceNameParts)-1)])
17 | project-away InstanceNameParts, InstanceName;
18KubePodInventory
19 | distinct ClusterName, Computer, Namespace, ContainerName
20 | extend InstanceNameParts = split(ContainerName, "/")
21 | extend PodUI = tostring(InstanceNameParts[(array_length(InstanceNameParts)-2)])
22 | extend PodName = tostring(InstanceNameParts[(array_length(InstanceNameParts)-1)])
23 | project ClusterName, Computer, Namespace, PodUI, PodName
24 | join kind= leftouter (nodeCapacity) on Computer
25 | join kind= leftouter (podRequestsAndLimits) on PodUI, PodName
26 // Pods without CPU Requests. If container cpu resource requests are not specified, cpuRequestNanoCores metric will not be collected
27 | extend CPURequests = isnotnull(cpuRequestNanoCores)
28 // Pods without CPU Limits. If container resource limits are not specified, node's capacity will be rolled-up as container's limit
29 | extend CPULimits = cpuAllocatableNanoCores != cpuLimitNanoCores
30 // Pods without Memory Requests. If container memory resource requests are not specified, memoryRequestBytes metric will not be collected
31 | extend MemoryRequests = isnotnull(memoryRequestBytes)
32 // Pods without Memory Limits. If container resource limits are not specified, node's capacity will be rolled-up as container's limit
33 | extend MemoryLimits = memoryAllocatableBytes != memoryLimitBytes
34 | distinct ClusterName, Namespace, PodName, CPURequests, CPULimits, MemoryRequests, MemoryLimits
35 | where not(CPURequests) or not(CPULimits) or not(MemoryRequests) or not(MemoryLimits)
36 | project ClusterName, Namespace, PodName, CPURequests, CPULimits, MemoryRequests, MemoryLimits
Hope it helps!!!
Please find the KQL file here
References:
Comments