Lesson: AKS Networking and Storage Design

Introduction

Azure Kubernetes Service (AKS) simplifies the deployment of managed Kubernetes clusters in Azure. However, the operational success of your applications depends heavily on how you architect the underlying infrastructure.

Networking in AKS determines how your pods communicate with each other, how they reach external services, and how they are secured. Storage in AKS ensures that your stateful applications (like databases or message queues) have persistent data access even when pods are rescheduled or nodes are patched. Designing these components correctly is vital for performance, scalability, and security.

1. Networking Design in AKS

AKS primarily utilizes the Azure CNI (Container Networking Interface) or Kubenet.

Azure CNI vs. Kubenet

• Azure CNI: Every pod gets an IP address from the virtual network (VNet) subnet. This provides high performance and direct routing but requires careful IP address planning to avoid exhaustion.
• Kubenet: Pods receive IPs from a virtual bridge. NAT is used to communicate outside the cluster. It is easier to set up but introduces complexity for external connectivity.

Practical Example: Ingress Controllers

To expose services to the internet, you should avoid using LoadBalancer services for every endpoint. Instead, use an Ingress Controller (like NGINX or AGIC).

Example: NGINX Ingress Resource

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80

Pro Tip: Use the Application Gateway Ingress Controller (AGIC) if you are already using Azure Application Gateway. It provides native WAF capabilities and seamless layer-7 load balancing.

2. Storage Design in AKS

Kubernetes treats storage as an abstraction. You define a StorageClass, which tells AKS how to provision the underlying Azure storage (Managed Disk, Azure Files, or Azure Blob).

Choosing the Right Storage

1. Azure Managed Disks (Premium/Standard): Best for high-performance, single-node access (ReadWriteOnce). Ideal for databases like PostgreSQL or SQL Server.
2. Azure Files (SMB/NFS): Best for shared storage across multiple pods (ReadWriteMany). Ideal for content management systems or shared configuration files.
3. Azure Blob Storage (BlobFuse): Best for massive, unstructured data sets.

Practical Example: Creating a StorageClass

To dynamically provision storage, define a StorageClass.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-premium-retain
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Using the storage in a PersistentVolumeClaim (PVC):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: managed-premium-retain
  resources:
    requests:
      storage: 10Gi

3. Best Practices

Networking Best Practices

• IP Planning: If using Azure CNI, calculate your pod density. Ensure your VNet subnet is large enough to accommodate the number of nodes multiplied by the maximum pods per node.
• Network Policies: By default, all pods can talk to all pods. Implement Azure Network Policies or Calico to enforce a Zero Trust model.
• Private Clusters: Always deploy production clusters as Private Clusters to ensure the API server is not exposed to the public internet.

Storage Best Practices

• Use CSI Drivers: Always use the Container Storage Interface (CSI) drivers. They are the standard for Kubernetes and offer better performance and feature parity compared to the older in-tree drivers.
• Reclaim Policy: Set your reclaimPolicy to Retain for production databases. If a developer accidentally deletes a PVC, the data will persist in Azure, allowing for manual recovery.
• Monitoring: Monitor storage latency using Azure Monitor Container Insights. High latency on disk operations can cause application crashes.

4. Common Pitfalls

1. IP Exhaustion: Choosing a subnet that is too small for Azure CNI. Once the subnet is full, you cannot scale your cluster.
2. Mounting Issues: Attempting to use ReadWriteOnce storage (Managed Disks) across multiple pods. This will cause "Multi-Attach" errors. Use Azure Files for shared access.
3. Over-provisioning: Creating massive persistent volumes that aren't being used, leading to unnecessary storage costs. Use dynamic provisioning to scale as needed.
4. Ignoring Network Security Groups (NSGs): Assuming AKS handles all firewalling. While AKS manages some NSGs, you must still configure your own NSGs or Azure Firewall to control egress/ingress traffic.

Key Takeaways

• Plan your IP space: Azure CNI requires significant IP overhead; calculate your needs before deployment.
• Decouple Storage: Use StorageClasses and PVCs to abstract the underlying storage provider, making your manifests portable.
• Security First: Use Network Policies to restrict pod-to-pod communication and Private Clusters to limit management surface.
• CSI is King: Always prefer CSI-based drivers for storage to ensure long-term support and improved feature sets.
• Operational Visibility: Integrate Azure Monitor and Log Analytics from day one to troubleshoot complex networking and storage performance issues.