Infrastructure Security: Securing EKS and ECS

Introduction: Why Compute Security Matters

In modern cloud architecture, compute services like Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS) serve as the backbone of application delivery. When we move workloads into containers, we shift the security boundary from traditional virtual machine hardening toward container runtime security, orchestration configuration, and identity-based access control. Securing these environments is not just about patching software; it is about establishing a "defense-in-depth" strategy that protects your application code, the container images, the orchestration platform, and the underlying infrastructure from unauthorized access, data exfiltration, and lateral movement.

If these services are misconfigured, the impact can be catastrophic. An exposed Kubernetes API server can allow an attacker to gain cluster-admin privileges, while an overly permissive ECS task role can grant an attacker access to sensitive S3 buckets or database credentials. Understanding the nuances of EKS and ECS security is essential for any engineer tasked with maintaining production-grade infrastructure. This lesson will guide you through the shared responsibility model, the specific security mechanisms unique to EKS and ECS, and the practical steps you can take to harden your compute environment today.

The Shared Responsibility Model in Containerization

Before diving into specific configurations, it is vital to understand where your responsibility ends and where the cloud provider's begins. In the context of compute security, the cloud provider manages the underlying hardware, the virtualization layer, and the physical security of the data centers. However, you are responsible for the security of the container images, the container configuration, the network policies, and the identity management for the services running inside those containers.

When using EKS, the cloud provider manages the Kubernetes control plane, but you are responsible for the worker nodes (unless using Fargate) and the configuration of the Kubernetes objects themselves. In ECS, the cloud provider manages the orchestration, but you are responsible for the task definitions, the Docker images, and the IAM roles assigned to those tasks. Understanding this distinction prevents the common mistake of assuming that "managed" services are inherently secure by default.

Callout: Managed vs. Self-Managed Security While EKS and ECS are managed services, they are not "secure by default" in a way that protects against application-level vulnerabilities or poor IAM design. The cloud provider ensures the API server is available and patched, but they cannot know if you have accidentally granted your pod access to delete all your production databases. You must configure the security controls to match your specific threat model.

Securing Amazon EKS (Elastic Kubernetes Service)

EKS is a powerful, complex environment. Because Kubernetes provides a massive surface area for configuration, security must be applied at every layer of the stack: the API server, the node pools, the pod network, and the identity provider.

1. Protecting the Kubernetes API Server

The API server is the brain of your cluster. If an attacker gains access to it, they control everything. By default, EKS clusters are often configured with public endpoints. You should prioritize restricting access to this endpoint.

• Make the API endpoint private: Configure your EKS cluster to only accept traffic from within your VPC. This eliminates the risk of brute-force attacks from the public internet.
• Use CIDR restriction: If you must have a public endpoint (e.g., for CI/CD pipelines outside the VPC), restrict access to specific IP addresses of your build servers or VPN gateways.
• Enable Audit Logging: Always enable Kubernetes audit logs in CloudWatch. These logs record every action taken against the API server, which is critical for incident response and forensics.

2. IAM Roles for Service Accounts (IRSA)

Historically, engineers would assign an IAM role to the entire EC2 instance hosting the Kubernetes nodes. This was a massive security flaw because every pod on that node inherited those permissions. IRSA solves this by allowing you to assign a specific IAM role to a specific Kubernetes service account.

To implement this, you create an IAM OIDC provider for your cluster. Then, you associate an IAM role with a service account using an annotation.

# Example: Kubernetes ServiceAccount with IAM role annotation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role

This ensures that my-app only has access to the specific resources defined in my-app-role, and no other pod in the cluster can assume those permissions.

3. Kubernetes Network Policies

By default, all pods in a Kubernetes cluster can talk to all other pods. This is a significant security risk. If a front-end pod is compromised, the attacker can scan the internal network to find back-end databases or management interfaces.

You should use a CNI (Container Network Interface) plugin that supports Network Policies, such as the Amazon VPC CNI or Calico. You must implement a "default deny" policy for all namespaces and then explicitly allow traffic between components.

# Example: Default Deny All Policy
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

4. Node Security

If you are using managed node groups, ensure that your nodes are running a hardened Amazon Linux or Bottlerocket image. Bottlerocket is particularly effective for security because it has a minimal file system, lacks a shell, and is designed specifically for running containers. Avoid SSH access to your nodes entirely. If you need to debug a container, use ephemeral containers or log aggregation tools rather than logging into the host.

Securing Amazon ECS (Elastic Container Service)

ECS is often perceived as simpler than EKS, but it requires equally rigorous security standards. The focus here shifts to Task Definitions, task roles, and VPC configuration.

1. The Principle of Least Privilege in Task Roles

ECS uses two types of roles: the Task Execution Role and the Task Role. The Task Execution Role is used by the ECS agent to pull container images and send logs to CloudWatch. The Task Role is used by the application code inside the container to interact with AWS services.

A common mistake is using the same role for both or granting overly broad permissions. Always audit your Task Roles to ensure they only have the s3:GetObject or dynamodb:PutItem permissions they absolutely require. Use condition keys in your IAM policies to restrict access further, such as limiting access to a specific S3 bucket prefix.

2. Secrets Management

Never hardcode credentials in your task definitions or environment variables. Environment variables are often logged in plain text or visible in the AWS console. Instead, use the integration between ECS and AWS Secrets Manager or Systems Manager Parameter Store.

In your task definition, you can reference a secret directly:

"secrets": [
  {
    "name": "DB_PASSWORD",
    "valueFrom": "arn:aws:ssm:region:account:parameter/db_password"
  }
]

This way, the ECS agent fetches the secret at runtime and injects it into the container memory, where it is never written to disk or stored in the task definition configuration.

3. Fargate vs. EC2 Launch Types

ECS provides two ways to run containers: Fargate (serverless) and EC2. From a security perspective, Fargate is generally safer because it provides workload isolation at the kernel level. You do not have to manage the underlying host or worry about patching the OS. If your security policy requires deep control over the underlying instance (e.g., specific kernel modules or custom agents), you might choose EC2, but be aware that you are now responsible for patching and hardening that host.

Comparing EKS and ECS Security Features

Note: When using ECS with Fargate, you gain the benefit of "security by isolation." Because there is no shared host, a compromise of one container cannot lead to a breakout into the underlying OS, as the abstraction is managed entirely by the cloud provider.

Best Practices for Image Security

Regardless of whether you use EKS or ECS, the security of your container images is the foundation of your compute security. A vulnerable application inside a secure container is still a vulnerable application.

1. Use Minimal Base Images

Avoid using general-purpose operating system images like Ubuntu or CentOS as your base. These images contain hundreds of packages, shells, and utilities that an attacker can use once they gain a foothold. Instead, use "distroless" images or minimal images like Alpine Linux. These images contain only your application and its dependencies, significantly reducing the attack surface.

2. Implement Image Scanning

Automate vulnerability scanning in your CI/CD pipeline. Use tools like Amazon ECR Image Scanning, which uses the Clair engine to identify known vulnerabilities (CVEs) in your image layers.

• Fail builds on high-severity CVEs: Configure your pipeline to stop the deployment if an image contains critical vulnerabilities.
• Regularly refresh images: Even if an image was secure last week, a new vulnerability might be discovered today. Set up a policy to re-build and re-deploy your services on a regular schedule.

3. Immutable Infrastructure

Treat your containers as immutable. Never "patch" a running container. If you need to update a library or a configuration, build a new image, push it to your registry, and redeploy your EKS deployment or ECS service. This ensures that your production environment always matches your tested, scanned images.

Common Pitfalls and How to Avoid Them

Pitfall 1: Overly Permissive IAM Roles

Many developers assign the AdministratorAccess policy to ECS Task Roles or EKS node groups because it "just works." This is the single most dangerous practice in cloud security.

• Solution: Use the IAM Access Analyzer to identify unused permissions and generate policies based on actual service usage.

Pitfall 2: Running Containers as Root

By default, many Docker images run as the root user. If an attacker manages to break out of the container, they are already root on the host.

• Solution: In your Dockerfile, always create a non-privileged user and switch to it using the USER directive.

# Dockerfile best practice
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

Pitfall 3: Ignoring Network Segmentation

Treating the internal network as a "trusted zone" is a relic of the past.

• Solution: Assume that every pod or task could be compromised. Use Security Groups to restrict traffic between services to only the ports and protocols necessary for communication. In EKS, ensure that allow-all policies are replaced with granular NetworkPolicy objects.

Pitfall 4: Exposing Sensitive Data in Environment Variables

Environment variables are visible to anyone who can describe the pod or task definition.

• Solution: As discussed, move all secrets to a dedicated secret storage service (Secrets Manager or Parameter Store) and inject them at runtime.

Detailed Step-by-Step: Hardening an EKS Deployment

If you are currently running an EKS cluster, follow this checklist to improve your security posture:

1. Audit the Control Plane:
   • Navigate to the EKS console and check the "Networking" tab. Ensure the cluster endpoint is private.
   • Enable all diagnostic logs: API server, Audit, Authenticator, Controller Manager, and Scheduler.
2. Verify IAM Roles:
   • Run kubectl get pods -A to list all pods.
   • For each deployment, check the associated service account. Ensure that the IAM role attached to that service account has the minimum necessary permissions.
3. Implement Pod Security:
   • Install the Kyverno or OPA Gatekeeper admission controller. These tools allow you to enforce policies, such as "all pods must run as non-root" or "all containers must have resource limits."
4. Scan Images:
   • Enable "Scan on push" in your Amazon ECR repositories.
   • Review the findings in the ECR console and prioritize patching images with Critical or High severity vulnerabilities.
5. Restrict Network Traffic:
   • Deploy a basic NetworkPolicy to deny all ingress traffic by default.
   • Gradually add "allow" rules for the specific services that need to talk to each other.

Advanced Security: Runtime Protection

While static analysis and IAM policies are essential, they do not protect you against zero-day exploits or runtime attacks. For high-security requirements, consider implementing runtime security tools.

Tools like Falco (for Kubernetes) monitor system calls at the kernel level. They can detect suspicious behavior such as:

• A shell being spawned inside a container.
• A container attempting to write to a sensitive path like /etc/.
• Unexpected outbound network connections to unknown IP addresses.

By integrating these tools into your EKS cluster, you receive real-time alerts when your security policies are violated, allowing you to respond to incidents before they escalate.

Callout: The Importance of Observability Security without observability is blind. You cannot protect what you cannot see. Ensure that your logs, metrics, and security alerts are centralized in a single pane of glass (like Amazon CloudWatch or an external SIEM). Without centralized logging, an attacker can delete the logs of their intrusion from the local container disk, leaving you with no evidence of the breach.

Key Takeaways for Compute Security

To wrap up this lesson, here are the most critical points to remember when securing your EKS and ECS workloads:

1. Principle of Least Privilege: Never assign broad IAM permissions to a compute resource. Always scope roles down to the specific actions and resources required by the application.
2. Identity-Based Security: Use IRSA for EKS and Task Roles for ECS. Never rely on the permissions of the underlying host instance to authorize your application's interactions with AWS services.
3. Network Segmentation is Mandatory: Do not trust the internal network. Use Security Groups and Network Policies to enforce strict traffic boundaries between your services.
4. Automate Image Security: Vulnerability scanning should be a non-negotiable part of your CI/CD pipeline. If an image is vulnerable, it should not reach production.
5. Minimize the Attack Surface: Use minimal or "distroless" base images to reduce the amount of code and utilities available for an attacker to exploit.
6. Secrets Management: Never store credentials in source code or task definitions. Use dedicated secret managers and inject secrets at runtime into memory.
7. Continuous Monitoring: Enable audit logging for your orchestration platforms and implement runtime security monitoring to detect anomalies that static analysis cannot catch.

By applying these principles, you move from a reactive security posture to a proactive one. Security in compute is not a one-time setup; it is a continuous cycle of assessment, configuration, and monitoring. As you continue to build and scale your infrastructure, keep these practices at the forefront of your design decisions, and you will significantly reduce the risk of a security incident.

Common Questions (FAQ)

Q: Should I use EKS or ECS for better security? A: Neither is inherently more secure. ECS is easier to secure because it is simpler and has fewer moving parts. EKS offers more granular control, which allows for deeper security customization but also introduces more opportunities for misconfiguration. Choose based on your team's expertise and the specific requirements of your applications.

Q: Is it safe to use public container images from Docker Hub? A: Generally, no. Public images often lack regular patching and may contain malicious code or hardcoded credentials. If you must use public images, pull them into your private ECR registry, scan them, and sign them before deploying them to your cluster.

Q: How do I handle SSH access to containers? A: You shouldn't. If you find yourself needing to SSH into a container, it usually means your logging or observability is insufficient. Use kubectl exec only for emergency troubleshooting, and ensure that these sessions are audited through your cluster's API logs.

Q: What is the biggest risk in Kubernetes security? A: The most common risk is an overly permissive API server combined with improperly configured RBAC (Role-Based Access Control). If a user or service account has cluster-admin privileges, they can effectively bypass every other security control in the cluster. Always audit your RBAC roles regularly.