Security Logging and Monitoring: ELB and CloudFront Logging

Introduction: Why Logging at the Edge Matters

In modern cloud architecture, the Elastic Load Balancer (ELB) and Amazon CloudFront serve as the primary gateways for incoming traffic. Whether your application is a simple web server or a complex microservices architecture, these services act as the first line of defense and the initial point of contact for every user request. Because they sit at the edge of your infrastructure, they are the most valuable sources of telemetry data for security, performance, and troubleshooting.

Logging at the ELB and CloudFront level is not merely a "nice-to-have" feature; it is a fundamental requirement for maintaining a secure and observable environment. When a security incident occurs, such as a distributed denial-of-service (DDoS) attack or an unauthorized data scraping attempt, your logs provide the forensic evidence needed to reconstruct the timeline of events. Without these logs, you are effectively flying blind, unable to distinguish between legitimate user patterns and malicious activity.

Beyond security, these logs provide deep insights into user experience and system health. You can track latency, identify broken links, monitor geographic traffic patterns, and optimize cache hit ratios. This lesson will guide you through the intricacies of enabling, managing, and analyzing logs from both Elastic Load Balancing and Amazon CloudFront, ensuring you have the visibility required to operate a professional-grade cloud environment.

Part 1: Elastic Load Balancing (ELB) Access Logs

Elastic Load Balancing automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses. To understand how your load balancer is handling traffic, you must enable Access Logs. These logs provide detailed information about requests sent to your load balancer, including the time the request was received, the client’s IP address, latencies, request paths, and server responses.

Enabling Access Logs on ELB

Access logging is an optional feature that is disabled by default. To enable it, you must have an S3 bucket configured to receive the logs. The load balancer requires permission to write to this bucket, which is handled via a bucket policy.

Step-by-Step Configuration:

1. Create an S3 Bucket: Create a dedicated bucket for your logs. It is best practice to keep logs in a separate account or a restricted bucket to prevent tampering.
2. Attach a Bucket Policy: You must allow the Elastic Load Balancing service principal to perform the s3:PutObject action on your bucket.
3. Enable Logging via Console or CLI: In the Load Balancer console, navigate to the "Attributes" tab and select "Edit attributes," then check the box for "Access logs." Specify the S3 bucket and an optional prefix.

Note: The ELB service principal account ID varies by region. Ensure you look up the specific account ID for the region where your load balancer resides to ensure your bucket policy is accurate.

Analyzing ELB Log Structure

ELB logs are stored as text files in your S3 bucket. Each line in the log file represents a single request. The format is standard across all ELB types (Application, Network, and Gateway Load Balancers), though fields may vary slightly.

A typical entry looks like this: type timestamp elb client:port target:port request_processing_time target_processing_time response_processing_time elb_status_code target_status_code received_bytes sent_bytes "request" "user_agent" ssl_cipher ssl_protocol target_group_arn trace_id

• timestamp: When the request was received.
• elb_status_code: The status code generated by the load balancer itself.
• target_status_code: The status code returned by the backend server.
• trace_id: A unique identifier added to the X-Amzn-Trace-Id header, which is essential for distributed tracing across microservices.

Callout: ELB vs. Target Status Codes It is common to see a difference between the ELB status code and the target status code. If the ELB returns a 503 but the target returns a 200, it usually indicates that the load balancer reached its connection limit or the target was unhealthy. Always look at both values to pinpoint where a request failed.

Part 2: Amazon CloudFront Logging

Amazon CloudFront is a content delivery network (CDN) that speeds up the distribution of your static and dynamic web content. Because CloudFront caches content at edge locations worldwide, logs are essential to verify that your cache policies are working and to track requests that never actually reach your origin servers.

Types of CloudFront Logs

CloudFront offers two distinct types of logging:

1. Standard Logs (Access Logs): These are generated for every request and stored in an S3 bucket. They are highly detailed and contain information about the edge location, the request method, the status code, and the object requested.
2. Real-time Logs: These are sent to an Amazon Kinesis Data Stream. They provide much lower latency (seconds instead of hours) and are ideal for real-time monitoring and alerting.

Configuring Standard Logs

To enable standard logging, you must specify an S3 bucket in the CloudFront distribution settings. Unlike ELB, CloudFront does not automatically create the necessary permissions; you must ensure your bucket policy allows the cloudfront.amazonaws.com service principal to write to the bucket.

Best Practices for CloudFront Logging:

• Prefixing: Always use a prefix (e.g., logs/cloudfront/) to organize your bucket files.
• Lifecycle Rules: CloudFront logs can accumulate quickly. Use S3 Lifecycle policies to transition logs to Glacier or delete them after a set period to manage costs.
• Compression: CloudFront automatically compresses logs in GZIP format. Ensure your log processing tools are configured to decompress these files before analysis.

Configuring Real-time Logs

Real-time logs are configured using a "Log Configuration" object in the CloudFront console or API. You define the fields you want to capture (such as c-ip, cs-method, sc-status, etc.) and point the configuration to a Kinesis Data Stream.

{
  "Name": "my-realtime-log-config",
  "Fields": ["timestamp", "c-ip", "cs-method", "sc-status", "sc-bytes"],
  "KinesisStreamConfig": {
    "StreamArn": "arn:aws:kinesis:us-east-1:123456789012:stream/my-log-stream",
    "RoleArn": "arn:aws:iam::123456789012:role/my-kinesis-role"
  }
}

Tip: Use real-time logs for security monitoring (e.g., detecting a sudden spike in 403 Forbidden errors) and standard logs for long-term audit trails and cost analysis.

Part 3: Practical Log Analysis Strategies

Having the logs is only half the battle. To derive value, you must have a mechanism to query and visualize the data.

Querying with Amazon Athena

Amazon Athena is the standard tool for querying S3-based logs using SQL. You do not need to move the data; Athena queries the files directly in S3.

1. Create a Database and Table: Define the schema that matches your log format.
2. Partitioning: Use partitioning (e.g., by date) to improve query performance and reduce costs.
3. Run SQL Queries: You can now perform complex analysis, such as identifying the top 10 IP addresses requesting your site or calculating the average latency of requests.

Example Query: Identifying High Latency Requests

SELECT client_ip, request_uri, target_processing_time
FROM elb_logs
WHERE target_processing_time > 1.0
ORDER BY target_processing_time DESC
LIMIT 10;

Visualizing with Amazon QuickSight

Once you have your queries optimized in Athena, you can connect Amazon QuickSight to visualize trends. Dashboards showing "Requests over Time," "Top Errors by Status Code," or "Geographic Traffic Distribution" are invaluable for operational teams.

Part 4: Security and Compliance Considerations

Logs are a goldmine for attackers if they fall into the wrong hands. They often contain sensitive information, such as IP addresses, user-agent strings, and sometimes even parts of the requested URI that might contain session tokens or PII (Personally Identifiable Information).

Protecting Your Logs

• Encryption at Rest: Ensure your S3 buckets use Server-Side Encryption (SSE-S3 or SSE-KMS).
• Access Control: Use IAM policies to implement the principle of least privilege. Only the security team or automated analysis tools should have s3:GetObject access.
• Integrity: Enable S3 Object Lock if you are in a regulated industry (such as finance or healthcare) to ensure that logs cannot be deleted or modified for a specific retention period.

Callout: PII in Logs Never log full authorization headers or sensitive body content. If your application design accidentally includes PII in the URL, you must implement a log scrubbing process (using AWS Glue or Lambda) before the logs are stored in your long-term repository.

Common Pitfalls to Avoid

1. Ignoring Log Volume Costs: S3 storage and Athena queries can become expensive if you log everything at high granularity. Be selective about the fields you log in CloudFront real-time configurations.
2. Missing Time Synchronization: Ensure all your log analysis tools account for UTC time. ELB and CloudFront logs are always in UTC; mixing them with local application logs that might be in different time zones is a frequent cause of confusion during incident response.
3. Lack of Alerting: Logs are useless if they just sit in S3. You should set up CloudWatch Alarms or Lambda triggers to notify your team when error rates (e.g., 5xx errors) exceed a specific threshold.
4. Failure to Rotate Logs: If you do not have an automated lifecycle policy, your S3 bucket will grow indefinitely, leading to unnecessary costs and difficulty in finding relevant data.

Part 5: Industry Best Practices

To ensure your logging infrastructure is effective, follow these industry-standard practices:

• Centralized Logging: If you have multiple AWS accounts, aggregate all logs into a single, dedicated "Security/Logging" account. This prevents an attacker who gains access to a production account from deleting the evidence of their actions.
• Automation: Use Infrastructure as Code (Terraform, CloudFormation, or CDK) to deploy logging configurations. Never configure logging manually in the console for production environments, as this leads to configuration drift.
• Log Enrichment: Consider using AWS Lambda to enrich your logs as they arrive. For example, you could append geolocation data to IP addresses, making it easier to identify traffic from unexpected regions.
• Retention Policies: Define your retention period based on compliance requirements (e.g., GDPR, PCI-DSS). Usually, 90 days of "hot" data and 1-7 years of "cold" archived data is sufficient for most organizations.

Comparison Table: ELB vs. CloudFront Logging

Part 6: Implementation Example – Automating Log Cleanup

To prevent your S3 log bucket from becoming a cost burden, you should implement an S3 Lifecycle Policy. You can do this using the AWS CLI.

Step 1: Create a JSON policy file (lifecycle.json)

{
  "Rules": [
    {
      "ID": "MoveToGlacierAfter30Days",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

Step 2: Apply the policy

aws s3api put-bucket-lifecycle-configuration \
    --bucket my-log-bucket \
    --lifecycle-configuration file://lifecycle.json

This policy ensures that logs are moved to cold storage after 30 days and deleted after one year, balancing accessibility with cost-efficiency.

Part 7: Troubleshooting Common Logging Issues

Even with a perfect setup, you may occasionally run into issues where logs aren't appearing. Here is a checklist for troubleshooting:

1. Check Bucket Permissions: The most common cause is a missing or incorrect bucket policy. Ensure the service principal has s3:PutObject access for the specific prefix you defined.
2. Validate Time Intervals: Remember that CloudFront logs are not instantaneous. If you are looking for logs from five minutes ago, they will not be there. Use real-time logs if you need immediate visibility.
3. Verify Encryption Settings: If your bucket requires KMS encryption, ensure the ELB or CloudFront service principal has permission to use the KMS key. Without this, the service cannot write to the bucket.
4. Check for "Empty" Logs: Sometimes, an S3 bucket will have a manifest file but no log files. This usually happens if there was no traffic during that time interval, or if the load balancer was misconfigured during the window.

Part 8: Advanced Monitoring Patterns

For organizations that need high-level security, logging is only the starting point. You should integrate your logs with services like Amazon GuardDuty or AWS Security Hub.

• CloudFront with Security Hub: By sending your logs to a central security hub, you can correlate edge traffic with other AWS security findings. For example, if GuardDuty detects a compromised EC2 instance, you can cross-reference your ELB logs to see if that instance was making suspicious external requests.
• Anomaly Detection: Use Amazon CloudWatch Logs Insights to run queries that look for deviations from the norm. If your average number of 404 errors is 50 per minute and it suddenly jumps to 5,000, your CloudWatch Alarm should trigger an automated incident response flow.

Warning: Do not rely solely on logs for security. Logs are reactive. Always implement proactive measures like AWS WAF (Web Application Firewall) to block known malicious patterns before they reach your ELB or CloudFront distribution.

Key Takeaways

Logging at the ELB and CloudFront layer is the cornerstone of a secure and observable cloud architecture. As you conclude this lesson, keep these core principles in mind:

1. Visibility is Mandatory: You cannot secure or optimize what you cannot see. Enable access logs immediately upon deploying any edge service.
2. Centralize and Protect: Treat your logs as sensitive data. Store them in a dedicated account, encrypt them, and implement strict access controls to prevent tampering.
3. Choose the Right Tool: Use standard logs for deep historical analysis and compliance, and real-time logs for immediate operational alerting and incident response.
4. Automate Lifecycle Management: Prevent storage costs from spiraling by using S3 Lifecycle policies to automatically archive or delete old log data.
5. Query, Don't Just Store: Use Amazon Athena to turn raw log files into actionable insights. A log file that is never queried is just an expensive digital paperweight.
6. Context Matters: Always correlate logs with other telemetry data (like application logs or infrastructure metrics) to gain a full understanding of your system's behavior.
7. Proactive vs. Reactive: While logs are essential for post-mortem analysis, combine them with automated alerting and WAF rules to detect and stop threats before they impact your users.

By mastering the configuration and analysis of ELB and CloudFront logs, you are not just gathering data; you are gaining the ability to understand the heartbeat of your application. This visibility allows you to respond faster to incidents, optimize your infrastructure for better performance, and maintain a robust security posture in an ever-changing threat landscape. Use the tools provided by AWS—Athena, QuickSight, and Kinesis—to turn these streams of data into your most powerful operational asset.