Lesson: Cross-Account Log Aggregation with CloudWatch

Introduction: Why Centralize Logs?

In modern cloud environments, infrastructure is rarely confined to a single account. Organizations typically split workloads across multiple AWS accounts to enforce strict security boundaries, manage billing costs, or separate development, staging, and production environments. While this isolation is excellent for security and operational hygiene, it creates a significant challenge for security teams: visibility. If your security operations center (SOC) team has to log into ten, fifty, or one hundred different AWS accounts to investigate a potential incident, the time to detect and respond to threats increases exponentially.

Cross-account log aggregation is the practice of streaming logs from multiple "source" accounts into a single, centralized "logging" account. By doing this, you create a "single pane of glass" for your security telemetry. This allows you to run centralized queries, trigger cross-account alerts, and maintain a long-term audit trail that remains intact even if an individual source account is compromised or deleted. This lesson will guide you through the architecture, implementation, and best practices of building a centralized logging strategy using AWS CloudWatch.

The Architecture of Centralization

To implement cross-account logging, you need to designate one account as the "Central Logging Account." This account acts as the final destination for all logs generated by your "Member" (or source) accounts. The communication between these accounts relies on CloudWatch Logs Destinations, which act as a bridge between accounts.

The core mechanism involves setting up a Kinesis Data Stream in the Central Logging Account. This stream acts as a buffer that receives logs from multiple sources. You then attach a CloudWatch Logs Destination to this stream. In the Member accounts, you configure Subscription Filters that point to this Destination. When a log event occurs in a Member account, it is automatically pushed through the stream and into your central account.

Why use Kinesis Data Streams?

You might wonder why we don't just stream logs directly from CloudWatch to CloudWatch. The primary reason is scalability and decoupling. By placing a Kinesis Data Stream in the middle, you provide a buffer that handles spikes in log volume without dropping data. Additionally, it allows you to fan-out logs to other services, such as an S3 bucket for long-term cold storage, an OpenSearch cluster for advanced analytics, or a Lambda function for real-time threat detection.

Callout: The Hub-and-Spoke Model The architecture described here is commonly referred to as the Hub-and-Spoke model. The Central Logging Account is the Hub, and all other accounts are the Spokes. This model is preferred because it minimizes the configuration required in the spokes, centralizes the security posture, and simplifies compliance auditing by ensuring all logs are stored in a hardened, restricted environment.

Step-by-Step Implementation

Setting up cross-account log aggregation involves three distinct phases: setting up the Hub (Central Account), establishing the cross-account permissions, and configuring the Spokes (Member Accounts).

Phase 1: Setting up the Hub (Central Account)

In the central account, you need to prepare the infrastructure that will receive the logs.

1. Create a Kinesis Data Stream: Create a stream with sufficient shards to handle your expected throughput.
2. Create an IAM Role: Create a role for CloudWatch to assume. This role must have a trust policy allowing logs.amazonaws.com to assume it, and a policy granting the kinesis:PutRecord and kinesis:PutRecords permissions on the stream you created.
3. Create the CloudWatch Logs Destination: Using the AWS CLI or SDK, create a destination that maps to the Kinesis stream and the IAM role.

Example: Creating a Destination via CLI

# First, create the Kinesis stream
aws kinesis create-stream --stream-name CentralLoggingStream --shard-count 2

# Create the IAM role (policy document omitted for brevity)
aws iam create-role --role-name CloudWatchToKinesisRole --assume-role-policy-document file://trust-policy.json

# Attach the policy allowing CloudWatch to write to Kinesis
aws iam put-role-policy --role-name CloudWatchToKinesisRole --policy-name CloudWatchKinesisPolicy --policy-document file://permissions.json

# Finally, create the destination
aws logs put-destination \
    --destination-name CentralLoggingDestination \
    --target-arn arn:aws:kinesis:us-east-1:123456789012:stream/CentralLoggingStream \
    --role-arn arn:aws:iam::123456789012:role/CloudWatchToKinesisRole

Phase 2: Defining Cross-Account Permissions

The destination you just created is private by default. You must explicitly allow the member accounts to send logs to it. This is done by attaching an access policy to the destination.

# Define the access policy
# This policy allows accounts 111111111111 and 222222222222 to use the destination
aws logs put-destination-policy \
    --destination-name CentralLoggingDestination \
    --access-policy '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "AWS": ["111111111111", "222222222222"] },
          "Action": "logs:PutSubscriptionFilter",
          "Resource": "arn:aws:logs:us-east-1:123456789012:destination:CentralLoggingDestination"
        }
      ]
    }'

Phase 3: Configuring the Spokes (Member Accounts)

In each member account, you must create a subscription filter for each log group you want to export. This filter tells CloudWatch to send incoming logs to the destination located in the hub account.

# Create a subscription filter in the member account
aws logs put-subscription-filter \
    --log-group-name "/aws/lambda/my-app-logs" \
    --filter-name "CentralLogSubscription" \
    --destination-arn "arn:aws:logs:us-east-1:123456789012:destination:CentralLoggingDestination" \
    --filter-pattern ""

Note: The filter-pattern is empty in the example above, which means all logs are sent. You should use this field to filter out noisy logs (like debug-level logs) to save on ingestion costs.

Best Practices for Log Aggregation

Implementing this architecture is only the first step. To make your logging effective, you need to follow industry-standard best practices regarding security, cost, and management.

1. Log Retention and Lifecycle Policies

Logs are expensive. Storing high-volume logs indefinitely in CloudWatch can result in a massive monthly bill. It is best practice to set a short retention period in CloudWatch (e.g., 7 or 14 days) and use a Kinesis Firehose delivery stream to archive those logs into S3. Once in S3, you can use S3 Lifecycle policies to move logs to cheaper storage classes like S3 Glacier or delete them after a set period.

2. Encryption at Rest and in Transit

Ensure that all logs are encrypted using AWS Key Management Service (KMS). When sending logs across accounts, use customer-managed keys (CMKs) to ensure you retain control over the encryption keys. This prevents unauthorized access to the logs even if the underlying storage media is compromised.

3. Monitoring the Pipeline

What happens if the pipeline breaks? If a misconfiguration occurs in a member account, logs might stop flowing. You should set up CloudWatch Alarms on the "IncomingBytes" metric of your Kinesis Data Stream and the "IncomingLogEvents" metric on the CloudWatch Logs Destination. If these metrics drop to zero unexpectedly, you should trigger an alert to your engineering team.

4. IAM Least Privilege

Do not use root or overly permissive credentials to configure these resources. Use Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation to manage these configurations. This ensures that the permissions required for cross-account access are documented and auditable, and that no human has permanent "admin" rights to the logging infrastructure.

Comparison Table: Log Collection Methods

Callout: Why Subscription Filters are Superior Subscription Filters are the preferred method for AWS-native services because they are fully managed. You don't have to install software on your EC2 instances or worry about agent crashes. They provide a reliable, low-latency stream of data directly from the AWS logging backend to your destination.

Common Pitfalls and How to Avoid Them

Pitfall 1: Circular Dependencies

A common mistake is creating a circular dependency where a log group in the central account is configured to send logs to the central account's own destination. This creates an infinite loop of log events, which will inflate your costs rapidly. Always ensure that the destination ARN is unique and carefully managed.

Pitfall 2: Ignoring Throughput Limits

Kinesis Data Streams have specific shard limits. If you are aggregating logs from hundreds of accounts, you might hit the throughput limit of a single stream. You must monitor the WriteProvisionedThroughputExceeded metric in Kinesis and scale your shard count accordingly. If your logs are bursty, consider using "On-Demand" mode for your Kinesis streams.

Pitfall 3: Incomplete IAM Policies

When setting up the cross-account destination policy, developers often use wildcards (*) to simplify the configuration. This is a security risk. Always specify the exact Account IDs that are permitted to send logs. Furthermore, ensure that the IAM role in the central account has the most restrictive policy possible—it should only have permission to write to the specific Kinesis stream created for logging.

Pitfall 4: Missing Log Metadata

When logs arrive in the central account, it can be difficult to tell which account they originated from if the log format doesn't include the Account ID. AWS CloudWatch logs sent through subscription filters include the owner field, but you should ensure your log processing logic (e.g., in Lambda) correctly parses this metadata so you can filter and query by origin.

Practical Troubleshooting Steps

When logs aren't appearing in your central account, follow this checklist to isolate the issue:

1. Check Subscription Filter Status: Run aws logs describe-subscription-filters --log-group-name <name> in the member account. Ensure the destination ARN is correct.
2. Verify IAM Permissions: Check the "Access Policy" on the destination in the central account. Does it include the member account's ID?
3. Check Kinesis Metrics: Look at the "IncomingBytes" metric in the central account's Kinesis dashboard. If it's zero, the issue is on the member side. If it's positive but the logs aren't in the final storage (like S3), the issue is on the Kinesis-to-S3 delivery pipeline.
4. Examine Lambda Logs (if applicable): If you are using a Lambda to process logs, check the CloudWatch logs for that specific Lambda function for any runtime errors or permission issues.

Advanced Considerations: Security Analytics

Once you have your logs centralized, you are in a prime position to perform security analytics. You can use Amazon Athena to query your logs stored in S3. Since S3 is the destination for your aggregated logs, you can write SQL queries to hunt for threats across your entire organization.

Example: Finding suspicious API calls with Athena

SELECT eventTime, userIdentity.arn, eventName, sourceIpAddress
FROM cloudtrail_logs
WHERE eventName LIKE 'Delete%'
AND eventTime > '2023-10-01'
ORDER BY eventTime DESC;

This query, when run against a table of centralized CloudTrail logs, can show you every "Delete" action performed across your entire cloud footprint. Without centralized logging, you would have to run this query in every single account individually, which is practically impossible at scale.

The Role of AWS Organizations

If you are using AWS Organizations, you can simplify this entire process using "CloudWatch Logs Centralized Logging" features that are now becoming integrated into the AWS Control Tower and Organization-level services. Instead of manually configuring every account, you can use a Service Control Policy (SCP) or a CloudFormation StackSet to deploy the subscription filters automatically to every new account that joins your organization.

Tip: Use CloudFormation StackSets to deploy your logging configuration. By creating a StackSet in your management account, you can "target" all accounts in your organization. This ensures that whenever a new account is created, it is automatically configured to stream its logs to your central account without any manual intervention.

Summary Checklist for Success

1. Centralize: Use a dedicated "Logging Account" to isolate and protect your logs.
2. Buffer: Use Kinesis Data Streams to handle high-volume traffic and provide a bridge between accounts.
3. Automate: Use Infrastructure as Code (StackSets) to apply subscription filters to all accounts in the organization.
4. Protect: Use KMS keys to encrypt logs at rest in the central account.
5. Lifecycle: Move logs from CloudWatch to S3 and use lifecycle policies to minimize storage costs.
6. Audit: Regularly review your IAM policies and destination access policies to ensure only authorized accounts can send data.
7. Monitor: Set up alerts on your Kinesis and Destination metrics to detect if the log stream stops.

Frequently Asked Questions

Q: Can I send logs from multiple log groups to the same destination? A: Yes. A single destination can receive logs from any number of log groups, even across different accounts.

Q: Are there costs associated with cross-account logging? A: Yes. You pay for Kinesis Data Stream ingestion, the amount of data stored in S3, and standard CloudWatch charges. However, this is usually cheaper than the operational cost of managing decentralized logging.

Q: How do I handle log format differences? A: Since CloudWatch logs are essentially JSON, you can use a Lambda function in the central account to normalize the log format before storing it in S3 or OpenSearch.

Q: Is there a limit to how many accounts I can aggregate? A: There is no hard limit on the number of accounts, but you must monitor your Kinesis shard limits and CloudWatch API quotas as your organization grows.

Final Key Takeaways

• Visibility is Security: You cannot protect what you cannot see. Aggregating logs is the first step in building a mature security monitoring program.
• Decouple for Reliability: Using a Kinesis Data Stream between your source accounts and your storage destination prevents data loss during traffic spikes and allows for flexible post-processing.
• Automation is Mandatory: Manual configuration of logging in a multi-account environment is prone to error and impossible to maintain. Use StackSets.
• Lifecycle Management: Logs are data, and data has a lifecycle. Plan for how you will store, archive, and eventually delete logs to control costs.
• The Power of SQL: Centralizing logs into S3 unlocks the ability to use query engines like Athena, turning raw text files into actionable security intelligence.

By following the patterns and practices outlined in this lesson, you will move from a fragmented, difficult-to-manage logging setup to a robust, scalable, and secure architecture that provides the visibility required to operate safely in the cloud. Remember that security is an iterative process; start with your most critical accounts, validate the flow, and then roll it out across your entire organization.