Lesson: Custom Threat Detection with Amazon EventBridge

Introduction: Why Custom Threat Detection Matters

In the modern landscape of cloud computing, static security rules are no longer sufficient to protect infrastructure. While managed security services provide a baseline of protection by monitoring for known signatures and patterns, they often struggle to identify context-specific threats unique to your organization’s environment. Custom threat detection bridges this gap by allowing you to create logic that interprets your infrastructure's specific behavior, user access patterns, and API interactions.

Amazon EventBridge serves as the central nervous system for this activity. It acts as a serverless event bus that receives streams of data from various sources—such as cloud infrastructure logs, application logs, and third-party SaaS providers—and routes them to designated targets for analysis. By leveraging EventBridge, you move away from reactive, manual log review and toward automated, real-time security orchestration. This lesson explores how to harness this capability to build a responsive, custom threat detection engine that evolves alongside your architecture.

Understanding the Event-Driven Security Architecture

At its core, an event-driven security architecture treats every administrative action, network connection, or configuration change as an "event." When a developer launches a new virtual machine, modifies a security group, or accesses a sensitive database, these actions generate events within the cloud provider's audit logs. EventBridge captures these events, evaluates them against defined rules, and triggers automated responses if a security policy is violated.

The Anatomy of an Event

Every event in EventBridge follows a structured JSON format. Understanding this structure is critical because your detection rules will rely on matching specific fields within this JSON object. A typical event contains:

• Source: The service that generated the event (e.g., aws.ec2).
• Detail-type: The specific category of the event (e.g., EC2 Instance State-change Notification).
• Detail: The payload containing the actual data about the occurrence.
• Time: The timestamp when the event occurred.
• Resources: A list of ARNs or identifiers of the resources involved.

Callout: Event-Driven vs. Polling-Based Detection Traditionally, security teams used "polling" models where scripts would periodically query logs to look for anomalies. This approach is inherently flawed because it introduces a time lag between the event and the detection. Event-driven detection using EventBridge operates in near real-time, meaning the moment an unauthorized action occurs, the detection logic is triggered, allowing for immediate mitigation before an attacker can escalate their privileges.

Building Your First Detection Rule: Monitoring Unauthorized API Calls

The most common use case for custom threat detection is monitoring for unauthorized API calls or sensitive configuration changes. For example, if a user attempts to modify a firewall rule that is supposed to be immutable, you need to know immediately.

Step-by-Step: Creating a Rule for Security Group Changes

1. Identify the Event Source: In this case, we use aws.ec2.
2. Define the Pattern: We need to look for AuthorizeSecurityGroupIngress or RevokeSecurityGroupIngress actions.
3. Set the Target: Route these events to an AWS Lambda function that performs the analysis or notification.

Example Event Pattern (JSON)

{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": [
      "AuthorizeSecurityGroupIngress",
      "AuthorizeSecurityGroupEgress"
    ]
  }
}

This pattern tells EventBridge to ignore everything except for the specific API calls that modify network ingress or egress rules. By narrowing the focus, you reduce the noise and ensure that your downstream analysis engine only processes relevant security data.

Implementing the Analysis Logic

Once the event hits your Lambda function, you need to apply logic to determine if the change is malicious. A simple script might check if the change opens a port to the entire internet (0.0.0.0/0).

import json

def lambda_handler(event, context):
    # Extract the detail from the event
    detail = event.get('detail')
    request_parameters = detail.get('requestParameters', {})
    
    # Check for wide-open ingress rules
    ip_permissions = request_parameters.get('ipPermissions', {})
    items = ip_permissions.get('items', [])
    
    for item in items:
        ip_ranges = item.get('ipRanges', {}).get('items', [])
        for ip_range in ip_ranges:
            if ip_range.get('cidrIp') == '0.0.0.0/0':
                # Trigger an alert or remediation
                print("ALERT: Security group modified to allow public access!")
                return {"status": "alert_triggered"}
                
    return {"status": "no_threat_detected"}

Note: Always ensure that your Lambda functions follow the principle of least privilege. The function only needs permissions to read the event data and publish notifications (e.g., to an SNS topic). It does not need administrative access to your entire cloud environment.

Advanced Detection: Correlating Multi-Source Events

Simple detection rules look at one event at a time, but sophisticated threats often span multiple actions. For instance, an attacker might first disable logging, then create a new user, and finally launch an expensive resource. Detecting this requires stateful analysis.

Using EventBridge Pipes and Step Functions

When a single event isn't enough, you can pipe events into an AWS Step Functions workflow. This allows you to maintain state across multiple events. You could create a "workflow" that tracks user behavior over a 15-minute window. If the user performs three "suspicious" actions within that window, the workflow triggers an incident response.

Pattern Matching for Sequence Detection

To detect a sequence, you can use EventBridge to route events to a database (like DynamoDB) that logs the activity. A second Lambda function can then query this database to see if the current event completes a known attack pattern.

Best Practices for Threat Detection Rules

Creating rules is easy, but maintaining a high-signal, low-noise environment requires discipline. Follow these industry-standard practices to ensure your detection engine remains effective.

1. Minimize False Positives

False positives are the primary cause of "alert fatigue." If your team receives hundreds of alerts every day, they will eventually stop paying attention. Before deploying a new detection rule, run it in a "log-only" mode for several days to analyze the volume of alerts generated.

2. Use Descriptive Metadata

When an alert is triggered, include as much context as possible in the notification. Instead of saying "Security Group Changed," your message should include:

• The ID of the user who made the change.
• The IP address the request originated from.
• The specific resource ARN that was modified.
• A link to the relevant internal security policy document.

3. Automate Remediation Where Possible

Detection is only half the battle. If a rule detects an unauthorized public S3 bucket, don't just alert the team—have the Lambda function automatically set the bucket to private. This is known as "auto-remediation."

Warning: Be extremely careful with automated remediation. If a rule is too broad, it could disrupt legitimate business operations. Always implement a "human-in-the-loop" approval process for critical infrastructure changes unless you are 100% confident in the rule's logic.

Common Pitfalls and How to Avoid Them

Even experienced security engineers fall into common traps when implementing custom detection. Being aware of these will save you countless hours of debugging.

The "Over-Filtering" Trap

Some engineers try to write complex event patterns that include every possible condition. While this seems efficient, it often leads to missing events due to subtle mismatches in the JSON structure. It is better to have a slightly broader pattern and filter the noise inside your Lambda function, where you have more control over the logic.

Ignoring Rate Limits

EventBridge has service quotas. If you configure a rule that triggers a Lambda function for every single API call in a high-traffic environment, you may hit concurrency limits. Always monitor your EventBridge metrics and ensure your downstream targets can handle the throughput.

Hardcoding Sensitive Information

Never hardcode API keys, database credentials, or secret identifiers inside your detection scripts. Use environment variables or a secrets management service. If your detection logic requires access to an external API (like a threat intelligence feed), use a secure store to manage those credentials.

Implementing a Real-World Scenario: The "Suspicious Login" Detector

Let's walk through a more advanced scenario. We want to detect when a user logs into our management console from an unrecognized geographic location.

Step 1: The Event Source

We focus on aws.signin events. These events occur when a user successfully authenticates.

Step 2: The Logic

1. The EventBridge rule triggers on ConsoleLogin.
2. The Lambda function parses the sourceIpAddress from the event.
3. The function uses an IP geolocation library (or an external API) to determine the country of origin.
4. If the country is not in a pre-defined "allowed" list, the function sends a notification to your security team via an SNS topic.

Code Snippet for Geolocation Check

import json
import requests # Note: You'll need to package this or use a Lambda layer

def lambda_handler(event, context):
    ip_address = event['detail']['sourceIPAddress']
    allowed_countries = ['US', 'CA', 'GB']
    
    # Use a mock service for IP lookup
    response = requests.get(f"https://ipapi.co/{ip_address}/json/")
    data = response.json()
    country = data.get('country_code')
    
    if country not in allowed_countries:
        # Trigger high-priority alert
        send_alert(f"Unauthorized login from {country} at IP {ip_address}")
        
    return {"status": "success"}

Callout: Why Not Use Managed Services? You might wonder why you would build this when cloud providers offer managed threat detection services. The answer is customization. Managed services are excellent at detecting generic threats, but they cannot know that your organization only operates out of three specific countries. Custom detection allows you to build rules that reflect the reality of your business, not just general security best practices.

Maintaining and Scaling Your Detection System

As your organization grows, so will the number of events. You must treat your security infrastructure as code.

Infrastructure as Code (IaC)

Use tools like Terraform or AWS CloudFormation to define your EventBridge rules. This ensures that your detection logic is version-controlled, auditable, and can be easily deployed across multiple environments (e.g., Development, Staging, Production).

Testing Your Rules

You cannot rely on real attacks to test your detection rules. You must implement "Game Days" or "Security Chaos Engineering." During these sessions, you intentionally perform suspicious actions to verify that your EventBridge rules trigger as expected. If an action doesn't trigger an alert, you have a gap in your coverage.

Monitoring the Monitoring

Who monitors the security system? You should have CloudWatch alarms set up to notify you if your Lambda functions fail or if your EventBridge rules fall behind in processing events. If the detection engine itself goes down, you are effectively blind to threats.

Comparison: EventBridge vs. Other Security Tools

FAQ: Common Questions

Q: Does EventBridge impact the performance of my applications? A: No. EventBridge operates asynchronously. When an event is generated, the source service does not wait for EventBridge to process it. Your application performance remains unaffected.

Q: Can I use EventBridge to detect non-AWS events? A: Yes. EventBridge supports custom event buses. You can configure your own applications, third-party SaaS tools, or even on-premises servers to send events to EventBridge via an API, allowing you to centralize security monitoring across your entire ecosystem.

Q: How do I handle very high volumes of events? A: If you expect millions of events, consider using EventBridge Pipes to filter and transform events before they reach your targets. This reduces the load on your downstream services and optimizes costs.

Key Takeaways

1. Event-Driven Security is Proactive: By using EventBridge, you move from periodic log checking to real-time, event-driven detection, allowing for immediate mitigation of threats.
2. Context is King: The power of custom detection lies in the ability to define rules that understand your specific business requirements, such as expected geographic locations or specific administrative workflows.
3. Signal-to-Noise Ratio: Focus on precision in your detection rules. Over-alerting leads to fatigue; always test your rules in a non-production environment before enabling full-scale automated responses.
4. Automation is Essential: Use Lambda functions not just to alert, but to auto-remediate. If you detect a common, low-risk misconfiguration, let the system fix it automatically so your security team can focus on complex, high-value threats.
5. Infrastructure as Code: Treat your detection rules as software. Use version control, automated testing, and deployment pipelines to manage your rules, ensuring consistency and reliability across all your environments.
6. Continuous Testing: Regularly validate your detection logic through security exercises. Don't assume a rule works just because it is deployed; verify it by simulating the conditions it is meant to catch.
7. Monitor the System: Ensure your detection infrastructure is itself monitored. A silent failure in your security engine is a major vulnerability, so always have health checks and alerts on your detection pipelines.

By integrating these strategies into your workflow, you will build a robust, scalable, and highly effective threat detection capability. EventBridge is not just a routing tool; it is the foundation of a modern, automated security posture that keeps your systems safe in an ever-evolving threat landscape. Remember that security is a process, not a destination—continue to refine your rules, learn from your events, and iterate on your detection logic as your organization evolves.