Threat Detection and Incident Response: Security Hub and the ASFF Standard

Introduction: The Challenge of Security Visibility

In modern cloud environments, security is rarely managed by a single tool. Organizations often deploy firewalls, identity management systems, vulnerability scanners, and endpoint protection software, each generating its own unique stream of log data. Without a centralized way to interpret these signals, security operations teams often face "alert fatigue," where critical indicators of compromise are lost in a sea of noisy, disparate data formats. This fragmentation makes it nearly impossible to maintain a clear picture of an organization’s security posture in real-time.

AWS Security Hub was designed to solve this exact problem by acting as a central dashboard that aggregates, organizes, and prioritizes security alerts from across your cloud infrastructure. However, simply collecting data is not enough; the data must be interpretable by both humans and automated systems. This is where the AWS Security Finding Format (ASFF) comes into play. ASFF is the standardized language that Security Hub uses to normalize disparate data, ensuring that a threat detected by a network firewall looks structurally identical to a threat detected by an identity provider.

Understanding how to work with Security Hub and the ASFF is a fundamental skill for any security engineer. By mastering these tools, you move away from manual log parsing and toward automated, structured incident response. This lesson will guide you through the architecture of Security Hub, the technical structure of the ASFF, and the practical implementation of these tools to harden your cloud environment.

Understanding AWS Security Hub: The Central Nervous System

AWS Security Hub functions as a cloud security posture management (CSPM) tool. It continuously monitors your AWS environment using security best practice checks, aggregates alerts from various services like Amazon GuardDuty, Amazon Inspector, and AWS IAM Access Analyzer, and provides a unified view of your security status.

The Role of Aggregation

The primary value of Security Hub is its ability to ingest data from multiple sources. When you enable Security Hub, it begins to pull findings from integrated AWS services automatically. Furthermore, it allows for third-party integrations, meaning your existing security stack—whether it be a web application firewall or a data loss prevention tool—can feed its findings directly into the same dashboard. This central repository allows your team to focus on a single pane of glass rather than jumping between consoles.

Compliance and Standards

Beyond simple threat detection, Security Hub automates compliance tracking. It maps your environment’s configuration against industry-standard frameworks such as the CIS AWS Foundations Benchmark or the PCI DSS standard. By comparing your current resource configurations against these benchmarks, Security Hub identifies specific misconfigurations—such as an S3 bucket with public read access or an IAM policy that is overly permissive—and reports them as findings in the ASFF format.

Callout: Security Hub vs. SIEM It is common to confuse Security Hub with a Security Information and Event Management (SIEM) system. While a SIEM is designed for log ingestion, long-term storage, and complex correlation across diverse data sources, Security Hub is focused on real-time security findings and posture management. Think of Security Hub as your immediate operational dashboard, whereas a SIEM serves as your historical archive and deep analytics platform.

Deep Dive: The AWS Security Finding Format (ASFF)

The ASFF is a JSON-based schema that defines how security findings are represented within Security Hub. By enforcing a strict structure, ASFF ensures that every finding contains the necessary context for an analyst or an automated script to make a decision.

Why Standardization Matters

If every security tool reported data in its own proprietary format, you would need to write custom parsers for every single integration. ASFF removes this burden. Whether the finding is a brute-force attack detected by GuardDuty or a patch missing from an EC2 instance detected by Inspector, the data follows a predictable structure. This allows you to write one automation script—perhaps a Lambda function that isolates compromised instances—that works regardless of which tool generated the initial alert.

Anatomy of an ASFF JSON Object

An ASFF object is a complex, nested JSON structure. While there are dozens of fields, they can be categorized into four primary buckets:

1. Metadata: Unique identifiers, the source of the finding, and the time it occurred.
2. Resource Context: The specific AWS resource (or resources) involved, including ARNs and tags.
3. Severity and Status: Numerical and qualitative rankings of how urgent the finding is.
4. Threat Information: Detailed data about the threat actor, such as IP addresses, malware signatures, or specific attack techniques.

Below is a simplified example of an ASFF finding:

{
  "SchemaVersion": "2018-10-08",
  "Id": "arn:aws:securityhub:us-east-1:123456789012:subscription/aws-guardduty/12345/finding/abc-123",
  "ProductArn": "arn:aws:securityhub:us-east-1::product/aws/guardduty",
  "GeneratorId": "aws/guardduty",
  "AwsAccountId": "123456789012",
  "Types": ["Software and Configuration Checks/Vulnerabilities/CVE"],
  "FirstObservedAt": "2023-10-27T10:00:00Z",
  "Severity": {
    "Label": "HIGH",
    "Normalized": 70
  },
  "Resources": [
    {
      "Type": "AwsEc2Instance",
      "Id": "arn:aws:ec2:us-east-1:123456789012:instance/i-0abcdef1234567890",
      "Partition": "aws",
      "Region": "us-east-1"
    }
  ],
  "Title": "Unauthorized access attempt detected",
  "Description": "An unauthorized user attempted to access the instance via SSH."
}

Explaining Key Fields

• SchemaVersion: Always use the latest version to ensure compatibility with new Security Hub features.
• Types: This follows a hierarchical taxonomy (e.g., Software and Configuration Checks/Vulnerabilities/CVE). Using this taxonomy allows you to filter your findings by category effectively.
• Severity: This field uses both a Label (INFORMATIONAL, LOW, MEDIUM, HIGH, CRITICAL) and a Normalized score (0-100). This dual approach allows for both human-readable categorization and machine-readable sorting.
• Resources: This is an array, meaning a single finding can be associated with multiple resources, which is vital for complex cross-resource attacks.

Implementing Automated Incident Response

The true power of Security Hub and ASFF is realized when you move from manual intervention to automated response. Because ASFF is standard, you can build "responder" pipelines using Amazon EventBridge and AWS Lambda.

The Response Pipeline Architecture

1. Detection: A security tool generates a finding in ASFF format and sends it to Security Hub.
2. Notification: Security Hub pushes the finding to the Amazon EventBridge default event bus.
3. Filtering: An EventBridge rule is configured to trigger only for specific types of findings (e.g., only "CRITICAL" severity findings).
4. Action: The rule invokes a Lambda function to perform an action, such as revoking an IAM credential, applying a restrictive Security Group, or taking a snapshot of an EBS volume for forensics.

Example: Automating Security Group Lockdown

Imagine you have a finding indicating that an EC2 instance is communicating with a known malicious IP address. You can write a Lambda function to automatically update the instance's Security Group to block all inbound and outbound traffic.

import boto3

def lambda_handler(event, context):
    # Extract the instance ID from the ASFF finding
    finding = event['detail']
    instance_id = finding['Resources'][0]['Id'].split('/')[-1]
    
    ec2 = boto3.client('ec2')
    
    # Logic to isolate the instance
    # 1. Create a new "isolated" security group
    # 2. Modify the instance to use this new group
    print(f"Isolating instance: {instance_id}")
    
    return {"status": "success"}

Note: When building automated responders, always include an "opt-out" or "manual override" mechanism. Blindly isolating production resources can lead to significant downtime. Consider tagging resources that should be excluded from automated remediation.

Best Practices for Managing Security Findings

Managing a high volume of security findings requires discipline. If you do not have a process for reviewing and clearing alerts, your Security Hub will quickly become a "graveyard of alerts" that no one checks.

1. Implement a Lifecycle Policy

Do not let old findings linger indefinitely. Use Security Hub's ability to archive findings. A good practice is to archive findings that have been addressed or deemed false positives after a set period.

2. Prioritize by Business Impact

Not all "CRITICAL" findings are created equal. A critical vulnerability on a public-facing web server is significantly more dangerous than the same vulnerability on an isolated, non-production test instance. Use tags and metadata to prioritize your remediation efforts based on the criticality of the resource.

3. Leverage Insights

Security Hub provides "Insights," which are saved aggregations of findings. Use them to identify recurring patterns. For example, create an insight for "Top 10 resources with the most findings." This helps you identify systemic issues, such as a misconfigured base image that is causing the same vulnerability to appear across fifty different instances.

4. Continuous Improvement

Review your findings monthly. Are you seeing a high number of false positives from a specific tool? If so, tune the configuration of that tool rather than simply archiving the alerts in Security Hub. High false-positive rates erode the trust of your security team.

Common Pitfalls and How to Avoid Them

Even with a robust tool like Security Hub, teams often encounter common traps that undermine their security posture.

Falling for the "Alert Fatigue" Trap

If you enable every single compliance check and third-party integration without filtering, you will be overwhelmed.

• The Fix: Start by enabling only the most critical standards (e.g., CIS Foundations). Once your team is comfortable managing those findings, incrementally add more checks.

Ignoring the "Resources" Field

Some teams write automation that only looks at the Title of the finding. This is dangerous because it ignores the actual context of the attack.

• The Fix: Always inspect the Resources array. Ensure your automation logic handles cases where multiple resources are listed, or where the resource type might not be what you expect.

Failing to Test Automation

Writing an automated remediation script and deploying it directly to production is a recipe for disaster.

• The Fix: Always test your Lambda functions in a sandbox environment. Use a dummy ASFF JSON object (you can copy one from your Security Hub console) to trigger your function and verify that it performs the expected actions without unintended side effects.

Not Updating the Schema

ASFF is updated periodically by AWS to support new features and service integrations.

• The Fix: Ensure your custom code, especially any parsers or regex-based extractors, is written to be resilient to new fields being added to the JSON. Use flexible JSON parsing libraries that ignore unknown fields rather than strict schema validation where possible.

Comparison: ASFF vs. Log Formats

Callout: The Importance of Context The biggest advantage of ASFF over traditional logs is context. A traditional log might tell you "Unauthorized login attempt from 192.168.1.1." An ASFF finding tells you "Unauthorized login attempt from 192.168.1.1, targeting IAM User X, which has administrative privileges, occurring on a production database server." This context is the difference between a minor event and an emergency.

Step-by-Step: Enabling and Customizing Security Hub

If you are just getting started, follow these steps to ensure you are capturing the right data:

1. Enable Security Hub: Navigate to the Security Hub console in your primary region. Ensure you have the necessary permissions.
2. Enable Standards: Choose the compliance frameworks relevant to your business (e.g., CIS AWS Foundations).
3. Integrate Sources: Go to the "Integrations" tab and enable Amazon GuardDuty, Amazon Inspector, and AWS IAM Access Analyzer.
4. Configure EventBridge: Create an EventBridge rule that targets your Security Hub findings. Set the event pattern to filter for Severity: { Label: ["CRITICAL", "HIGH"] }.
5. Create a Responder: Deploy a Lambda function that receives the event and logs the finding details to a dedicated "Incident Response" Slack channel or email list for testing.
6. Refine: Review the findings that come in over the first week. If you notice noise, adjust your EventBridge filter or tune the source tool.

Advanced ASFF: Custom Findings

While Security Hub is designed to ingest data from AWS services, you can also inject your own custom findings. This is incredibly useful for internal security tools or custom scripts that check for business-specific security requirements.

To send a custom finding, you use the BatchImportFindings API call. You must ensure your JSON payload strictly adheres to the ASFF specification, or the API will reject it.

import boto3
import datetime

client = boto3.client('securityhub')

finding = {
    'SchemaVersion': '2018-10-08',
    'Id': 'custom-finding-001',
    'ProductArn': 'arn:aws:securityhub:us-east-1:123456789012:product/custom-tool',
    'GeneratorId': 'my-custom-scanner',
    'AwsAccountId': '123456789012',
    'Types': ['Software and Configuration Checks/Custom Check'],
    'FirstObservedAt': datetime.datetime.now().isoformat() + 'Z',
    'Severity': {'Label': 'MEDIUM'},
    'Title': 'Custom Security Check Failed',
    'Description': 'A custom check found a configuration issue.',
    'Resources': [{'Type': 'Other', 'Id': 'custom-resource-id'}]
}

client.batch_import_findings(Findings=[finding])

This capability allows you to bring your entire security ecosystem into the Security Hub dashboard, creating a truly unified view of your organization’s security health.

Summary and Key Takeaways

As we conclude this lesson, remember that security is an ongoing process of visibility and response. AWS Security Hub and the ASFF provide the foundation for this by normalizing data and enabling automated action.

Key Takeaways:

• Normalization is Essential: The ASFF provides a common language for security data. Without this, your security tools are just silos of information that are difficult to manage.
• Automation is Mandatory: At cloud scale, manual incident response is impossible. Use EventBridge and Lambda to turn security findings into immediate, automated actions.
• Context is King: Always prioritize findings that provide context. A high-severity finding on a critical resource is always more important than a similar finding on an isolated test resource.
• Start Small, Scale Up: Don't enable every check at once. Begin with the most critical standards and tune your environment to minimize noise before expanding.
• Lifecycle Management: Treat your security findings like tickets. If a finding is not actionable, archive it. Keep your dashboard clean so that when a real threat appears, it is immediately obvious to your team.
• Customization is Powerful: Don't be afraid to use the BatchImportFindings API to bring your internal, proprietary security checks into the same dashboard as your AWS-native alerts.

By applying these principles, you will transform your security operations from a reactive, manual process into a proactive, automated, and highly visible capability. The journey to a mature security posture begins with understanding the data you have, and Security Hub is the best starting point for that journey.

Frequently Asked Questions (FAQ)

Q: Can I use Security Hub across multiple AWS accounts? A: Yes, Security Hub supports a multi-account architecture using AWS Organizations. You can designate a "Security Hub Administrator" account to aggregate findings from all member accounts, providing a centralized view for your security team.

Q: What happens if I make a mistake in my ASFF JSON format? A: If you attempt to import a finding that does not match the ASFF schema, the BatchImportFindings API will return an error, and the finding will not be created. Always validate your JSON structure against the official AWS documentation before attempting to push custom findings.

Q: Does Security Hub store my logs? A: No. Security Hub stores "findings," which are summaries of security events. It is not intended to be a long-term log storage solution. If you need to retain the raw logs for compliance or forensic purposes, you should continue to send those to Amazon S3 or a dedicated SIEM.

Q: How often are the compliance checks updated? A: AWS updates the compliance standards and checks in Security Hub regularly to reflect changes in industry benchmarks (like CIS) and new AWS service capabilities. You should check the Security Hub console periodically to see if new controls are available for you to enable.

Q: Is there a cost associated with Security Hub? A: Security Hub pricing is based on the number of security checks performed and the volume of findings ingested. Always review the pricing page for your region, as costs can grow quickly if you have an extremely high volume of findings or if you enable a large number of compliance standards.