How to Respond to an Incident on a Linux VM: IR Guide

Warning signs

›Unknown processes or processes with disguised names. Processes with names similar to legitimate binaries (e.g., 'systemd-' with a space, 'kworker' in an unusual directory), abnormal CPU/memory consumption detected via 'top' or 'ps auxf', processes running from temporary directories such as /tmp, /dev/shm or /var/tmp — a classic pattern of malware and cryptocurrency miners on Linux.
›Unauthorized network connections or C2 traffic. Outbound connections to unknown IPs on unusual ports (4444, 8080, 1337), especially in the ESTABLISHED state with no process clearly mapped via 'ss -tulnp' or 'lsof -i'. Anomalous DNS traffic with long subdomains may indicate DNS tunneling for data exfiltration.
›SSH keys added or users created. New entries in ~/.ssh/authorized_keys of privileged users or service accounts, users created with UID 0 or with a valid shell that should not exist in /etc/passwd, changes to /etc/sudoers for silent privilege escalation — these indicate persistence following credential compromise.
›Suspicious cron jobs and systemd services. Entries in /etc/cron.d/, /var/spool/cron/crontabs/ or .service files in /etc/systemd/system/ that run scripts from /tmp or /dev/shm, download remote payloads via curl/wget, or periodically re-activate — a persistence mechanism frequently used after initial exploitation of a vulnerability.
›Modifications to system binaries or configuration files. Binaries in /bin, /usr/bin, /sbin with recent modification dates inconsistent with the system's update history, detected via 'find /bin /usr/bin -newer /var/log/dpkg.log' or via 'rpm -Va'/'debsums -c'. Changes to /etc/ld.so.preload to inject malicious libraries are a classic sign of a userspace rootkit.
›Mass authentication failures followed by success. Hundreds of SSH attempts with different users and passwords (brute force) in /var/log/auth.log, followed by a successful login from an external IP — a pattern of credential stuffing or successful brute force. Also: logins from service accounts (www-data, postgres, nobody) via SSH that should never have an interactive shell.

Step by step

1
1. Do not shut down — preserve volatile memory
Before any other action, capture RAM with LiME (Linux Memory Extractor) or via /proc/kcore. Memory contains hidden processes, encryption keys in use, open network sessions and resident malware that disappears on shutdown. Use 'insmod lime.ko path=/mnt/evidencias/mem.lime format=lime' and document the SHA-256 hash immediately to ensure forensic integrity and chain of custody.
2
2. Take a snapshot/image of the disk before touching any file
Create a VM snapshot at the hypervisor level (AWS: aws ec2 create-snapshot; GCP: gcloud compute disks snapshot) or use 'dd if=/dev/sda | gzip | ssh backup@host dd of=disco.img.gz' for a bit-for-bit image. Compute and store the MD5 and SHA-256 hashes of each image. This immutable image is the foundation of the chain of custody — without it, any later analysis can be challenged.
3
3. Isolate the VM without destroying it
Change the Security Group (AWS), firewall rules (GCP/Azure) or iptables rules to block all inbound and outbound traffic, except your investigation SSH session originating from a trusted management IP. Do not delete the instance, do not restore the backup yet. The goal is to cut off communication with the attacker (C2, exfiltration) while keeping the forensic state intact for analysis.
4
4. Triage triad: processes, connections and persistence
Run in quick succession, redirecting output to timestamped files: (a) Processes — 'ps auxf', 'top -bn1', 'lsof -nP'; (b) Network connections — 'ss -tulnp', 'netstat -antp', 'lsof -i'; (c) Persistence — 'crontab -l', 'cat /etc/cron.*/*', 'systemctl list-units --type=service', 'grep -r CRON /var/spool/cron/', 'cat ~/.bashrc ~/.profile', 'cat ~/.ssh/authorized_keys'. Map everything before altering any file.
5
5. Audit users, logins and authentication logs
Check 'cat /etc/passwd | awk -F: \"$3==0\"' for improper UID 0 accounts; 'last -F' and 'lastb -F' for successful and failed logins with IPs; 'who' and 'w' for active sessions; 'cat /var/log/auth.log' or 'journalctl -u ssh --since "48 hours ago"' for authentication attempts. Identify users created by the attacker, SSH keys inserted and privilege escalations via sudo/su.
6
6. Verify binary integrity and hunt for rootkits
On RPM systems: 'rpm -Va 2>/dev/null | grep -E "^..5|^.M"' detects altered binaries. On Debian/Ubuntu: 'debsums -c'. Run 'chkrootkit' and 'rkhunter --check --skip-keypress' from external media (a USB drive or a clean mounted AMI) to avoid false negatives caused by already-compromised system binaries. Also check 'find / -perm -4000 -type f 2>/dev/null' for improper SUID files.
7
7. Contain, eradicate via reprovisioning and recover from a clean image
After collecting evidence, do not trust any manual cleanup — the attacker may have implanted persistence at levels you will not see. The practice recommended by NIST SP 800-61 is to reprovision the VM from a verified baseline AMI/image, restore only data (not binaries) from a pre-compromise backup, and only then reintegrate it into the environment with heightened monitoring (IDS/EDR, centralized logs).
8
8. Document, notify and record lessons learned
Prepare the incident report with: timeline, TTPs mapped to MITRE ATT&CK (tactics, techniques, procedures), IoCs (hashes, IPs, domains), root cause, estimated impact and actions taken. Notify internal parties (CISO, legal, communications) and mandatory external ones (ANPD if personal data was exposed, per LGPD Art. 48). Schedule a lessons-learned meeting within 5 business days after closure.

What NOT to do

✕Do not shut down or reboot the VM immediately. Shutting down the machine destroys RAM and, with it, running processes, open connections, encryption keys and malware artifacts that exist only in memory. Rebooting can also trigger the attacker's cleanup scripts. Preserve the volatile state before any shutdown.
✕Do not run analysis tools directly on the compromised system's binaries. Commands like 'ps', 'ls', 'netstat' and 'find' may have been replaced with trojanized versions that hide the attacker's processes, files and connections. Always use trusted binaries from external media (a live CD, a clean AMI mounted read-only) or transfer the artifacts to an isolated forensic workstation.
✕Do not delete logs or suspicious files before copying them. Logs in /var/log/, files in /tmp/ and any artifact identified as malicious are evidence. Deleting or moving them before collecting and computing hashes breaks the chain of custody and can invalidate legal actions, regulatory notifications and in-depth forensic analysis.
✕Do not try to clean the system manually and return it to production. Manual cleanup of a compromised system is rarely complete. The attacker may have installed kernel-level rootkits, backdoors in shared libraries, or modified the bootloader. The only safe way to return to production is to reprovision from a verified, known-clean base image.
✕Do not communicate about the incident over channels that may be compromised. If the affected VM hosts email, Slack or other communication services, assume those channels are compromised. Use a completely separate, out-of-band means of communication to coordinate the response — phone, Signal or an isolated incident workspace — to prevent the attacker from monitoring your investigation.

Fundamentals: why the order of actions matters in Linux incidents

NIST SP 800-61 (Computer Security Incident Handling Guide) establishes four phases for incident response: Preparation, Detection and Analysis, Containment/Eradication/Recovery, and Post-Incident Activity. On Linux VMs, the sequence of actions in the containment phase is critical because each action modifies the state of the system — and the wrong order can destroy evidence before it is collected.

Evidence volatility follows a hierarchy: CPU registers and process state (most volatile), RAM, network connections and routing tables, running processes, temporary file systems (/tmp, /dev/shm) and, last, files on disk (least volatile). This order — known as the 'Order of Volatility' — must guide the sequence of forensic collection before any containment action.

Common mistakes that compromise investigations include: an immediate reboot 'to clean the system', running 'rm -rf /tmp/*' before collecting evidence, restoring the backup before analysis, and communicating about the incident over potentially compromised channels. Teams without specific IR training tend to make these mistakes under pressure, which is a solid technical argument for engaging specialists from the outset.

Forensic triage: a complete map of artifacts on a Linux system

Systematic triage on Linux covers five analysis surfaces. (1) Processes and memory: 'ps auxf --forest' for the process tree, 'lsof -nP +L1' for deleted files still open (a common malware technique), '/proc/[PID]/maps' and '/proc/[PID]/exe' to locate the binaries of suspicious processes, 'strings /proc/[PID]/mem' to extract artifacts from a specific process's memory.

(2) Network: 'ss -tulnp' lists all sockets with their associated PID, 'iptables -L -n -v' and 'ip rule show' reveal firewall rules the attacker may have inserted, '/proc/net/tcp' and '/proc/net/udp' show connections even if 'netstat' is compromised. (3) Persistence: in addition to cron and systemd already mentioned, check '/etc/rc.local', '/etc/init.d/', 'at -l' (scheduled jobs), 'find / -name ".bashrc" -o -name ".profile" 2>/dev/null | xargs grep -l "curl\|wget\|nc "'.

(4) Authentication and access control: beyond /etc/passwd, analyze '/etc/shadow' for recently changed hashes, '/etc/group' for unexpected members of privileged groups (sudo, wheel, adm), and the PAM logs in /var/log/auth.log. (5) File system integrity: 'find / -mtime -1 -type f 2>/dev/null' lists files modified in the last 24h, 'stat' on critical binaries reveals modification timestamps inconsistent with the patch history.

Assess your company for free

See in minutes what is already exposed from your business.

Decripte's free Threat Management plan maps vulnerabilities, monitors threats and shows leaked credentials — no credit card and no technical team.

Start free now

Chain of custody and forensic documentation: minimum requirements

Chain of custody is the uninterrupted, verifiable record of who collected, stored and accessed each piece of evidence, and when. Without it, evidence is inadmissible in court and can be challenged in regulatory audits (LGPD, PCI DSS, ISO 27001). For each piece of evidence collected, record: date and time (UTC), the person responsible for collection, the method and tools used, the SHA-256 hash before and after any transfer, and the storage location.

For disk and memory images, use 'sha256sum' immediately after capture and store the hash alongside the evidence on separate media (e.g., a digitally signed email sent to yourself or to a trusted third party). Tools like Guymager or dc3dd generate hashes automatically during acquisition. For logs, use 'tar czf --numeric-owner logs_$(date +%Y%m%d_%H%M%S).tar.gz /var/log/' and compute the hash of the compressed file.

The incident report should follow a standardized structure: executive summary (for management), a detailed technical timeline (for the IR team and legal), IoCs (for blocking and threat intelligence), root cause analysis (for remediation), estimated impact (for insurers and regulators) and prioritized remediation recommendations. Mapping TTPs to the MITRE ATT&CK framework — such as T1053 (Scheduled Task/Job), T1136 (Create Account) or T1059 (Command and Scripting Interpreter) — facilitates communication between technical teams and enables correlation with known threats.

Eradication and recovery: why reprovisioning is the only safe option

The temptation to 'clean' a compromised server is understandable — it seems faster than reprovisioning. In practice, it is the decision that most often leads to re-compromise within days or weeks. Kernel-level rootkits (LKM rootkits) modify system calls directly in the kernel, becoming invisible to any tool running on the same operating system. Bootkits modify the bootloader. Implants in shared libraries (/usr/lib/) affect every process that loads them.

The correct eradication process begins with identifying the exploited vulnerability (CVE, misconfiguration, leaked credential) and fixing it in the base image before redeploying. Without fixing the root cause, the same attacker — or others who bought the access — will re-compromise the system within hours. Use infrastructure as code (Terraform, Ansible, CloudFormation) to ensure the new instance is provisioned in a reproducible and auditable way.

During recovery, restore only business data (databases, user uploads) from verified backups predating the incident — never restore binaries, configurations or scripts from compromised sources. Reintegrate the server into the production environment under heightened monitoring: active EDR, logging of all system calls (auditd), file integrity alerts (AIDE, Wazuh) and manual review of the first logs for 48 to 72 hours. Document the date and time of return to production as part of the incident timeline.

When to engage a specialized incident response team

Some signs indicate that the incident exceeds internal response capacity and requires external specialists immediately: (1) there is evidence of sensitive data exfiltration (personal data, intellectual property, trade secrets), which may trigger legal obligations to notify the ANPD within 72 hours; (2) the compromise has reached multiple systems or appears to be a targeted attack (APT) with sophisticated persistence; (3) the root cause has not been identified after initial triage; (4) there is critical operational impact (payment systems, healthcare infrastructure, industrial control).

Other criteria for immediate engagement: (5) the internal team lacks adequate forensic tools or specific Linux IR training; (6) there is a risk of reputational or regulatory impact requiring forensic documentation for legal defense; (7) the attacker is still active in the environment — a specialized team can perform threat hunting and coordinated eviction without alerting the intruder prematurely.

Engaging specialists does not mean losing control of the incident — it means having at your side a team with tools, playbooks and experience across hundreds of similar incidents. The difference between an unstructured internal response and a professional response can be measured in hours of downtime avoided, data not exfiltrated and regulatory fines not applied. Decripte operates with 24x7 on-call coverage and an SLA to start service within 4 hours for clients with a contracted incident response plan.

Decripte performs incident response and digital forensics for companies of all sizes — from the sole proprietor to groups with more than 100,000 employees — with a certified team, documented chain of custody and reports accepted by Brazilian regulatory and judicial bodies. If your Linux VM has been compromised or you suspect a compromise, go to /plano/resposta-incidentes to activate service or request a free assessment to evaluate your environment's level of exposure.

Lessons learned and post-incident hardening: turning crisis into maturity

The lessons-learned phase — mandatory in NIST SP 800-61 — is frequently skipped by teams under pressure to return to normal. This is a strategic mistake: incidents in organizations without this process tend to recur with variations, because the structural root cause is not addressed. The post-incident meeting should take place within 5 business days after closure, with the participation of technical staff, managers and business representatives.

The central questions of the retrospective: how did the attacker get in (initial vector)? How long did they go undetected (dwell time)? Which existing controls should have detected the attack and did not? What worked well in the response? Which procedures failed? The answers feed a hardening plan prioritized by real risk — not theoretical compliance.

Typical post-incident controls for Linux environments: implementing MFA for SSH access (keys + TOTP via PAM), disabling password authentication over SSH, using a bastion host or VPN for administrative access, deploying AIDE or Wazuh for real-time file integrity, centralizing logs in an immutable SIEM (local logs are the first targets of attackers), network segmentation by function (databases not directly reachable from the internet), and a patch management program with an SLA defined by CVE criticality.

Key terms

Chain of Custody: A documented, chronological and verifiable record of who collected, transferred, accessed and stored each piece of digital evidence in a forensic investigation. It ensures the integrity and authenticity of evidence for use in court, regulatory audits and contractual disputes. It is established by computing cryptographic hashes (SHA-256) of the evidence at the moment of collection and maintaining a record of the entire subsequent access chain.
Dwell Time: The time elapsed between the initial compromise of a system and its detection by the security team or the incident response team. The greater the dwell time, the greater the potential for damage: the attacker has more time for lateral movement, data exfiltration, privilege escalation and establishing multiple persistence points. Reducing dwell time is one of the main indicators of the maturity of a security program.
Order of Volatility: A hierarchy of digital evidence ordered from most to least volatile, which defines the sequence of forensic collection. In decreasing order of volatility: CPU registers and process state, RAM, network connections and routing tables, running processes, temporary file systems, hard disk and, last, logs and archived backups. Collecting in the correct order preserves evidence that would be lost over time or through subsequent actions.
MITRE ATT&CK: A public framework maintained by the MITRE organization that categorizes and describes the tactics, techniques and procedures (TTPs) used by advanced threat groups in real attacks. In Linux incident response, it is used to map the artifacts found (e.g., a malicious cron job -> T1053.003 Cron, an inserted SSH key -> T1098.004 SSH Authorized Keys) to known techniques, facilitating communication between teams and the identification of attack patterns specific to APT groups.

Frequently asked questions

Can I shut down the compromised Linux VM to 'stop the attack' immediately?

No. Shutting down the VM destroys RAM and, with it, critical evidence: running malicious processes, active connections to C2 servers, encryption keys and malware artifacts that exist only in memory. The correct approach is to isolate the VM on the network (by changing Security Groups or firewall rules at the cloud provider level) without shutting it down, and only then begin collecting evidence. Shutdown can occur after memory capture and disk snapshot.

How can I tell if the Linux system binaries have been replaced with malicious versions?

On RPM systems (RHEL, CentOS, Amazon Linux): run 'rpm -Va 2>/dev/null' and filter for lines beginning with '..5' (MD5 hash changed) or '.M' (permissions changed). On Debian/Ubuntu systems: use 'debsums -c' to verify the hashes of all installed package files. For both: run chkrootkit and rkhunter from external media or a clean AMI mounted read-only, because tools running on a compromised system may be trojanized and return false negatives.

What is chain of custody in incident response and why does it matter?

Chain of custody is the documented, verifiable record of who collected, transferred, accessed and stored each piece of digital evidence, when, and by what method. It is mandatory for evidence to be accepted in court (cybercrime, contractual disputes) and in regulatory audits (LGPD, PCI DSS). Without it, the attacker's defense may claim the evidence was tampered with. Chain of custody is established by computing SHA-256 hashes of each piece of evidence at the moment of collection and documenting the entire subsequent access chain.

How long does an attacker typically stay on a Linux VM undetected (dwell time)?

According to industry reports, average dwell time — the time between initial compromise and detection — ranges from 16 to 24 days in environments without EDR or continuous monitoring. In environments with a SIEM and configured alerts, that number drops to hours or days. Sophisticated attackers (APTs) can remain undetected for months, performing slow reconnaissance and gradual exfiltration below alert thresholds. Implementing centralized, immutable logging and behavioral analysis is the main way to reduce dwell time.

Should I pay the ransom if my Linux VM was infected by ransomware?

There is no universal answer, but IR specialists recommend not paying as a general rule for the following reasons: payment does not guarantee data recovery (on average, 30% of organizations that pay do not recover everything), it funds criminal groups, and it does not resolve the original vulnerability — the same environment can be re-compromised within days. Before any decision, engage an IR specialist to assess whether there are valid pre-compromise backups, whether the files can be recovered by forensic means, and the real extent of the compromise.

Am I required to notify the ANPD if personal data was exposed on a compromised Linux VM?

Yes. Under Article 48 of the LGPD, controllers of personal data must notify the ANPD and the affected data subjects about security incidents that may cause relevant risk or harm to the subjects, within a period the ANPD has set at up to 72 hours after becoming aware of the incident. The notification must include: the nature of the affected data, the number of data subjects involved (actual or estimated), the technical and security measures adopted, the risks related to the incident and the measures that have been or will be taken. It is recommended to involve the DPO and legal immediately.

What is the difference between chkrootkit and rkhunter for rootkit detection on Linux?

Both are open source rootkit detection tools for Linux, but with complementary approaches. chkrootkit (Chuck Rootkit) checks for known signatures of rootkits, trojans and backdoors in system binaries and in command output. rkhunter (Rootkit Hunter) performs more comprehensive checks: MD5 hashes of critical binaries, file attributes, network configurations, strings in commands, and compares them against updatable signature databases. The recommendation is to run both — from external media — because each detects different subsets of threats. No tool is 100% effective against advanced kernel-level rootkits.

Security for businesses

Decripte protects companies of every size — from solo founders to Enterprise.

A full platform and services: threat management, 24x7 SOC, incident response, pentest and compliance. Start free and see what has already leaked from your business.

Start free now See plans

How to respond to a security incident on a Linux VM: a step-by-step technical guide