Chapter 6: Host-Based Security and Endpoint Protection

1.1 HIDS vs. HIPS: Detection vs. Prevention

Every network ultimately connects to endpoints — workstations, servers, laptops, and containers. The endpoint is where users work, credentials are entered, code executes, and where most attackers ultimately want to be. Host-based security places detection, prevention, and visibility capabilities directly on the machine rather than relying solely on network chokepoints.

Host-Based Intrusion Detection System (HIDS) monitors system activity — file system changes, log entries, running processes, network connections — and generates alerts when anomalies are detected. It is a passive observer: it records and reports but does not block. HIDS is well suited to audit environments where stopping false positives from disrupting production systems is a higher priority than automatic prevention.

Host-Based Intrusion Prevention System (HIPS) adds an enforcement layer on top of detection. When the system identifies a match against a known attack pattern or behavioral threshold, it can terminate the process, quarantine the file, or block the network connection in real time. HIPS is active — powerful but requires careful tuning.

Think of HIDS as a smoke detector — it sounds the alarm when something burns. HIPS is the automatic sprinkler system — it sounds the alarm and douses the fire simultaneously.

Modern Endpoint Detection and Response (EDR) platforms subsume both HIDS and HIPS capabilities into a unified agent while adding continuous telemetry recording and cloud-backed AI analysis.

1.2 Antimalware Detection Methods

Signature-Based Detection compares file hashes, byte sequences, or known malicious strings against a database of known threats. Fast and precise for known malware, but blind to anything not yet catalogued.

Heuristic Detection looks for behaviors or structural characteristics that resemble malicious code — an executable that unpacks itself at runtime, injects into a running process, and reaches out to a random-looking domain, even if that specific file has never been seen before.

Predictive AI/ML Detection uses machine learning models trained on millions of samples to score new files and activities probabilistically. Rather than a binary match, the model outputs a confidence score. With 79% of 2024 detections being malware-free, behavioral and predictive methods are no longer optional — they are the primary defense layer.

1.3 Host-Based Firewalls and EDR

A host-based firewall enforces traffic policy at the OS level, independent of any network firewall. This matters for three reasons: (1) east-west traffic between systems on the same segment never passes through a perimeter firewall; (2) laptops leave the corporate network; (3) defense in depth provides a second enforcement point if the perimeter is bypassed.

EDR platforms like CrowdStrike Falcon and SentinelOne collect continuous behavioral telemetry: process creation (including full command-line arguments), parent-child process relationships, memory injection indicators, network connections, and file system modifications. The key insight is that correlation of multiple signals — not any single indicator — drives high-confidence detections.

A classic EDR catch: outlook.exe spawning powershell.exe with -EncodedCommand -WindowStyle Hidden that immediately initiates an outbound HTTPS connection. No single step is conclusive; the sequence is damning.

2.1 The Authentication Stack: SAM, LSA, and LSASS

Understanding Windows authentication internals is essential for both securing systems and investigating compromises, because credential theft almost always targets these components.

• Security Accounts Manager (SAM) — the database at C:\Windows\System32\config\SAM that stores local user NTLM password hashes. Locked by the OS while Windows is running. NTLM hashes enable pass-the-hash attacks — the attacker doesn't need the plaintext password.
• Local Security Authority (LSA) — the security subsystem handling authentication policy, token generation, and enforcement. It checks credentials against SAM for local accounts or forwards to a domain controller for Kerberos validation.
• LSASS (lsass.exe) — the process hosting the LSA. It maintains authenticated user sessions in memory — which is why it is the primary target of credential-dumping tools like Mimikatz (sekurlsa::logonpasswords).

Think of the SAM as a vault of keys, the LSA as the security guard checking your badge, and LSASS as the guard's active memory of everyone currently inside the building. Credential dumping is like pickpocketing the guard's memory rather than breaking into the vault.

Defending LSASS: Enable Credential Guard (virtualizes LSASS using VBS/hypervisor isolation); configure RunAsPPL (Protected Process Light) so only digitally signed processes can access LSASS memory; monitor Sysmon Event 10 (process accessed LSASS).

2.2 Windows Event Log and Critical Event IDs

The Windows Security event log (C:\Windows\System32\winevt\Logs\Security.evtx) is the primary source for authentication, authorization, and policy change monitoring. SOC analysts route this log to a SIEM and build correlation rules around high-value event ID sequences.

The classic SIEM correlation chain for brute-force to privilege escalation: multiple 4625 (failed logon) events from the same source IP → 4624 (successful logon) → 4672 (special privileges assigned). Event 1102 (audit log cleared) is a standalone critical alert — there is rarely a legitimate reason for this.

2.3 Windows Defender and the Registry

Windows Defender Antivirus is the built-in antimalware engine in modern Windows. In enterprise environments it is managed through Microsoft Defender for Endpoint, which provides cloud-delivered protection, automatic sample submission, and Attack Surface Reduction (ASR) rules — blocking high-risk behaviors like Office applications spawning child processes, independent of signatures.

Key Registry Persistence Locations:

3.1 Users, Groups, and SELinux

Linux access control begins with the Unix model: every file, process, and resource has an owner, a group, and permissions (read/write/execute). Root (UID 0) bypasses most checks, making privilege escalation to root the primary goal of a Linux attacker.

Key files for investigation:

• /etc/passwd — user account information (no longer stores hashed passwords in modern Linux)
• /etc/shadow — hashed passwords, readable only by root
• /etc/group — group definitions and membership
• /etc/sudoers — defines which users can run commands as root via sudo; a persistence target

SELinux (Security-Enhanced Linux) is a Mandatory Access Control (MAC) framework developed by the NSA and integrated into the Linux kernel. While traditional permissions are discretionary (owner decides access), SELinux enforces system-wide policy that even root cannot override without explicit policy permission. Every process and file receives a security context label (e.g., system_u:system_r:httpd_t:s0).

Think of traditional Linux permissions as an office where each employee decides who enters their own office. SELinux is a central building management system controlling every door — even the CEO cannot override those access rules without a policy change approved by the security team.

3.2 Linux Logging: syslog, journald, and auditd

auditd is the most powerful: it can audit any system call at the kernel level, track privilege escalation (setuid, capabilities), and generate AUID (Audit User ID) that persists across su/sudo — even if a user switches to root, the original login UID is preserved in audit records.

auditctl -a always,exit -F arch=b64 -S execve           # Log all executions
auditctl -w /etc/passwd -p rwxa                          # Monitor /etc/passwd access

A deleted-file process technique: malware may delete its binary after execution. On Linux, the process continues running but ls -la /proc/<PID>/exe shows (deleted). The binary can be recovered: cat /proc/<PID>/exe > /tmp/recovered_binary.

3.3 iptables and nftables

iptables rules are organized in tables (filter, nat, mangle) and chains (INPUT, OUTPUT, FORWARD). The filter table handles host-based firewall policy.

# Block all inbound except established sessions and SSH
iptables -P INPUT DROP
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

nftables is the modern replacement, providing a unified framework replacing iptables/ip6tables/arptables/ebtables with a single coherent syntax and better performance. The LOG target in both allows logging matched packets to syslog — essential for detecting lateral movement on systems without a dedicated EDR agent.

4.1 Asset Identification and Inventory

Effective investigation begins with knowing what you have. You cannot determine whether an endpoint's behavior is anomalous without a baseline. A complete asset record includes: hardware identity (MAC, serial, asset tag), OS version and patch level, installed applications, network identifiers (IP, hostname, FQDN), assigned users and their role, normal traffic patterns, and security tools installed (EDR agent, AV version).

Asset inventory feeds detection context: a development workstation running a debugger that injects into processes is normal. A finance workstation doing the same thing is a critical alert.

4.2 Threat Actor Profiling

Attribution associates an attack with a specific threat actor — nation-state, criminal organization, hacktivist, or insider. True attribution is difficult, but profiling observed TTPs against MITRE ATT&CK narrows the field.

4.3 IoC vs. IoA

Indicators of Compromise (IoCs) are forensic artifacts that signal a past breach — file hashes, known C2 IP addresses, registry persistence keys, malicious user agent strings, mutex names. They are retrospective: "what happened?"

Indicators of Attack (IoAs) are behavioral signals indicating an attack is in progress, regardless of specific tools used — privilege escalation outside business hours, lateral movement via rapid multi-system authentication, Office applications spawning command interpreters. They are real-time: "what is happening now?"

Ransomware scenario: IoC-based response identifies the LockBit binary hash, known C2 IPs, and the ransom note filename — useful for remediation. IoA-based detection would have caught the pre-encryption step 48 hours earlier: vssadmin delete shadows /all /quiet executed by a service account from a non-standard workstation. The IoA should have triggered containment before encryption began.

4.4 Chain of Custody

Chain of custody is the documented, unbroken record of who collected evidence, how it was handled, stored, and transferred — from initial collection through legal or disciplinary proceedings. Without it, technically correct findings may be legally worthless.

Requirements:

1. Documentation at collection — who, when, from where, and what method (disk image, memory dump, log export)
2. Hash verification — compute SHA-256 of all digital evidence at collection; any copy must produce the same hash
3. Secure storage — physical evidence in tamper-evident bags; digital evidence in write-protected forensic containers (E01, AFF4)
4. Access logging — every person accessing evidence is logged with date, time, and purpose
5. Transfer receipts — both parties sign when evidence moves between custodians

Think of chain of custody as the property room in a police station — every item is logged in and out, who touched it is recorded, and the log proves the evidence has not been tampered with between the crime scene and the courtroom.