Chapter 5: Endpoint Attacks, Evasion Techniques, and Cryptography

Endpoints are rich attack targets — they hold credentials, process sensitive data, and typically trust the user sitting in front of them. Attackers exploit this trust through memory corruption, command infrastructure abuse, and malicious software designed to persist, exfiltrate, or extort.

1.1 Buffer Overflow Attacks

A buffer overflow occurs when a program writes more data to a fixed-size memory region than it can hold. The overflow spills into adjacent memory, allowing an attacker to overwrite the return address and redirect program execution.

The classic exploitation sequence:

1. Attacker identifies a vulnerable input (e.g., a call to gets() or strcpy() with no bounds check)
2. Crafted input fills the buffer, overflows into the saved RBP and return address (RIP/EIP)
3. Return address is overwritten with attacker shellcode, a ROP gadget, or a libc function
4. When the vulnerable function returns, execution jumps to attacker-controlled code

Modern Mitigations and Bypass Techniques:

1.2 Command-and-Control (C2) Frameworks

Once an attacker has initial access, they need persistent, bidirectional communication between compromised hosts (implants/beacons) and an operator's server. C2 frameworks provide this infrastructure.

C2 channels include HTTP/HTTPS beaconing (port 80/443 blending with web traffic), DNS-based C2 (commands encoded in DNS queries), and social media/cloud storage (APTs embedding instructions in tweets or GitHub commits).

1.3 Malware Taxonomy

Fileless malware injects into legitimate process memory (svchost.exe), uses PowerShell/WMI as execution engines, and stores configuration in the Windows registry. Detection requires memory forensics and behavioral EDR with AMSI integration.

1.4 Ransomware

Ransomware encrypts victim files and demands payment for the decryption key. Modern "Big Game Hunting" combines data exfiltration with encryption (double extortion). Ransomware-as-a-Service (RaaS) allows affiliates to perform intrusions for a 70–80% revenue share.

Maintaining access without triggering detection requires evading firewalls, IDS/IPS, EDR, DLP, and SIEM. Modern evasion targets the visibility gaps in these controls rather than defeating them head-on.

2.1 Protocol Tunneling

Protocol tunneling encapsulates one protocol's traffic inside another — specifically one that security controls allow through without inspection.

DNS Tunneling: DNS is almost universally permitted outbound. Attackers encode arbitrary data inside DNS query names and response records, creating a bidirectional covert channel over port 53.

DNS Tunneling Detection Signals:

2.2 Encrypted C2 Channels

Attackers abuse encryption to make malicious traffic indistinguishable from legitimate communications. HTTPS C2 over port 443 uses valid TLS certificates (free from Let's Encrypt); DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) encrypt DNS queries inside HTTPS, bypassing traditional DNS monitoring.

Detection approaches for encrypted C2:

• JA3/JA3S fingerprinting: Hash TLS ClientHello parameters — malware libraries produce distinctive JA3 hashes
• Behavioral analytics: Beacon periodicity (connections every N seconds with low variance) is detectable even when encrypted
• Threat intelligence: IOC feeds for known C2 domains/IPs
• TLS inspection: Decrypt traffic at a proxy — legally complex, requires trust store management

2.3 Domain Generation Algorithms (DGAs) and DNS Fast Flux

DGAs are algorithms embedded in malware that generate hundreds of pseudo-random domain names using a seed (often the current date). Blocklisting one domain is useless — hundreds of alternatives are generated daily. DNS Fast Flux rapidly rotates A records (low TTL, ~60 seconds) using a botnet as proxy tier, defeating IP-based blocking.

2.4 Proxy Chains

Proxy chains route malicious traffic through a series of intermediate hosts, obfuscating the true origin. Each proxy knows only the previous and next hop. Common techniques include TOR (multi-hop onion routing), bulletproof hosting, legitimate cloud infrastructure (AWS/Azure/GCP relay nodes), and residential proxy networks.

2.5 Polymorphic and Metamorphic Malware

Cryptography underpins every security control in this course. The four foundational primitives — symmetric encryption, asymmetric encryption, hashing, and key exchange — are composed together in real-world protocols like TLS.

3.1 Symmetric Encryption

Symmetric encryption uses a single shared secret key for both encryption and decryption. It is fast and suitable for bulk data, but has a key distribution problem: N parties require N×(N−1)/2 unique keys.

AES Block Modes: CBC (sequential, legacy TLS) → GCM (authenticated encryption AEAD, TLS 1.3/IPsec) → CTR (parallelizable, disk encryption). GCM provides both encryption and integrity in one pass.

3.2 Asymmetric Encryption

Asymmetric encryption uses a mathematically linked key pair. Data encrypted with the public key can only be decrypted by the private key. It solves the key distribution problem but is orders of magnitude slower than symmetric encryption — used only for handshake/authentication, not bulk data.

3.3 Cryptographic Hashing

A cryptographic hash maps arbitrary input to a fixed-length digest with three key properties: pre-image resistance (can't reverse H(x) to find x), collision resistance (can't find x≠y where H(x)=H(y)), and avalanche effect (one-bit change → completely different digest).

3.4 Diffie-Hellman Key Exchange

DH solves the problem of establishing a shared secret over an untrusted channel without transmitting the secret. Both parties compute the same value using public and private components; an eavesdropper sees only the public values and cannot compute the shared secret without solving the discrete logarithm problem.

Ephemeral DH (DHE/ECDHE) generates a new key pair for each TLS session and discards it after use — providing Perfect Forward Secrecy (PFS): past sessions remain secure even if the server's long-term private key is later compromised.

A Public Key Infrastructure (PKI) is the system of policies, processes, hardware, software, and people that manages digital certificates and public-key encryption — turning raw asymmetric cryptography into a scalable, trusted identity system.

4.1 PKI Architecture

The core problem PKI solves: how does Alice know a public key actually belongs to Bob and not an attacker performing MitM? A trusted third party — a Certificate Authority (CA) — signs a certificate binding a public key to an identity.

4.2 X.509 Certificate Structure

X.509v3 is the current ITU-T standard for public-key certificates, extended by RFC 5280 for Internet use.

Critical X.509v3 Extensions:

4.3 Cipher Suites

A cipher suite is a named combination of four cryptographic algorithms negotiated during the TLS handshake.

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 |    |     |       |    |   |    |
 |    |     |       |    |   |    +-- HMAC/PRF hash
 |    |     |       |    |   +------- Mode (AEAD)
 |    |     |       |    +----------- Key size (256 bits)
 |    |     |       +---------------- Bulk cipher (AES)
 |    |     +------------------------ Authentication (RSA cert)
 |    +------------------------------ Key exchange (ECDHE)
 +----------------------------------- Protocol

TLS 1.3 removes key exchange and authentication from the suite name (negotiated separately). Only AEAD cipher + hash remain: TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256.

Weak/Prohibited Cipher Suites to Block:

• TLS_RSA_* — No PFS; key exchange via RSA encipherment
• *_RC4_* — RC4 is cryptographically broken
• *_3DES_* — Sweet32 birthday attack; deprecated
• *_NULL_* — No encryption; plaintext
• *_EXPORT_* — 40/56-bit keys; intentionally weak (Cold War era)

4.4 PKCS Standards

Certificate Revocation: