HollowByte: 11-Byte Payload Triggers Memory Bloat DoS on OpenSSL Servers

Security researchers have disclosed a denial-of-service vulnerability in OpenSSL, nicknamed HollowByte, that allows an unauthenticated attacker to exhaust server memory using a crafted payload of just 11 bytes. The flaw affects servers processing TLS connections and requires no prior authentication or established session to trigger.

What Is HollowByte?

HollowByte is a memory exhaustion vulnerability rooted in how OpenSSL handles a specific malformed TLS record early in the handshake process. When the server receives the crafted 11-byte packet, it enters a code path that performs an allocation proportional to an attacker-controlled length field in the packet header — without first validating whether the declared length is reasonable.

The result is an unbounded or disproportionately large memory allocation triggered by a single, tiny network packet. Under sustained sending — trivially achievable from a single machine or a distributed botnet — server memory fills rapidly, causing the process to be terminated by the operating system's out-of-memory killer or to become unresponsive.

Technical Breakdown

The vulnerability lies in the parsing of TLS record headers before authentication or session establishment. TLS records carry a length field indicating the size of the payload to follow. A well-implemented parser validates this length against reasonable bounds before allocating a buffer to receive the remainder of the record.

In the vulnerable code path, the allocation occurs before bounds validation — a classic TOCTOU-adjacent pattern where the check comes too late. Since TLS sessions begin with unauthenticated exchanges, any internet-facing server running a vulnerable OpenSSL version is exposed without needing a credential or prior TCP session beyond the initial SYN/ACK.

Why 11 bytes? The 11-byte payload is the minimum structure needed to reach the vulnerable parsing logic: 5 bytes for the TLS record header (content type, protocol version, length) plus enough bytes to satisfy earlier parsing guards. Everything beyond the length field value itself can be garbage — the allocation triggers regardless.

Affected Systems

Any server-side application linked against a vulnerable OpenSSL version and accepting inbound TLS connections is at risk. This includes:

• Web servers — Apache, Nginx, Lighttpd compiled with system OpenSSL
• Mail servers — Postfix, Dovecot, Exim using STARTTLS or SMTPS
• VPN endpoints — OpenVPN and other TLS-based VPN services
• Database servers — PostgreSQL, MySQL with SSL mode enabled
• Application servers — any backend accepting TLS directly

Cloud load balancers and TLS-terminating proxies that offload SSL from backend servers provide a natural layer of protection — backend services behind such infrastructure are not directly reachable by the raw 11-byte packet.

Exploitation in the Wild

At time of publication, there are no confirmed reports of HollowByte being exploited in active campaigns. However, the simplicity of the exploit — a single-packet trigger requiring no authentication — makes weaponization trivial. Proof-of-concept code is expected to circulate quickly following disclosure.

Defenders should treat this as an imminent threat and prioritize patching, particularly for internet-exposed TLS endpoints.

Mitigations and Remediations

Immediate actions:

1. Apply the OpenSSL patch — update to the fixed version released alongside the CVE disclosure. Check your distribution's security advisories for backported patches.
2. Restart OpenSSL-linked services after patching — in-memory instances of the library are not updated by a package upgrade alone.
3. Deploy rate limiting — configure upstream firewalls, load balancers, or WAFs to rate-limit new TLS connection attempts per source IP. A sustained HollowByte attack requires high packet volume; rate limiting raises the cost significantly.
4. Enable connection throttling at the kernel level — iptables/nftables rules limiting new TCP connections per second from a single IP provide defense-in-depth.
5. Monitor memory usage on TLS-serving processes — sudden spikes in sslh, nginx, or httpd memory consumption may indicate active exploitation.

Architectural mitigations:

• Place internet-facing TLS termination behind a reverse proxy or CDN that handles the TLS layer independently (e.g., Cloudflare, AWS ALB, HAProxy). Backend services behind these layers are not directly reachable.
• Segment high-value internal services so they are not internet-accessible, even if they use TLS.

Broader Context: OpenSSL's Attack Surface

OpenSSL is foundational to internet security — it underpins a significant fraction of all encrypted connections globally. Its attack surface is correspondingly vast and has produced several high-profile vulnerabilities over the years, including Heartbleed (CVE-2014-0160), DROWN (CVE-2016-0800), and various padding oracle attacks.

HollowByte joins this history as a reminder that even mature, heavily-audited codebases can contain subtle parsing flaws. The simplicity of the trigger — 11 bytes, no authentication — highlights how a small logic error in an early, unauthenticated code path can have disproportionate impact.

Operators running OpenSSL at scale should maintain a patching cadence that allows rapid response to disclosures like this. Automated vulnerability scanning and Software Composition Analysis (SCA) tools that flag OpenSSL version mismatches are valuable here.

References

• BleepingComputer: HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload
• OpenSSL Security Advisories
• MITRE ATT&CK: T1499 — Endpoint Denial of Service