A recently disclosed vulnerability, now known as **HollowByte**, poses a significant threat to **OpenSSL** servers. This flaw enables unauthenticated attackers to initiate a denial-of-service (DoS) attack using an incredibly small, 11-byte payload. The **OpenSSL** team has quietly addressed this issue, backporting the fix to older releases. While no specific identifier was assigned, the implications for secure internet communication, where **OpenSSL** serves as a critical backbone, necessitate immediate action from organizations. ### Understanding HollowByte **Okta**'s Red Team provided a detailed advisory earlier this week, explaining the mechanics and real-world impact of the **HollowByte** DoS vulnerability. Their research highlights a critical flaw in how vulnerable **OpenSSL** versions handle TLS handshakes. During a TLS handshake, each message includes a 4-byte header that declares the size of the incoming message. However, affected **OpenSSL** versions would pre-allocate the declared memory size *before* the actual payload was received and its size validated. As **Okta** explains, "The worker thread then blocks, waiting indefinitely for data that will never arrive." This means an attacker can simply open a TLS connection, send an 11-byte malicious input with a header falsely claiming a much larger message body, and the server will allocate substantial memory based on this claim. By repeatedly executing this process across multiple connections, an attacker can force the server to allocate vast amounts of memory using minimal transmitted data. ### Memory Bloat and Persistent Impact **Okta**'s researchers also noted that while **OpenSSL** typically frees buffers when a connection is dropped, the **GNU C Library (glibc)** manages memory differently. **Glibc** doesn't immediately return small-to-medium allocations to the operating system, instead holding them for potential reuse. This behavior is exploited by **HollowByte**. "By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks," **Okta** states. "The heap fragments heavily, causing the server’s Resident Set Size (RSS) to climb continuously. Even after the attacker disconnects, the server remains permanently bloated." The only effective way to reclaim this permanently bloated memory is by restarting the affected process. ### Widespread Impact and Necessary Fixes The open-source **OpenSSL** library is deeply embedded across a wide range of popular software projects. This includes widely used web servers like **NGINX** and **Apache**, language runtimes such as **Node.js**, **Python**, **Ruby**, and **PHP**, and databases like **MySQL** and **PostgreSQL**. It is also a standard component in most **Linux** distributions for TLS encryption and certificate handling. **Okta**'s tests on **NGINX** demonstrated that environments with low capacity could be easily depleted of memory by **HollowByte**. Higher-spec servers might lose up to 25% of their memory, all while the attack bandwidth remains below typical security alerting thresholds. While DoS vulnerabilities are often considered less severe than those leading to data theft or code execution, they can cause significant operational disruptions, service outages, and reputational damage. The **HollowByte** DoS issue has been resolved in **OpenSSL 4.0.1** and backported to versions **3.6.3**, **3.5.7**, **3.4.6**, and **3.0.21**. These patched versions now grow the buffer only when data actually arrives, effectively ignoring false header claims. Despite being classified as a "hardening fix" rather than a critical security vulnerability, **Okta** strongly advises organizations to "upgrade your distribution's **OpenSSL** packages immediately."