Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade

Buffer Overflows: Attacks and Defenses for the Vulnerabilty of the Decade – Cowan et al. 2000

Some of you may recall reading “Smashing the Stack for Fun and Profit” (hard to believe that was published in 1996!), which helped to raise consciousness of buffer overflow attacks. In this paper from 2000 Cowan et al. provide a very readable breakdown of how they work and the potential defenses against them.

Buffer overflows have been the most common form of security vulnerability in the last ten years (1990-2000). More over, buffer overflow vulnerabilities dominate in the area of remote network penetration vulnerabilities, where an anonymous Internet user seeks to gain partial or total control of a host. Because these kinds of attacks enable anyone to take total control of a host, they represent one of the most serious classes security threats.

A buffer overflow attack enables the attacker to inject and execute attack code, which runs with the privileges of the vulnerable program (which is why you should always run processes with the least privileges possible).

Crafting a Buffer Overflow Attack

Typically an attacker is attacking a root program, and immediately executes code similar to exec(sh) – but not always. To achieve their goal, an attacker has to accomplish two objectives:

  1. Arrange for suitable code to be available in the program’s address space.
  2. Get the program to jump to that code, with suitable parameters loaded into registers and memory

Arranging for suitable attack code to be available

Again, there are two ways to get suitable code into a program’s address space: either inject it, or simply use whatever is already there.

When injecting code:

The attacker provides a string as input to the program, which the program stores in a buffer. The string contains bytes that are actually native CPU instructions for the platform being attacked. Here the attacker is (ab)using the victim program’s buffers to store the attack code.

Note that you don’t need to overflow any buffers to do this, sufficient payload can be injected into perfectly reasonable buffers. The buffer itself can be anywhere – on the stack, the heap, or in the static data area.

If you want a shell, there’s really no need to worry about injecting code though. Since nearly all C programs link with libc the code you need is already present.

Often, the code to do what the attacker wants is already present in the program’s address space. The attacker need only parameterize the code, and then cause the program to jump to it. For instance, if the attack code needs to execute “exec(“/bin/sh”)”, and there exists code in libc that executes “exec(arg)” where “arg” is a string pointer argument, then the attacker need only change a pointer to point to “/bin/sh” and jump to the appropriate instructions in the libc library

Jumping to the attack code

The basic method is to overflow a buffer that has weak or non-existent bounds checking on its input with a goal of corrupting the state of an adjacent part of the program’s state, e.g. adjacent pointers, etc. By overflowing the buffer, the attacker can overwrite the adjacent program state with a near-arbitrary sequence of bytes, resulting in an arbitrary bypass of C’s type system and the victim program’s logic.

Most attacks seen in the wild attack code pointers. The distinguishing factors among overflow attacks are the kind of state corrupted, and where in the memory layout the state is located. Activation Records, Function Pointers, and Longjmp buffers are all vulnerable.

Each time a function is called, it lays down an activation record on the stack that includes, among other things, the return address that the program should jump to when the function exits, i.e. point at the code injected. Attacks that corrupt activation record return addresses overflow automatic variables, i.e. buffers local to the function. By corrupting the return address in the activation record, the attacker causes the program to jump to attack code when the victim function returns and dereferences the return address. This form of buffer overflow is called a “stack smashing attack” and constitutes a majority of current buffer overflow attacks.

If an overflowable buffer is adjacent to a function pointer, then the pointer can be overwritten and the next time the program calls the function it will jump to the attack code. Likewise a setjmp/longjmp buffer adjacent to an overflowable buffer can also be overwritten, changing the destination the program jumps to.


The simplest and most common form of buffer overflow attack combines an injection technique with an activation record corruption in a single string. The attacker locates an overflowable automatic variable, feeds the program a large string that simultaneously overflows the buffer to change the activation record, and contains the injected attack code. This is the template for an attack outlined by Levy. Because the C idiom of allocating a small local buffer to get user or parameter input is so common, there are a lot of instances of code vulnerable to this form of attack. The injection and the corruption do not have to happen in one action. The attacker can inject code into one buffer without overflowing it, and overflow a different buffer to corrupt a code pointer. This is typically done if the overflowable buffer does have bounds checking on it, but gets it wrong, so the buffer is only overflow-able up to a certain number of bytes. The attacker does not have room to place code in the vulnerable buffer, so the code is simply inserted into a different buffer of sufficient size.

Defending Against Buffer Overflows

There are four basic mechanisms of defense against buffer overflow attacks: writing correct programs; enlisting the help of the operating system to make storage areas for buffers non-executable; enhanced compilers that perform bounds checking; and performing integrity checks on code pointers before dereferencing them.

“Writing correct code is a laudable but highly expensive proposition:”

Despite a long history of understanding of how to write secure programs vulnerable programs continue to emerge on a regular basis… Even defensive code that uses safer alternatives such as strncpy and snprintf can contain buffer overflow vulnerabilities if the code contains an elementary off-by-one error. For instance, the lprm program was found to have a buffer overflow vulnerability, despite having been audited for security problems such as buffer overflow vulnerabilities.

The operating system approach is to make data areas for buffers non-executable. Many older computer systems were designed this way, but in the name of performance, “more recent UNIX and MS Windows
systems have come to depend on the ability to emit dynamic code into program data segments to support
various performance optimizations.” However, you can make the stack segment non-executable. This protection is highly effective against attacks that depend on injecting code into automatic variables, but offers no protection against other forms of attack.

While injecting code is optional for a buffer overflow attack, the corruption of control flow is essential. Thus unlike non-executable buffers, array bounds checking completely stops buffer overflow vulnerabilities and attacks. If arrays cannot be overflowed at all, then array overflows cannot be used to corrupt adjacent program state.

With array bounds checking, you need to check all reads and writes to ensure that they are within range. As of 2000, the best approaches either had severe limitations in what they could actually check for, or severe performance penalties. Of course, another approach is simply to use a type-safe language in the first place!

All buffer overflow vulnerabilities result from the lack of type safety in C. If only type-safe operations can be performed on a given variable, then it is not possible to use creative input applied to variable foo to make arbitrary changes to the variable bar. If new, security-sensitive code is to be written, it is recommended that the code be written in a type-safe language such as Java or ML. Unfortunately, there are millions of lines of code invested in existing operating systems and security-sensitive applications, and the vast majority of that code is written in C. This paper is primarily concerned with methods to protect existing code from buffer overflow attacks. However, it is also the case that the Java Virtual Machine (JVM) is a C program, and one of the ways to attack a JVM is to apply buffer overflow attacks to the JVM itself. Because of this, applying buffer overflow defensive techniques to the systems that enforce type safety for type-safe languages may yield beneficial results.

Code pointer integrity checking seeks to detect that a code pointer has been corrupted before it is dereferenced (and thus prevent the pointer from being used).

Code pointer integrity checking has the disadvantage relative to bounds checking that it does not perfectly solve the buffer overflow problem; overflows that affect program state components other than code pointers will still succeed. However, it has substantial advantages in terms of performance, compatibility with existing code, and implementation effort.

The authors’ StackGuard project implements this approach. It works by placing a ‘canary’ word next to the return address on the stack.

The enhanced function tear down code first checks to see that the canary word is intact before jumping to the address pointed to by the return address word. Thus if an attacker attempts a “stack smashing,” the attack will be detected before the program ever attempts to dereference the corrupted activation record.

To ensure that an attacker cannot forge a canary you can ensure that the canary is comprised of common termination symbols for C strings which thwarts string-based attacks. You can also use a random canary (just a 32 bit number) chosen when the program starts. StackGuard was shown to provide effective protection against stack-smashing attacks while still preserving virtually all of the system performance and compatibility.

At the time StackGuard was built, the “stack smashing” variety formed a gross preponderance of buffer overflow attacks. It is conjectured that this resulted from some “cook book” templates for stack smashing attacks released in late 1996. Since that time, most of the “easy” stack smashing vulnerabilities have been exploited or otherwise discovered and patched, and the attackers have moved on to explore the more general form of buffer overflow attacks. PointGuard is a generalization of the StackGuard approach designed to deal with this phenomena…

PointGuard places canaries next to all code pointers (function pointers and longjmp buffers). Both allocating and checking the canary are difficult problems though.

What Happened Next?

There is a good summary in the Wikipedia article on Buffer Overflow Protection:

  • StackGuard was suggested for inclusion in GCC 3.x in 2003, but this never happened
  • IBM developed ProPolice which improved on StackGuard by placing buffers after local pointers and function arguments in the stack frame.
  • In 2005 RedHat identified problems with ProPolice and re-implemented stack-smashing protection for GCC 4.1 introducing the -fstack-protector and -fstack-protector-all flags
  • In 2012, Google added -fstack-protector-strong striking a better balance between performance and security. It has been in GCC since version 4.9.