C Abstract Machine

Hello, we meet again. Sounds crazy but the C you write doesn’t describe your CPU. It describes a fiction the standard has held since 1989 the abstract machine, and the only thing your compiler owes you is a binary that reproduces that machine’s observable behavior. Observable behavior is a short list you got volatile accesses, data written to files, I/O ordering at sequence points. Everything else, every add, load, and branch, the compiler can shred, reorder, duplicate, or delete, as long as a conforming observer can’t tell. That’s the “as-if” rule, and it’s a licence to do almost anything.

That part is just optimization, and optimization is fine. It turns dangerous when as-if meets the standard’s other favorite phrase yep undefined behavior. The standard marks somewhere around two hundred constructs as undefined, and for each one it “imposes no requirements.” An execution that hits UB constrains the generated code nowhere not from the UB onward, but across the whole run, forward and backward.

So UB doesn’t happen “at a line.” It poisons the entire execution it belongs to. If the optimizer proves an input leads to UB, it owes that input nothing, including the parts that already ran.

The shift is to stop asking “what will the CPU do when this overflows or derefs null?” That has no answer, because your source never reaches the CPU as written. It reaches the optimizer first, and the optimizer doesn’t think in registers and flags. It thinks in theorems. Two things fall out:

  • you can spot the checks the compiler is about to delete from your own code.
  • or you get a bug class that slips past source review, because the bug isn’t in the source as written. It’s in the gap between the source and what the toolchain decided the source was allowed to mean.

This came out of a long ass convo with a friend the other day we got into provenance, poison values, and “time travel” optimizations, but that is for another piece. Today is about one narrower consequence of the gap checks that are right there in your C, then gone from the optimized binary.

int aght(int size) {
    if (size > size + 1)   /* reject if size+1 overflows */
        return 0;
    return 1;
}

size is signed, signed overflow is UB, so the compiler assumes size + 1 never overflows. That makes it always greater than size, the condition always false, the branch dead. GCC 15, -O2:

aght:
        mov     eax, 1
        ret

https://godbolt.org/z/518TEeTjz

Gone. The same check on unsigned size survives, because unsigned overflow is defined to wrap. The whole vulnerability is one letter of signedness, which no reviewer reads as security-relevant.

int deref(int *p) {
    int v = *p;         /* dereference */
    if (!p) return -1;  /* then check  */
    return v;
}

Dereferencing null is UB, so once *p has run the compiler treats p as non-null. The check can’t fire. GCC 15 and Clang 18 agree:

deref:
        mov     eax, DWORD PTR [rdi]
        ret

https://godbolt.org/z/KnP97jvMx

This is the shape of an old vuln (CVE-2009-1897), the Linux tun driver, where a null check got deleted because the pointer was dereferenced above it. The source had the check. The shipped kernel didn’t.

Those two are folklore. This next one is the same trick, but the outcome flips on the compiler version, and that’s the part people miss.

vuln.c reassembles two attacker-controlled chunks into a fixed buffer, with length validation up front:

#include <string.h>
static char pool[256];

int reassemble(int l1, int l2, const char *a, const char *b) {
    int total = l1 + l2;

    if (l1 < 0) return -1;
    if (l2 < 0) return -1;
    if (total < l1)                /* did the sum wrap? */
        return -2;
    if (total > (int)sizeof pool)  /* capacity guard */
        return -3;

    memcpy(pool, a, l1);
    memcpy(pool + l1, b, l2);
    return 0;
}

The driver reads l1/l2 off argv and hands over fixed 64-byte buffers, so the declared lengths are the attack surface, not the data. Two sanity checks, an overflow guard, a capacity check. You’d sign off on this. I would too.

Honest packet first, then one crafted so l1 + l2 overflows int and wraps negative, which is what total < l1 is there to catch:

$ gcc-14 -O2 -fno-stack-protector -o v vuln.c
$ ./v 100 100
reassemble(l1=100, l2=100) = 0
$ ./v 100 2147483647
reassemble(l1=100, l2=2147483647) = -2      # rejected

https://godbolt.org/z/4hEGbhcnf

Guard fires. Now the same source, same flags, GCC 15:

$ gcc-15 -O2 -fno-stack-protector -o v vuln.c
$ ./v 100 2147483647
Segmentation fault (core dumped)            # exit 139, SIGSEGV

https://godbolt.org/z/ean6cEx79

Nobody touched the code. The overflow guard is gone, and you can watch it leave: the -2 return path (0xfffffffe) is in the GCC 14 object and not the GCC 15 one.

$ objdump -d <gcc14 build> | grep -c 0xfffffffe   # 1
$ objdump -d <gcc15 build> | grep -c 0xfffffffe   # 0

See it in the asm (GCC 15, no -2 path): https://godbolt.org/z/bYbfa7413 Flip the compiler dropdown to GCC 14 and the -2 return comes back.

With the guard removed, the crafted length passes the capacity check (total wrapped negative at runtime, and negative isn’t > 256) and reaches memcpy(pool + l1, b, l2) with l2 In this toy that just faults a huge read from b and write into pool + 100. In real stuff a deleted guard like this can become a memory corruption primitive, but whether it’s actually exploitable, and how far, depends on memory layout, allocator and global placement, copy direction, when the fault lands, and mitigations. The narrow, certain part is that the check you were relying on is gone.

Why 15 and not earlier? To kill total < l1 the compiler has to prove l1 + l2 >= l1, which means combining l2 >= 0 from the earlier check with the no-overflow assumption. GCC’s value-range analysis only got that sharp at 15. Older GCC keeps the check. I spot-checked GCC 4.x through 16 and a range of Clang releases the GCC boundary sits at 15 in plain don’t assume a clean version switch. Check the compiler that builds your target.

-fwrapv brings the guard back. It makes signed overflow defined, so there’s no assumption left to exploit:

$ gcc-15 -O2 -fwrapv -fno-stack-protector -o v vuln.c
$ ./v 100 2147483647
reassemble(l1=100, l2=2147483647) = -2      # rejected again

https://godbolt.org/z/q639fhEMo

The check is right there in the source. Read the binary.

  • Build the TU at -O0 and -O2, disassemble both, diff them. A compare-and-branch present at -O0 and gone at -O2 is a candidate.
  • Build with -fsanitize=undefined (UBSan) and run it over your tests and fuzz corpus. By default it reports UB at runtime and keeps going (add -fsanitize-trap=undefined to abort on it), so it flags the inputs that would feed a deletion. It can’t see a check that’s already gone.

If you’re the foo on the other side, this runs in reverse. Fingerprint the thingy from build IDs, package metadata, or the GCC: string in the ELF .comment, and you can guess which class of check is probably already deleted before reading a line, then confirm it in the disassembly.

Never write a check whose condition is only true when something is UB.

  • Test the operands, not the result: if (a > INT_MAX - b), not if (a + b < a).
  • Stay unsigned where wraparound is defined and you want it, but don’t mix signed and unsigned lengths carelessly.
  • Validate a pointer before you dereference it.
  • Compare lengths as integers, not by building an out-of-bounds pointer and testing it: if (n > (size_t)(end - buf)).
  • Initialize variables before you read them.

When you can’t touch the source, change what the compiler is allowed to assume:

  • -fwrapv: signed overflow wraps instead of being UB.
  • -fno-strict-overflow: drops the same overflow assumption.
  • -fno-delete-null-pointer-checks: keeps null checks after a deref.
  • -fno-strict-aliasing: allows type punning. The Linux kernel builds with it.
  • -ftrivial-auto-var-init=zero: zero-fills locals.

These can cost performance or block optimizations, and how much is workload-dependent, so measure before you put them across a whole build use 'dm on code where a deleted check is a pain in ass

The funny thing is I do none of this shit when I write C. It’s not that I’m unwilling (I’m a shitty programmer) so this is my way to carve that into memory maybe one day, who know also keep in mind this covers very little of UB and doesn’t cover every UB class don’t take this as a panacea keep it as a mental model, but it needs adaptation.

  • J. Regehr, A Guide to Undefined Behavior in C and C++. blog.regehr.org
  • C. Lattner, What Every C Programmer Should Know About Undefined Behavior. LLVM Project Blog, 2011
  • R. Jung, Undefined Behavior deserves a better reputation. ralfj.de, 2021
  • X. Wang et al., Towards Optimization-Safe Systems. SOSP 2013 (the STACK checker, “unstable code”)
  • Project Ranger and The new oracles of GCC. Red Hat Developer, 2021 and 2023
8 Likes

Thank you for good information, I will learn this

1 Like

This is what I got for one of my C++17 projects:

# Security hardening compiler flags
target_compile_options(meshtastic-bridge PRIVATE
    # Warnings
    -Wall -Wextra -Wformat -Wformat=2 -Wconversion -Wimplicit-fallthrough
    -Werror=format-security
    -Wno-deprecated-declarations  # Keep for protobuf
    # Runtime hardening
    -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3
    -D_GLIBCXX_ASSERTIONS
    # Code hardening
    -fstrict-flex-arrays=3
    -fstack-protector-strong
    -fstack-clash-protection
    -fPIE
    -fno-delete-null-pointer-checks
    -fno-strict-overflow
    -fno-strict-aliasing
    -ftrivial-auto-var-init=zero
    -fexceptions
)

# Security hardening linker flags
target_link_options(meshtastic-bridge PRIVATE
    -Wl,-z,nodlopen
    -Wl,-z,noexecstack
    -Wl,-z,relro
    -Wl,-z,now
    -Wl,--as-needed
    -Wl,--no-copy-dt-needed-entries
    -pie
)

I forgot where I pulled most of them from, but I’ll defiantly add some more lol. Nice read.

3 Likes

Awesome read, What’s the prevalence of this in other languages ?

2 Likes

Nice write up @0xf00s . But I think your conclusion is a bit misleading.

If you program in C, you have to know what’s UB and what not and you just don’t do it. Using compiler flags to force a specific behavior is the wrong call and makes your code not portable. The right call is to fix your code and remove the UB.

A good resource about these (and other tricky C areas) from a security point of view is the book:

Secure Coding in C and C++. Robert C. Seacord

It is a bit old, but UB is still UB :slight_smile:

This document contains a good summary of these and other C caveats and how to fix them in your code.

Or check their wiki and browse the rules online

You will find a few more curious things about C. There are also guides for C++ and Java. Interesting reading.

The original C FAQ is also a great resource about UB and other topics.

5 Likes

Thanks @pico yep agreed on the goal rm the UB that’s why

is in there, and the flags come after it, for code where a deleted check is a security problem not as the fix.

“just know what’s UB and don’t do it” is the exact thingy this piece argues against everyone who shipped the reassemble guard knew C It still read as correct and still got deleted knowing isn’t the failure point, spotting it in code that reads fine is.

and -fwrapv isn’t there to make bad source correct. It’s for the build you don’t fully own, the legacy tree you can’t audit line by line mf just use AI now right ? but yeah that’s why the kernel ships -fno-strict-aliasing and -fno-delete-null-pointer-checks portability you control at the build a deleted bounds check you find in a coredump but yeah,

and thanks again for the resources, man much appreciated.

And like you said, there’s a few more curious things about C I seem to always learn something new. I don’t know if this is something you guys notice, but I’ve found that reading ‘old’ books or articles has more depth and insight into the concept than newer stuff. Maybe I’m wrong, or just haven’t read a lot of new stuff :laughing: but yeah, it’s very interesting.

4 Likes

I think this is mostly a C/C++ disease other langs either don’t have UB, or box it somewhere the optimizer can’t weaponize.

I overlooked that sentence. Great post @0xf00s . I really enjoy it!

2 Likes

Yeah and not everyone has the time to review the outputted compiled machine code via disassembly. Obviously this is why tests happen and should happen, code quality checks are a thing and other stuff, but the flags are there more for the compiler than anything else.

2 Likes