Understanding x86-64 Flags and Conditional Branches

These blog posts are written as lessons. Examples use Intel syntax. Performance behavior must be measured on the target processor and real input distribution.

cmp eax, edx stores no visible result, so how can a following jl know which value is smaller? Why can the same bits 0xFFFFFFFF be treated as -1 in one comparison and 4,294,967,295 in another?

x86-64 comparisons and branches communicate through status bits in RFLAGS. The practical task is to identify which instruction produced the flags and whether the consumer applies a signed or unsigned condition.

Common status flags

RFLAGS contains more state, but ordinary compiler-generated comparisons mainly combine ZF, CF, SF, and OF.

cmp behaves like a discarded subtraction

cmp rax, rdx

Conceptually it computes rax - rdx, updates flags, and leaves both operands unchanged.

cmp eax, 10
je  equal_ten
jl  signed_less_than_ten

je tests ZF = 1; jl tests the signed-less condition SF != OF. A branch does not have a grammatical attachment to the nearest cmp. Any intervening flag-setting instruction replaces the state.

cmp eax, 10
add ecx, 1
jl somewhere      ; observes ADD's flags, not CMP's

test behaves like a discarded AND

test rax, rax
jz is_zero

test eax, 1
jnz is_odd

test updates flags from a bitwise AND without storing the result. It is common for zero and bit-mask checks. It clears CF and OF and defines ZF, SF, and PF from the result. A zero test may branch like cmp eax, 0, but do not assume that every flag has identical semantics.

Equality is independent of signedness

The paired mnemonics are aliases for the same condition code; one name simply reads better after a comparison and the other after a zero test.

Ordering requires signed or unsigned interpretation

After cmp left, right:

mov eax, 0FFFFFFFFh
cmp eax, 1
jl signed_less       ; taken: -1 < 1
jb unsigned_below    ; not taken: 4294967295 is not below 1

The register has no signed type tag. The condition used to interpret the same subtraction flags makes the difference.

Array sizes and indices are often unsigned:

cmp rdx, rcx
jae out_of_range     ; index >= count

Coordinates or temperatures may need signed conditions. A particular jcc is evidence about the source range, not a perfect recovery of the original declaration.

CF and OF describe different overflow

In eight bits, 0xFF + 1 = 0x00 has unsigned carry but no signed overflow: -1 + 1 is zero. 0x7F + 1 = 0x80 has no unsigned overflow, but signed 127 + 1 exceeds the range.

add al, 1
jc unsigned_overflow
jo signed_overflow

C and C++ define unsigned arithmetic modulo a power of two, while signed overflow is undefined behavior. The existence of a hardware OF bit does not define the language operation.

Arithmetic can directly feed a condition

add, sub, inc, dec, neg, and logical operations update flags, with important differences. inc and dec, for example, preserve CF; replacing an add 1 in a multi-precision carry chain can change meaning.

sub rcx, 1
jnz loop_begin

The branch consumes sub's ZF; no separate compare is required.

setcc materializes a Boolean

cmp edi, esi
setl al
movzx eax, al
ret

al becomes one if signed edi < esi, otherwise zero. Because setcc only writes one byte, clear or extend the upper bits.

xor eax, eax
cmp edi, esi
setl al
ret

cmovcc selects without changing control flow

; eax = a, edx = b
cmp eax, edx
cmovl eax, edx      ; if a < b, choose b

Conditional moves can implement small selections such as max. They can also lengthen dependency chains, and a memory source must not be treated as a safe way to suppress an invalid access. They are not automatically faster than branches.

Branchless is not automatically faster

Modern processors predict branches. A highly predictable jcc can be cheap. Irregular input and tiny alternate paths may benefit from setcc, cmovcc, or masks. Judge using real branch direction distribution, misprediction rate, work on each path, added instructions and dependencies, compiler output, and measurements on the target CPU.

Counting if or jmp statements is not a performance measurement.

Reconstruct a loop

xor eax, eax
xor edx, edx
loop_test:
    cmp rdx, rsi
    jae loop_done
    add eax, dword ptr [rdi + rdx*4]
    inc rdx
    jmp loop_test
loop_done:
    ret

The control-flow graph contains a test block, a body/back-edge block, and a return block. jae performs an unsigned i >= count check. Optimization may rotate, unroll, or vectorize the loop into a very different shape.

Short-circuit conditions are ordered safety checks

if (p != nullptr && p->ready)
    use(p);

test rcx, rcx
jz done
cmp byte ptr [rcx + READY_OFFSET], 0
je done
call use
done:

The first branch prevents a null dereference. Reversing the conditions while reading loses that safety property. The compiler may remove the first test only if it proves the pointer non-null.

A switch may use a jump table

sub eax, 3
cmp eax, 4
ja default_case
jmp qword ptr [table + rax*8]

The code normalizes the minimum case, performs an unsigned range check, and makes an indirect branch through a table. The range check is security-relevant because it keeps the table index in bounds.

Flags are not preserved across ordinary function calls

cmp eax, edx
call helper
jl old_result       ; helper may have changed the flags

Materialize the condition before the call or compare again afterward.

Common mistakes

• Treating jl and jb, or jg and ja, as equivalent.
• Thinking cmp stores the subtraction into its left operand.
• Missing an instruction that overwrites flags before a branch.
• Assuming inc and add 1 update every flag identically.
• Equating C signed overflow with hardware wraparound.
• Assuming branchless code is inherently faster.
• Ignoring upper bits after setcc al.

Exercise

Compile signed and unsigned variants of these functions:

int less_i32(int a, int b) { return a < b; }
int less_u32(unsigned a, unsigned b) { return a < b; }
int clamp_zero(int x) { return x < 0 ? 0 : x; }
unsigned count_nonzero(const int* p, size_t n);

Find the selected jcc or setcc, see whether clamp_zero uses a branch or cmov, compare loop shapes across optimization levels, and test signed and unsigned boundary values.

Summary

cmp and test produce flags without saving their arithmetic or logical result. Equality uses ZF, while ordering must distinguish signed SF/OF conditions from unsigned CF/ZF conditions. Trace the last flag-defining instruction, the signedness of the consumer, and the input range on every edge.

Official references

• Intel 64 and IA-32 Architectures Software Developer's Manual
• AMD64 Architecture Programmer's Manual Volume 3
• Intel 64 and IA-32 Architectures Optimization Reference Manual