Practical x86-64 Function Calls and ABIs

These blog posts are written as lessons. This article compares the common System V AMD64 and Microsoft x64 ABIs. OS and toolchain extensions, kernel calls, syscalls, and conventions such as vectorcall need their own target documentation.

The processor's call instruction only stores a return address and transfers control. The ABI defines where arguments and results live, who preserves each register, how the stack is aligned, and how a frame is unwound.

ISA, calling convention, and ABI

• The ISA defines hardware instructions and register behavior.
• A calling convention defines arguments, results, preserved state, and stack responsibility.
• An ABI includes calling conventions plus type layout, object format, symbols, relocation, and unwind rules.

extern "C" controls C++ language linkage and name mangling. It does not turn a Windows function into a System V function or simplify every structure and exception boundary.

Comparison of the two common x86-64 ABIs

Aggregates, vectors, variadic functions, and hidden return pointers need more detailed classification than this table.

Caller-saved and callee-saved state

A volatile or caller-saved register may change across a call; the caller saves a live value if necessary. A nonvolatile or callee-saved register must be restored by a callee that uses it.

push rbx
mov rbx, rcx
; use rbx
pop rbx
ret

Preservation does not mandate push. A function that does not touch rbx saves nothing, and another may use a stack slot with mov. Only the state observable at the function boundary matters.

A simple System V call

For long combine(long a, long b, long c):

mov edi, 10
mov esi, 20
mov edx, 30
call combine
; result in rax

Remember that C long is normally 64 bits in the LP64 System V model but 32 bits in Windows LLP64. “x64” does not force every source-language long to 64 bits.

The equivalent Windows x64 pattern

sub rsp, 40
mov ecx, 10
mov edx, 20
mov r8d, 30
call combine
add rsp, 40

Thirty-two bytes are the caller-provided shadow space. The extra eight bytes account for alignment after the return address pushed by call, making 40 bytes a common allocation before a nested call. The caller reserves shadow space even with fewer than four arguments; the callee need not actually spill all register parameters there.

Stack alignment is a point-in-time contract

Distinguish:

1. the caller immediately before call;
2. the callee immediately after the return address was pushed;
3. the callee after its prologue;
4. the callee immediately before a nested call.

Applying one point's modulo rule to another creates the familiar eight-byte confusion. Microsoft x64 requires rsp to stay 16-byte aligned outside prologues and epilogues, with limited leaf-function exceptions. System V describes its call boundary with the return-address push included in the entry relationship.

The System V red zone

Eligible System V user-space leaf functions may use 128 bytes below rsp without first moving the stack pointer.

mov qword ptr [rsp - 8], rax

The storage cannot safely survive a nested call, and the same assumption does not apply to Windows, kernel code, or builds that disable the red zone.

Windows unwind metadata

A Windows x64 non-leaf function that changes nonvolatile state or rsp needs metadata that allows exceptions, debuggers, and profilers to reconstruct the caller.

Work PROC FRAME
    push rbx
    .pushreg rbx
    sub rsp, 32
    .allocstack 32
    .endprolog

    ; body

    add rsp, 32
    pop rbx
    ret
Work ENDP

.pushreg and .allocstack describe the real instructions; they do not execute them. A mismatch can pass ordinary return tests while breaking exception handling and stack walking.

Stack arguments require a time reference

Windows passes positions beyond the first four on the stack. At callee entry, the return address is followed by the 32-byte home area, then later arguments. If the prologue changes rsp, every rsp-relative offset changes. Never memorize [rsp+40] as “the fifth argument” without saying whether it is an entry or post-prologue offset.

System V also uses the stack when register classes or available registers are exhausted. Aggregate classification is more complex than “argument seven onward.”

Floating-point arguments use position and type

For Windows x64:

void f(int a, double b, int c, float d);

The positions map approximately to a -> rcx, b -> xmm1, c -> r8, and d -> xmm3. Do not count only floating-point arguments and put b in xmm0. System V scalar FP arguments primarily use xmm0 through xmm7 under its own classification rules.

Variadic functions add rules

For calls such as printf:

• System V uses al to communicate information related to vector-register arguments.
• Microsoft x64 duplicates variadic floating-point arguments into the matching integer-register position.
• C default argument promotions turn float into double and promote small integers.

The format string must agree with the promoted types. A variadic callee cannot recover a missing type contract from raw register bits.

Structures need ABI classification

System V classifies values in eightbyte units and may split suitable aggregates across integer or SSE registers. Microsoft x64 passes certain aggregate sizes as integer-like values and often passes others through a caller-created temporary pointer.

For a handwritten assembly boundary:

1. Prefer fixed-width scalar parameters.
2. Prefer a simple extern "C" free function.
3. Assert sizeof, alignof, and offsetof in C++.
4. Inspect both caller and callee disassembly.
5. Test Debug, Release, and LTO/LTCG configurations.

Hidden return pointers

A large result that cannot be returned in registers may use caller-allocated storage and a hidden pointer argument. That pointer can shift every explicit argument by one apparent register position. Member-function this, constructors, and aggregate returns make source-level argument counting unreliable; inspect the actual lowering.

Tail calls reuse the current return path

wrapper:
    ; arrange arguments for target
    jmp target

When the last action is returning another function's result and ABI conditions permit it, a compiler can replace call plus ret with jmp. target then returns directly to the original caller.

Calling C++ from Windows assembly

EXTERN AddBias:PROC
PUBLIC AsmTwiceThenBias

AsmTwiceThenBias PROC FRAME
    sub rsp, 40
    .allocstack 40
    .endprolog

    add ecx, ecx
    call AddBias

    add rsp, 40
    ret
AsmTwiceThenBias ENDP

extern "C" int AddBias(int value, int bias);
extern "C" int AsmTwiceThenBias(int value, int bias);

The boundary still needs range, overflow, exceptional-path, and stack-walk tests appropriate to the real project.

Debugging order

1. Are caller and callee built for the same architecture and ABI?
2. Do symbol spelling and language linkage match?
3. Do argument bits and types occupy the correct locations immediately before call?
4. Is every stack offset relative to the correct rsp moment?
5. Are alignment and Windows shadow space correct?
6. Does the callee restore every nonvolatile register it changed?
7. Do Windows frame directives match the real prologue?
8. Is the result read from the correct width of rax or from xmm0?

Common mistakes

• Assuming x86-64 has one OS-independent calling convention.
• Believing the Windows callee allocates its own shadow space.
• Using the System V red zone on Windows or in kernel code.
• Confusing entry alignment with pre-call alignment.
• Expecting volatile registers to survive a nested call.
• Restoring a nonvolatile register on only some return paths.
• Treating successful normal return as proof of correct unwind data.

Exercise

Create a C function receiving six integers, four doubles, and two small structures. Compare a GCC/Clang System V caller with an MSVC x64 caller. Record every argument location, trace rsp to the first nested call, list save/restore operations, detect any hidden structure-return pointer, and explain one variadic call's promotions and extra ABI rules.

Summary

call and ret are only the hardware skeleton. The ABI defines argument and result locations, preservation, stack alignment, aggregate and variadic passing, and unwinding. System V uses six main integer argument registers and offers a red zone; Microsoft x64 uses four position-based register slots, 32 bytes of shadow space, nonvolatile XMM state, and strict unwind metadata. Prove the boundary with caller and callee disassembly, tests, and stack walking.

Official references

• System V AMD64 ABI project and latest specification
• Microsoft x64 calling convention
• Microsoft x64 ABI conventions
• MASM for x64 and unwind directives
• Intel 64 and IA-32 Architectures Software Developer's Manual