Calling Conventions

When a function is called, the arguments are typically passed to it, and the return value is retrieved. A calling convention describes how the arguments are passed and values returned by functions. It also specifies how the function names are decorated.

As an example for the calling conventions:

int sum (int a, int b)
{
    return a + b;
}

// Call to above function
int c = sum (2, 3);

cdecl calling convention

This convention is the default for C/C++ programs. If a project is set to use some other calling convention, we can still declare a function to use __cdecl:

int __cdecl sum (int a, int b);

The main characteristics of __cdecl calling convention are:

1. Arguments are passed from right to left, and placed on the stack.
2. Stack cleanup is performed by the caller.
3. Function name is decorated by prefixing it with an underscore character ‘_’ .

// Example of a __cdecl call:

; // push arguments to the stack, from right to left
push        3    
push        2    

; // call the function
call        _sum 

; // cleanup the stack by adding the size of the arguments to ESP register
add         esp,8 

; // copy the return value from EAX to a local variable (int c)
mov         dword ptr [c],eax

stdcall calling convention

This convention is usually used to call Win32 API functions. In fact, WINAPI is nothing but another name for __stdcall:

// Declaration of function to use the __stdcall convention
int __stdcall sum (int a, int b);

The main characteristics of __stdcall calling convention are:

1. Arguments are passed from right to left, and placed on the stack.
2. Stack cleanup is performed by the called function.
3. Function name is decorated by prepending an underscore character and appending a ‘@’ character and the number of bytes of stack space required.

// Example of __stdcall convention
; // push arguments to the stack, from right to left
  push        3    
  push        2    
  
; // call the function
  call        _sum@8

; // copy the return value from EAX to a local variable (int c)  
  mov         dword ptr [c],eax

Functions with the variable number of arguments (like printf()) must use __cdecl, because only the caller knows the number of arguments in each function call; therefore only the caller can perform the stack cleanup.

fastcall calling convention

Fast calling convention indicates that the arguments should be placed in registers, rather than on the stack, whenever possible. This reduces the cost of a function call, because operations with registers are faster than with the stack.

We can explicitly declare a function to use the __fastcall convention as shown:

// Declaration of function to use the __fastcall convention
int __fastcall sum (int a, int b);

The main characteristics of __fastcall calling convention are:

1. The first two function arguments that require 32 bits or less are placed into registers ECX and EDX. The rest of them are pushed on the stack from right to left.
2. Arguments are popped from the stack by the called function.
3. Function name is decorated by by prepending a ‘@’ character and appending a ‘@’ and the number of bytes (decimal) of space required by the arguments.

// Example of __fastcall convention

; // put the arguments in the registers EDX and ECX
  mov         edx,3 
  mov         ecx,2 
  
; // call the function
  call        @Sum@8
  
; // copy the return value from EAX to a local variable (int c)  
  mov         dword ptr [c],eax