System Call Instructions


User Mode threads transition into Kernel Mode when they make system calls. There are 3 ways a user mode thread can perform this transition:

Int 0x2e

"int 2e" is the legacy way of performing user to kernel mode transitions and is supported by all x86 CPUs existing today. The call to "int 2e" results in the interrupt service routine registered in the interrupt descriptor table (IDT) for vector 0x2e (i.e. nt!KiSystemService) being invoked.

"int 2e" being a generic software interrupt and has all the overhead associated with interrupt handling. User to kernel mode transition is a very frequently executed operation (on a reasonably busy system this can be about 5000-6000 times a second). So there needs to be an optimized or fast path way to perform these transitions. To address this issue both Intel and AMD added CPU instructions to speed up user to kernel mode transitions. Intel added "sysenter" and AMD added "syscall".

On X86
0:000> uf ntdll!NtCreateFile
ntdll!ZwCreateFile:
7c90d0ae b825000000      mov     eax,25h
7c90d0b3 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
7c90d0b8 ff12            call    dword ptr [edx]
7c90d0ba c22c00          ret     2Ch

0:000> uf poi(7ffe0300)
ntdll!KiIntSystemCall:
7c90e520 8d542408        lea     edx,[esp+8]
7c90e524 cd2e            int     2Eh
7c90e526 c3              ret

sysenter/syscall

The sysenter/syscall instruction is supported on the Pentium and on later versions of the CPU and is called "Fast Transition to System Call Entry Point". Bit 11 (SEP) of the EDX register from the result of a CPUID instruction tells if the the processor supports sysenter/syscall instructions.

The instruction uses a special path to invoke the kernel mode system service handler i.e. nt!KiFastCallEntry. The kernel registers this function with the CPU by storing a pointer to this function into the CPU Model Specific Register (MSR) SYSENTER_EIP_MSR (0x175) during system startup. The switch to kernel mode using sysenter/syscall is 3 times faster than using the legacy "int 2e" method.

Since particular NTDLL.dll binary can execute on all types of X86 CPUs, NTDLL has to support all of the mechanisms to perform user to kernel mode transitions. The decision as to which mechanism to use is made at run time depending on the type of CPU in the system. On x64 systems, however, NTDLL always uses the syscall instruction.

On X64
0:000> uf ntdll!NtCreateFile
ntdll!ZwCreateFile:
00000000`76eb02a0 4c8bd1          mov     r10,rcx
00000000`76eb02a3 b852000000      mov     eax,52h
00000000`76eb02a8 0f05            syscall
00000000`76eb02aa c3              ret
On X86

0:000> uf ntdll!NtCreateFile
ntdll!ZwCreateFile:
7c90d0ae b825000000      mov     eax,25h
7c90d0b3 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
7c90d0b8 ff12            call    dword ptr [edx]
7c90d0ba c22c00          ret     2Ch

0:000> uf poi(7ffe0300)
ntdll!KiFastSystemCall:
7c90e510 8bd4            mov     edx,esp
7c90e512 0f34            sysenter
7c90e514 c3              ret