System setup for kernel development and debugging

System setup instructions for CodeMachine training courses' hands-on lab exercises.

© CodeMachine Inc. | codemachine.com | @codemachineinc

This article provides detailed instructions to set up your host system and a guest VM for the hands-on labs for CodeMachine training courses.

You will need a host system (desktop or laptop) that is virtualization capable and must have the necessary software installed. The hands-on lab exercises will not damage or negatively impact your host system. All the hands-on lab exercises will be performed inside the virtual machine (VM) guest OS. The following figure shows the relationship between the host system, the Hyper-V platform, and the guest VM.

FIG#1
Figure 1: Host system and guest VM

The table below describes the recommended hardware configuration for your host system.

CPU Intel or AMD CPU with support for hardware virtualization. CPU must be fast enough to run at least one VM without any performance degradation.
Memory At least 8 GB RAM, preferably 16 GB RAM.
Storage Preferable SSD/NVMe with at least 50 GB of free.
Display Resolution of at least 1920x1080 HD. Display size at least 14 inches diagonal.
Network WiFi or Ethernet. The system must have reliable Internet access.
USB Ports At least one free USB port for copying course material (if necessary).
Host OS Windows 10 64-bit Enterprise Edition latest version. Ensure that no Endpoint Security or Antivirus software is installed. Windows Defender is fine. You will disable the required features as needed.
Virtualization You can choose to use any virtualization software such as Hyper-V, VMWare, or Virtual Box. The steps described in the Host Setup section of this document assume that you are using Hyper-V which is built into Windows 10 Professional and Enterprise editions.

The instructions in this document assume that you will be using a serial port over a named pipe (KDCOM) as the kernel debugging transport. Alternatively, you could use a different kernel debugging mechanism such as KDNET or VirtualKD. If you choose to do so, please make the necessary changes to the instructions in the following sections.

  • Hyper-V VM configuration
  • Kernel debugging configuration
  • Testing the kernel debugging setup

Some students prefer to use macOS with Windows VMs, where one VM acts as the host system and the second VM acts as the guest system. Windows kernel debugging over named pipes does work in this setup. However, this document assumes you are using a Windows host and a single guest VM.

Host System

1. Host OS

The latest version of Windows 10 64-bit Professional or Enterprise edition must be installed on the host system and the latest updates must be applied. OS installation images can be obtained from MSDN. Windows need not be activated. A trial version of Windows will work fine.

2. Internet access

The host system must have reliable internet access throughout the duration of the course to look up MSDN and for the debugger to download symbols from Microsoft’s public symbol server.

3. Source code editor

The hands-on labs for most CodeMachine courses involve programming in C/C++. Therefore, so a good code editor must be installed on your host system. You can use your favorite source code editor. If you prefer to use Visual Studio Code, you can download Visual Studio Code System Installer 64-bit for Windows 7, 8, 10 and run the installer with default settings. Alternatively, you can use Visual Studio 2019 with the Windows Driver Kit (WDK Version 2004) Extension. Visual Studio 2019 and the WDK are completely optional and all the code in the class can be built using the Enterprise Windows Driver Kit (EWDK).

4. Enterprise Windows Driver Kit (EWDK)

The EWDK contains all the tools required to build and debug kernel software drivers on Windows. Download the "EWDK for Windows 10, version 2004 with Visual Studio Build Tools 16.7" and extract the contents of the .ISO file into the folder c:\EWDK. The EWDK uses XCOPY deployment, therefore no installation is necessary. The instructions in the rest of this document assume that you are using the EWDK and it is available at c:\EWDK.

5. Windows Debugger

Download the WinDBG preview from the Windows App Store and install it, if it is not already installed.

6. Debugging symbols

Run the following command in an administrative command (cmd.exe) window to set up the environment variable for the debugger to download symbols from Microsoft's public symbol server.

setx _NT_SYMBOL_PATH SRV*c:\symbols*https://msdl.microsoft.com/download/symbols

7. Virtualization software

You can use any virtualization Software such as Hyper-V, VMWare, or Virtual Box. We recommend using Hyper-V as it is built-in Windows 10 Professional and Enterprise Editions. The instructions in steps 8 through 12 are specific to Hyper-V. Please modify appropriately if you are using different virtualization software.

NOTE:If you are using Hyper-V, steps 8 through 10 have been automated using a PowerShell script. The script will create a Hyper-V Gen 2 VM with the name WINLABVM. If you prefer a different name, you can edit the script to specify a VM name of your choice. If you do so, please use that same name in all steps in this document which has a reference to WINLABVM. If you prefer to continue creating and configuring your VM using this script, download Create-VM-WINLABVM.ps1 to the host. Run the script in an administrative PowerShell command window and proceed to step 11.

8. Hyper-V RDP access

You will spend a significant amount of time in the class inside the kernel debugger debugging the Hyper-V guest VM. During this time, the guest VM will be stalled and hence any RDP sessions to the guest VM will timeout and disconnect. Hence, "Enhanced Session Mode" must be turned off in Hyper-V. This can be achieved by running the following command in an administrative PowerShell command window.

Set-VMHost -EnableEnhancedSessionMode $False

9. Hyper-V virtual switch configuration

The guest VM must be in a predictable state for the hands-on lab exercises, so Windows updates must not be applied after installing the guest OS. The best way to ensure this is to prevent the guest VM from connecting to the Internet. However, the guest VM must be on a network to share files with the host system. Modern virtualization software provides an internal network switch that allows only guest-to-host networking. In Hyper-V, this is the "Internal Network" virtual switch. This switch facilitates communication between Hyper-V guests and the Hyper-V host using Automatic Private IP Addressing (APIPA) (i.e. IP address in the 169.254.X.Y range). Such a network switch can be created by running the following command from an administrative PowerShell command window. The network switch has been named LABNET.

New-VMSwitch -SwitchName "LABNET" -SwitchType Internal

If you are using Hyper-V, avoid adding any "External Network" virtual switches as they may cause problems with some DHCP servers.

10. Hyper-V VM configuration

You will need a guest VM with at least 2 virtual CPUs, 2GB virtual RAM, and a 30GB virtual hard drive. You must configure the virtual network adapter to connect to the "Internal Network" virtual switch, turn off UEFI Secure Boot in the guest VM BIOS and configure the VM to boot from a Windows 10 20H2 64-bit bootable ISO image.

To create such a VM in Hyper-V, run the following set of PowerShell commands from an administrative PowerShell command window. The name of the Hyper-V VM is set to WINLABVM. You can customize the variables $vmname, $switch, $vhdpath, and $isopath, as per your requirements.

$vmname = "WINLABVM"
$switch = "LABNET"
$vhdpath = (Get-VMHost).VirtualHardDiskPath + $vmname + ".vhdx"
$isopath = $env:userprofile + '\Downloads\en_windows_10_business_editions_version_20h2_x64.iso'
$pipename = "\\.\pipe\" + $vmname
New-VHD -Dynamic -SizeBytes 30GB -Path $vhdpath
New-VM -Name $vmname -MemoryStartupBytes 2GB -Generation 2 -SwitchName $switch -VHDPath $vhdpath
$dvddrive = Add-VMDvdDrive -VMName $vmname -Path $isopath 
Set-VM -VMName $vmname -AutomaticCheckpointsEnabled $False -CheckpointType Standard -StaticMemory -ProcessorCount 2
Set-VMFirmware -VMName $vmname -EnableSecureBoot Off -BootOrder $dvddrive
Set-VMCOMPort -VMName $vmname -Number 1 -Path $pipename
Set-VMVideo -VMName $vmname -ResolutionType Maximum -HorizontalResolution 1280 -VerticalResolution 720

11. Connect to the VM console

Since Enhanced Session mode has been disabled, you must connect to the VM is through a local console. This can be achieved by running the following command in an administrative PowerShell or CMD command window.

vmconnect.exe localhost WINLABVM

12. Start the VM

At this point, you can start the VM, so it can boot up from the Windows bootable ISO image to begin the Windows installation process. This can be achieved by running the following command in an administrative PowerShell command window.

Start-VM WINLABVM

VM guest system

13. Guest OS installation

Install Windows 10 Version 20H2 X64 Professional or Enterprise Edition in the guest VM from the OS installation image (.ISO). You can also obtain this .ISO from MSDN. Windows need not be activated. A trial version of Windows will also work fine. When setting up the user account, please specify a password since Windows does not allow shared folders to be accessed with accounts that do not have a password. You can use student as the account username, password, and the answers to the security questions in the VM.

NOTE: Steps 14 through 24 have been automated using a PowerShell script. The script will change the computer name of the Guest VM to WINLABVM. If you prefer a different name, you can modify the script file. Steps 14 through 24 are independent of the virtualization software you are using since these steps are executed in the Guest VM. If you prefer to continue configuring your VM using the script, download Setup-VM-WINLABVM.ps1 to the host.

If you are using Hyper-V, you can execute the script in the Guest VM using the PowerShell Direct service by running the following command in an administrative PowerShell window on the host. When prompted enter the account name and password which you used while installing Windows in the Guest VM.

$vmcreds = Get-Credential
Invoke-command -VMName WINLABVM -Credential $vmcreds -FilePath Setup-VM-WINLABVM.ps1

If you are using VMWare or Virtual Box, you can copy Setup-VM-WINLABVM.ps1 to the Guest VM, start an administrative PowerShell window on the Guest VM, change to the directory containing the script, and run the following commands:

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser -Force
.\Setup-VM-WINLABVM.ps1

Once you have successfully executed the script Setup-VM-WINLABVM.ps1, you can proceed to step 25 in the Setup verification section.

14. Enable PowerShell script execution

You will need to run PowerShell scripts (.ps1) in the guest VM. So, you must enable script execution by running the following command in an administrative PowerShell command window in the guest VM.

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser -Force

15. Enable File and Printer Sharing

In the hands-on lab exercises, you will need to copy files from the host to the guest VM. Therefore, you must enable the Windows Firewall "File and Printer Sharing" rules for the Public network profile which should have been automatically assigned to the network interface in the VM. To enable all the firewall rules for the display group "File And Printer Sharing" run the following command in an administrative PowerShell command window in the guest VM.

Set-NetFirewallRule -Group "@FirewallAPI.dll,-28502" -Profile Public -Enabled true

16. Shared folder setup

Create a folder c:\pub in the guest VM. Share this folder with the host by running the following commands in an administrative PowerShell command window in the guest VM.

New-Item -ItemType "directory" -Path "c:\pub"
New-SmbShare -Name "pub" -Path "c:\pub" -FullAccess "Everyone"

17. Setting up the debugger symbol path

Since the Guest VM will not be connected to the Internet, the symbols for the binaries on the guest required by the debugger and other tools in the guest VM will have to be manually copied from the host to the c:\pub folder. The environment variable _NT_SYMBOL_PATH must be configured to point to c:\pub\sym, in the guest VM.

[System.Environment]::SetEnvironmentVariable('_NT_SYMBOL_PATH', "SRV*c:\pub\sym", 'Machine')

18. Windows Defender exclusions

Throughout the training, you will be copying tools from the host to the Guest VM that may be blocked by Windows Defender. Therefore, you must add the shared folder c:\pub to the Windows Defender list of paths excluded from scanning. You can do so by running the following command in an administrative PowerShell command window in the guest VM.

Add-MpPreference -Exclusion Path "c:\pub"

19. Kernel debugging configuration

Examining the system through a kernel debugger is an integral part of CodeMachine training courses. The guest VM must be configured for kernel debugging through the boot configuration database (BCD). The commands listed below create three different boot manager profiles using BCDEDIT:

  • The "original" profile does not have any debugging settings enabled.
  • The "testsign" profile has test-signing enabled.
  • The "debugger" profile has kernel debugging enabled and is the default profile.

Kernel debugging is configured to work over serial port COM1 at a baud rate of 115200. The following commands, which must be run in an Administrative CMD prompt in the Guest VM, configure the system for live kernel debugging.

bcdedit.exe /copy {current} /d "Windows 10 [original]"
bcdedit.exe /set {current} testsigning  ON
bcdedit.exe /copy {current} /d "Windows 10 [testsign]"
bcdedit.exe /debug {current} ON
bcdedit.exe /set {current} description "Windows 10 [debugger]"
bcdedit.exe /dbgsettings serial debugport:1 baudrate:115200
bcdedit.exe /set {current} recoveryenabled No
bcdedit.exe /set {current} bootstatuspolicy IgnoreAllFailures

20. Debug message settings

In the programming-focused training courses, you will call the classic kernel API DbgPrint() to display messages in the kernel debugger. For the kernel debugger to display these messages, the appropriate "Debug Print Filter" must be set in the registry. To enable this registry setting, run the following commands in an administrative PowerShell command window in the guest VM.

New-Item -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager' -Name 'Debug Print Filter'
New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Debug Print Filter' -Name 'DEFAULT' -Type DWORD -Value 0xffffffff 

21. Non-Maskable Interrupt Configuration

Run the following commands in an administrative PowerShell command window in the guest VM to configure the guest to bugcheck upon receiving an interrupt (NMI) from the host.

New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl' -Name 'NMICrashDump' -Type DWORD -Value 1

22. System memory dump settings

Run the following commands in an administrative PowerShell command window in the guest VM to configure the guest OS for memory dump retention, turn off auto-reboot during bug-check, generate complete memory dumps, and always allow memory dumps to be created.

New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl' -Name 'AlwaysKeepMemoryDump' -Type DWORD -Value 1
Get-WmiObject Win32_OSRecoveryConfiguration | Set-WmiInstance -Arguments @{ AutoReboot=$False }
Get-WmiObject Win32_OSRecoveryConfiguration | Set-WmiInstance -Arguments @{ DebugInfoType=1 }
Get-WmiObject Win32_OSRecoveryConfiguration | Set-WmiInstance -Arguments @{ OverwriteExistingDebugFile=$True }

23. Increase the size of the pagefile

Turn off system-managed pagefile and increase the size of the paging file to accommodate a complete system memory dump.

Get-CimInstance Win32_ComputerSystem | Set-CimInstance -Property @{AutomaticManagedPagefile=$false}
Get-CimInstance Win32_PageFileSetting | Set-CimInstance -Property @{InitialSize=2064;MaximumSize=2064}

24. Renaming the guest OS

Upon installation, Windows automatically assigns a computer name to the guest VM which is not too memorable or user friendly. It is best to change the computer name of the Guest to something that is easier to remember, considering that you will use this computer name often to copy files from the host to the guest. You can change the SMB name of the guest to WINLABVM by running the following command in an administrative PowerShell command window in the guest VM. DO NOT RESTART the guest VM yet.

Rename-Computer -NewName "WINLABVM"

Setup verification

25. Shared folder verification

Verify that the share ("pub") is accessible from the host system by attempting to access \\169.254.X.Y\pub from the host. Where 169.254.X.Y is the APIPA IP address assigned to the guest by the internal virtual network switch. You can obtain this IP address by running the following commands in an administrative PowerShell command window in the guest VM.

Get-NetIPAddress -AddressFamily Ipv4 | Select-Object InterfaceAlias, IPv4Address

26. Testing the kernel debugging setup

On the host system, start WinDBG Preview with administrative privileges. Download the workspace settings file WINLABVM.debugtarget. Click on "File" -> "Open Workspace" and load the workspace file WINLABVM.debugtarget. This workspace file configures WinDBG to connect to the named pipe WINLABVM which is mapped to the serial port COM1 of the Guest VM. If you have used a different pipe name for the serial port, then change the following line in WINLABVM.debugtarget.

<Property name="ConnectionString" value="com:port=\\.\pipe\WINLABVM,baud=115200,pipe,reconnect" />

Restart the guest VM. As the guest VM starts up, ensure that WinDBG displays "Kernel Debugger connection established" in the main Window.

A working kernel debugging setup with proper symbols is critical to the hands-on lab exercises. CodeMachine courses are intensive and run on a tight schedule. There is little time during the class to deal with system or software configuration issues. So, please make sure you test your kernel debugging setup before the class.

Look forward to seeing you in class.