NOTE: It is recommended that you use VMware for this I have NOT tested this on any other virtualisation software.
1. VirtualKD:
Giving the fact that this is a straight forward installation we will not be covering this in depth, you can find information and the download link on the official website at:-
http://virtualkd.sysprogs.org/
Important notes:
- This only works between a HOST and Virtual Machine it will NOT WORK between two virtual machines.
- Very fast debugging compared to Serial Ports.
- Only works on a Windows host so if you with to do this on OS X or Linux this will not help you. (There might be other similar software to help you achieve this however I am not aware of any)
2. Serial Ports:
This method even though it's documented I found that most of the online sources I found were missing different steps or were covering an older version of VMware. For the following example we will use the following names:
"DEBUGEE" - Machine to be debugged.
"DEBUGGER" - Machine which runs the debugger.
Make sure you have Windows Debugging tools installed on the debugger, if you do not you can download and install it at the following url:-
http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx
The next step is to edit the .vmx file of both debugger and debugee, before saving the changed make sure you have no serialport0 line before.
WINDOWS:
* DEBUGGER:
serial0.present = "FALSE"
serial1.present = "TRUE"
serial1.fileType = "pipe"
serial1.yieldOnMsrRead = "TRUE"
serial1.startConnected = "TRUE"
serial1.fileName = "\\.\pipe\D:\windbg"
serial1.pipe.endPoint = "client"
* DEBUGEE:
serial0.present = "FALSE"
serial1.present = "TRUE"
serial1.fileType = "pipe"
serial1.yieldOnMsrRead = "TRUE"
serial1.startConnected = "TRUE"
serial1.fileName = "\\.\pipe\D:\windbg"
Of course the fileName should be a valid path.
OS X / LINUX:
* DEBUGGER:
serial0.present = "FALSE"
serial1.present = "TRUE"
serial1.fileType = "pipe"
serial1.yieldOnMsrRead = "TRUE"
serial1.startConnected = "TRUE"
serial1.fileName = "/private/tmp/windbg"
serial1.pipe.endPoint = "client"
* DEBUGEE:
serial0.present = "FALSE"
serial1.present = "TRUE"
serial1.fileType = "pipe"
serial1.yieldOnMsrRead = "TRUE"
serial1.startConnected = "TRUE"
serial1.fileName = "/private/tmp/windbg"
The same thing applies here, fileName should be a valid path.
Now there is only one step left to do and that is to edit the c:\boot.ini on the debugee and add a line as follows:
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows Server 2003 Debug" /fastdetect /NoExecute=OptIn /debug /debugport=com2 /baudrate=115200
The above method only applies to Windows XP/2003 for further version you can use bcdedit.exe as follows:
bcdedit /debug on
bcdedit /enum (to see if debug mode is on)
bcedit /dbgsettings Serial debugport:<com_port> baudrate:115200
The last thing you need to do is open Windbg on your debugger and go to "File -> Kernel Debugging" (make sure you select com2 on port) and reboot your debugee machine.