VBox DEP issue.

First of all here is a quick demo about the issue:

DEP Issue on VBox from sickness on Vimeo.

Ok now that you have seen the demo:

A lot of people Enable DEP on VirtualBox but did anyone test it to see if it works properly ? Well guess what it doesn't !

*How ?
Well I wanted to test some DEP bypass methods, and I just have this habit before actually trying to bypass DEP, I turn it off and test that app with a simple exploit (usually to launch calc.exe) but I forgot to turn DEP off, it remained on and when I launched the exploit I did not receive a DEP error and the calculator got executed, you can take a look at the quick video demo to check it out.

*What I did.
I started trying different things to make it work like upgrading to the latest VBox, made sure that my CPU supported NX, Enable PAE/NX from Vbox, reinstall the guest OS with PAE/NX enabled from the beginning, and others. Everything seemed ok but DEP was not working, I tried using tools like NXTEST which actually told me that DEP wasn't enabled so I tried the same configuration on Vmware and what do you know … it worked!
After saying that Vmware DEP works I asked a few people to help me confirm this Vbox issue. From the feedback of these tests I learned that only 32 bit CPU's are affected by this, DEP works on 64.

*Reason for this issue.
So after saying this issue I have reported it to the guys at Vbox who after a while told me that they have figured out what is causing the problem, it's normal but they have not documented it yet. (Great, because DEP not working is not such a big deal, what could happen !?)
QUOTE:
“For raw mode we do NOT enable NX protection by default. I'm currently not
aware of the exact reason but I believe this is to keep the code simpler
or there are some compatibility issues.”
(Yet this is not documented)

*Fix.
In order to fix this you need the following:
-NX and PAE support ( cat /proc/cpuinfo and check the flags for nx pae )
-A PAE enabled kernel ( which doesn't make much sense to me, Vmware DEP works without a PAE enabled kernel and NX and PAE are 2 different things from my point of view, but I might be wrong. )
-You also need to invoke a command from the terminal to enable NX because apparently the option “Enable PAE/NX” from Vbox doesn't work ( not sure why they included it in the first place)

Ok so in order to fix DEP in Vbox, your CPU must support NX and PAE, you must run a PAE enabled kernel and:
Open a terminal with the same privileges as your Virtual machines and type in:

"VBoxManage list vms"

You will get something like this: "Windows" {xxxxxx-xxxx-xxxx-xxxx-xxxxxxxx}
Now issue the following command considering Windows as the name of your VM (Virtual Machines must be stopped ):

"VBoxManage setextradata “Windows” VBoxInternal/CPUM/EnableNX 1"

Now start the VM and DEP should work.

Author: sickness