Yes, I know that this is an old thread. I read all messages in all threads that contain "NTVDM" and this is the one that seemed most appropriate. I have avast 4.1 Pro.
No, I don't have any other antivirus and never did. This machine is less than 2 months old. NTVDM was working before, including after installing SP2. The error started immediately after installing avast. (I know it now because this is the second time that I install everything, so I was testing the NTVDM at every step.)
I have another machine (one year old) also running WinXP, on which I have NAV instead of avast and there is problem with NTVDM on that one. But I was planning on buying avast Pro for this machine.
I modified the VDD value in the ...CurrentControlSet... only, because that is how it is specified in your .reg file, and the value was changed automatically in the other ControlSetnnn keys, as expected. I have only ControlSet001 and ControlSet003. The VDD value is still correct in all three keys after a reboot.
As I said before, after modifying the VDD value, I no longer get the error message about NTVDM having "encountered an illegal operation". When I launch a program that requires NTVDM, the avast icon activates, but nothing else happens. There is no message from avast, even though I turned on the option to "Display detailed info on current action" [translating from French] in every module that has this option.
I hope I won't have to uninstall avast. I had already started recommending it to friends and colleagues.