Hi,
Rogue antiviruses are ALWAYS packed with,usually UPX or other static packers(e.g mystic compressor),but most of the times they use custom packers.So,antiviruses' hashes fail greatly,in an epic way(i am serious).I guess,pattern recognition could work,the chances are too low though.The next way to stop it is,malware's presence in memory,but then it would actually be way too late.
I don't feel like discussing anything about Sandboxes and virtual machines since those variants have been in the wild for more than 5 years(Generally speaking about rogue av's).The concept is that,the viruses install themselves,you don't install them,therefore you don't sandbox them.