I tried using just Defender for a few days and it still suffers from the same problem that has been present from MSE days. I keep the installation files, mostly .exe's, for all of my installed programs in my Downloads folder. The first time that folder is opened after a restart, it takes a very long time for the contents to be shown. I get a small message saying "working on it" while the green progress bar creeps slowly along. With Defender off or disabled by another product, the folder loads much faster. The only real solution if you want to keep using Defender is to exclude the Downloads folder from being scanned. Of course this is a less than desirable scenario.
Defender has made big strides forward since the first MSE and is much lighter and faster while at the same time being more effective. The problem I gave, however, prevents me from using it as the default protection. The problem has been reported many times by many people with still no resolution.