There's not much point in removing any of the main shields, if there not being used then there is no resource usage and you have the added protection that they will still pick up abnormalities in there area of protection, say if you don't use a mail client it is still beneficial to keep the mail shield just in case you end up with a spambot being installed.
Note that Network shield also scans web traffic so it shouldn't be removed, the auto sandbox shouldn't become annoying if it is left in it's default state of auto decision making, I never hear from it and it adds to the file reputation protection.