Yes, that's right. Everything else is fine--you can go in if physically connected to the console. It's clearly one of the shields, maybe the network shield that is somehow blocking me from coming in (establishing a new remote session). I haven't tried it when leaving the remote desktop session running (turns out this is microsofts perfered way of doing it).
Problem is, it's hard to experiment since leaving the sheilds on I risk not being able to get back on the server.