The fact that you aren't playing a game doesn't matter as you have set it to 'silent/gaming' mode and that is the effect.
The other option which is enabled by default automatically detects when you are using full screen mode (normally for gaming). This applies gaming mode selectively, when there is no full screen application running it drops back to normal mode.