Update...
It started doing it again, but this time no VPN had been connected since reboot (the OpenVPN TAP adapter is enabled but "Network cable unplugged")
Apparently disconnecting and reconnecting my wireless connection was enough.
In this case it actually stopped by itself after around 5 minutes.