Correct, you MUST disable the setting in the console, otherwise the client settings are overwritten at the next refresh interval.
its always good to test on one client (for exclusions, etc) but always remember to make the change in the console to make it permanent