Avast WEBforum
Business Products => Avast Business => Cloud Management Console & Clients => Topic started by: REDACTED on May 18, 2017, 06:21:31 AM
-
Hi All
I'm trying to diagnose a problem with System State backups on a Windows 2012 R2 server that has Avast for Business Cloud (Free).
Some time ago, the Windows July 2016 update rollup KB3172614 was installed but somehow was corrupted and caused a number of issues with the server, one of which was an issue successfully starting a System State backup.
Since then, the patch has been fixed but the System State backups fail for a different reason (different error) than previously. I have diagnosed the probable cause to be an bad enumeration of service imagepaths in the registry, and all the problematic paths seem to be related only to Avast service files.
The Avast program version was upgraded before the Windows patch issue was fixed, so I am wondering if the program upgrade introduced a new problem that I didn't have using the older Avast program version (ie when System State backups was still working). Since I had the problem with the patch for so long, it's difficult to say if this alternate error was introduced by the Avast update.
So, rather than bombard you with error logs and technical detail, my question is simply does anyone else use the latest AfB cloud (free) on Windows Server 2012 R2 and can still perform successful system state backups? I don't want to go reinstalling just now (production server), particularly if it turns out to be a bug.
Look forward to any results. 8)
-
I jumped ahead a little and found a VirtualBox image of Windows Server 2012 R2 and tested my theory. Here's my results:
Vanilla Windows Server install: System State backup works
Installed Avast Business Cloud 17.2.3419: System State backup fails to enumerate files
Uninstall Avast: System State backup works
Installed Avast Business Cloud 12.3.2515: System state backup works
I hope there are others out there that can validate this so I can report it as a bug with some evidence to back up my testing :)
-
Hi, one of our sub-resellers found this issue about 1 month ago in some of his customers. We communicated the problem then, using the support system, but due to the tests that they asked us to perform and to certain problems that were with the ticket system, the matter has been delayed more than anticipated.
Finally, yesterday we have received a confirmation that the issue has integrated into the system with an issue-ID (AV-13802). I hope this will mean that will be solved soon.
In any case, maybe it would be interesting to write to support providing your own data and evidence, and mentioning this thread and the ID I said.
-
Thank you Juanjo, glad to know its not just me going crazy ;D
Will definitely contact support and see what they say and use this issue ID as reference.
Did you happen to get a workaround for this?
If not, I'll see if I can figure one out. I'd really like to get this server back on track after the mess the bad Microsoft patch left behind. Like you say we might be waiting some time for a binary fix. I'm surprised there aren't more reports of this problem.
-
Hi:
No, we don't have a workaround right now, sorry.
"I'm surprised there aren't more reports of this problem" --> same here!!!
-
I contacted support and they confirmed the issue too, so I asked if there was a workaround and this was the reply...
Unfortunately, there is not a work around at the moment. However, with the soon release of the new version the bug will be fixed.
No workaround? Hogwash :)
I was also told no ETA on new version... oh please you can do better, Avast. Make a guess. 1 week, 1 month, 1 year?
So the problem I experience is the system state backup errors while enumerating a list of files to backup and gets part-way into a backup of a poorly enumerated path and fails with Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.
The workaround I suggest for anyone experiencing this problem is to uninstall v17.2 and install an earlier version. I tested with v12.3 and system state backup still works on my virtual test host. Make sure if you do this that the host is using a cloud settings template that does not perform automatic program updates or it will just upgrade again :) If system states still do not work for you, then there might actually be another problem with the host which is not possible to diagnose in this forum. Remember, always test on non-production first for your specific environment!
If you don't have an earlier version and cannot obtain one from support (usually they can supply if you need it) and cannot wait for a new version, the other workaround I found is modifying the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services registry. Because of the dangers here, I have opted not to provide specific instruction on how to change this. Doing an incorrect backup and edit of this part of the registry can make your system unbootable and is for experienced systems admins only. You have been warned, and I take no responsibility for any losses you might suffer by using the information I provide here! You'll also need to reboot for this registry edit to take effect and so downgrading the program version is just as disruptive on a production host and is why it is probably the better (supported) choice.
Here are the following keys that I discovered cause the system state to fail:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswbidsh
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswblog
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswbuniv
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswRvrt
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswVmm
You either need to change the ImagePath values to have a fixed path before the filename (eg C:\windows\system32\drivers\aswbidsha.sys), or, a more correct fix, to change the data type of the current ImagePath value from REG_SZ by deleting the value and recreating it as a REG_EXPAND_SZ.
I'm no expert and can't explain why loading on boot (ie, the value Start=0) does not have a problem with the current REG_SZ but system state does. And so for that reason a program downgrade is still my preferred choice as I expect when the fix is released in an upgrade it is going to upgrade properly.
I hope someone finds this useful.
-
Only a few words to say thank you very much for sharing all this useful information. Very interesting investigation and workarounds.
-
I have the same problem. I have Windows Server 2012 R2 and Avast for Bussines v17.4.2520.
I had discovered, watching the logs, that the faulty Avast driver was aswvmm.sys.
Excuse my english, i don't write it very well.
Look forward to an answer from avast support.
Greetings from Argentina.
-
Yep, I agree, it still fails, although I think on brief inspection it is only affected by aswRvrt and aswVmm now and not all the ones previously listed. So I'm not sure if the developers actually did try to address their bug, or if their testing is just inadequate.
I've emailed Tech Support back to ask what's going on, I suggest anyone experiencing this problem to do the same as the resolution to this really should be a high priority. Breaking a built-in OS component is bad enough but to then fail to fix it as promised is almost inexcusable.
Welcome to the forum @Maxi
-
Response from Tech Support:
This is a known issue that the development team is working on resolving at this time. We should have a fix pushed out with the next version update within the next business week.
Nice quick response too, I haven't lost all faith ;) If it's true, it sounds like it is being taken seriously. Considering they are planning another update soon there might be a couple of other serious bugs in the current version they are looking to fix too. Might hold off on my full site deployment for now :)
-
Well, at least 1 business week has passed, and no new version ... not surprised really :P
I contacted support again and was told the bug is still being worked on. So I guess I am just being told what they think I want to hear.
They did ask me to gather debug logs and send them in to be attached to the bug because they said: "the more we report a bug the higher it goes on the priority list".
So if you are experiencing this issue and have not done so already, contact support to create a case number, and ask to be added to the bug and request instructions on how to upload debug logs. The more people report it, the sooner it will get fixed... maybe :D
-
For those interested, the 17.5 release hasn't fixed the issue. I asked support about the current status and they said:
The developers have identified the specific error that causes this problem, and the fix was sent to QA for review yesterday. We don't have a release date for this fix just yet, but it should pass QA review and be included in our next major release (17.5) or the next release after that (17.6).
Since 17.5 was released before the QA review, it should be coming in the following release. I haven't seen a consumer beta yet, so that might still be a few months away. :-\
Unfortunately the only workarounds I know are either:
- keep running Avast 12.3 and ignore the danger message in the console (my current preference)
- do a full volume backup of the system drive without System State selected (but can complicate restore types)
- use a 3rd party tool
- use my registry fixes above (not recommended)
I'd be keen to hear about how others avoid the problem, eg what 3rd party tools they use for System State backups. :)
-
I have got the same issue. I have got 17.6 on a server and it is still an issue.
I have just contacted support about it as well.
Steve
-
I have got the same issue. I have got 17.6 on a server and it is still an issue.
I have just contacted support about it as well.
Yes I can confirm this still occurs on my test server too.
Thanks for logging it with support. The more people that report it, the higher priority it gets in being fixed.
-
I have got the same issue. I have got 17.6 on a server and it is still an issue.
I have just contacted support about it as well.
Yes I can confirm this still occurs on my test server too.
Thanks for logging it with support. The more people that report it, the higher priority it gets in being fixed.
Its a real pain. I have got Avast on about 15-20 servers currently.
Steve
-
Its a real pain. I have got Avast on about 15-20 servers currently.
Are all your servers 2012 R2 or this affects other server versions you have too? I assumed it was just a 2012 R2 issue given the low priority Avast have put on the bug.
-
Are all your servers 2012 R2 or this affects other server versions you have too? I assumed it was just a 2012 R2 issue given the low priority Avast have put on the bug.
I have got the issue with Server 2016, Server 2012r2, Server2012. I had contact from Avast last Tuesday/Wednesday but nothing since.
Steve
-
i have the same problem on 2008 r2, 2012r2 and 2016 servers. Is there any stable workaround?
-
i have the same problem on 2008 r2, 2012r2 and 2016 servers. Is there any stable workaround?
No, there is no official workaround, however downgrading to v12 works for me. Please log this fault with Avast support so that they prioritize this bug fix. The more reports they get the faster they will fix it, or so they tell me :-X
-
This works (with disable selfprotection) for now at least
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswRvrt]
"ImagePath"="system32\\drivers\\aswRvrt.sys"
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswVmm]
"ImagePath"="system32\\drivers\\aswVmm.sys"
-
Yes it works, but it's a risky unsupported fix and could render a server unbootable if done improperly :) Doing a product upgrade might also fail later on because of the changes and leave the server in a worse state. Thats the reason I chose not to do it on my server. So do take care and have a good alternate backup and recovery plan if this is a critical server.
-
The developers have identified the cause of the issue in the installer configuration and should be releasing a corrected installer as part of our next major update patch, or the major update patch after next. You shouldn't experience this issue again for this system unless you re-install the AV. Please let me know if these registry entries revert back to the broken path again.
I got this, it looks like allmost the same as you get) Lets see, what happens next version.
-
Does anyone know what is going on with this?
I havent had any update from Avast support for 3 weeks.
The way its going I will end up moving Anti-Virus software.
-
I just tried 17.7.2526 today; same problem on my test VM. Can anyone else confirm?
Very disappointing that a company with data security as it's core will consciously allow this problem to continue 6 months later. System State is an OS core feature, also with the aim of data security. It's not like it's a 3rd party backup Avast can blame.
I too will be shopping around for another vendor, but first I might post something and see what happens in the https://forum.avast.com/index.php?board=15.0 (https://forum.avast.com/index.php?board=15.0) Beta board since the cloud beta board is dead.
-
Please follow topic https://forum.avast.com/index.php?topic=210048.0 (https://forum.avast.com/index.php?topic=210048.0)
Looks like the target for the fix is 17.8
Exciting! ;D
-
Perfect. I will wait for that version.
Thank you.
Maxi
-
Its look the new version 7.8 has sorted the issue.
-
Yes I can confirm my Test VM is able to complete a System State backup now with Avast v17.8
Looking forward to finally installing to my Production host!
-
Perfect. Glad to hear it!
-
Unfortunately I have to post again in this thread. Our same reseller who discovered the bug months ago, informs us that since days ago he has observed that the problem is happening again in one of his customers.
This customer had the problem before, then it was fixed when it was announced here that it had been solved, and now it seems to happen again.
Have any of you noticed that the problem is happening again? Thanks for your information.
-
I am still running 17.8 and still working fine.
I assume you have 17.9. I refuse to let program autoupdates run on my network for reasons like this. :)
I suggest you set a policy to turn off program updates and add the server to it, then downgrade to 17.8 and see if the issue goes away. If not, you might have different issues (particularly if it's just one customer).
Thanks for the heads-up. I'll definitely test before I upgrade and report back if I have anything to report.
If you are convinced 17.9 regresses, please report the issue to support, as will I!
-
Hi, first of all, thank you very much for your reply.
The new bug (or the same repeated) was reported by one of our resellers (we are Avast Distributor), is not from a direct customer. We have asked this reseller again and he inform to us that the problem is happening now in 2 of his customers, both with Windows Server 2012. Because of this, and because he was one of the first persons to report this bug months ago, we give value to his words and we believe that it is possible that in some of the latest AfB versions there is a bug (again) with the backup of Windows Server.
Thank you for your tips, we will take them into account. And please, if you do some test with the latest version of AfB, I would appreciate it if you could tell us the result here.
-
I installed 17.9 to a test server and no problem. It was a fresh install and not an upgrade from 17.8 but previously it made no difference anyway.
It sounds like the customers are having different problems unrelated to this old bug, but uninstalling Avast could help prove or disprove it as being a cause.
Check the event logs for clues and the Windows Backup logs, as it might be failing but with a slightly different error. Make sure Windows Backup still works without system state to narrow down the problem. PM me if you need some help.
I'll eventually upgrade my production server and if I have a problem I'll be sure to let you know.
Good luck :)
-
Hi friend, thank you very much for your valuable information, it's very useful!
We will be in contact here if there are more to say about this.
Once again, thank you for your time and help.
-
system: Windows Server 2008 R2
Avast console on premise: 7.29.911
There is an error while trying to use Windows System State Backup:
wbadmin start systemstatebackup -backuptarget:f:
this command returns an error:
Error in backup of C:\program files\avast software\management console\\console\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.
I guest the problem is in doubled "\", but how to fix this?