Friday update for 12.2.2276 - “You are unprotected!” bug - what happened?

[petr.chytil]

What happened?
On Friday, July 29, Avast released a hotfix for the Avast user-interface (UI) component via the emergency update channel for Avast version 12.2.2276. This hotfix contained a bug which disrupted communication between the Avast Service and the Avast UI. Thus, the warning message "You are unprotected! Avast Background service is not running." was incorrectly displayed in the Avast UI. Despite this, the Avast service was running and you continued to be protected.

Why did it happen?
The emergency update channel is one of our regular update channels. We use it to release critical fixes and security updates between major version updates such as the above mentioned emergency update which rectified a troublesome behavior of the Avast UI. All hotfixes are subject to automated and manual testing by our QA engineers before being released via this channel. In this particular case, however, a slight difference between the testing environment and the production environment made it possible for this defective hotfix to enter production.

When did everything return to normal?
On Saturday afternoon at 2pm, a fix for the issue was released via the emergency update channel. This immediately resolved the issue for those who performed a clean installation after this time. For others, the fix took effect later, however, a reboot was still required to apply it. If you’re an existing user, and you’re yet to receive the fix, the fastest way to rectify the situation is to run the C:\Program Files\AVAST Software\Avast\AvastEmUpdate.exe file then reboot your PC.

Why won’t it happen again?
Over the weekend, we worked very hard to enhance the release, improve our quality assurance process, and resolve the bug. Despite this, I understand that it was even worse for some of you who endured a sleepless night trying to understand what was wrong while waiting patiently for Avast to release a meaningful explanation. At Avast, it is my responsibility to ensure bugs like this never make it to production. On this occasion, I obviously failed, and I sincerely apologize for such an error on my part. But apologies will not provide you with a better security product, so please let me share some of the changes we’ve made today and what we plan to change in the near future:

- Our rollback process for released emergency updates has been simplified and now takes just a few minutes to stop the distribution of an emergency update.
- Manual tests now include assessments within a production-like environment and automated tests are being improved in similar way.
- Monitoring of the production environment is being improved and we’re implementing immediate escalation to responsible personnel.
- The Avast UI will be improved to recognize whether there really is an issue with the protection or just a problem with the service <-> UI communication.

Last but not least, so that we are always available to respond to and rectify errors of this nature in a timely manner, we will ensure that releases are never performed in the evenings or before weekends unless absolutely critical.

Have a good day,
Petr

[RejZoR]

I've been mentioning "Self Diagnostics & Repair" for quite a while. An interval based diagnostics of all avast! subsystems (lets say every hour, maybe even less often) that can be connected to your servers for telemetry needs. If it finds an anomaly it should report to your service and try to fix it with "Repair" function (same as the one users can invoke manually). Then report again to your servers if status hasn't changed.

I think you guys could spot severe issues when all of a sudden huge numbers of clients become "unprotected", far outside of usual numbers of clients with such issues. Is this doable on your end? Doy ou think it could be beneficial to both, users and avast! team?

[Be Secure]

What happened?
On Friday, July 29, Avast released a hotfix for the Avast user-interface (UI) component via the emergency update channel for Avast version 12.2.2276. This hotfix contained a bug which disrupted communication between the Avast Service and the Avast UI. Thus, the warning message "You are unprotected! Avast Background service is not running." was incorrectly displayed in the Avast UI. Despite this, the Avast service was running and you continued to be protected.

Why did it happen?
The emergency update channel is one of our regular update channels. We use it to release critical fixes and security updates between major version updates such as the above mentioned emergency update which rectified a troublesome behavior of the Avast UI. All hotfixes are subject to automated and manual testing by our QA engineers before being released via this channel. In this particular case, however, a slight difference between the testing environment and the production environment made it possible for this defective hotfix to enter production.

When did everything return to normal?
On Saturday afternoon at 2pm, a fix for the issue was released via the emergency update channel. This immediately resolved the issue for those who performed a clean installation after this time. For others, the fix took effect later, however, a reboot was still required to apply it. If you’re an existing user, and you’re yet to receive the fix, the fastest way to rectify the situation is to run the C:\Program Files\AVAST Software\Avast\AvastEmUpdate.exe file then reboot your PC.

Why won’t it happen again?
Over the weekend, we worked very hard to enhance the release, improve our quality assurance process, and resolve the bug. Despite this, I understand that it was even worse for some of you who endured a sleepless night trying to understand what was wrong while waiting patiently for Avast to release a meaningful explanation. At Avast, it is my responsibility to ensure bugs like this never make it to production. On this occasion, I obviously failed, and I sincerely apologize for such an error on my part. But apologies will not provide you with a better security product, so please let me share some of the changes we’ve made today and what we plan to change in the near future:

- Our rollback process for released emergency updates has been simplified and now takes just a few minutes to stop the distribution of an emergency update.
- Manual tests now include assessments within a production-like environment and automated tests are being improved in similar way.
- Monitoring of the production environment is being improved and we’re implementing immediate escalation to responsible personnel.
- The Avast UI will be improved to recognize whether there really is an issue with the protection or just a problem with the service <-> UI communication.

Last but not least, so that we are always available to respond to and rectify errors of this nature in a timely manner, we will ensure that releases are never performed in the evenings or before weekends unless absolutely critical.

Have a good day,
Petr

+1

[Asyn]

Why won’t it happen again?
Over the weekend, we worked very hard to enhance the release, improve our quality assurance process, and resolve the bug. Despite this, I understand that it was even worse for some of you who endured a sleepless night trying to understand what was wrong while waiting patiently for Avast to release a meaningful explanation. At Avast, it is my responsibility to ensure bugs like this never make it to production. On this occasion, I obviously failed, and I sincerely apologize for such an error on my part. But apologies will not provide you with a better security product, so please let me share some of the changes we’ve made today and what we plan to change in the near future:

- Our rollback process for released emergency updates has been simplified and now takes just a few minutes to stop the distribution of an emergency update.
- Manual tests now include assessments within a production-like environment and automated tests are being improved in similar way.
- Monitoring of the production environment is being improved and we’re implementing immediate escalation to responsible personnel.
- The Avast UI will be improved to recognize whether there really is an issue with the protection or just a problem with the service <-> UI communication.

Last but not least, so that we are always available to respond to and rectify errors of this nature in a timely manner, we will ensure that releases are never performed in the evenings or before weekends unless absolutely critical.

Nice, let's hope for the best.

[Rednose]

Thnx Petr,

Among the Evangelists there was a discussion if we should have used the emergency procedure available to us to notify you guys.
As that procedure is already several years old, maybe it is time to re-evaluate it and give us clarification when to use it and when not.

Greetz, Red.

[lukas.hasik]

Yes, let's do it! Imo, you're doing great job.

Lukas

[Alikhan]

It's great to see an improvement in the way QA is handled.

I'd like to see staged updates too so if anything wrong did happen, it will be noticed and won't cause issues at such a large scale.

There could also be an option to roll back to a previous working version within Avast UI (troubleshooting) if an issue happens and then it auto updates when a newer version is released.

[RejZoR]

Mistakes and errors happen, but they learn from them. System wide false positives also yielded drastic measures to eliminate FP's on system files, enhancing their methods and mechanics to prevent them in the future. This is similar. A mistake was made, they acknowledge it and try to improve the whole process. Can't argue with that.

[REDACTED]

Petr, I just want to compliment you for owning up to the issue, investigating what went wrong, and instituting improvements!  This update summary is most informative and demonstrates a sincere commitment to full disclosure.  A lot of companies will just move on as if nothing has happen and not a word to the customers.  Communications is key in maintaining customer confidence.  We're all human, so things will go wrong from time to time.  Again, kudos!

[schmidthouse]

Yes, nicely done.
Good work!

[mchain]

Thank you for the information.  Mistakes happen.

[bob3160]

Thank you for the information.  Mistakes happen.

Thanks for the explanation. Part of the problem was a lack of communication between Avast USA
and Avast Prague. I was able to contact someone here in the states via phone but, that person apparently
had a problem getting the info. relayed to Prague.
As Red already mentioned, time to revamp the emergency communications.

[Staticguy]

@RejZor: Great, great idea. You the man 
To Avast Team: Please put this system into Avast 2017 

[NON]

Thank you for the explanation.

Thank you for the information.  Mistakes happen.

Thanks for the explanation. Part of the problem was a lack of communication between Avast USA
and Avast Prague. I was able to contact someone here in the states via phone but, that person apparently
had a problem getting the info. relayed to Prague.
As Red already mentioned, time to revamp the emergency communications.

+1

Petr, I just want compliment you for owning up to the issue, investigating what went wrong, and instituting improvements!  This update summary is most informative and demonstrates a sincere commitment to full disclosure.  A lot of companies will just move on as if nothing has happen and not a word to the customers.  Communications is key in maintaining customer confidence.  We're all human, so things will go wrong from time to time.  Again, kudos!

+1

[noelrojasc]

I just want to say "thanks" for your effort to offer us excellent tools to keep our Computers safe from malware.

You guys in Avast are doing a great job, congratulations!