Avast WEBforum
Other => General Topics => Topic started by: chocholo on May 02, 2012, 11:21:48 AM
-
Hello,
As I've promised in the very brief error message, I am going to explain what happened and why it took us two days to get the avast! Forum back.
Neither our forum nor our provider was hacked or DDOSed. The forum is currently hosted on our own servers, connected to our own network (we have our own AS numbers and peerings) in our rack, currently in our HQ in Prague.
On Monday morning, the forum server crashed during backup, and it unfortunately somehow corrupted the mapped iscsi volume with the forum data (we are investigating with the vendor how that could happen). What was worse, the backup iscsi volume also was mapped to the forum server and corrupted. So, because of this, we didn't have a fresh (few hours old) backup to restore. We had only a backup a few weeks old.
However, I found that it would be possible to get access to the freshest (online) data from the time when the server crashed. Because of this, I didn't restore the old backup and waited to see how the restoration would finish. Fortunately, it finished OK, so we now have the forum back as it was on Monday morning, just before crash.
I was called on Monday around 11:00 that the forum was not behaving normally, and it took approximately two hours to investigate what exactly was wrong. Some eight hours I was trying to find out how we could fix it by ourselves, then I contacted the vendor. That was the most time-intensive aspect, and after a few e-mails we managed to fix the iscsi volumes. That was some 30 minutes ago.
I apologize for all the troubles,
Petr Chocholous
jabber/xmpp: chocholous at avast.com
email: chocholous at avast.com
IT crowd
AVAST Software a.s.
-
I forgot to thank you all who notified me about the problem.
-
I forgot to thank you all who notified me about the problem.
NP Petr. ;)
-
Thanks for letting us know Petr :)
-
Thanks for the update, though I was getting the server is under severe load, try again later for searches/show last topics, etc. error message when I first managed to get on-line.
This was also happening some time before the crash, so I don't know if that was a prelude/warning of any imminent problem/crash.
-
Thanks for letting us know. :)
-
Petr... Thanks for your hard work. Glad that the data is safe.
But... Would we go for a Plan C backup scheme?
-
Thanks Chocholo.
Hell of a labor day, ah!
-
I only had Vlk in my contacts that i could remember right away so i've sent him a mail. :)
-
Yes, missed the reading. ;)
Hard work,
THanks
-
Thanks for the hard work chocholo, we are glad to see the forum is back. :)
-
Petr... Thanks for your hard work. Glad that the data is safe.
But... Would we go for a Plan C backup scheme?
Yep, we already changed the backup scheme.
-
***
My thanks also, Chocholo, for the hard work and quick response by you and those who helped.
***
-
Hi chocholo,
That meant some struggling with the back-up "table". Thanks for bringing it back up again,
polonus
-
Many thanks from me too, and welcome back. Still getting your error message yesterday was surprising, since repairs don't usually take that long, but we certainly appreciate the detailed explanation.
-
Thanks Chocholo.
Having my two favorite forums down was not a pleasing experience for me.
DSL Reports has been down for over two weeks. We have gotten a promise that it will be back up today. Fingers crossed.
-
Thanks for letting us know! :) I was worried that i was the only one seeing it. ???
-
Thanks for the info chocholo! :D
But it was a trauma and exhausting to not be able to check the forum when it's was down... ;)
But now I'm relaxed as the forum is working! :)
Keep up the good work!
-
Happy to see again the forum,
Thank's for restoring the forum chocholo! :)
-
Glad we can trust someone that maintains the forums as theirself. :)
For an unsuspected crash, 2 days was fast! ;)
-
1. Glad we can trust someone that maintains the forums as theirself. :)
2. For an unsuspected crash, 2 days was fast! ;)
Hi!
1: Excepted. :)
2: What happened really? Is it secret? ;) :-X
Greetings from
HDW38
-
What happened really? Is it secret? ;) :-X
Greetings from
HDW38
Only for you... :-*
http://forum.avast.com/index.php?topic=97902.0
Best regards
TerraX
-
<snip>
2: What happened really? Is it secret? ;) :-X
<snip>
You have read the first post haven't you ???
On Monday morning, the forum server crashed during backup, and it unfortunately somehow corrupted the mapped iscsi volume with the forum data (we are investigating with the vendor how that could happen). What was worse, the backup iscsi volume also was mapped to the forum server and corrupted. So, because of this, we didn't have a fresh (few hours old) backup to restore. We had only a backup a few weeks old.
-
thanks for the explaining and good job on the restoration :)