Author Topic: Huge temp files are created and not removed  (Read 16591 times)

0 Members and 1 Guest are viewing this topic.

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #15 on: May 08, 2008, 10:43:11 AM »
Uhm... let me explain:

1.  AVAST scanning mailboxes, interrupted, I saw the temp files not removed.
2. I manually deleted the temp files.
3. In order to not keep on manually deleting those files I changed AVAST options so now the "mail" folder is excluded.
4. No more orphan temp files.

I say it again: mail excluded, no more files left in _avast_.

Now you may wonder why I am posting here if my local issue is "solved".
Simply because if I am having the issue it means other people can have it as well and IMHO it should be addressed.

I am happy you are not having the same issue but, you see, that is not particularly meaningful. Even if there were other 100 people who come here and say they aren't having any problem with it, still it doens't mean much.

I have a relatively small "mail" folder, the boxes are compacted, spam deleted, trash deleted, I've got only incoming and sent folders. In those folders there are my mail and some messages do have attachments. So what, can't I receive attachments? Those attachment are relatively small documents and images, some zipped, not a whole DVD content.

About skipping files, as I said above it would be much smarter to NOT scan MBOX files at all instead of decompressing, extracting, whatever it is done and then skip the contents. Incoming mail is already scanned, even if an attachement is executed the "standard shield" should stop it if infected. So it seems to me the scanning of the mailboxes is just a waste of resources/time. Expecially given the difficulties I am seeing.

Last thing: it is a little annoying when you are told between the lines that the issue you are reporting is not a flaw in the software but something wrong in you. Ok, I am not paying for AVAST but in the same time I am not payed to testing, bug finding, reporting and such.

I am seeing temp files more than 100MB left in _avast_.
I can reproduce the issue at will.
You say the issue doesn't exist? Ok, fine.

« Last Edit: May 08, 2008, 10:53:53 AM by LorenzoC »

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #16 on: May 08, 2008, 10:51:56 AM »
I will say once more and then leave you to your belief.

You can disbelieve the clear information I posted that shows you an exclusion - correctly set - causes avast to skip the Thunderbird mail files - it does not unpack them; it does not scan them. 

If you want to go on unpacking them - as your post above clearly shows you are - it is your system and your choice.  avast gives you a way to stop that - I showed you the evidence above.

There is no such thing as a standard for an MBOX file - it is just a plain text file.

     
 

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #17 on: May 08, 2008, 11:05:12 AM »
Ok, I am stupid

Now let people fill their disks with "_avast4_\unp*.tmp files".

When somebody complains of it you tell them you aren't having any problem.

Case closed.

On a side note, I may be stupid but  it seemed obvious to me that when I was asked to generate the report file I DID NOT exclude the mailbox, otherwise I could not generate any report on it. The exclusion I set is on the whole "mail" folder so nothing is scanned inside. The files "skipped" are reported with NO EXCLUSION set (and NO blocker either). Otherwise I would not say it is a joke (the irony to generate huge temp file to skip their content) either suggest to exclude them as default.

a. no exclusion -> scanning cancelled -> temp files left
b. exclusion -> no scanning -> no temp files left
c. testing to generate the report -> no exclusion -> scanning completed -> no temp files left -> skipped contents (due to scanner settings) in the report.

My current status b.
That translates in "better if AVAST doesn't see Thunderbird mailboxes".

Edit: speaking of me being stupid, I find a little difficult to understand the idea of a software that can detect rootkits but can't recognize a directory from a file (see blocker blocking directories because their names) or recognize a Thuderbird MBOX file.
« Last Edit: May 08, 2008, 11:46:28 AM by LorenzoC »

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #18 on: May 08, 2008, 11:44:30 AM »
I said:

Quote
I suspect that any abnormal termination of the avast scan might well leave one or more temporary files around that will not get cleaned up.

Quote
I do not doubt that you experienced those files but reproducing the condition does not appear to be easy.

I have not said that you did not experience this problem. I have not said that the problem does not exist.

The stopping the scans of the Thunderbird mail is a separate issue. 

In the end I was simply trying to assist you to really stop scanning the Thunderbird files and asking you to take a look at the avast4.ini file in Programs File\Alwil Software\Avast4\Data.

When you look at the exclusion to avoid scanning the Thunderbird Mail accounts in the Mail folder then please make sure that the exclusion setting ends  ....\Mail\* and that there is no space between the \ and the *.  I believe you will find that if you then create a report of your avast scan the part_..... files will not be created and no longer appear in the report.
« Last Edit: May 08, 2008, 11:46:15 AM by alanrf »

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #19 on: May 08, 2008, 12:05:49 PM »
Just some parting thoughts ...

I think that you misunderstand.  I was trying to recreate your problem because I support a bunch of people using Thunderbird and avast too.  If you can have the problem then so could they, if I could recreate the problem it would help me and them should they ever experience the problem too.

I could probably point you to the post long ago where I said something very similar about avast being able to recognize a Thunderbird MBOX in this forum.  I thought it would be easy ... but then I remembered that it is just a plain text file of email messages and that those messages can be in every language on the planet. 

I have actually written code to manipulate Thunderbird mail files for myself and I know it is a Thunderbird MBOX.  I would hate to have to write the code to recognize a Thunderbird MBOX in every language possible - since there is nothing in it to say it is a Thunderbird MBOX.  Thunderbird mail files do not need to have "Thunderbird" in their folder names anywhere either if you care to use the settings available in the Thunderbird client. 

 

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #20 on: May 08, 2008, 12:24:34 PM »
Ok, I put aside all the misunderstandings and summarize:

I was requested to create a report showing AVAST accessing the mailboxes during normal operations.
That is what I tried to do, with default settings, no exclusions or anything.

Pressing the button "cancel" in the scanner window is not what I would call "abnormal termination".

I guess "abnormal termination" is when you kill/terminate the process via task manager or such, which I never tried.

This kind of line:
\Mail\pop.gmail.com\Inbox\PartNo_0#599384397\PartNo_0#38486210\PartNo_0#3100651197 [E] File was skipped because of scanner settings. (42016)

Seems to show AVAST finds the "Inbox" MBOX file, extract/decompress something that is somehow nested, then decides to skip what it finds, my guess is because of the "standard scan" settings.

To extract the "PartNo*" things it creates the huge temp files in _avast_ and for some reason when you abort the scanning (pressing "cancel" button in the scanner window) before it comes to the true end AVAST isnt' able to properly terminate all operations and leaves the temp files in place.

My reasoning is:
1. I don't see anything strange in my mailboxes so I don't see any reason why AVAST should not be able to handle it, expecially if it is supposed to scan the mailboxes every scan as default.
2. It is not "good/normal" that, whatever AVAST is doing, it can't properly close the scanning task leaving the temps files behind (regardless the size).

Now, if the same issue or other issues have been reported about mail scanning and they had not been addressed, I guess it makes even more sense to exclude those dir/files from scanning.
I belive AVAST should be able to detect the mailboxes, but in my poor understanding you could simply add a procedure during install that scans the HD, finds mail client profiles and writes exclusion rules as part of the default settings. It is basic, you find "\Thunderbird\Profiles\" and exclude (with all the refining), you don't need to really detect the MBOX files.
« Last Edit: May 08, 2008, 12:29:25 PM by LorenzoC »

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #21 on: May 08, 2008, 12:51:40 PM »
Would you be willing please to post here the complete exclusions line from your avast4.ini file?

Then we will have a better idea of what is happening in scanning of your Thunderbird files, why it is creating the part_ files and then not scanning them.  I am fairly sure that you had other exclusions set when you posted the test results above, like excluding .msf files and part_ files. Not all the files were excluded in your report above (the Junk and Drafts folders were reported as scanned OK).

Again I wish I understood how you can recreate the orphaned avast files in the temp folder just by canceling the scan. I do not have a Win2K system or any FAT partitions to test on with my systems but perhaps this is a difference of avast installations in different operating systems 

avast knows nothing at all about any mail clients (and there are a lot of them) - with the exception of MS Outlook where it works with an interface provided specially for that product.  There are no standards in existence for email file structures within mail clients and no standards for the actual mail files themselves.  The only standards that every mail client must conform to are the POP and SMTP standards and it is those that the avast Internet Mail provider works with to ensure that the mail is clean before it is committed to whatever mail file format by whatever mail client you use.  avast simply does not know or care that these files belong to Thunderbird and it does not know Thunderbird is a mail client.     

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #22 on: May 08, 2008, 01:10:16 PM »
Not it is becoming funny... I made the report WITHOUT ANY exclusion, how many times should I tell? :)

Currently in the "ini" file there is only this line:
Exclude=C:\Documents and Settings\Administrator\Dati applicazioni\Thunderbird\Profiles\mc135pum.default\Mail\*

That is what I've already said, intended to exclude the whole "mail" directory and downwards.

I added the exclusion after I noticed the issue with temp files and reproduced it some times, that means the issue happens WITHOUT ANY exclusion, in default condition.

Now, "default" means the "standard scan" and it should detect file types and scan against the appropriate virus types. I don't know if this can make the scanner "skip" during mail scanning.

About AVAST not knowing anything, well, but we know something more. I understand you can't provide coverage for any possibile mail client but you could at least exclude Thunderbird or other KNOWN troubled clients. Like I said, I would simply write the exclusion rules during the installing of AVAST, simply looking to the users profiles.

Edit: the junk, draft, etc "folders", that are OK, are obviously EMPTY.
« Last Edit: May 08, 2008, 01:27:55 PM by LorenzoC »

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #23 on: May 08, 2008, 09:06:45 PM »
Ok, I made another round test with the reporting ON and NO EXCLUSION, this time I switched on "thorough" instead "standard", here is the report (I shortened the path):

* Task 'Simple user interface' used
* Started on giovedì 8 maggio 2008 20.56.50
* VPS: 080507-0, 07/05/2008
*

\Mail\pop.gmail.com\Inbox\PartNo_0#599384397\PartNo_0#38486210\PartNo_0#3100651197 is OK
\Mail\pop.gmail.com\Inbox\PartNo_0#599384397\PartNo_0#38486210\PartNo_1#3325552817 is OK
\Mail\pop.gmail.com\Inbox\PartNo_0#599384397\PartNo_0#38486210 is OK
\Mail\pop.gmail.com\Inbox\PartNo_0#599384397 is OK
\Mail\pop.gmail.com\Inbox is OK
\Mail\pop.gmail.com\Templates is OK
\Mail\pop.gmail.com\msgFilterRules.dat is OK
\Mail\pop.gmail.com\popstate.dat is OK
\Mail\pop.gmail.com\Inbox.msf is OK
\Mail\pop.gmail.com\Trash is OK
\Mail\pop.gmail.com\Trash.msf is OK
\Mail\pop.gmail.com\Drafts is OK
\Mail\pop.gmail.com\Drafts.msf is OK
\Mail\pop.gmail.com\Junk is OK
\Mail\pop.gmail.com\Junk.msf is OK
\Mail\pop.gmail.com\Templates.msf is OK
\Mail\pop.gmail.com\Sent\PartNo_0#1731396481 is OK
\Mail\pop.gmail.com\Sent is OK
\Mail\pop.gmail.com\Sent.msf is OK

Infected files: 0
Total files: 19
Total folders: 1
Total size: 421,0 MB

*
* Task stopped: giovedì 8 maggio 2008 20.59.32
* Run-time was 2 minute(s), 42 second(s)
*

Compare with the previous report line:
\Mail\pop.gmail.com\Inbox\PartNo_0#599384397\PartNo_0#38486210\PartNo_0#3100651197 [E] File was skipped because of scanner settings. (42016)

The only difference is the different scanning method.

In this test I made AVAST complete the scanning.

Then I tried stopping it before it completes and the two huge temp files are still left behind in the _avast_ folder, so there isn't any change about the main issue of this thread.

So now I see 2 issues:

1. in "standard scanning mode" it seems the mailboxes aren't actually scanned because it ends in the "skipping" of every file. In "thorough scanning mode" it seems the mailboxes are scanned instead.
2. in what ever mode, cancelling the scanning makes AVAST leaving the temp files in _avast_ folder.


« Last Edit: May 08, 2008, 09:14:30 PM by LorenzoC »

Offline igor

  • Avast team
  • Serious Graphoman
  • *
  • Posts: 11754
    • AVAST Software
Re: Huge temp files are created and not removed
« Reply #24 on: May 08, 2008, 09:19:23 PM »
Igor is familiar with the format of Thunderbird files ... he and I have discussed them a  number of times over the years in this forum.

I'm not very familiar with the format of Thunderbird files... but as far as I remember, there's actually nothing to be familiar about - cause there's no "format" here.
The mailboxes are just a concatenation of the messages (pretty stupid, if you ask me) - so if they contain some strange messages (I mean messages with strange structure of parts, attachments, ...) - so it's possible that the MIME unpacker, responsible for extraction of one message, gets confused and crosses the border between the messages, and extracts something strange.

If it extracts a huge file, it's an operation hard to interrupt. So, if you cancel the scan and the on-going operation doesn't stop within some time limit, it's terminated in a "hard way"... and that's when the temporary files are kept in the TEMP folder, I'm afraid.

The limit could be enlarged, but it doesn't really solve the problem. Maybe the MIME unpacker could be improved somehow (not to extract what it shouldn't - though I'm not 100% sure it's the case here) - but not without inspecting the particular Inbox file.

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #25 on: May 08, 2008, 09:29:31 PM »
Well, sorry but at this point I don't see what scanning the MBOX format is good for. Besides the fact the MBOX can't be properly handled, even if something wrong is detected It seems AVAST could not do anything about it in any case, besides trashing the whole file, so why even bother?

On a side note, I am not an expert but the "compacting" of the MBOX in Thunderbird should make sure the file is "cleaned" by any "strangeness", that is why I told I regularly compact the mailboxes.
For example long time ago I was using an antivirus who alerted me about viruses when the original message was  deleted and it did not once the mailbox was compacted, meaning before compacting the deleted messages were still present inside the actual MBOX file.

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #26 on: May 09, 2008, 03:24:20 AM »
Two main sections to this thread:

1) the scanning issue.

We who have been here a long time and have our own exclusion list built can sometimes forget what it is to be a new user of avast. Based on that we can also find our long held assumptions about the differences in the levels of scanning in avast challenged, as mine have been by LorenzoC.  So I must express my thanks to LorenzoC for making re-learn some of those differences.

I believe that, with the exclusion you posted earlier, you will find that none of the files or folders (ie for Thunderbird in this case the accounts) will be scanned by avast at all (in any on demand scan).

I hope we can agree on the basic issue in the scanning function - that avast has no value in scanning our Thunderbird mail folders in its on demand scans.

2) the "orphaned" temporary files

I think Igor gave us a very clear pointer in this:

Quote
If it extracts a huge file, it's an operation hard to interrupt. So, if you cancel the scan and the on-going operation doesn't stop within some time limit, it's terminated in a "hard way"... and that's when the temporary files are kept in the TEMP folder, I'm afraid.

All of my testing to try to reproduce your concern was done on a relatively new multi-processor system with fast disks. Even with this system when I interrupted the scan of the Thunderbird files I noticed a very distinct lag before avast cleaned up the temporary files and I heard the sound associated with the end of the avast scan.  Given the information from Igor I can believe that on an older system with slower hardware there could be a timeout condition in which the scan would terminate before all of the disk activity (including cleanup) could occur.  In such a case since the slowness of the hardware would be a constant then the problem would be easily reproducible - as you have reported.

In support of Igor's comment on the MBOX issue.  When I started to write my own code to manipulate Thunderbird's MBOX mail folders I was surprised by the lack of consistency in the way messages are concatenated in the MBOX file.  Sometimes there is a blank line between messages, sometimes not and I can easily understand how the avast unpacker can be confused by this inconsistency.  It takes real hard work to separate the messages properly especially when plain text messages are simply run together.  I think Igor is referring to this problem rather than any issue of compacting which simply removes the deleted messages from the file.

There is a constant tussle among the Thunderbird developers many of whom believe MBOX was a bad decision and want to replace MBOX with MailDir.  We will have to see how this plays in the development of Thunderbird now that this very small player in the email world has itself been orphaned by Mozilla casting it out from the organization. 

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #27 on: May 09, 2008, 09:11:19 AM »
Ok, understood.

Anyway the point here it is not how much Thunderbird sucks :) but how to make it live together with AVAST without issues and possibly without destroying your mail.

Inform users, work around limitations.

Offline alanrf

  • Avast Evangelist
  • Massive Poster
  • ***
  • Posts: 3867
  • Just an avast user
Re: Huge temp files are created and not removed
« Reply #28 on: May 09, 2008, 09:38:16 AM »
I have made my comments in the other thread you posted in.

Let's get one thing clear absolutely clear. avast has not come anywhere close to destroying your mail.

In this thread and now in others you appear determined to distort comments to fit your concerns.  I am saddened by your stance in this and I suspect that it may affect the level of assistance you may expect.
« Last Edit: May 09, 2008, 09:50:21 AM by alanrf »

Offline LorenzoC

  • Full Member
  • ***
  • Posts: 134
Re: Huge temp files are created and not removed
« Reply #29 on: May 09, 2008, 10:10:47 AM »
"you appear determined to distort comments"
This is a false accusation, you are simply being TECHNICALLY wrong in this thread and elsewere. Mail is not destroyed once the "Inbox" is moved to the "chest" but for the user it is not accessible until you move the "Inbox" back in place. Is that fun?

"I suspect that it may affect"
This is some sort of menace but I could care less. Infact your help wasn't either useful or requested. My post were not meant to ask for assistance, they were only ISSUEs reporting. I don't need any assistance, thanks.