Please Remember To Fix The Outbound Email Scanning Problem

[hgratt]

Just a friendly reminder for the Avast team to not forget the problem with severe transmit speed degradation when outbound email scanning is enabled. See the following link for details:

http://forum.avast.com/index.php?topic=13520.0

Thanks,
Harvey

[lukor]

Harvey,

I have prepared a special optimized build for you to test it. As we don't see the same network speed slowdown it's a little bit hard to say if this modification would make any difference.

Please download this: http://www2.asw.cz/~rypacek/ashmaisv.zip

Stop Internet Mail Provider and unzip the file into the avast4 folder. Zipped are two files (ashMaiSv.exe and ashmaiSv.exe.sum). The .sum file is there to prevent file replacement during avast! integrity checks.

Then please edit the avast4.ini and add this line to [MailScanner] section:

[MailScanner]
BurstTransfer=1

and start Internet Mail again.

I have made some testings just now at my home DSL connection. Some were pretty nice but sending the mail again and again I got the high speed results for both the old and this modified transfer modes. So, maybe it will make any difference at your setup and maybe not.

Thanks for helping.
Lukas.

[hgratt]

O.K., did the patch you gave. No difference. With outbound email scanning enabled, I only get around 35-37KBS transfer rate. With outbound email scanning disabled, I get around 75KBS. I should be getting around 92KBS, apparently just having Avast installed is causing a degradation.

What upload speed does your DSL connection have? The 35-37KBS is pretty much near the maximum obtainable if your upload speed is 384kbs. I think you need a much higher upload speed to really see the degradation.

Thanks,
Harvey

[lukor]

Harvey, the transfer is done via one system call, TransmitFile(). According to the documentation it is the best possible method of transferring data when they reside in a file (a tempfile after being scanned in this situation). It involves just one transition to the kernel mode and uses file caching to the maximum possibly extend. Of course the network speed is rather slow when compared to the disk IO speed possibilities but with the TransmitFile() in this latest test-build there is nothing left in this area to optimize.

It must be something else!

To complete the testing I suggest you try to uninstall the Network Shield from avast! It is not monitoring outgoing connections (and their content) but no other avast! component is being involved during the SMTP send.

[alanrf]

Lukas,

in Avast mail scanning are there any "optimised interfaces" for differing mail clients?  I have the Microsoft clients in mind as I ask the question.  I assume they must feature pretty strongly in your development and testing.

If not, would you expect the scanning and transmission of the same message sent by different clients to be just about the same?

[lukor]

Alan, I don't know what exacly do you have in mind. Perhaps Vojtech might give us some in-depth info. But as far as I know there are no optimization for specific mail clients. The POP3 and SMTP interfaces (protocols) are described by their respective RFCs and avast! follows this standard.

Concerning the outgoing scan - it works like this:

1) client connects to the Internet Mail provider (either manualy configured or via redirect)
2) sends the mail via SMTP comamnds
3) Internet Mail stores it into temporary file and scans it
----------< up to here this should be very quick, if I neglect the time for scanning it is all on localhost

4) the Internet Mail provider connects to the SMTP server and starts sending mail
----------< here is the step when your's and Harvey's slowdown is observed

In this latest test-build (as I have written just in my previous message) the step 4 is replaced by single API call - TransmitFile(). This is the function used by IIS to transmit files (on Web server) - a highly optimized method of network transfers. And it does not involve any SMTP commands at all, it is just a plain byte burst on the network interface on a single line of source code. If this is not fast enough someone else is getting into the way...

[alanrf]

Harvey and I were both conducting our tests using Thunderbird.

I assumed that there would be no difference if I used another mail client to send the same message, but I thought I should test that assumption.

So I repeated the tests using Outlook Express.  The results surprised me.
I conducted the same tests multiple times to make sure they were consistent. 

Without Avast mail scan OE is about 40% slower in transmitting the same file attachment (pdf) than Thunderbird.

With Avast mail scan the slowdown for OE is much less than the slowdown for Thunderbird and OE becomes the clear winner.

One interesting note.  The transfer of the SMTP stream from OE to completion of caching by Avast took less than 2 seconds, the transfer of the SMTP stream from Thunderbird to completion of caching by Avast took 8 -9 seconds.     

I will repeat my tests today followng the instructions you gave Harvey to see if I can see any difference.

[hgratt]

Harvey, the transfer is done via one system call, TransmitFile(). According to the documentation it is the best possible method of transferring data when they reside in a file (a tempfile after being scanned in this situation). It involves just one transition to the kernel mode and uses file caching to the maximum possibly extend. Of course the network speed is rather slow when compared to the disk IO speed possibilities but with the TransmitFile() in this latest test-build there is nothing left in this area to optimize.

It must be something else!

To complete the testing I suggest you try to uninstall the Network Shield from avast! It is not monitoring outgoing connections (and their content) but no other avast! component is being involved during the SMTP send.

I uninstalled Network Shield -  no difference. I then uninstalled Avast - still only averaged around 83KBS.

I then re-installed NAV 2004 to test it out again. Amazingly, I was now back to getting around 92KBS!

All I can think is that NAV also fixed some bad registry entries since I'm getting better speeds with it installed than with no AV installed at all!

Sure would like to know what is going on.

Thanks,
Harvey

[alanrf]

Here are my test results for today.

Same message sent in each test (time in seconds).


                      Mail Scanner       Mail Scanner           Mail Scanner 
                            off                   4.6.665      Optimized,BurstTransfer=1

ThunderBird            88                    134                      131
 
Outlook Express    136                    130                      128
 
                   
The numbers for OE are in the correct columns.

[alanrf]

Lukas and/or Vojtech,

The Avast Internet Mail provider is in the unenviable position of having to emulate both POP3 server and client as well as SMTP server.

I say unenviable because the number of real products you are standing in for on both sides must be considerable. 

I understand the comment about compliance with RFC standards made by Lukas, but are you settling for "as good as Microsoft" or are you looking at "best of breed" candidates if another product might have found a more efficient way of getting the job done?

I will certainly understand if you come back and tell us that "it needs to be this way to meet the needs of most clients and most servers" - a kind of lowest common denominator. 

Frankly, in the cold light of day, while this is an interesting technical issue for me it is not a burning day to day performance issue for me or for those I support.  Most of us do not send enough large messages that even a 100% slowdown in transmission of the messages would warrant any real concern and - to return to an old refrain - especially at the price point many of us pay for Avast. 

I do not seek to downplay the real issue this may represent to other Avast users who may be running listservers etc. and who need to optimize mail transmission with the assurance of virus prevention.  I will leave this issue to them and the smart minds of the Avast team. 

I think I have another issue with the Internet Mail provider I need to devote some research time to.   
 
     
 

[hgratt]

Lukas, it would still be interesting to know what your DSL upload speed is. As I indicated before, I think that anything at 384kbs or lower will mask the problem.

Thanks,
Harvey

[Vlk]

Unfortunately Lukor is currently on a vacation so I guess I'll follow up.
I think he tested it here with 4mbit uplink.

alanrf is suggesting that the problem does not appear when using OE as email client (that is, with OE, it is always slow - no matter if avast is installed or not). Is that statement valid even on your machine?


Also, do I understand it correctly that you're getting these results even with outbound scanning disabled. What if you uninstall the Internet Mail provider completely - does that make any difference? And other providers...?

Somehow, we still don't know what component is causing the problem, exactly...


Cheers
Vlk

[hgratt]

Unfortunately Lukor is currently on a vacation so I guess I'll follow up.
I think he tested it here with 4mbit uplink.

alanrf is suggesting that the problem does not appear when using OE as email client (that is, with OE, it is always slow - no matter if avast is installed or not). Is that statement valid even on your machine?


Also, do I understand it correctly that you're getting these results even with outbound scanning disabled. What if you uninstall the Internet Mail provider completely - does that make any difference? And other providers...?

Somehow, we still don't know what component is causing the problem, exactly...


Cheers
Vlk

I have tried it uninstalling the components with the same results. If outbound scanning is disabled and/or pertinent components uninstalled, I still only get about 83KBS (kilobytes/sec) transfer rate (35KBS with Avast outbound email scanning enabled).

Even if I uninstall Avast completely, I still only get about 83 KBS. As I indicated in a previous post, I now have re-installed NAV 2004 and, amazingly, my outbound email speeds (with NAV outbound scanning enabled) is back up to 92KBS.

This would seem to imply that the NAV installation either fixed some bad registry entries created by Avast and/or NAV somehow handles transmissions more effectively. My guess is some registry settings (or equivalent) were "fixed" since I always get around 92KBS upload speeds when I test with various speed sites regardless of what AV is installed.

This is a very strange situation.

Also, FWIW:

1. I am using the Mozilla 1.7.8 email client
2. Several other individuals on a Comcast site also see the same problem.

I appreciate your looking into this.

Thanks,
Harvey

[Vlk]

I tried to simulate it here (with ThunderBird) but don't see any considerable slowdowns.
I'm measuring only the time from the Thunderbird progress bar reaching 100% to its closing - it's practically the same as with outbound scanning disabled (on this test machine).
How fast does the progress bar go on your machine? It should go really fast because it is transferring data only to the localhost avast proxy.

BTW

seem to imply that the NAV installation either fixed some bad registry entries created by Avast

I doubt that because that would mean that after you actually uninstall NAV, it would work just as fast.
Anyway, IMHO differences < 10% can be caused by just about anything...

[hgratt]

I tried to simulate it here (with ThunderBird) but don't see any considerable slowdowns.
I'm measuring only the time from the Thunderbird progress bar reaching 100% to its closing - it's practically the same as with outbound scanning disabled (on this test machine).
How fast does the progress bar go on your machine? It should go really fast because it is transferring data only to the localhost avast proxy.

BTW

seem to imply that the NAV installation either fixed some bad registry entries created by Avast

I doubt that because that would mean that after you actually uninstall NAV, it would work just as fast.
Anyway, IMHO differences < 10% can be caused by just about anything...

Need to make sure we're talking about the same thing.

When scanning is enabled, I first see a progress bar which, I believe, indicates the time being taken for the scanning to be performed (this process is relatively quick).

After that bar reaches 100% (I assume that the scanning is now completed), the actual outbound transmission starts up. It is then that I monitor my speeds via NetPerSec (and a stopwatch) - these are the speeds that I am reporting.

Thanks,
Harvey