Web Shield with provider-proxy

[Fract504]

VLK: I try to see if this also happens with other proxies and not only with my provider, but I have to search a bit to find a public proxy I can use to test things further.
Until you have sorted things out, I will use your workaround. And by the way: Web Shield scanning is too cool!

[Fract504]

I tried a public proxy with the Web Shield turned on and this time all went fine.
But as one can imagine this public proxy was damn slow and not nearly as fast as the proxy from my provider. But at least all web pages loaded completely so far...
So maybe the problem is somehow speed related...

[Vlk]

Well I don't think so.
We've been stress-testing the WebShield proxy for months, and I can say it can very well handle loads MUCH MUCH higher than you're probably having with your home connection.

It's definitely something else - which is in fact proven by the fact that if you configure the things manually, everything works OK.

I'd wait for Lukor to come back from the mountains (he's in GaPa - on Tuesday. He'll be probably able to tell us more (he's the actual dev responsible for the WebShield, I'm just his (dumb) manager, y'know ).

Cheers
Vlk

[Fract504]

I'd wait for Lukor to come back from the mountains (he's in GaPa - on Tuesday. He'll be probably able to tell us more (he's the actual dev responsible for the WebShield, I'm just his (dumb) manager, y'know ).

PM me if you (or Lukor) have any questions, or if I should test something for you. I would be glad to help!

[essexboy]

Also having a problem with web shield on firefox,ie6 and yahoo browser where some sites will not load at all,  e.g. PCPitstop, beta news etc mainly sites with lots of graphics, but some with lots of graphics eg sg1archive .com loads OK .  Yet when I turn web shield off they all load fine and dandy.  However when I go into customize and delete the redirected port the pages load OK.  The question is do I need a port number in that section?

[Fract504]

Hi essexboy,

please read the topic here completely. Currently there only seems to be a workaround available by modifying the avast.ini and pointing the http-proxysettings of your browser to localhost:12080.

Seems that the Web Shield somehow treats proxy requests wrong. Will know more next week, when Alwil investigated.

[Fract504]

ok folks,

I did some in depth analysis using TCPView from sysinternals.com and packetmon from analogx.com.

Here are the hexdumps of the conversation between my browser and the proxy:

A good Request should look like this (Web Shield off):

Browser:
47 45 54 20 68 74 74 70 3A 2F 2F 77 77 77 2E 67  GET http://www.g
6F 6F 67 6C 65 2E 64 65 2F 67 72 70 68 70 3F 68  oogle.de/grphp?h
6C 3D 64 65 26 74 61 62 3D 77 67 26 71 3D 20 48  l=de&tab=wg&q= H
54 54 50 2F 31 2E 31 0D 0A 41 63 63 65 70 74 3A  TTP/1.1..Accept:
20 69 6D 61 67 65 2F 67 69 66 2C 20 69 6D 61 67   image/gif, imag
65 2F 78 2D 78 62 69 74 6D 61 70 2C 20 69 6D 61  e/x-xbitmap, ima
.....

Proxy:
48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D  HTTP/1.1 200 OK.
0A 44 61 74 65 3A 20 53 61 74 2C 20 31 39 20 46  .Date: Sat, 19 F
65 62 20 32 30 30 35 20 32 33 3A 31 32 3A 30 37  eb 2005 23:12:07
20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 2D 4C 65   GMT..Content-Le
6E 67 74 68 3A 20 31 37 36 30 0D 0A 43 6F 6E 74  ngth: 1760..Cont
......

The "Bad-Request" (Web Shield On and trapping proxy communicatio to port 8080):

Browser (notice the missing server in the first line):
7 45 54 20 2F 69 6E 74 6C 2F 64 65 5F 41 4C 4C  GET /intl/de_ALL
2F 69 6D 61 67 65 73 2F 67 72 6F 75 70 73 5F 68  /images/groups_h
70 2E 67 69 66 20 48 54 54 50 2F 31 2E 31 0D 0A  p.gif HTTP/1.1..
41 63 63 65 70 74 3A 20 2A 2F 2A 0D 0A 52 65 66  Accept: */*..Ref
65 72 65 72 3A 20 68 74 74 70 3A 2F 2F 77 77 77  erer: http://www
2E 67 6F 6F 67 6C 65 2E 64 65 2F 67 72 70 68 70  .google.de/grphp
.....

Proxy:
48 54 54 50 2F 31 2E 31 20 35 30 30 20 53 65 72  HTTP/1.1 500 Ser
76 65 72 20 45 72 72 6F 72 0D 0A 44 61 74 65 3A  ver Error..Date:
20 53 61 74 2C 20 31 39 20 46 65 62 20 32 30 30   Sat, 19 Feb 200
35 20 32 33 3A 31 37 3A 32 35 20 47 4D 54 0D 0A  5 23:17:25 GMT..
43 6F 6E 74 65 6E 74 2D 4C 65 6E 67 74 68 3A 20  Content-Length:
32 31 32 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 70  212..Content-Typ
65 3A 20 74 65 78 74 2F 68 74 6D 6C 0D 0A 53 65  e: text/html..Se
72 76 65 72 3A 20 4E 65 74 43 61 63 68 65 20 28  rver: NetCache (
4E 65 74 41 70 70 2F 35 2E 31 52 32 44 31 34 29  NetApp/5.1R2D14)
0D 0A 0D 0A 3C 48 54 4D 4C 3E 0A 3C 48 45 41 44  ....<HTML>.<HEAD
3E 3C 54 49 54 4C 45 3E 35 30 30 20 53 65 72 76  ><TITLE>500 Serv
65 72 20 45 72 72 6F 72 3C 2F 54 49 54 4C 45 3E  er Error</TITLE>
3C 2F 48 45 41 44 3E 0A 3C 42 4F 44 59 3E 0A 3C  </HEAD>.<BODY>.<
48 31 3E 53 65 72 76 65 72 20 45 72 72 6F 72 3C  H1>Server Error<
2F 48 31 3E 0A 3C 48 34 3E 0A 54 68 65 20 66 6F  /H1>.<H4>.The fo
6C 6C 6F 77 69 6E 67 20 65 72 72 6F 72 20 6F 63  llowing error oc
63 75 72 72 65 64 3A 3C 50 3E 0A 43 6F 75 6C 64  curred:<P>.Could
20 6E 6F 74 20 63 6F 6E 6E 65 63 74 20 74 6F 20   not connect to
74 68 65 20 73 65 72 76 65 72 0A 3C 2F 48 34 3E  the server.</H4>
0A 3C 48 52 3E 0A 50 6C 65 61 73 65 20 63 6F 6E  .<HR>.Please con
74 61 63 74 20 74 68 65 20 61 64 6D 69 6E 69 73  tact the adminis
74 72 61 74 6F 72 2E 0A 3C 2F 42 4F 44 59 3E 0A  trator..</BODY>.
3C 2F 48 54 4D 4C 3E 0A                          </HTML>.       


It seems that the request of the browser gets malformed or modified by Web Shield. The web-server is missing in the get request, so we get an error answer of the proxy.... But sometimes I also saw that the proxy didn't answer at all to this kinds of requests!

Alwil: Please also check, what happens to ssl and ftp requests, if they are also handled by the web shield and then forwarded to the provider-proxy... Most users (like myself) are lazy and simply check the mark "use the same proxy for all protocols" in their web browser...

But also maybe there isn't another solution for proxy users and we have to use the workaround of localhost:12080 and the avast.ini change. The workaround itself is quite logical and runs very fine indeed!

hmm... let's think about it....
In fact, if the proxy provider destination port is trapped to Web Shield (8080 in my case), then Web Shield has to handle and distinguish between the different protocols (http, ssl, ftp, gopher), because most of the provider-proxies use the same port for all protocols. Think of a person in a corporate lan, with only one proxy and only one port to talk to the internet... So, if Web Shield really expects only pure http-traffic to the "trapped" (or "redirected") port/ports, then there will be no other way, but the workaround....

Hope that Alwil can replay the problem.

[Vlk]

I didn't want to sink into the technical details previously - but since you already started...

The presence or absence of the host name (http://www.google.com) in the GET request is (by definition in the RFC) governed by the fact that the browser is, or is not, configure to use a proxy server. I mean, this is how proxy servers work: instead of getting

GET /index.html

they get

GET http://www.google.com/index.html

(the browser sends this) and they know they have to connect to google.com and request the page /index.html from it.
But if you configure the WebShield to use (monitor) port 8080 then it thinks that the http://www.google.com prefix was added because of himself and will strip it, as every proxy server should. Of course, UNLESS you speficy an upstream proxy, which OTOH tells the WebShield to forward the whole request unmodified to the host speficied in the upstream proxy settings.

So this is definitely nothing that would be surprising. To use a proxy server in conjunction with WebShield, you should configure the UpstreamProxyXXX settings in avast4.ini and reconfigure your browser to use proxy localhost:12080, and NOT tell WebShield to simply filter port 8080. Again, the reason is that port 8080 traffic is NOT a regular HTTP traffic, it's a proxy-HTTP traffic, which is different (has the full host names in the URLs).

I hope this makes sense to you.

BTW the WebShield should handle SSL and FTP requests if it is configured in the browser as a proxy. There's no way to do this when in transparent mode though (this applies mainly to SSL) - beucase then it's acting as a man-in-the-middle - something that SSL was designed to prevent in the first place.


Cheers
Vlk

[Fract504]

Hello VLK and good morning!

I totally agree with you and using the ini-settings and pointing the http-proxy to localhost:12080 works perfectly and I can live with this solution very well. I am not a programmer, but I think I understand how the Web Shield works. It simply wants to get pure http-requests on its redirected port (80 normally) and not messages which are designed for an upstream proxy.

[Vlk]

It can handle both but it has no way of knowing what to send upward. Whether to strip the host name from the URL (which is OBLIGATORY) for the case of direct connection (no upstream proxy)) or leave the host name in the URL (which is OBLIGATORY when an upstream proxy is in place).

[essexboy]

hi Fract 504 I have re-read the rest of the posts and tried the avast.ini fix but for some reason it does not appear to make any difference, however I must admit that I am a bit of a numpty where these files are concerned.  Ok this is what I did first I found out my IP address and inserted that where it says proxy then set web shield to port 8080 and got it to show all actions 
This is my avast ini section
 [WebScanner]
UpstreamProxyHost=host81-132-71-192.range81-132.btcentralplus.com
UpstreamProxyPort=8080

I then went to the troublesome sites and the pages loaded but there was no indication that web shield was working and the pages loaded ok.

I then removed the section from the ini file and reset the port to 80 the webshield showed that it was working but I could no longer access the problem sites unless I allready had the pages in my browser cache.  Additionally if I refreshed the page it all worked ok as long as I had a page in my cache.

Running WinXPsp2  Zone Alarm Pro  Firefox 1.0  IE6  Spyware Blaster Spyware Guard  All upto date as of this am


EDIT webshield has just caught a VBS trojan so obviously when you can get it working it is good.  Currently I am just suspending the service to load the first instance of problem sites then re-enabling

[Fract504]

Hi essexboy,

I can still not quite understand your problem, but hope I can help you nevertheless.

Here is how to use web shield:

1. If you have a direct connection to the internet, then just enabling web shield works fine.

2. If you are using a proxy to access the internet, then insert the address of that proxy into the avast4.ini:

[WebScanner]
UpstreamProxyHost=blabla.proxyprovider.com (put your proxy provider here)
UpstreamProxyPort=8080 (or whatever port the provider uses)

Point your webbrowser to use localhost:12080 as a proxy for the http-service.
Point all other services (ftp, ssl, gopher) to the proxy of the provider.

Just leave the redirected ports of web shield as they are (port 80 only).

[Vlk]

essexboy, the following was not very clear from your original post

1. What exactly is the problem? That web pages that are not in the local cache don't load (at all)?
2. Normally (without avast) - are you using a proxy server? (I mean, do you have a proxy server configured in your browser)?


Thanks
Vlk

[essexboy]

Hi vlk after investigation  (and questioning others) I find that I am not using a proxy I connect direct using DSL.   Therefore do I need a port number in the basic tab?

Fract504 Yes, unless the page is in the local cache before web shield is enabled it will not load, on the popup webshield just displays the main web address but  then ceases to load the remainder of the page.  If the page was previously in the cache the popup shows all the items as they load.  For example when I go to www.betanews.com the popup shows scanning www.betanews. com and the fails to download anything else. 

This problem only appears on a few web pages some of the others are pcpitstop.com, download.com,  yet there is no problem with grc.com, gateworld.net  or majorgeeks.com

Thanks for your help in this problem it's greatly appreciated. 

[Vlk]

Aha, essexboy, so it seems that your problem is actually unrelated to the proxy issues of Fract504 and frampo...

Maybe it would be better if you could start a new thread about your problem (some web pages not loading completely) - to keep this (bloated anyway) thread on-topic...

Thanks
Vlk