Win XP Kernel Paged Pool Memory Leak

[Geoffk]

Hello,

I recently upgraded from Win XP SP2 to SP3.  Since upgrading, I have had a problem with kernel paged pool memory being exhausted, resulting in the need to reboot.  I enabled symbols and pool tagging and determined through poolmon that the processes which were leaking had tags pSsA and AswR.  Both eventually grow to 133 Mb each, exhausting the pool of 264 mb.  These tags seem to correspond to the Avast Free drivers aswSP.sys and aswrlr.sys.

I uninstalled Avast, reinstalled it and upgraded to the latest version.  However, the kernel leak seems to still be present.  It takes a day or two, but the processes still grow out of control, requiring a reboot.

I use fairly few applications--mostly web browsing and videos from my local file server (with VLC).  I did a boot level virus scan and confirmed no viruses are present.

Any suggestions for how I can resolve this issue?

Thanks!

[Geoffk]

Some additional info:  This is XP Pro 32 bit SP3 on a Thinkpad R500 (Core2 Duo 1 GB RAM).  Local networking (Windows workgroup only) and internet.  Avast Free 7.0.1474.

[Geoffk]

Um, none of the Tech guys have any thoughts?  I'll be happy to provide additional info if you need it. 

After the two pools mentioned, the only other kernel paged pool with significant usage is Keys (registry key entries) whch seems to be about 50% of the size of the other two (i.e. growing to a max of about 70 Mb).  All others are tiny and stable.  Process Explorer shows the growth in the kernel paged pool also.  In Process Explorer, the Avast service process uses relatively little memory (20 Mb) and about 300 Mb of swap, which is not outrageous.  Stopping and starting the Avast service doesn't seem to free any kernel memory.

This problem is especially annoying as my file server PC, which is also XP 32 bit SP3, has no problems with the same version of Avast.  So I know it should work ok, but something is wrong here.  No other AV,  firewall (except Windows internal firewall) or spyware software is running.

Help!

[DavidR]

This is certainly above my general knowledge and many of the other volunteers.

I will try to attract some attention to your topic.

[Lisandro]

Geoffk, memory leaks are very tricky. pk is the one who helped me to troubleshoot.
You'll need to learn and use:
http://msdn.microsoft.com/pt-br/library/windows/hardware/ff550442%28v=vs.85%29.aspx
http://support.microsoft.com/kb/177415/en-us

[pk]

Geoffk: Can you please upload your kernel dump to our ftp? (https://support.avast.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=18) I'd examine pSsA/AswR blocks (probably related to registry filtering in behavior shield). Thanks.

[DavidR]

Thanks for joining the topic pk.

[Geoffk]

Thanks PK and to the others who replied as well.  I do have the system set up for full memory dumps, but the kernel error doesn't result in a BSOD--the system just stops being able to do networking and disk writes, the display starts messing up and it gets generally unusable.  However, I can still initiate a reboot, which will work (eventually) without a full-blown crash.  Hence, I don't think that I have a relevant dump that I can upload right now.

I can manually generate a dump for you when things are getting tight.  I haven't tested this, but I should have it set now so that CTRL-SCRLK manually fires off a dump.  Please confirm that this is Ok.  If so, you'll need to wait a day or so, as I have recently rebooted, and will need to wait for the processes to balloon up again.  Note that the Keys kernel process seems to increase in parallel to the other two, so you may be right about it being registry related.  The behavior shield is running, but just with the default settings.

Thanks again for your help!

[pk]

I haven't tested this, but I should have it set now so that CTRL-SCRLK manually fires off a dump.  Please confirm that this is Ok.

Thanks right, but it's little complicated & registry settings is different for ps2/usb keyboards.
You can crash your OS manually with this tool:
- http://public.avast.com/~kurtin/osrbang.exe
- http://public.avast.com/~kurtin/osrbang_x64.exe
run the tool & click on "CRASH" button

[Geoffk]

Thanks!  That's handy.  Right now I have
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters\CrashOnCtrlScroll set to 1, which is supposed to do it, but I don't crash my machine (on purpose) very often, so I haven't actually tested it yet...

As soon as the kernel processes get excessively large, I'll crash it and upload the dump for you.  Last time I checked, they were already up to 40 Mb each and easily the top two kernel processes in memory usage, but they'll get much bigger soon.  The one good thing about this problem is that it's consistant and easy to replicate.

Thanks again!

[Geoffk]

That was Fun!  I liked the 0xDEADDEAD stop code, although I think the 0xDEADBEEF code in AIX is more clever...

Anyway, the file Geoffk-11-21-2012-MEMORY.DMP has been uploaded to the ftp server.  I think the file transfer went ok.  The XP ftp command-line client is useless, so I eventually grabbed Filezilla, which seems tp have worked fine.  My system wasn't quite ready to crash when the dump was done..the kernel memory usage was about 250Mb, but it should look similar to the final state.  DUMPCHK says it's a good file.

Thanks again!

[pk]

The problem is, one of your process (vsnp2uvc.exe, CameraMonitor) opened 220k+ registry handles and it didn't close them (probably a bug in this software). As you can see, Windows allocated 223163 "Key" objects and avast allocated handle contexts (pSsA + AswR) with registry path (~ 239bytes per block => ~110 unicode chars per handle). Yes, these contexts are allocated from two separated parts in behavior shield (we'll reduce it to use only one handle context).

Tag       Allocs       Frees      Diff         Used
 pSsA     108005254   107782227    223027     54315168   
 AswR     108198978   107975600    223378     53407944   
 Key       50949280    50726117    223163     23208912   Key objects
 CMpb        634194      203475    430719     20674512   registry post blocks , Binary: nt!cm

PROCESS 833a8da0  SessionId: 0  Cid: 096c    Peb: 7ffdf000  ParentCid: 0494
    DirBase: 06980400  ObjectTable: e2b9ca18  HandleCount: 221462.
    Image: vsnp2uvc.exe

List of opened keys for vsnp2uvc.exe:

161e4: Object: e1c05628  GrantedAccess: 00020019 Entry: e41483c8
Object: e1c05628  Type: (8639e1e0) Key
    ObjectHeader: e1c05610 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \REGISTRY\MACHINE\SYSTEM\CONTROLSET002\CONTROL\CLASS\{6BDD1FC6-810F-11D0-BEC7-08002BE2092F}\0000\SETTINGS

161e8: Object: e503b720  GrantedAccess: 00020019 Entry: e41483d0
Object: e503b720  Type: (8639e1e0) Key
    ObjectHeader: e503b708 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \REGISTRY\MACHINE\SYSTEM\CONTROLSET002\CONTROL\CLASS\{6BDD1FC6-810F-11D0-BEC7-08002BE2092F}\0000

161ec: Object: e4905690  GrantedAccess: 00020019 Entry: e41483d8
Object: e4905690  Type: (8639e1e0) Key
    ObjectHeader: e4905678 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \REGISTRY\MACHINE\SYSTEM\CONTROLSET002\CONTROL\CLASS\{6BDD1FC6-810F-11D0-BEC7-08002BE2092F}\0000\SETTINGS

...

[Geoffk]

Wow, that's great!  Thanks for figuring this out.  I really need to learn how to read kernel dumps.  The program that you found was preinstalled by Lenovo as part of the integrated web-camera driver.  I killed the process, deleted all the copies of the file and applied the latest updated version of the camera driver from Lenovo's web page.  Interestingly, the latest version of the driver has a vsnp2uvc.dll, but they seem to have eliminated the exe file altogether.  If I still see problems, I'll uninstall the camera driver altogether.

I'm a little surprised that behavior shield doesn't really show that this process was misbehaving.  It seems like allocating hundreds of thousands of registry entries should be the kind of thing tht it flags and warns about.  But the entry history is mostly clean, with nothing suspicious indicated.  In this case, I'm glad that I found the problem anyway, but I wish it had warned me about it.

Anyway, thanks again for all of your help.  I really appreciate it!!