System crashes after 24-48 hours

[timn]

Alex:

Thanks for your response. I will follow your recommendations.

I really appreciate the help you've given and I understand Host Monitor is not responsible for all the buggy drivers in the world

This could be a slow process but at least the failure is predictable now -- I simply have HM warn me when NP Pool reaches about 75% of crash size.

This has been a tougher problem than most - I suspect you are right about it being some newer driver.

I will pursue this on my end and report back when I learn more.

Tim

[timn]

OK, some additional info. Using:

"How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks"

http://support.microsoft.com/kb/177415/EN-US/

The following values were recorded over a 2-hour period. Clearly, the 3 big leakers of nonpaged pool bytes, in terms of Pool Tags are :

1. "Ddk"
2. "File"
3. "TCPt"

(Host Monitor currently has a 'burn rate' on nonpaged pool bytes of aroung 22MB per hour)

Code: Select all

 Tag  Type     Allocs            Frees            Diff   Bytes      Per Alloc
-------------------------------------------------------------------------------
 Ddk  Nonp     474780 ( 277)    396318 ( 249)    78462 30033696 (  9536)    382
 File Nonp     353989 ( 250)    274572 ( 200)    79417 12712608 (  8000)    160
 TCPt Nonp     631812 ( 602)    554436 ( 553)    77376 5034432 (  3136)     65

 Ddk  Nonp     487286 ( 213)    406779 ( 181)    80507 30753536 ( 11904)    381
 File Nonp     362523 ( 157)    281055 ( 126)    81468 13040768 (  4960)    160
 TCPt Nonp     647413 ( 384)    567986 ( 351)    79427 5165696 (  2112)     65

 Ddk  Nonp     523442 ( 297)    436931 ( 275)    86511 32862144 ( 12864)    379
 File Nonp     390995 ( 205)    303228 ( 186)    87767 14054848 (  3040)    160
 TCPt Nonp     695328 ( 364)    610001 ( 329)    85327 5543296 (  2240)     64

 Ddk  Nonp     551141 ( 419)    460084 ( 355)    91057 34463936 ( 25088)    378
 File Nonp     418760 ( 281)    326465 ( 228)    92295 14779328 (  8480)    160
 TCPt Nonp     732218 ( 483)    642376 ( 422)    89842 5832256 (  3904)     64

 Ddk  Nonp     570983 ( 190)    476651 ( 171)    94332 35614816 (  9568)    377
 File Nonp     433504 ( 173)    337929 ( 158)    95575 15304096 (  2400)    160
 TCPt Nonp     759642 ( 404)    666559 ( 381)    93083 6039680 (  1472)     64

 Ddk  Nonp     623375 ( 311)    520441 ( 253)   102934 38631840 ( 19456)    375
 File Nonp     472297 ( 221)    368142 ( 165)   104155 16676896 (  8960)    160
 TCPt Nonp     828733 ( 416)    727094 ( 363)   101639 6587264 (  3392)     64
 
 Ddk  Nonp     661179 ( 350)    552130 ( 314)   109049 40782400 ( 11072)    373
 File Nonp     499772 ( 246)    389567 ( 186)   110205 17644928 (  9600)    160
 TCPt Nonp     878199 ( 436)    770508 ( 382)   107691 6974592 (  3456)     64

 Ddk  Nonp     739239 ( 214)    617587 ( 168)   121652 45213536 ( 13632)    371
 File Nonp     556377 ( 187)    433548 ( 137)   122829 19664864 (  8000)    160
 TCPt Nonp     979615 ( 415)    859325 ( 373)   120290 7780928 (  2688)     64
 
 Ddk  Nonp     852702 ( 210)    712447 ( 175)   140255 51758592 ( 12640)    369
 File Nonp     640032 ( 168)    498669 ( 136)   141363 22630240 (  5120)    160
 TCPt Nonp    1129774 ( 330)    990932 ( 297)   138842 8968256 (  2112)     64 

According to MS:

Pool tags:

Code: Select all

Tag    Driver         Purpose
-----------------------------------------------------
Ddk  - <unknown>    - Default for driver allocated memory (user's of ntddk.h)
File - <unknown>    - File objects
TCPt - tcpip.sys    - TCP/IP Network Protocol

"Ddk" is the biggest loser of all. If I understand this, though, this is Microsoft's 'default driver kit' - an SDK for driver writers. So, seeing this tag show up still doesn't tell us a whole lot.

Next, using:

"How to Find Pool Tags That Are Used By Third-Party Drivers"

http://support.microsoft.com/kb/298102/EN-US/

I discovered that the drivers in C:\WINNT\System32\Drivers\ that reference Pool Tag "Ddk" are:

Code: Select all

   CV2K1.SYS       TamoSoft (CommView Driver)                 2.3.0.1
   DMBOOT.SYS      VERITAS (NT Disk Manager Startup Driver)   2195.6655.297.3
   DMIO.SYS        VERITAS (NT Disk Manager I/O Driver)       2195.6655.297.3
   FDC.SYS         Microsoft (Floppy Disk Controller Driver)  5.0.2195.6655
   MF.SYS          Microsoft (Multifunction Enumerator)       5.0.2195.6668
   NPF.SYS         CACE Technologies (WinPCap Driver)         3.1.0.27
   NULL.SYS        Microsoft (Null Driver)                    5.0.2134.1
   PTILINK.SYS     Parallel Technologies (DirectParallel IO)  1.1.0.0
   RDPWD.SYS       Microsoft (RDP Terminal Stack Driver)      5.0.2195.7055
   TDASYNC.SYS     Microsoft (Serial Transport Driver)        5.0.2195.6692
   TDI.SYS         Microst (TDI Wrapper)                      5.0.2195.6655
   TDIPX.SYS       Microsoft (IPX Transport Driver)           5.0.2195.6692
   TDNETB.SYS      Microsoft (NetBIOS Transport Driver)       5.0.2195.6697
   TDPIPE.SYS      Microsoft (Named Pipe Transport Driver)    5.0.2195.6692
   TDSPX.SYS       Microsoft (SPX Transport Driver)           5.0.2195.6692
   TDTCP.SYS       Microsoft (TCP Transport Driver)           5.0.2195.6692
   TS_LB.SYS       TamoSoft (CommView Loopback Driver)        1.1.0.4

Since the problem increases linearly with TCPIP traffic, we can probably eliminate from suspicion, any non-TCP driver in the above list.

The WinPCAP driver was recent addition so it is not the culprit. The CommView drivers are similar to Ethereal/WinPCap stuff

FYI, I have just uninstalled both Commview and Ethereal/WinPCap and rebooted... we'll see if that makes any difference.

Any thoughts on how to further trouble-shoot this? Seems we have confirmed that the memory leak is in a driver that is associated with TCP/IP

Tim

[timn]

Found it!

The source of the trouble appears to have been the CommView driver.

After uninstalling CommView and rebooting, here's the last hour's nonpaged pool (keep in mind - prior to uninstalling CommView, we were burning through the nonpaged pool bytes at 22MB/hour):

Code: Select all

[1/25/2006 5:43:30 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:44:31 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:45:33 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:46:35 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:47:37 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:48:39 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:49:40 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:50:43 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:51:47 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:52:48 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:53:51 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:54:54 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:55:56 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:56:58 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:58:04 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 5:59:07 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:00:09 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:01:12 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:02:13 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:03:15 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:04:19 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:04:58 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:06:00 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:07:04 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:08:05 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:09:08 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:10:14 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:11:17 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:12:19 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:13:24 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:14:26 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:15:28 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:16:33 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:17:34 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:18:36 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:19:38 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:20:40 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:21:42 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:22:47 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:23:48 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:24:54 PM]		10780672.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:25:56 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:26:58 PM]		10784768.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:28:01 PM]		10801152.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:29:02 PM]		10792960.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:30:04 PM]		10792960.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:31:12 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:32:14 PM]		10792960.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:33:16 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:34:19 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:35:22 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:36:24 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:37:27 PM]		10788864.00	 (Pool Nonpaged Bytes)
[1/25/2006 6:38:30 PM]		10788864.00	 (Pool Nonpaged Bytes)

I also unistalled Ethereal/WinPcap at the same time but because this was only installed 3 days ago, it could not possibly have been the culprit.

BTW, the version of CommView was 5.1. (Build 493)

-Tim

[Yoorix]

It's a very good you was able to solve the problem by yourself.

But you recently said, that you have no additionall software, installed on your PC, except PCAnywhere.

Whatever, the problem is solved.

Regards,
Yoorix

[KS-Soft]

So, you had non-standard tcp/ip driver. Such programs aften lead to problems. That's why we do not recommend to install real-time antivirus monitors, content monitoring software, personal firewall...
I am glad you have solved this problem

Regards
Alex

[timn]

But you recently said, that you have no additionall software, installed on your PC, except PCAnywhere.

Yes sir, guilty as charged! Thirty lashes!

A packet sniffing tool is an essential part of my tool set so I'll need to find a replacement for CommView if the vendor (TamoSoft) is not able to quickly respond or is unable to locate the problem.

And, Alex was correct, we had recently upgraded to the latest version of CommView (5.1). FWIW, further investigation shows this leak did not occur under CommView 3.1 (build 161), but definitely occurs under versions 5.0 and 5.1

We use this utility everywhere and it explains a number of memory-related issues we were seeing - not just with the Host Monitor machine.

Apparently the CommView driver is 'always in play' -- even when Commview itself is not running.

BTW, I turned on several hundred more tests last night and checkedmy nonpaged pool bytes this morning - holding very steady at 11MB (essentially 0 MB/hour increase)

Feels like a whole new day.

-Tim

[timn]

In fairness to TamoSoft, the vendor of CommView, I wanted to post this followup.

I notified TamoSoft of the problem and they responded promptly -- it appears that this was a known issue -- one that had been resolved earlier, but for some reason, we ended up with mismatched drivers...

...we are aware of the problem and it was resolved some
time ago. According to the driver listing you have:

TS_LB.SYS TamoSoft (CommView Loopback Driver) 1.1.0.4
CV2K1.SYS TamoSoft (CommView Driver) 2.3.0.1

Then you write "BTW, the version of CommView was 5.1. (Build 493)". This
doesn't match, as CommView 5.1.493 comes with TS_LB.SYS 1.2.0.9 and
CV2K.SYS 3.0.1.2.

So, TS_LB.SYS 1.1.0.4 indeed leaks memory, but TS_LB.SYS 1.2.0.9 that comes
with CommView 5.1.493 doesn't...

I've also now verified that, on two other machines where we were seeing an unexplained memory leak, the CommView drivers were also mismatched.

[KS-Soft]

Thank you for information. Perhaps it helps somebody...

Regards
Alex