System crashes after 24-48 hours

[timn]

Update from the originator of this thread:

[KS-Soft]

Regarding corrupted DLL: my colleague says he had similar problem on his home computer. Problem was caused by cold air near the window! When system was moved to warm place, problem gone.

Regarding resources problem. I don't think its hardware problem either.
Probably you have configured some new test items that lead to such problem? E.g. ODBC Query check or ODBC Logging? Some ODBC driver may cause resource leakage.
Or may be you have added Performance Counters to check some 3rd party software? Ok, you said there are no 3rd party software.
So, what test methods, alerts do you use? Do you use ODBC Logging? What ODBC driver do you use? Have you updated that driver before problem started? Microsoft security patches often leads to various problems as well, but our Windows 2000 works fine (the worst experience we have with Windows XP).

1st step to track the problem should be - check what kind of resources are leaking. And what process causes this leakage. Its pretty simple - start standard Windows Task Manager and check Memory, Handles, GDI Objects and USER Objects for each process.

Regards
Alex

[timn]

[timn]

Alex:

Just a quick update. HM crashed again this morning but I discovered that I had not disabled ALL OBDC tests - there were 3 still active and 2 were getting continuos connect errors.

I have now disabled ALL ODBC tests (as shown by View | Estimate Load... 0 active, 15 disabled)

We'll let this run for a period and see what happens...

[KS-Soft]

What about resource usage?

[timn]

[KS-Soft]

1,600 handles is not a problem unless total umber of handles allocated by other processes below 10-12 thousand.
My Windows 2000 works stable until 11,000 - 12,000 handles allocated.

What about GDI, User objects? Have you checked resource usage at the moment when application crashed? Or you have checked resources when everything works fine? Its good to perform 2 checks and compare resource usage.

Regards
Alex

[timn]

[KS-Soft]

We spent some time on microsoft.com and various tech forums, found some articles with similar problems but did not find any article with solution. My colleague (and some articles) says it can be problem with active directory replication, corrupted sysvol or corrupted computer account...
Do you see error messages in security event log? BTW: system with the problem - domain controller? If not, check event logs on domain controler as well.
E.g. if the computer account for the computer listed is corrupted, following steps fix the problem: join the computer to a workgroup named WORKGROUP, and then restart the computer, rejoin the computer to the domain, and then restart the computer again.
http://support.microsoft.com/?kbid=329708

Regards
Alex

[timn]

Alex:

I think you are on to something here.

The Host Monitor box (WATCHDOG) is not a domain controller itself but belongs to our local domain.

The security Eevent log on WATCHDOG did not show anything strange, BUT there were several messages in the DC's security event log indicating login failures from WATCHDOG.

So, I unjoined WATCHDOG from the domain (joining workgroup WORKGROUP) but I then received a warning message saying (roughly) "You have unjoined the domain but the computer account on the Domain Controller could not be disabled"). So I went to the DC, and manually deleted the computer account for WATCHDOG.

Then I rebooted WATCHDOG. Then attempted to rejoin domain. But...WATCHDOG rejoined the domain just a little too quickly - it usually takes several seconds to join a machine to the domain but this 'join' came back almost immediately. Then I rebooted WATCHDOG again.

After rebooting, I noticed that the name WATCHDOG did not appear in the list of computer accounts on the DC. I waited several minutes but the name never shows up.

So, I unjoined WATCHDOG from the domain once again (back to WORKGROUP) and again received the strange dialog warning about being unable to disable computer account.

Next, I changed the name of the Host Monitor machine to WATCHDOG7, then rebooted. Then rejoined domain. Then rebooted again. This time, WATCHDOG7 appears in the list of computer accounts on the DC.

Now that you've pointed me in this direction, I recall running into a similar problem a couple of years ago (not involving Host Monitor) where problems resulted from replacing hardware and naming it the same as the old hardware.

Anyway, given all this weirdness, I strongly suspect you are correct about this. We should know more in a day or 2. Thanks!

-Tim

[timn]

OK, problem still exists but.. after solving previous problem, we are getting more information - the problem appears to be with resource \Memory\Pool Nonpaged Bytes.

Immediately after a reboot, \Memory\Pool Nonpaged Bytes is

13,094,912 (13 MB)

The value steadily increments for the next 24 hours. System
crashes when \Memory\Pool Nonpaged Bytes reaches:

266,874,880 (266 MB)

Event log now shows Event ID 2019, src "Srv":

"The server was unable to allocate from the system
nonpaged pool because the pool was empty. "

Strange thing is, Task manager shows no 'hog' of this resource. Largest is SERVICES.EXE at about 1/2 megabyte

I maintain a private text log on this and can see this resource is being eaten up at at 12 MB / hour. The increase appears pretty linear - right up to crash point.

Next. how to find?

None of the processes seem to be taking up much:

Currently

\Process(_Total)\Pool Nonpaged Bytes = 1.1 MB

but

\Memory\Pool Nonpaged Bytes = 54 MB

Where's Waldo? (i.e the missing 53 MB of Non-Paged bytes)??

[Yoorix]

Strange problem. Maybe, following links would be usefull:

http://support.microsoft.com/kb/888928/en-us or
http://support.microsoft.com/kb/833266/en-us.

Are you sure you have not installed any antivirus software like Norton Antivirus or McAfee?

Could you, please, review this link to match particular problem?
http://eventid.net/display.asp?eventid=2019&eventno=661&source=Srv&phase=1

[timn]

Yoorix:

Definitely no AV installed at this point.

I have also uninstall the Dell OpenManage software and drivers but this has made no difference.

Also, just FYI, from the console, I restarted all service (well, all that would let me) hoping the restart would release the pool resource and I'd see the counter drop, but alas, this yielded no result.

I am currently exploring the other links you've suggested and will post more when finished.

Thanks for your help.

[timn]

Yoorix:

I've gone through most of the info at the links you provided -- I haven't found much that seems relevant yet -- but I've still got a couple of things to follow up on.

But my latest testing seems to confirm that the resource leak is associated with some driver on my system that HM needs to use to perform testing.

1. Stopping the Host Monitor process and the HM Web Service does NOT restore the memory resource to the pool. And no (visible) process appears to be hogging the NP Pool. Thus, it is NOT Host Monitor itself that is the problem.
   
   
2. I cretaed a new test list with only 1 test - i.e. monitor PerfCounter: \Memory\Pool Nonpaged Bytes. For the past 2 hours the return value has stayed constant at 16,015,360
   

So, resource leak is most likely in a system or third-party driver that HM must use in order to perform tests. But with over 3,000 active tests, pinning down which tests are driving the resource leak could take some time.

A large percentage of my tests use either RMA or SNMP. I think I'll ignore tests actually performed by RMA and focus on tests performed by Host Monitor itself.

I also will check to make sure I am using latest drivers and firmware for NIC, SCSI, etc. (they should be the lastest because I just rebuilt this machine from scratch and downloaded the various latest drivers at that time - but maybe I missed something). Also consider, what if the latest driver contains the resource leak but older driver -- on older Host Monitor machine -- did not? Hmmmm.

[KS-Soft]