HostMonitor unexpectedly quits!

[JuergenF]

From time to time I've the same problem (every 7 to 10 days)
- HP ProLiant DL360 G3
- - 2* Intel Xeon 3.066Ghz
- - 2 GB RAM
- Windows Server 2003 SE SP1
- HostMonitor is started as service
- TrendMicro ServerProtect 5.58
- No Personal firewall
- No Content monitoring software
- No Non-standard winsock components
- No Network packet analyzer
- No ODBC tests or Logging
- No SQL tests or logging

When HM is stopped (has died) I see:
Total Handels: 35265
Threads: 912
Processes: 56
Avail Memory: 1434288

Top Users:
Runner.exe uses 21438 Handels (and raising)
svchost.exe uses 32834K memory
serversalive.exe uses 9:08:21 hours CPU time
serversalive.exe uses 121 USER Objects
explorer.exe uses 196 GDI Objects

In Eventlog:

Code: Select all

Error - Service Control Manager - Event 7034
The KS Advanced Host Monitor service terminated unexpectedly.
It has done this 2 time(s).

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Last entries from Log-File

Code: Select all

[2006-12-22 13:54:47]	Sys	Command "E:\mrtg\work\TrimDashboard.exe" executed
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_AT_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_AT_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_AUS_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_AUS_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CH_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CH_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CN_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CN_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CZ_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_CZ_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DK_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DK_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DL_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DL_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DU_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_DU_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_ES_Bad.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_ES_Full.htm)
[2006-12-22 13:54:47]	Sys	HTML report created (file: e:\mrtg\work\Rep_FR_Bad.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_FR_Full.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_GB_Bad.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_GB_Full.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_IT_Bad.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_IT_Full.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_US_Bad.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_US_Full.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_ZA_Bad.htm)
[2006-12-22 13:54:48]	Sys	HTML report created (file: e:\mrtg\work\Rep_ZA_Full.htm)

No more reports to process - so report generation finished from my pov

I can restart HM Service - and it works for another 7 days without rebooting or restarting any other Software.
All tests are run several times a day (so there is no test running only once a week that may cause the crash)

[KS-Soft Europe]

Runner.exe uses 21438 Handels (and raising)

Ups. Too many handles, I think. Windows 2003 can handle about 15-17 thousand handles. But system behavior is unpredictable when this limit is reached. Could you disable application, that starts process named "Runner.exe"? Have you tried to disable TrendMicro Antivirus?

Regards,
Max

[JuergenF]

Runner.exe uses 21438 Handels (and raising)

Ups. Too many handles, I think. Windows 2003 can handle about 15-17 thousand handles. But system behavior is unpredictable when this limit is reached. Could you disable application, that starts process named "Runner.exe"?

It's a kind of scheduler that starts my perl Scripts for MRTG.
Runner.exe is started at System-Startup and continuously running
I'll try to change that to Windows Scheduler

Have you tried to disable TrendMicro Antivirus?

Not yet. The Server guys may not allow that - for good reason.

I'll start with replacing runner.exe

[JuergenF]

Runner.exe uses 21438 Handels (and raising)

Ups. Too many handles, I think. Windows 2003 can handle about 15-17 thousand handles. But system behavior is unpredictable when this limit is reached. Could you disable application, that starts process named "Runner.exe"?

It's a kind of scheduler that starts my perl Scripts for MRTG.
Runner.exe is started at System-Startup and continuously running
I'll try to change that to Windows Scheduler

I'll start with replacing runner.exe

Just for information:
Runner.exe was replaced.
Still the problem that HM Service terminates from time to time.

Regards

Juergen

[KS-Soft]

Have you disabled TrandMicro? What about resource usage?

Regards
Alex

[JuergenF]

Have you disabled TrandMicro?

Not yet. The Server guys do not allow that - for good reason.
Normally it takes more than a week until it stops.
I'm monitoring the handles HM uses now. (see other thread)

What about resource usage?

What exactly can I tell you and where can I find that ?

Regards

Juergen

[KS-Soft]

Not yet. The Server guys do not allow that

According to other customers - TrendMicro is the reason of the problem.

What exactly can I tell you and where can I find that ?

GDI and User objects, Handles, memory usage. Total usage and what process uses the most of resources.
Just like you did last time. I assume you have used standard Windows Task Manager to obtain this information

Regards
Alex

[JuergenF]

Not yet. The Server guys do not allow that

According to other customers - TrendMicro is the reason of the problem.

That is possible, but we are running about 170 Servers - with TrendMicro

What exactly can I tell you and where can I find that ?

GDI and User objects, Handles, memory usage. Total usage and what process uses the most of resources.
Just like you did last time. I assume you have used standard Windows Task Manager to obtain this information

When HM is running I see:
Totals
Handles: 15055
Threads: 966
Processes: 57
Avail Physical Memory(K): 1555920

Top Users:
System uses 3137 Handles
svchost.exe uses 1362 Handles
-
svchost.exe uses 21272K Mem Usage
hostmon.exe uses 14100K Mem Usage
-
serversalive.exe uses 26:19:12 hours CPU time
System uses 10:09:12 hours CPU time
-
hostmon.exe uses 280 USER Objects
serversalive.exe uses 120 USER Objects
-
hostmon.exe uses 332 GDI Objects
explorer.exe uses 156 GDI Objects

By the way: What about a little tool to get that information

Regards

Juergen

[KS-Soft]

Resource usage looks Ok (for Windows 2003). Could you check this again when problem appears?

By the way: What about a little tool to get that information

We have such task in "to do" list

Regards
Alex

[JuergenF]

HM Service is down now

Totals
Handles: 15237
Threads: 957
Processes: 59
Avail Physical Memory(K): 1411412

Top Users:
System uses 3127 Handles
svchost.exe uses 1344 Handles
-
svchost.exe uses 63052K Mem Usage
sqlservr.exe uses 54312K Mem Usage
-
serversalive.exe uses 30:51:45 hours CPU time
System uses 11:39:35 hours CPU time
-
iexplore.exe uses 133 USER Objects
taskmgr.exe uses 96 USER Objects
-
iexplore.exe uses 331 GDI Objects
explorer.exe uses 185 GDI Objects

[KS-Soft]

Everything is fine but what the total GDI and USER object usage?

Regards
Alex

[JuergenF]

I don't know.

Let's see what happens next time.