|
| View previous topic :: View next topic |
| Author |
Message |
CH
Joined: 23 Sep 2010
Posts: 3
Location: New Hamsphire
|
 Posted: Thu Sep 23, 2010 10:12 am Post subject: PerfMon Counters broken..error: the wait operation timed out |
 |
|
We are currently running hostmon 8.68c on our windows 2003 server that suddenly started giving us errors within the perfmon counters test on our citrix servers.
This error did not occur before the upgrade to 8.68, we were on 8.64 previously. We also tested perfmon on a windows box that does not run citrix on it and we experienced the same error " Error: The wait operation timed out".
The test we're running is the performance counters \\%path%\Terminal Services\Active Sessions on all our citrix servers. We currently disabled all of the test for perfmon, but we would like to get some help asap to monitor how heavily utilized our environment is.
If we go into the test itself and click on get current value, it gets it the value, but hostmon errrors out.
Thanks for the quick reply! |
|
| Back to top |
|
 |
KS-Soft
Joined: 03 Apr 2002
Posts: 12793
Location: USA
|
| Posted: Thu Sep 23, 2010 11:11 am Post subject: |
|
|
Well, HostMonitor does not change any system setting so HostMonitor update could not effect perfmon.
I assume you made some other changes on the system beside HostMonitor update.
Is this test performed directly by HostMonitor or you are using RMA installed on remote system?
Could you please check the following
1) you are using account with admin rights for this test
2) RPC service is started on both systems (system where HostMonitor is running and target host)
3) Remote Registry service should be started on target system as well
4) Check if account that you are using for the test has KEY_READ access to the following registry key on target system: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows_NT\CurrentVersion\Perflib.
To view/change permissions to the registry key you can use the registry editor utility (Regedt32.exe).
5) no firewall between 2 system blocks TCP ports above 1024
6) Performance DLLs should be enabled on target system
Microsoft provides "Extensible Counter List" utility.
This utility displays list of DLLs that are used to support Performance Counters, using the utility you may disable/enable specific DLLs.
http://www.microsoft.com/downloads/details.aspx?familyid=96A35011-FD83-419D-939B-9A772EA2DF90&displaylang=en
| Quote: | | If we go into the test itself and click on get current value, it gets it the value, but hostmon errrors out |
Not sure I understand. HostMonitor gets value from target system or it shows error?
May be it shows error when it tries to perform several tests at the same time? Could you try to refresh such test items one by one?
Quote from the manual
==================
Windows implementation of performance counters has bugs. E.g., Windows 2000 (Professional, Server, and Advanced Server editions) can produce memory leak in PDH.DLL when user (application) querying performance counter that does not exist. This bug fixed in SP2. Also PDH.DLL does not work correctly with multithread applications.
That's why in HostMonitor we have implemented several different methods to work with pdh.dll:
- MultiThread mode: HostMonitor works almost according to Microsoft documentation with some workaround to avoid most likely problems. HM loads pdh.dll at once and uses it all the time. This method fast because HM can start several tests simultaneously. If everything will work correctly on your system, use this method (by default HostMonitor uses this method).
- OneByOne mode: Using this method HM will start Performance Counter tests one by one This method is slow (when you setup Performance Counter test using Test Properties dialog program even can hang for 1-2 min) but using this method you may avoid some problems due to a buggy pdh.dll
- Smart mode: With this method HM will try to detect when pdh.dll has to be reloaded.
- External mode: HostMonitor uses external (perfobj.exe) utility to perform the tests. This is fast and most reliable method.
==================
If you can perform tests one by one, I would suggest to
- use OneByOne mode if you do not have a lot of such tests
- use External mode when you have a lot of Performance Counter tests
Also, we recommend to use WMI test instead of Performance Counter when possible. Microsoft WMI implementation works much more reliable
Regards
Alex |
|
| Back to top |
|
|
CH
Joined: 23 Sep 2010
Posts: 3
Location: New Hamsphire
|
| Posted: Thu Sep 23, 2010 12:07 pm Post subject: |
|
|
| KS-Soft wrote: | Well, HostMonitor does not change any system setting so HostMonitor update could not effect perfmon.
I assume you made some other changes on the system beside HostMonitor update.
Is this test performed directly by HostMonitor or you are using RMA installed on remote system? Directly on Hostmon
Could you please check the following
1) you are using account with admin rights for this test =Yes
2) RPC service is started on both systems (system where HostMonitor is running and target host) = Yes
3) Remote Registry service should be started on target system as well = Yes
4) Check if account that you are using for the test has KEY_READ access to the following registry key on target system:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows_NT\CurrentVersion\Perflib. =Yes
To view/change permissions to the registry key you can use the registry editor utility (Regedt32.exe).
5) no firewall between 2 system blocks TCP ports above 1024 =No firewall port blocking 1024 or above
6) Performance DLLs should be enabled on target system
Microsoft provides "Extensible Counter List" utility.
This utility displays list of DLLs that are used to support Performance Counters, using the utility you may disable/enable specific DLLs.
http://www.microsoft.com/downloads/details.aspx?familyid=96A35011-FD83-419D-939B-9A772EA2DF90&displaylang=en
| Quote: | | If we go into the test itself and click on get current value, it gets it the value, but hostmon errrors out |
Not sure I understand. HostMonitor gets value from target system or it shows error? If we click and go into an actual perfmon test then click on the get current value button, it replies in the value the current active sessions. When we exit the test and go back to the console view we get the error.
May be it shows error when it tries to perform several tests at the same time? Could you try to refresh such test items one by one? The tests ran fine before the upgrade and they run every 10min. We reenabled each test one-by-one and still getting this error
Quote from the manual
==================
Windows implementation of performance counters has bugs. E.g., Windows 2000 (Professional, Server, and Advanced Server editions) can produce memory leak in PDH.DLL when user (application) querying performance counter that does not exist. This bug fixed in SP2. Also PDH.DLL does not work correctly with multithread applications.
That's why in HostMonitor we have implemented several different methods to work with pdh.dll:
- MultiThread mode: HostMonitor works almost according to Microsoft documentation with some workaround to avoid most likely problems. HM loads pdh.dll at once and uses it all the time. This method fast because HM can start several tests simultaneously. If everything will work correctly on your system, use this method (by default HostMonitor uses this method).
- OneByOne mode: Using this method HM will start Performance Counter tests one by one This method is slow (when you setup Performance Counter test using Test Properties dialog program even can hang for 1-2 min) but using this method you may avoid some problems due to a buggy pdh.dll
- Smart mode: With this method HM will try to detect when pdh.dll has to be reloaded.
- External mode: HostMonitor uses external (perfobj.exe) utility to perform the tests. This is fast and most reliable method.
==================
If you can perform tests one by one, I would suggest to
- use OneByOne mode if you do not have a lot of such tests
- use External mode when you have a lot of Performance Counter tests
Also, we recommend to use WMI test instead of Performance Counter when possible. Microsoft WMI implementation works much more reliable I just created a WMI test and it worked! I'll just do a mass import of the citrix servers with the replicator, but still wondering what caused PerfMon to break
Thanks again for the quick response and keep up the great job with this app and in supporting it.
|
|
|
| Back to top |
|
|
KS-Soft
Joined: 03 Apr 2002
Posts: 12793
Location: USA
|
| Posted: Thu Sep 23, 2010 4:36 pm Post subject: |
|
|
| Quote: | | If we click and go into an actual perfmon test then click on the get current value button, it replies in the value the current active sessions. When we exit the test and go back to the console view we get the error. |
Are you using RCC to setup tests? Is RCC running on different system?
If yes, then it looks like system where HostMonitor is running cannot connect to target system while RCC system does not have any problems to perform such request.
Is there any antivirus monitor installed on target system or system where HostMonitor is running? Real-time antivirus monitors often lead to various problems 
| Quote: | | The tests ran fine before the upgrade and they run every 10min. |
We did not change code used by Performance Counter test for years, so upgrade could not cause this problem and it could not cause any problems with Windows Perfmon 
Regards
Alex |
|
| Back to top |
|
|
CH
Joined: 23 Sep 2010
Posts: 3
Location: New Hamsphire
|
| Posted: Fri Sep 24, 2010 7:06 am Post subject: |
|
|
| Quote: | | Are you using RCC to setup tests? Is RCC running on different system? |
No, tests are always created on the Hostmon Server.
| Quote: | | Is there any antivirus monitor installed on target system or system where HostMonitor is running? Real-time antivirus monitors often lead to various problems |
Yes, but it hadn't interfered before on the tests.
I am quite confused why this happened, but we are going to do WMI tests to get the info needed. Thanks again.  |
|
| Back to top |
|
|
mrfrosty
Joined: 14 Feb 2008
Posts: 21
|
| Posted: Tue Jan 04, 2011 5:35 am Post subject: |
|
|
CH, are you still experiencing this issue?
I have also had this problem for a number of months now, since upgrading Host Monitor and affected agents to 8.68c I get this problem. Any RMA agents (active / passive) that have not been updated are not affected.
Since this issue appeared I have not updated any more RMA agents, just in case more servers are affected.
I'm currently at 8.80 and updated the affected RMA agents to the latest version and still have this issue.
Alex I originally spoke to you about this on the 16/08/2010 via email.
Marcus |
|
| Back to top |
|
|
KS-Soft
Joined: 03 Apr 2002
Posts: 12793
Location: USA
|
| Posted: Tue Jan 04, 2011 12:54 pm Post subject: |
|
|
| Quote: | | Alex I originally spoke to you about this on the 16/08/2010 via email |
As I see you reported different problem.
Quote
==================
The error message I get is “RMA: 301 – Error: The specified object was not found on the computer.”
==================
And you said problem was fixed when you reboot server.
Does this problem come back again and again and you have to reboot system all the time? Then could you check resource usage for each process? You may use standard Windows Task Manager to check Handles, Threads, GDI and USER objects. What is the total resource usage on the system? How many handles/threads/GDI objects used by hostmon.exe/rma.exe process?
We don't see how HostMonitor/RMA update can lead to this problem because we did not change code that performs Performance Counter test for 3 years.
Also it works just fine on our systems and as I see several thousand customers already updated to version 8.68, 8.72, 8.80 without any problems
Regards
Alex |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in
