Out of Memory on WMI tests

Kris · Post by **Kris** » Tue Nov 03, 2020 8:52 am

Hi all,

Several times a day I get a 'Out of Memory' error on WMI tests.
It's not one specific test or machine, it looks totally random, but on WMI tests only.
When I instantly refresh the test it will be fine.
No big deal, but maybe we're doing something wrong...

Specs:
HM 12.50 on Windows 10 64bit
on VMware 6.7 with 2 cores @3.2GHz and 8GB memory.

KS-Soft · Post by **KS-Soft** » Tue Nov 03, 2020 10:03 am

Does Auditing Tool show some warnings/errors (menu View)?

Could you check resource usage for each process? You may use standard Windows Task Manager to check Handles, GDI and USER objects. What is the total resource usage on the system? How many handles/threads/GDI objects used by hostmon.exe process?

Do you use ODBC logging or ODBC test method? If yes, what ODBC driver do you use?
Some ODBC drivers have many bugs and leads to various problems.

Number of handles/threads/user objects not always can help because problem may depend on what exactly kind of handle is used, e.g. how much memory this resource used in Windows internal buffers.
Sometimes 200,000 handles is not a problem; other time 80,000 is a problem. However its a starting point that may tell us what process leads to the problems.

Regards
Alex

Kris · Post by **Kris** » Wed Nov 04, 2020 2:15 am

Hi Alex,

Thanks for your quick response.
Please see below.

----------
Auditing Tool:
Errors: No errors other than Syslog file size and password strength
Workload: Low, avg. 2 tests/sec

System:
CPU load: 10-50%
Mem usage: 30-50%

HostMonitor process:
HM #Threads: 30-50
HM #Handles: 1500-2000
HM #GDI Handles: 0
HM #User Handles# 0
HM #Wait Chain Handles: 5

We don't use ODBC logging or tests.
----------

Hope this is useful info...

KS-Soft · Post by **KS-Soft** » Wed Nov 04, 2020 5:42 am

Please check ALL processes running on the system, some may use too much Windows resources

Regards
Alex

Kris · Post by **Kris** » Wed Nov 04, 2020 8:37 am

Hi Alex,

There are plenty system resources available.
On average 5GB of 8GB free, around 20% cpu load.

I ran a logging with ProcessExplorer, and even when HM threw a 'Out of Memory' error, system resources were availabe in abundance.

Can't get my head around it

KS-Soft · Post by **KS-Soft** » Wed Nov 04, 2020 8:44 am

Total free memory means nothing in this case.
System may have 60 GB of "free" memory and still thow "out of memory" errors because there are various internal Windows buffers (e.g. desktop heap). If there is no room in such buffer, system will not work properly. Unfortunately there is no way to check status of such buffers (there are some Microsoft tools but only for Windows XP/2003).

Please check my 1st reply.

Could you check resource usage for each process? You may use standard Windows Task Manager to check Handles, GDI and USER objects. What is the total resource usage on the system? How many handles/threads/GDI objects used by hostmon.exe process?

Number of handles/threads/user objects not always can help because problem may depend on what exactly kind of handle is used, e.g. how much memory this resource used in Windows internal buffers.
Sometimes 200,000 handles is not a problem; other time 80,000 is a problem. However its a starting point that may tell us what process leads to the problems.

or just stop couple processes with significant number of Handles and see if this helps.

Regards
Alex

Kris · Post by **Kris** » Wed Nov 04, 2020 8:48 am

Wasn't this what you asked?

HostMonitor process:
HM #Threads: 30-50
HM #Handles: 1500-2000
HM #GDI Handles: 0
HM #User Handles# 0
HM #Wait Chain Handles: 5

Kris · Post by **Kris** » Wed Nov 04, 2020 9:27 am

Oh wait,

Total # handles: about 60000
Total # threads: about 2200

KS-Soft · Post by **KS-Soft** » Wed Nov 04, 2020 10:11 am

Could you check resource usage for each process? You may use standard Windows Task Manager to check Handles, GDI and USER objects. What is the total resource usage on the system? How many handles/threads/GDI objects used by hostmon.exe process?

Number of handles/threads/user objects not always can help because problem may depend on what exactly kind of handle is used, e.g. how much memory this resource used in Windows internal buffers.
Sometimes 200,000 handles is not a problem; other time 80,000 is a problem. However its a starting point that may tell us what process leads to the problems.

"Could you check resource usage for each process" means - please check ALL processes.
This way you may find most "hungry" one and try to stop or restart it.

Also you may setup Dominant Process tests with full logging mode (better use private log file for these tests), this may help to find some process with "leaks".

Also you may start Auditing Tool (menu View), it may detect some problems.
What the average load (tests/sec)?
WMI+DominantProcess+Memory+HyperV tests/sec rate?

Regards
Alex