Monitor VMware esxi using SOAP

All questions related to installations, configurations and maintenance of Advanced Host Monitor (including additional tools such as RMA for Windows, RMA Manager, Web Servie, RCC).
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Monitor VMware esxi using SOAP

Post by mrw »

Hi,
I´ve been struggling with monitoring a couple of ESXi servers(4.1 and 5.1) using SNMP for some time until I discovered that SNMP support was removed when ESX became ESXi. All I can get using SNMP is uptime which isn´t particularly useful. ;)

But then I found out that other monitoring systems monitor ESXi uses SOAP.
And I´ve read some documentation from VMware but I cant´get anything to work using SOAP/XML test in Hostmonitor.
Does anyone have any tip on how to get monitoring values out using SOAP(or any other way for that matter)? Like CPU usage, memory usage, If a VM is shut down or other stuff like that.
KS-Soft
Posts: 13012
Joined: Wed Apr 03, 2002 6:00 pm
Location: USA
Contact:

Post by KS-Soft »

There are some information regarding VMWare CIM classes

CIM Storage Management API Programming Guide PDF
http://www.vmware.com/support/developer ... 0_prog.pdf

Hardware Helth Monitoring via CIM
http://blogs.vmware.com/vsphere/2010/04 ... a-cim.html

CIM Classes
http://www.vmware.com/support/developer ... apirefdoc/

For example the following test item will check Processor HealthState.
(you may copy/paste the following text into text file, replace IP address, set correct password and import test using HostMonitor menu File->Import from text file)

Code: Select all

Method      = SOAP
;--- Common properties ---
;DestFolder = Root\333\
Title       =  VMWare Processor HealthState
Comment     = SOAP https://192.168.10.10:5989/
RelatedURL  = 
NamePattern = SOAP %object%
CmntPattern = SOAP %path%
ScheduleMode= Regular
Schedule    = 
Interval    = 600
Alerts      = 
Alerts2     = 
ReverseAlert= No
UnknownIsBad= Yes
WarningIsBad= Yes
UseCommonLog= Yes
PrivLogMode = Default
CommLogMode = Default
;--- Test specific properties ---
URL         = https://192.168.10.10:5989/
UrlUseMacros= No
SoapVersion = 1.2
username    = root
password    = testpassword
PostData    = <?xml version="1.0"  encoding="utf-8"?>^M<CIM CIMVERSION="2.0"  DTDVERSION="2.0">^M<MESSAGE ID="34422"  PROTOCOLVERSION="1.0">^M<SIMPLEREQ>^M<IMETHODCALL NAME="EnumerateInstances">  ^M<LOCALNAMESPACEPATH>^M<NAMESPACE NAME="root"/>^M<NAMESPACE NAME="cimv2"/>^M</LOCALNAMESPACEPATH>^M<IPARAMVALUE  NAME="ClassName">^M<CLASSNAME  NAME="CIM_Processor"/>^M</IPARAMVALUE>^M</IMETHODCALL>^M</SIMPLEREQ>^M</MESSAGE>^M</CIM>
is302ok     = Yes
IgnoreUnknCA= Yes
XPath       = /CIM/MESSAGE/SIMPLERSP/IMETHODRESPONSE/IRETURNVALUE/VALUE.NAMEDINSTANCE/INSTANCE/PROPERTY[@NAME="HealthState"]/VALUE
RowMode     = any row
CompareMode = DifferentFrom
CompareVal  = 5
DisplayTime = No
Regards
Alex
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

Thanks Alex for your help!
Unfortunately it doesn´t quite work. When I imported the test it complained about "wrong parameter line #13" and then Errors: 3.

Before the import I edited the ip and username/password.
After the test was added I also changed the "Test By" to the appropriate RMA.

And now I get the Status "No Answer".
We have checked on the ESX that port 5989 is open.

//Andreas..
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Post by KS-Soft Europe »

When I imported the test it complained about "wrong parameter line #13" and then Errors: 3.
What HostMonitor version do you use ? RMA agent (active/passive/version) ?
This parameter can be ignored.
And now I get the Status "No Answer".
We have checked on the ESX that port 5989 is open.
If there are some problems with server certificates, you will get "No answer" status even if port 5989 is opened.
You may mark all options, located in "Advanced" section of SOAP test properties dialog:
- HTTP/HTTPS: Code 302 (redirect) is Ok
- HTTPS: Ignore unknown certificate authority problems
- HTTPS: Accept certificates with invalid host name
- HTTPS: Accept certificates with invalid dates
Please note, some of these options could be "grayed". They should be marked (not "grayed").
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

HostMonitor version is 9.16 and I´m trying to use an active RMA agent to perform the tests on the ESXi server.

All the checkmarks in the Advanced sections were already set and I tried changing all of them but the results were always "No answer"

When I use a web browser to connect to https://<ip of esx host>:5989 I get certificate errors and then just a blank page.
We have checked the firewall inside ESX and its turned off and the "any" ip is allowed to access port 5989.

How can we debug this to see where the error lies?
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Post by KS-Soft Europe »

All the checkmarks in the Advanced sections were already set and I tried changing all of them but the results were always "No answer"
Two of these options were marked and grayed out. That means HostMonitor used default values for these options. They should be marked and not grayed out.
You may try to restart Management Network or entire ESXi.
When I use a web browser to connect to https://<ip of esx host>:5989 I get certificate errors and then just a blank page.

Thats, because this is SOAP interface and it requires XML requests.
Actually you should get HTTP error 501 (Not implemented) in this case.
We have checked the firewall inside ESX and its turned off and the "any" ip is allowed to access port 5989.
Do you have any Antivirus/Firewall software on system where active RMA agent is installed ?
How can we debug this to see where the error lies?
You may use some network sniffer to check if there is any reply from ESXi.
If you'll see reply, that means ESXi SOAP API interface works properly.
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

Now I´ve managed to go through your list of things to check/verify

1. The 2 bottom options are greyed with checkmarks and these 2 can be clicked and changed. I´ve restarted the Management Network on the ESX but that didn´t do any difference.

2. No, there´s no firewall or antivirus on the RMA server. I also tried from another server and the results were the same. Ping and SNMP work fine using the same RMA.

3. I used Wireshark to see some traffic and there is indeed some answers from the ESX when the SOAP test initiates. Wireshark lists it as WBEM-HTTPS and in one of the replies from ESX I can see some text in the package that contains information about the certificate and so on.

So I guess that confirms that ESX in answering on the SOAP tests so there´s nothing blocking the traffic, but obviously not with the correct answer. Can there be something wrong with the tests parameters?

I have tested all above on another ESX and the results are the same. Both are running ESXi v5.1.
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Post by KS-Soft Europe »

1. The 2 bottom options are greyed with checkmarks and these 2 can be clicked and changed.
These options should NOT be greyed.
You may also enable these option globally using menu Options -> "Misc" page -> "Settings for URL tests":
- Accept SSL/PCT certificates with invalid host name
- Accept SSL/PCT certificates with invalid dates
I used Wireshark to see some traffic and there is indeed some answers from the ESX when the SOAP test initiates. Wireshark lists it as WBEM-HTTPS and in one of the replies from ESX I can see some text in the package that contains information about the certificate and so on.

So I guess that confirms that ESX in answering on the SOAP tests so there´s nothing blocking the traffic, but obviously not with the correct answer. Can there be something wrong with the tests parameters?
Looks like problems with invalid certificates.
I have tested all above on another ESX and the results are the same. Both are running ESXi v5.1.
We have checked VMWare manuals, but couldnt find any useful information regarding these problems.
We have tested SOPA/XML test with fresh installations of VMware ESXi 4.1 and VMware ESXi 5.1 without any problems.
May be you shoud ask VMware support about this ?
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

What could be the reason those two options are greyed out? Network problems? Authentication issues? Does Hostmonitor or the RMA need some kind of framework installed for this to work like .NETv4? I assume that HostMonitor performs some initial test when I click the Edit button on the SOAP test?
Is there a debug log or something that might provide some answers to this?

AFAIK I´m using the correct root-password for the esx but without a log I can´t tell. Or should I add an entry in the Connection Manager?

And I will submit a supportcase to VMware to see if they can help.
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Post by KS-Soft Europe »

What could be the reason those two options are greyed out? Network problems? Authentication issues?
If these two options are grayed out - HostMonitor uses default values.
Please, click on these option - they become active.
If they are grayed out - HostMonitor will not accep connections (when there are problems with ESXi certificates)
Does Hostmonitor or the RMA need some kind of framework installed for this to work like .NETv4?
No.
I assume that HostMonitor performs some initial test when I click the Edit button on the SOAP test?
No.
Is there a debug log or something that might provide some answers to this?
You may open SOAP test properties dialog and use "Test" button to check server reply headers and content.
AFAIK I´m using the correct root-password for the esx but without a log I can´t tell. Or should I add an entry in the Connection Manager?
You can't use Connection Manager for URL/SOAP/XML test methods.
"Password propected page" section on SOAP test properties dialog should be used.
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

Ok, I clicked those options so they are not grayed out and all 4 are marked.
Still no difference.
And no matter what I do, everytime I click the Test button I always get the dialog box saying "Status: No Answer"

I got a reply from VMware on this and they stated: "it appears to be an issue with third party application configuration" :wink:

This is really frustrating. All I know is that there are some problem with the communication between my ESXi servers and Hostmonitor and the RMAs. This could probably be solved pretty easy if I had much more detailed logs regarding the communications. VMwares logs suck bigtime, which I have stated in my reply to their support technician. I have my hopes up that HostMonitor might give me more answers to exactly whats going on.

The dialog states "No Answer", but using Wireshark I can see that that's not entirely true. There are answers and replies on those answers and so on. What those answers contain I do not know, but Hostmonitor should be able to translate the answers to a debuglog or something.

Anyway I really appreciate the help so far on this from you!
KS-Soft
Posts: 13012
Joined: Wed Apr 03, 2002 6:00 pm
Location: USA
Contact:

Post by KS-Soft »

I got a reply from VMware on this and they stated: "it appears to be an issue with third party application configuration"
H'm, may be but web browser does not work with your server as well
but Hostmonitor should be able to translate the answers to a debuglog or something.
Not really :(
We use our own code for HTTP test so its easy to debug but HostMonitor uses Windows wininet/winhttp API for URL test and HTTPS requests so we limited in options and cannot get much information if there is some SSL/TLS related problem.
We will check what possible can be done on our side...

In some cases WireShark can decrypt HTTPS traffic. It needs PEM file from the server.

Regards
Alex
KS-Soft
Posts: 13012
Joined: Wed Apr 03, 2002 6:00 pm
Location: USA
Contact:

Post by KS-Soft »

Can we access your server thru internet (by HTTPS protocol)?

Regards
Alex
mrw
Posts: 195
Joined: Mon Oct 08, 2012 6:11 am

Post by mrw »

Hi Alex,

Neither the Hostmonitor server or the ESXi server are accessible from the Internet without VPN.
So I´m afraid I can´t let you have access to it. If this turns out to be impossible to fix without your intervention I´ll try to get an approval "from above" ;)

But I had forgotten about the browser access. I tested it some more and when I use firefox I only get a blank page, but in IE I get the certificate warning and then "HTTP 501/HTTPS 505 Not Implemented". Was that how it should be?

And I don´t mind doing some protocol analyzing in wireshark if you can point me in the right direction?
I actually have sent some packet captures to Watchguard since I have some trouble polling certain SNMP OIDs from their firewalls. In that case the problem lies with Watchguard and not with Hostmonitor.
If you want I could send you packet captures of the SOAP conversation between Hostmonitor and one of the ESXi servers if that would help?

Thanks!
//Andreas..
KS-Soft
Posts: 13012
Joined: Wed Apr 03, 2002 6:00 pm
Location: USA
Contact:

Post by KS-Soft »

501 means - The server does not support the functionality required to fulfill the request. This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource.
505 is more specific - The server does not support, or refuses to support, the HTTP protocol version that was used in the request message. The server is indicating that it is unable or unwilling to complete the request using the same major version as the client, as described in section 3.1, other than with this error message. The response SHOULD contain an entity describing why that version is not supported and what other protocols are supported by that server.

Try to disable "Use HTTP 1.1" option in IE (or enable if its already disabled). Menu Tools->Internet Options->Advanced...

Regards
Alex
Post Reply