Hostmonitor monitoring Hostmonitor

agressiv · Joined: 06 Oct 2002 Posts: 35

We have several licenses for hostmonitor, and we use hostmonitor to monitor the peers. Since there is no built-in function, we use the "check service" routine.

Well, hostmonitor hung the other day, yet the service was still running. Hostmonitor also hangs every time we do our SQL backup and re-indexing since SQL is blocking any inserts to the table.

1) Can you implement a routine that actually checks functionality of a remote hostmonitor server? I'll set up some sort of a perf counter test for now, but its not too desireable long term since its difficult to guage threshold on that.

2) Can you allow an option when the remote ODBC database is not allowing inserts, to keep on running after a, say, 2 minute timeout? The timeout currently is currently way too long, if it times out at all, and basically hostmonitor is useless if, say, the remote ODBC database server goes down.

Thanks,
Greg

KS-Soft · Joined: 03 Apr 2002 Posts: 12795 Location: USA

>Well, hostmonitor hung the other day, yet the service was still running. Hostmonitor also hangs every time we do our SQL backup and re-indexing since SQL is blocking any inserts to the table.

Do you use MS SQL test? What version of MS SQL client do you use? Clients v. 6.5 and 2000 without SP (may be some other versions) have bug that can crash application when server is not in normal operational mode.

>1) Can you implement a routine that actually checks functionality of a remote hostmonitor server? I'll set up some sort of a perf counter test for now, but its not too desireable long term since its difficult to guage threshold on that.

You may use Remote Control Interface that is implemented in HostMonitor 4 (only beta version available now). Enable interface on RCI page in the Options dialog, set TCP port, and you may use TCP test to check functionality of the HostMonitor.

>2) Can you allow an option when the remote ODBC database is not allowing inserts, to keep on running after a, say, 2 minute timeout? The timeout currently is currently way too long, if it times out at all, and basically hostmonitor is useless if, say, the remote ODBC database server goes down.

Are you talking about ODBC logging option?

Regards
Alex

agressiv · Joined: 06 Oct 2002 Posts: 35

>Do you use MS SQL test? What version of MS SQL client do you use? Clients v. 6.5 and 2000 without SP (may be some other versions) have bug that can crash application when server is not in normal operational mode.

No, this is not what I am talking about. We log via ODBC to MSSQL. When that SQL database is being backed up (its very large) - Hostmonitor cannot insert into the SQL database. When it can't insert, Hostmonitor freezes.

The same thing occurs if I have ODBC logging turned on, but the remote logging source goes down. So, if my ODBC database server goes down, Hostmonitor freezes (Doesn't perform any tests) until it can send the "Insert into hmlog" command.

I was just hoping it would time out sooner so that it can actually perform tests, even if the SQL inserts fail.

agressiv

Marcus · Joined: 18 Nov 2002 Posts: 367

>No, this is not what I am talking about. We log via ODBC to MSSQL. When that SQL database is being backed up (its very large) - Hostmonitor cannot insert into the SQL database. When it can't insert, Hostmonitor freezes

If you use online backups for your database, it's not MSSQL which blocks HM. Backup and data insertions are possible (that's why they made a 'online' backup procedure).

Are you running any other procedures, before or after the backup??

We make online backups and re-indexing on a 250 MB database for HM. I have never seen any problems with data insertion.

The other database on the same machine is 30+ Gb. When running the backup (and re-indexing) no problems occur and data insertions continue.

Most likely you use a procedure which require single user mode. If you can't prevent this, you must set a lock time out (default is no time out) for the transaction that updates the table. However, since HM does not have an error handler for this, data wil be lost.

Another option would be to downsize the database (or use a seperate database for HostMonitor).

[ This Message was edited by: Marcus on 2003-07-18 12:57 ]

agressiv · Joined: 06 Oct 2002 Posts: 35

Our database is about 45gb, which is only about 2-3 months worth of data. We are dumping to .BAK files, not using some "hot" online backup. Hostmonitor does not function when it's doing this backup.

The database server is a Quad Xeon box, and Hostmonitor is running on a dual 2.4ghz. I doubt it's a pure performance issue.

In any case, if that database goes down, hostmonitor freezes, which is really the question I have. It needs to time out MUCH sooner than it does and just continue monitoring without attempting to do that insert. At some point, every 5 minutes, it should probably test connectivity in the background. If it is responding, then it could start doing inserts again.

agressiv

agressiv · Joined: 06 Oct 2002 Posts: 35

I will check on the time out, didn't see that at first, but it still doesn't help if the DB server is down.

[ This Message was edited by: agressiv on 2003-07-18 13:10 ]

KS-Soft · Joined: 03 Apr 2002 Posts: 12795 Location: USA

HostMonitor has Timeout option for ODBC logging (by default timeout is 10 sec). However I see 2 problems:
1) many ODBC drivers return error code after specified timeout (e.g. application sets timeout to 5 sec, ODBC driver returns error code after 30 sec)
2) if ODBC driver works correctly but HostMonitor performs a lot of tests, it will spend 10 sec for each test. Of course it will slow down application a lot.

IMHO possible solution can be some option like "Disable ODBC logging for [M] min in case of [N] consecutive errors".

Regards
Alex

agressiv · Joined: 06 Oct 2002 Posts: 35

> IMHO possible solution can be some option like "Disable ODBC logging for [M] min in case of [N] consecutive errors".

This sounds like a great idea.

Thanks,

Greg

Marcus · Joined: 18 Nov 2002 Posts: 367

>We are dumping to .BAK files, not using some "hot" online backup

Dumping your database is an online backup. So inserting data must be possible. There are however options which you could perform after (or before) a backup, which require single user mode. In that case updates are not allowed. The only other thing I can think of (apart from the single user) is you are using ide drives which are taking all the processor time, but on a Quad Xeon box I can't imagine that's the case.

And 45GB for 2 to 3 months?? You must have a huge amount of tests :smile:

[ This Message was edited by: Marcus on 2003-07-18 15:00 ]

agressiv · Joined: 06 Oct 2002 Posts: 35

I just checked, we're down to just keeping 1 month of data, which after a dbcc shrink database brings it down to 12gb. At 8pm, we do a backup and re-index.

Hostmonitor is frozen for about 30 minutes during this time, but I haven't timed it for exact numbers.

The quad xeon database has other stuff on it, but the hostmonitor DB probably taxes it more than everything else. If I get my way, it will be part of an 8proc 2-way cluster on a SAN, but that won't happen anytime soon.

I'm not a SQL guy (I know just a little bit) so I'm hoping its just an I/O problem since the box itself isn't taxed from a CPU standpoint.

Greg

Marcus · Joined: 18 Nov 2002 Posts: 367

> I just checked, we're down to just keeping 1 month of data, which after a dbcc shrink database brings it down to 12gb.

This is still big for a HostMonitor database. How many test do you run? And what do you store in your database?

Currently we have 3500+ tests and 12 Gb would be sufficient for us to store over a year of data. Which we don't , we archive every month and this wil keep the database well under the 1 Gb and retain historical data (in the same database).

[ This Message was edited by: Marcus on 2003-07-20 09:40 ]

agressiv · Joined: 06 Oct 2002 Posts: 35

Currently, we have 3575 tests exactly, altough probably up to 15-30 of those are disabled.

The dependancies are heavily used, and most tests that are local go every 3 minutes. Remote tests go every 5-10 minutes depending on how "remote" they are. And pretty much everything is monitored 24x7, less than 10 tests are only monitored during the day.

Granted, our database is 12GB to the filesystem, but our .BAK file is only about 9GB.

Our insert command:
Insert into hmlog (eventtime, server, testname, status, reply) VALUES ('%DateTime%', '%Folder%', '%TestName%', '%Status%', '%Reply%')

Greg

[ This Message was edited by: agressiv on 2003-07-20 11:29 ]

Marcus · Joined: 18 Nov 2002 Posts: 367

>Insert into hmlog (eventtime, server, testname, status, reply) VALUES ('%DateTime%', '%Folder%', '%TestName%', '%Status%', '%Reply%')

This is less than what we use. Either you user very long folder and or testsnames, or you have very big indexes. Check your indexes. Most likely they are bigger than the data in your table.

KS-Soft · Joined: 03 Apr 2002 Posts: 12795 Location: USA

>Currently, we have 3575 tests exactly ... most tests that are local go every 3 minutes. Remote tests go every 5-10 minutes ... Granted, our database is 12GB

If average test interval 5 min, HM performs
12*24*30*3565 = 3,0801,600 tests per month. 12*2^30/30801600 = 418 bytes per record.

HostMonitor 3.70 allows to use different common log settings for different tests. Probably you may log all information about some test items and log status/reply changes for another tests. It can reduce database size..

Regards
Alex

Marcus · Joined: 18 Nov 2002 Posts: 367

>If average test interval 5 min, HM performs
12*24*30*3565 = 3,0801,600 tests per month. 12*2^30/30801600 = 418 bytes per record

pff 418 bytes, doesn't sound as much....

But then again: 4 bytes for an id, 8 bytes for date time, leaves 406 bytes for the rest. Avereging the status to 16 bytes (could be way off, but gives a nice number

), this leaves 390 bytes for folder, testname and reply. Which means an average of 130 characters (assuming you use varchar, not nvarchar and absolutely no char type). This sounds as a lot of bytes.

You could replace %Status% with %StatusID%, which could be hold in a tiny integer of just 1 byte. Giving you 405 bytes for folder, test name and reply.

[ This Message was edited by: Marcus on 2003-07-21 13:18 ]