log file consolidator

terje · Post by **terje** » Tue Oct 03, 2006 1:29 am

One of the great things about hostmonitor is the log analyser. By running private logs and calling the log analyser via a series of scripts we have web pages that show the history of different tests on a stack of servers across many sites.

One test that we have found very useful in this area is CPU testing. By logging every 5 minutes we have been able to create some long term charts that give us a very good baseline of CPU usage. We intend keeping three years data so we can monitor progress over time. Although usually we are only looking at charts for the last few weeks. Often we can spot when a process has hung.

The problem with all this however is that we end up with some very large log files. They are quite compressible so storage is not so much of an issue, however once they are large they take a very long time to process.

What would be ideal would be to execute a program that went through a specified log file and based on some parameters could do things such as the following:-

For all data more than 12 weeks old, then replace all logs for each week with the average for that week. (Set time for represenative average as 12 noon Monday).

For all data more than 90 days old, then for each day keep only the peak value for that day and purge other entries.

For all data more than 21 days old, then for each day keep only the first log entry after 4pm.

For all data more than 1000 days old then purge that data.

Such a tool would allow for log files to be stripped down to a more manageable size whilst preserving the historical trends or baselines required.

Note that in the examples given running a well designed "consolidator" more than once would not change anything the second time through. Assuming of course that the current date and the parameters have not changed. This would mean that progressive subsequent consolidations on later dates (using the same parameters) would not change anything. Of course for the first example this depends on carefully defining the start and finish boundary points for a week.

The "consolidator" would also want to be able to filter on things such as test type or test name. For instance you may only want to average old CPU tests, but not old ping tests.

And for scripting it should all be executable from a command line with the parameters passed as command line switches.

eg:-

CONSOLIDATE /l=C:\log3.log /t=CPU /c=weeklyaverage /a=12w /r=Mo1200

logfile /l=C:\log3.log
test type /t=CPU
consolidation type /c=weeklyaverage
consolidate only data over a given age /a=12w
representative time /r=Mo1200

KS-Soft · Post by **KS-Soft** » Tue Oct 03, 2006 4:03 pm

Yes, we have such task in the "to do" list

Regards
Alex

terje · Post by **terje** » Tue Oct 03, 2006 11:13 pm

You guys are really on the ball.

terje · Post by **terje** » Thu Mar 13, 2008 7:15 pm

Has this been implemented?

Also it would be good to be able to prune a log file such as:-

for test "CPU \\serverX" in privatelog1.htm where test is prior to 2007-10-01 then delete log entry

KS-Soft · Post by **KS-Soft** » Fri Mar 14, 2008 11:07 am

We have created test version of Log Compressor but we do not want to release it for now. Instead we created another application that provides some "compressor" functions as well and it may show charts for the test items using these compressed logs. So you will be able to store old huge logs in archive and work with compressed logs + some monthly logs (just for example).
Log Visualizer will be included in next version of HostMonitor.

Regards
Alex

terje · Post by **terje** » Tue Mar 25, 2008 5:56 pm

Log Visualizer will be included in next version of HostMonitor.

Does that mean in version 8 or in the next release?

KS-Soft · Post by **KS-Soft** » Wed Mar 26, 2008 12:14 am

Version 6.18

Regards
Alex

terje · Post by **terje** » Wed Mar 26, 2008 12:58 am

Sorry I am confused.

How can the next version of Hostmonitor be version 6.18 when the current version is 7.10.

KS-Soft Europe · Post by **KS-Soft Europe** » Wed Mar 26, 2008 1:14 pm

Sorry for confusing. Alex meant version 7.18 of course.
Version 7.18 has just been released: http://www.ks-soft.net/hostmon.eng/news.htm

You should take a look at the new "Log Visualizer" application: http://www.ks-soft.net/hostmon.eng/lv/index.htm

Regards,
Max

terje · Post by **terje** » Wed Mar 26, 2008 4:42 pm

Thanks for the clarification.

Having read the overview this application does not seem to solve the storage problem I am concerned with. It seems to be a tool for making charts quickly on the fly. Which isn't what I require. At the moment I create all the charts I need using a batch process and they are never more than six hours old which is fine for us. The batch process would be quicker if the logs could be pruned but it isn't predominantly a speed issue. Mostly it is a storage issue. Although the way these logs are growing I can see it becoming a speed issue for the batch processes in a year or so.

If I log CPU speed for every server across about 30 sites and I do this every 5 minutes for 3 years I get a very, very large set of log file. I need to prune and consolidate these logs.

KS-Soft · Post by **KS-Soft** » Wed Mar 26, 2008 5:29 pm

1) If you need to get some historical overview only rarely (not on daily basis), you may use "Automatically change the log every month" logging option and "special file processing" option to zip your old log files.
In such case you don't need any other software, just HostMonitor
http://www.ks-soft.net/hostmon.eng/mfra ... processing

2) If you need to check history pretty often, you may add Log Visualizer to your scheme.
You may use the same 2 options to manage your logs:
- "Automatically change the log every month" to create new log files
- "special file processing" to zip old log files
Also you may start Log Visializer to process log file before zipping. In such case Log Visualizer will store some compact information about test results in "repository" and will be able to show historical charts even when log file is removed/zipped/moved.

BTW Log Visualizer provides compact HTML files as well, check LogInfo directory after log processing.

Regards
Alex

terje · Post by **terje** » Wed Mar 26, 2008 5:32 pm

Thanks for the clarification.

Having read the overview this application does not seem to solve the storage problem I am concerned with. It seems to be a tool for making charts quickly on the fly. Which isn't what I require. At the moment I create all the charts I need using a batch process and they are never more than six hours old which is fine for us. The batch process would be quicker if the logs could be pruned but it isn't predominantly a speed issue. Mostly it is a storage issue. Although the way these logs are growing I can see it becoming a speed issue for the batch processes in a year or so.

If I log CPU speed for every server across about 30 sites and I do this every 5 minutes for 3 years I get a very, very large set of log file. I need to prune and consolidate these logs.

KS-Soft · Post by **KS-Soft** » Wed Mar 26, 2008 5:50 pm

Have you read my answer? What exactly you do not like? You do not like to use zip to compress old log files? You don't like "consolidated" HTML files provided by Log Visualizer?
Please do not post the same text 3rd time.

Regards
Alex

terje · Post by **terje** » Wed Mar 26, 2008 6:49 pm

Have you read my answer?

Actually no, I hadn't. I'm not sure why my comment got posted twice but it was in response to KS-Soft Europe, not to your latest comment. And it was not meant to be posted twice so I'm not sure what happened in this instance. Sorry if it confused.

What exactly you do not like?

I don't dislike the application proposed. I merely said it does not appear to solve the problem I have.

You do not like to use zip to compress old log files?

Given that the logs are currently stored in a folder on an NTFS volume with compression enabled this would not appear to offer any advantage. I don't want to zip or compress the logs I want to prune them.

You don't like "consolidated" HTML files provided by Log Visualizer?

They seem to be additonal to the log files, rather than replacing the log files. However I may have misunderstood. At the moment we place all log entries for a given customer into a single log file. I might want to discard 3 month old data for some types of tests but for other tests I want to keep the data for three or more years. If I do understand the new approach correctly I will need to separate each type of test into a separate file and then break the logs into separate files based on timeframe. I will need to know ahead of time what timeframes are meaningful. However this isn't always the case. And it would seem that if I want to keep three years worth of data on CPU usage I still have a lot of log entries all be it spread across multiple files. I want to prune and consolidate, not fragment and compress. However maybe I am missing something.

KS-Soft · Post by **KS-Soft** » Thu Mar 27, 2008 6:16 am

Lets clarify this topic. If you want to say that Log Visualizer does not completely replace "Log Consolidator" then I agree with you. Its not a replacement however it may help in some (many?) cases.
As I said it "provides some "compressor" functions as well and it may show charts for the test items using these compressed logs. So you will be able to store old huge logs in archive and work with compressed logs + some monthly logs (just for example)"
Note: here "compressed" does not mean "zipped"/"archived"

If you want to say that you need to keep your logs uncompressed all the time, you do not want to use Log Visualizer because you want to use your own script to generate charts, well in such case Log Visualizer will not help you.

If I do understand the new approach correctly I will need to separate each type of test into a separate file and then break the logs into separate files based on timeframe

No, you don't need to separate each type of test into a separate file.

I will need to know ahead of time what timeframes are meaningful

No, you don't need to know ahead of time what timeframes are meaningful.

And it would seem that if I want to keep three years worth of data on CPU usage I still have a lot of log entries all be it spread across multiple files. I want to prune and consolidate, not fragment and compress

As I said Log Visualizer may create compact HTML files with compact/consolidated/average information about all tests items.
I think first you should look at software. If you will not like format of that compact files, then please tell us what exactly you do not like.

Regards
Alex