KS-Soft. Network Management Solutions
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister    ProfileProfile    Log inLog in 

log file consolidator
Goto page 1, 2  Next
 
Post new topic   Reply to topic    KS-Soft Forum Index -> Wish list
View previous topic :: View next topic  
Author Message
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Tue Oct 03, 2006 1:29 am    Post subject: log file consolidator Reply with quote

One of the great things about hostmonitor is the log analyser. By running private logs and calling the log analyser via a series of scripts we have web pages that show the history of different tests on a stack of servers across many sites.

One test that we have found very useful in this area is CPU testing. By logging every 5 minutes we have been able to create some long term charts that give us a very good baseline of CPU usage. We intend keeping three years data so we can monitor progress over time. Although usually we are only looking at charts for the last few weeks. Often we can spot when a process has hung.

The problem with all this however is that we end up with some very large log files. They are quite compressible so storage is not so much of an issue, however once they are large they take a very long time to process.

What would be ideal would be to execute a program that went through a specified log file and based on some parameters could do things such as the following:-

Quote:
For all data more than 12 weeks old, then replace all logs for each week with the average for that week. (Set time for represenative average as 12 noon Monday).


Quote:
For all data more than 90 days old, then for each day keep only the peak value for that day and purge other entries.


Quote:
For all data more than 21 days old, then for each day keep only the first log entry after 4pm.


Quote:
For all data more than 1000 days old then purge that data.


Such a tool would allow for log files to be stripped down to a more manageable size whilst preserving the historical trends or baselines required.

Note that in the examples given running a well designed "consolidator" more than once would not change anything the second time through. Assuming of course that the current date and the parameters have not changed. This would mean that progressive subsequent consolidations on later dates (using the same parameters) would not change anything. Of course for the first example this depends on carefully defining the start and finish boundary points for a week.

The "consolidator" would also want to be able to filter on things such as test type or test name. For instance you may only want to average old CPU tests, but not old ping tests.

And for scripting it should all be executable from a command line with the parameters passed as command line switches.

Quote:
eg:-

CONSOLIDATE /l=C:\log3.log /t=CPU /c=weeklyaverage /a=12w /r=Mo1200

logfile /l=C:\log3.log
test type /t=CPU
consolidation type /c=weeklyaverage
consolidate only data over a given age /a=12w
representative time /r=Mo1200
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Tue Oct 03, 2006 4:03 pm    Post subject: Reply with quote

Yes, we have such task in the "to do" list

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Tue Oct 03, 2006 11:13 pm    Post subject: Reply with quote

You guys are really on the ball.
Back to top
View user's profile Send private message
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Thu Mar 13, 2008 7:15 pm    Post subject: Reply with quote

Has this been implemented?

Also it would be good to be able to prune a log file such as:-

Quote:

for test "CPU \\serverX" in privatelog1.htm where test is prior to 2007-10-01 then delete log entry
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Fri Mar 14, 2008 11:07 am    Post subject: Reply with quote

We have created test version of Log Compressor but we do not want to release it for now. Instead we created another application that provides some "compressor" functions as well and it may show charts for the test items using these compressed logs. So you will be able to store old huge logs in archive and work with compressed logs + some monthly logs (just for example).
Log Visualizer will be included in next version of HostMonitor.

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Tue Mar 25, 2008 5:56 pm    Post subject: Reply with quote

Quote:
Log Visualizer will be included in next version of HostMonitor.


Does that mean in version 8 or in the next release?
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Wed Mar 26, 2008 12:14 am    Post subject: Reply with quote

Version 6.18

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Wed Mar 26, 2008 12:58 am    Post subject: Reply with quote

Sorry I am confused.

How can the next version of Hostmonitor be version 6.18 when the current version is 7.10.
Back to top
View user's profile Send private message
KS-Soft Europe



Joined: 16 May 2006
Posts: 2832

PostPosted: Wed Mar 26, 2008 1:14 pm    Post subject: Reply with quote

Sorry for confusing. Alex meant version 7.18 of course.
Version 7.18 has just been released: http://www.ks-soft.net/hostmon.eng/news.htm

You should take a look at the new "Log Visualizer" application: http://www.ks-soft.net/hostmon.eng/lv/index.htm

Regards,
Max
Back to top
View user's profile Send private message Send e-mail Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Wed Mar 26, 2008 4:42 pm    Post subject: Reply with quote

Thanks for the clarification.

Having read the overview this application does not seem to solve the storage problem I am concerned with. It seems to be a tool for making charts quickly on the fly. Which isn't what I require. At the moment I create all the charts I need using a batch process and they are never more than six hours old which is fine for us. The batch process would be quicker if the logs could be pruned but it isn't predominantly a speed issue. Mostly it is a storage issue. Although the way these logs are growing I can see it becoming a speed issue for the batch processes in a year or so.

If I log CPU speed for every server across about 30 sites and I do this every 5 minutes for 3 years I get a very, very large set of log file. I need to prune and consolidate these logs.
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Wed Mar 26, 2008 5:29 pm    Post subject: Reply with quote

1) If you need to get some historical overview only rarely (not on daily basis), you may use "Automatically change the log every month" logging option and "special file processing" option to zip your old log files.
In such case you don't need any other software, just HostMonitor
http://www.ks-soft.net/hostmon.eng/mframe.htm#options.htm#logprocessing

2) If you need to check history pretty often, you may add Log Visualizer to your scheme.
You may use the same 2 options to manage your logs:
- "Automatically change the log every month" to create new log files
- "special file processing" to zip old log files
Also you may start Log Visializer to process log file before zipping. In such case Log Visualizer will store some compact information about test results in "repository" and will be able to show historical charts even when log file is removed/zipped/moved.

BTW Log Visualizer provides compact HTML files as well, check LogInfo directory after log processing.

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Wed Mar 26, 2008 5:32 pm    Post subject: Reply with quote

Thanks for the clarification.

Having read the overview this application does not seem to solve the storage problem I am concerned with. It seems to be a tool for making charts quickly on the fly. Which isn't what I require. At the moment I create all the charts I need using a batch process and they are never more than six hours old which is fine for us. The batch process would be quicker if the logs could be pruned but it isn't predominantly a speed issue. Mostly it is a storage issue. Although the way these logs are growing I can see it becoming a speed issue for the batch processes in a year or so.

If I log CPU speed for every server across about 30 sites and I do this every 5 minutes for 3 years I get a very, very large set of log file. I need to prune and consolidate these logs.
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Wed Mar 26, 2008 5:50 pm    Post subject: Reply with quote

Have you read my answer? What exactly you do not like? You do not like to use zip to compress old log files? You don't like "consolidated" HTML files provided by Log Visualizer?
Please do not post the same text 3rd time.

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
terje



Joined: 25 Jul 2005
Posts: 88
Location: Sydney

PostPosted: Wed Mar 26, 2008 6:49 pm    Post subject: Reply with quote

Quote:
Have you read my answer?


Actually no, I hadn't. I'm not sure why my comment got posted twice but it was in response to KS-Soft Europe, not to your latest comment. And it was not meant to be posted twice so I'm not sure what happened in this instance. Sorry if it confused.

Quote:
What exactly you do not like?


I don't dislike the application proposed. I merely said it does not appear to solve the problem I have.

Quote:
You do not like to use zip to compress old log files?


Given that the logs are currently stored in a folder on an NTFS volume with compression enabled this would not appear to offer any advantage. I don't want to zip or compress the logs I want to prune them.

Quote:
You don't like "consolidated" HTML files provided by Log Visualizer?


They seem to be additonal to the log files, rather than replacing the log files. However I may have misunderstood. At the moment we place all log entries for a given customer into a single log file. I might want to discard 3 month old data for some types of tests but for other tests I want to keep the data for three or more years. If I do understand the new approach correctly I will need to separate each type of test into a separate file and then break the logs into separate files based on timeframe. I will need to know ahead of time what timeframes are meaningful. However this isn't always the case. And it would seem that if I want to keep three years worth of data on CPU usage I still have a lot of log entries all be it spread across multiple files. I want to prune and consolidate, not fragment and compress. However maybe I am missing something.
Back to top
View user's profile Send private message
KS-Soft



Joined: 03 Apr 2002
Posts: 12791
Location: USA

PostPosted: Thu Mar 27, 2008 6:16 am    Post subject: Reply with quote

Lets clarify this topic. If you want to say that Log Visualizer does not completely replace "Log Consolidator" then I agree with you. Its not a replacement however it may help in some (many?) cases.
As I said it "provides some "compressor" functions as well and it may show charts for the test items using these compressed logs. So you will be able to store old huge logs in archive and work with compressed logs + some monthly logs (just for example)"
Note: here "compressed" does not mean "zipped"/"archived"

If you want to say that you need to keep your logs uncompressed all the time, you do not want to use Log Visualizer because you want to use your own script to generate charts, well in such case Log Visualizer will not help you.

Quote:
If I do understand the new approach correctly I will need to separate each type of test into a separate file and then break the logs into separate files based on timeframe

No, you don't need to separate each type of test into a separate file.

Quote:
I will need to know ahead of time what timeframes are meaningful

No, you don't need to know ahead of time what timeframes are meaningful.

Quote:
And it would seem that if I want to keep three years worth of data on CPU usage I still have a lot of log entries all be it spread across multiple files. I want to prune and consolidate, not fragment and compress

As I said Log Visualizer may create compact HTML files with compact/consolidated/average information about all tests items.
I think first you should look at software. If you will not like format of that compact files, then please tell us what exactly you do not like.

Regards
Alex
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    KS-Soft Forum Index -> Wish list All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

KS-Soft Forum Index