Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Endpoint Management Community Blog

{CWoc} Log Analyser version 0.1.8 Tagged, Now with Detailed Analysis (starting)

Created: 19 Dec 2009 • Updated: 21 Dec 2009
Ludovic Ferre's picture
0 0 Votes
Login to vote

So as time goes by I'm now in my 2nd week-end on {CWoc}. Definitely a week end that doesn't entice anyone to get out for long (-4C right now) as we had quite a bit of snow (and only the main streets are cleared).

In all case I have further progress the II Log analyser for Altiris solutions. I now displays detailed information such as error codes and hourly statistics. This is still not the full details implementation but it's a major milestone for me.

Here's the output I now have (I did a little bit of formatting I must admit it) from the many test I've run on my base workload (6 IIS files from a few production servers totaling ~ 800MB):

ludovic@ub-x64:~/dev/altiris-ns-tooling/logAnalyser$ time ./v018 --file log_files/382-08

Mime type analysis summary results:
	File type= htm ,  page hits=   5,611
	File type= js  ,  page hits=   1,561
	File type= css ,  page hits=     530
	File type= asp ,  page hits= 221,542
	File type= aspx,  page hits= 568,599
	File type= asmx,  page hits=  25,234
	File type= other, page hits= 118,977

Altiris Agent request analysis summary results:
	Agent request= Reg Client,	page hits=      30
	Agent request= Get Policies,	page hits=  43,099
	Agent request= Get Pkg Info,	page hits=  13,865
	Agent request= Get Snapshot,	page hits= 507,205
	Agent request= Post Event ,	page hits= 221,485
	Agent request= Other, 		page hits= 156,370

IIS Web-applications analysis summary results:
	Webapp= /Altiris/NS/Agent/, dir hits = 785,852
	Webapp= /Altiris/NS/NSCap/, dir hits =     340
	Webapp= /Altiris/NS/,		dir hits =  28,157
	Webapp= /Altiris/Resource/,	dir hits =     375
	Webapp= /Altiris/IRA[1]/,	dir hits =   4,170
	Webapp= Others,			dir hits = 123,160

[1] IRA is an abbreviation of InventoryRuleManagement/Agent

Detailed IIS status code analysis results:
	IIS Status code= Success	(1xx,2xx),	hits count = 887,429
	IIS Status code= Redirected 	(3xx),		hits count =  41,648
	IIS Status code= Client error	(4xx),		hits count =  12,973
	IIS Status code= Server error	(5xx),		hits count =       4

Detailed IIS status code analysis results:
	Sub Status code= 0, hits count = 930,353
	Sub Status code= 1, hits count =   9,161
	Sub Status code= 2, hits count =   2,538
	Sub Status code= 9, hits count =       2
[9] means sub status code > 9

Detailed IIS Win32 status code analysis results:
	Win32 Status code= Win32 Success,	hits count = 923,159
	Win32 Status code= Win32 Failure < 0,	hits count =   2,533
	Win32 Status code= Win32 Failure > 0,	hits count =  16,362

24 hour hit counters:
	Hits counted during hour  0 to  1  was 29,408
	Hits counted during hour  1 to  2  was 20,850
	Hits counted during hour  2 to  3  was 21,029
	Hits counted during hour  3 to  4  was 15,733
	Hits counted during hour  4 to  5  was 22,524
	Hits counted during hour  5 to  6  was 19,795
	Hits counted during hour  6 to  7  was 24,261
	Hits counted during hour  7 to  8  was 33,728
	Hits counted during hour  8 to  9  was 58,475
	Hits counted during hour  9 to 10  was 52,183
	Hits counted during hour 10 to 11  was 51,016
	Hits counted during hour 11 to 12  was 49,001
	Hits counted during hour 12 to 13  was 53,959
	Hits counted during hour 13 to 14  was 61,097
	Hits counted during hour 14 to 15  was 65,136
	Hits counted during hour 15 to 16  was 78,953
	Hits counted during hour 16 to 17  was 74,775
	Hits counted during hour 17 to 18  was 42,575
	Hits counted during hour 18 to 19  was 33,054
	Hits counted during hour 19 to 20  was 27,292
	Hits counted during hour 20 to 21  was 28,061
	Hits counted during hour 21 to 22  was 23,421
	Hits counted during hour 22 to 23  was 26,310
	Hits counted during hour 23 to 24  was 29,418

Brought to you by {Connect Winter of Code}

real	0m22.175s
user	0m22.010s
sys	0m0.180s

Note that even with the added functionallities the timing remain quite constant. 22 seconds to parse close to 1 million lines is pretty good imho :D.

I have attached a pdf snapshot of the code (at version 0.1.9 and revision 53 actually, as I've done major re-ordering and out-sourcing to function to simplify the code), and I was quite surprised to see that it's 13 pages long, with just over the 800 lines mark (loads of comments in there of course).