[mrtg] Re: content (site) checking

Tue Jul 17 16:00:32 MEST 2001

I had to do exactly the same thing a couple of weeks ago and after a lot of
searching and trying lots of different programs, I finally went for httrack
since is the most configurable one and has tons of options that make
tweaking much easier, as a result I managed to download a web page almost
exactly as you see it in a browser, it even downloaded css files parsed some
javascript included in the page and managed cookies, with no extra nesting,
wsiwyg. Great tool, you can download it from www.httrack.com , if you need
some further help let me know cause I had to make some scripts to adjust
this to mrtg that you might find usefull.

Greetings,
Luis

Luis Martin Albinati
N.O.C.
Prima S.A. - Argentina
Ciudad Internet / Datamarkets
Phone #: (54)11-4370-0024
E-Mail: albo at prima.com.ar <mailto:albo at prima.com.ar> 

-----Original Message-----
From: Garnel, Eric [mailto:egarnel at Question.com]
Sent: Viernes, 29 de Junio de 2001 02:21 PM
To: 'Daniel J McDonald'
Cc: Mrtg (E-mail)
Subject: [mrtg] Re: content (site) checking

I took a different route... (on Windows 2000)

Found a script (below) on http://www.leh.net/~kevin/docs/mrtg/index_1.html
that I hacked up a bit.
Installed wget for Windows from wget
http://www.interlog.com/~tcharron/wgetwin.html

Now I just have to tweak the graph a little and I am all set... time for me
to RTFM

#!/usr/local/bin/perl 
# Copywrite: 
#  This Work is copywrite Barry Irwin , Moria Research 1999
#  All Rights Reserved
#
# Permission is given for derivitive works to be freely distributed provided
# Original Author is acknowledged, and this copywrite remains intact
#
# DISCLAIMER: Your milage may vary. NO WARRANTY provided. USE, Enjoy, but if
#   it blows up , or suchlike, badluck. Author Assumes NO responsibility for
#   this work
# VERSION=0.1
# Outputs the %used for a disk partition. parameter can be either disk slice
# name (/dev/da2s1f) or symbolic name ( eg /var)
# probably a good idea to set AbsMax to somethign like 115( since file
# systems can get >100% full.  guage type is also recomended
#
# OCTOBER 2000
# I have majorly hacked this script to get load times from a web server.
# ---kevin at leh.net

#root at rucus:/home/system/quark/www# df -k /var | tail -1
#/dev/da2s1f     595383   263439   284314    48%    /var

#if ( $ARGV[0] eq "" )
#{
#  print "/dev/hda7 \n";
#  exit;
#}

#uptimestuff
#$tmp =`uptime`;
#$tmp =~ s/,//g;
#@utime = split /\s/, $tmp;

#Kevin added line to generate the text file

$percent = 0;

# Kevin - Here, insert the url of what site/page you are measuring. (you
will need wget)
#         Capturing load time of the page, via wget's reporting. (you have
to delete the page it retrieves in 
#         cron or something, otherwise you will have a bunch of copies of
the same page.

foreach $_ (`wget question.crv.com -o E:\\wwwroot\\crvload\\saved.txt | more
saved.txt | grep saved`)
# grab the data out of the file
{
  ($device, $size, $used, $free, $percent, $mount) = split(/\s+/);
 # if ( $ARGV[0] eq $device || $ARGV[0] eq $device) 
         { chop($used);  last;}  #bail out
}

# Kevin adding new stuff
$percent = 0;

# Kevin - I do the exact same command again to grab different information
from the same line I grep
#         I am sure there is a better way to do this, not the master
scripter. Greping for page size.

foreach $_ (`wget question.crv.com -o saved.txt | more saved.txt | grep
saved`)
# grab the data out of the file
{
  ($device, $size, $used, $free, $percent, $mount) = split(/\s+/);
 # if ( $ARGV[0] eq $device || $ARGV[0] eq $device) 
         { chop($used);  last;}  #bail out
}

# Kevin adding new stuff
foreach $_ (`more saved.txt | grep Length`) 
# grab the data out of the file
{
  ($length, $size1, $used1, $free1, $percent1, $mount1) = split(/\s+/);
 # if ( $ARGV[0] eq $device || $ARGV[0] eq $device)
         { chop($used1);  last;}  #bail out
}

# This is chopping off the last bit of data on the load time, one character
at a time.
#  someone please email me and tell me how to do this quicker. -Kevin
chop ($device);
chop ($device);
chop ($device);
chop ($device);
chop ($device);
chop ($device);

chop ($size1);
chop ($size1);
chop ($size1);
chop ($size1);

# Kevin - The first value I wanted, just used the old variable name of
$device, not really relevant.
print "$device\n";
# Kevin - The second value I wanted, again just altered the variable name.
#print "$size1\n";
#print "$utime[3] days $utime[5] hours\n";
#print "\n";

One of these days, I will try to learn perl, in the meantime, I am just
dangerous!

Eric Garnel
Network Engineer CCNA, MCSE
Question Technologies
Turning Questions into Commerce

701 Brazos Street, Suite 1200
Austin, TX 78701
512/391-4120 direct
512/480-9136 fax
egarnel at question.com
http://www.question.com/

 -----Original Message-----
From: 	Daniel J McDonald [mailto:dmcdonald at digicontech.com] 
Sent:	Friday, June 29, 2001 11:21 AM
To:	Garnel, Eric
Subject:	RE: [mrtg] content (site) checking

look at the LWP library in perl... 
#!/usr/bin/perl
#

use LWP::UserAgent;

$ua = new LWP::UserAgent;
$URL ='http://printserver2.austin-fso/cgi-bin/testserver.pl';
$req = new HTTP::Request GET => $URL;
$foo = $ua->request($req);
$response = $$foo{_content};
$response =~ /<BODY>(.+)<\/BODY>/;
$data = $1;

Then add some timers and checking to your hearts content...

-----Original Message-----
From: mrtg-bounce at list.ee.ethz.ch [mailto:mrtg-bounce at list.ee.ethz.ch]On
Behalf Of Garnel, Eric
Sent: Friday, June 29, 2001 10:12 AM
To: Mrtg (E-mail)
Subject: [mrtg] content (site) checking

Forgive me if I have missed this, but I would like to monitor web site
response with MRTG using content verification. (perhaps using a post or get
command)
I could use ping probe or measure http on that particular server, but that
wouldn't tell me how long it takes for a page retrieval.
Some of our pages are generated by a database rather than flat files.

Eric Garnel
Network Engineer CCNA, MCSE
Question Technologies
Turning Questions into Commerce

701 Brazos Street, Suite 1200
Austin, TX 78701
512/391-4120 direct
512/480-9136 fax
egarnel at question.com
http://www.question.com/ <http://www.question.com/> 

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi