[rrd-developers] Re: "remote RRD" - A prototype

[Todd Caine]

This is very cool.  I just thought I'd share some ideas on the kind of
functionality that I've seen implemented with a "remote rrdtool".

I'm using a very similar piece of software which I call the RRDUpdater which was
developed by a co-worker of mine.  It handles update requests but also creates
RRD files from predefined sources to predefined locations (I know this is kind of
scary).  I'm getting ready to add a basic challenge/response authentication to
the updater soon so currently it can only accept update requests from localhost.
I also want to add RRD::tune() functionality.

The reason we wrote the software was because we have thousands of RRD files that
need to be updated every few minutes and we weren't able to finish writing the
*interesting* data within the polling period.  One thing we noticed was that a
large majority of the files were being updated with non-interesting values such
as zero, so we decided to implement low and high priority queues so that the
interesting data gets written as soon as possible.  Also, if a second update
request is received for an RRD file, the update requests are linked together and
promoted to the high priority queue; then when the updater can get to the
request, both are done at the same time, as opposed to having to open the file
twice.  The beauty of the RRDUpdater is that we get the important data written to
disk as fast a possible and the un-interesting data written when ever we have
time, the collectors and updaters can be distributed, and you can send update
requests to multiple updaters for redundancy, etc.....

The current protocol looks something like:

PREFIX /tmp  <- must be one of the servers predefined paths for storing rrdfiles
UPDATE <priority> <file> <template> <time:value[s]>

Currently the software is written in Perl and has some things hard coded into it
like the RRD create args, but the protocol could be easily extended to add named
templates and then use the template's name in the update request.

ie.
TEMPLATE <name> <template>
UPDATE <priority> <file> <template_name> <time:value[s]>

Oh, there's also a STATUS message that gives all sorts of stats regarding the
work queues, the number of updates, high/low watermarks, etc.  We have a script
which connects and issues the STATUS request to the RRDUpdater and then stores
this data into another RRD file!!!  So we are using RRDs to monitor our
RRDUpdater.  It's definitely a work in progress but I thought that the concepts
may be useful for the C version, since signal handling under Perl isn't safe yet.

Regards,
tOdd


Poul-Henning Kamp wrote:

> http://phk.freebsd.dk/rrrd/
>
> README:
> =======
>
> rrrd version 1.0
>
> Man 13 Aug 2001 22:21:34 CEST
>
> ----------------------------------------------------------------------------
> "THE BEER-WARE LICENSE" (Revision 42):
> <phk at FreeBSD.ORG> wrote this file.  As long as you retain this notice you
> can do whatever you want with this stuff. If we meet some day, and you think
> this stuff is worth it, you can buy me a beer in return.   Poul-Henning Kamp
> ----------------------------------------------------------------------------
>
> This is a tiny thing I've written because it annoyed me that there were
> no structured remote interface to Tobi's RRDtool.
>
> Basically this consists of a server and a client.
>
> The server is a small C program which received UDP datagrams with
> stuff for "RRDupdate" in it, and if it looks right tries to do the
> update.  The server will only act on existing files in the current
> directory, so security should be OK if you make a dedicated directory
> for storing your .rrd files.
>
> The client can be used either like a C procedure or a program which
> reads stdin for lines like "rrdtool -", with the footnote that it
> only accepts "update" lines, sticks them in an UDP packet and ships
> it off to the server.
>
> A Perl and a Tcl binding for the C version of the client will be nice
> but I have no idea how to write a Perl function so expect only the
> Tcl thing from my hand.
>
> This file is all the doc there is for now.
>
> There is no attempt made to provide any kind of security.  This
> puts it on par with syslog.  The first two bytes of the packet is
> a "protocol version number" and currently equal to "00", this allows
> for future packet formats to contain hash or public/private key
> style authentications.
>
> Have fun,
>
> Poul-Henning


--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi