[rrd-developers] Re: Request: Dynamic creation/deletion of datasources..

[Alex van den Bogaerdt]

Steve Fulton wrote:

>   It was suggested by the rrd-users list that I ask here, so:  I really like
> RRDTool, but one of the major tasks I need to do involves dealing with
> multiple datasources -- and additions will be occuring daily.. Would it be
> possible in a (near?) future version of RRDTool?

The best thing to do is not to add DSes to an existing RRD but rather add
a new RRD with one DS.  There's no problem in combining information from
more than one RRD when you want to create an image, just specify which
DS you'd like to use from which RRD.

Finding out which RRD to use is as hard, or maybe even more simple, than
finding out which DSes to use when all DSes are in one big file.

Maybe I don't see your special needs.  If you have a good reason to use
one RRD with multiple DSes, please share.

>   Just so I'm clear, I'll explain exactly what one of these tasks is:  I
> work for an ISP managing mail servers, and I've written some code to parse
> the logfiles to determine who is sending out the most amount of mail, who is
> sending our systems the most amount of mail.  This is done for the purpose
> of finding spammers, preferably in real time.  Anyway, obviously the users
> sending the most amount of mail changes day to day, week to week, month to
> month.  I want to import this into RRDTool and graph it accordingly.   The
> problem is that RRDTool only allows me to create N datasources when I create
> the initial DB -- and I would like to add or delete datasources on the fly..
> is that possible?

I guess this means many users, many mutations in the userbase etcetera,
all you can expect with a larger ISP (or: set of ISPs).  This also reflects
on the number of DSes in your RRA.  I wish you good luck.


Anyway, if you need to add (and probably remove) DSes on a daily basis
(probably more frequent: "preferably in real time.") I guess there is
only one option and that is to do it using a (yet unwritten) part of
rrdtool.  Assuming many updates, using dump and restore will probably
prove to be too slow.

Have a look at rrd_format.h and see what needs to be done:
A new DS specifier needs to be inserted along with some related
stuff such as pdp_prep.  For each RRA there's a lot to be done as
the data is stored as:

   row 0: <value for DS 0> <value for DS 1> ... <value for DS n>
   row 1: <value for DS 0> <value for DS 1> ... <value for DS n>
   ...
   row n: <value for DS 0> <value for DS 1> ... <value for DS n>

This means *virtually every* single value needs to be moved:

(R0D0 means row 0 DS 0 and so on)
(view in fixed width font !)


   R0D0 R0D1 R1D0 R1D1 R2D0 R2D1 R3D0 R3D1
      |    |    |    |    |    |    |    |
      |    |    |    |    |    |    |    +-----------+
      |    |    |    |    |    |    +-----------+    |
      |    |    |    |    |    +------+         |    |
      |    |    |    |    +------+    |         |    |
      |    |    |    +-+         +    |         |    |
      |    |    +-+    |         |    |         |    |
      |    |      |    |         |    |         |    |
   R0D0 R0D1 R0D2 R1D0 R1D1 R1D2 R2D0 R2D1 R2D2 R3D0 R3D1 R3D2
              |              |              |              |
              |              |              |              |
          values to be inserted inbetween already present values

I think this is *very* intensive and needs good justification.
If you think you can do it using rrdtool dump and rrdtool restore,
please try.  The format of the intermediate xml file makes it not
to hard to do it as it presents the data per row which reduces the
problem of inserting into a problem of appending.

IMHO you didn't give a good reason for this addition and you can
(should) use one DS per RRD.  I didn't look back into the rrd-users
list and I can't recall such a reason from memory.

cheers,
-- 
   __________________________________________________________________
 / alex at slot.hollandcasino.nl                  alex at ergens.op.het.net \
| work                                                         private |
| My employer is capable of speaking therefore I speak only for myself |
+----------------------------------------------------------------------+
| Technical questions sent directly to me will be nuked. Use the list. | 
+----------------------------------------------------------------------+
| http://faq.mrtg.org/                                                 |
| http://rrdtool.eu.org  --> tutorial                                  |
+----------------------------------------------------------------------+

--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi