[rrd-users] Re: I know this is repetitive... but, is there a way to add a new DS to

[Przemyslaw Karwasiecki]

On Tue, 2002-06-25 at 06:08, Hamish Marson wrote:
> 
> Alex van den Bogaerdt wrote:
> 
> >Francisco Rivas C wrote:
> >
> ><snip long story>
> >
> >There currently is no way to add a DS without doing quite some work.
> >I can see four ways to approach your problem.  I present them in the
> >order which I think are best to worst:
> >
> >1: rethink your collecting/updating script.  I find it hard to believe
> >   you cannot alter it so that in one pass it generates multiple rrdtool
> >   update lines, one DS per RRD would be best.
> >
> 
> The problem with this is keeping the data in step. Eventually the 
> updates take so long to process that the data is too skewed to be useful 
> when you want it to mean something. If you have 200 DS's to be updated 
> every 10 seconds, and it takes 100ms to update each one, then you are 
> going to be falling behind... Having to open, seek & update even 32 
> files (My current number of DS's in my mail rrd's) is going to take way 
> more resources than just updating 32 columns in just one file.
> 
> There is a limit of course when you should split instead of adding a DS 
> to an existing rrd. But just blindly thinking you should just add rrd's 
> instead of adding DS's to an existing rrd is just as bad.

I am either missing something or I need to disagree with you.
I understand your comment related to performance, but I don't 
understand "The problem with this is keeping the data in step"

Basically, the process of data acquisition, 
can be split into 2 steps:
1. collecting data with SNMP,
2. storing data into some archive - RRD in this case.

Ad 1.
First, during data collection, you need to associate your data,
with specific moment in time, when they have been collected.
There are several ways to do this:

Simplistic -- based on (assumption that there is no latency 
             in agent/network:
Gather all data in single or multiple SNMP packets, and timestamp
them as they arrive.

Realistic -- based on assumption that SNMP agent will not be
             preempted, while processing single packet response:
Gather all data in single or multiple SNMP packets, but add to the
list of OIDs you are retrieving, sysUpTime.0 -- This will be your 
timestamp.

Ad 2.
Second, once you have your data properly timestamped, there is 
nothing which will prevent you from building RRD update string
with proper timestamp -- hence actual time, when you perfrom 
RRD update is irrelevant -- RRD will use their interpolation
mechanism based on update time passed to it in update string,
-- not the actual time when update operation is being performed.

Any comments?

Przemek
PS -- As far as performance is concerned, I have a script which 
is updating 1300 RRD files:

[karwas at cflow nm]$ ls -l /disk/store0/RRD/*.rrd | wc
   1309   11781  158471
[karwas at cflow nm]$ 

and it takes less then couple second to do this.

I know that this is not a scientific performance measurement,
but I would be more concerned and troubled with rebuilding RRDs
online, (dump to XML --> modify --> compile/rebuild ), 
then with highly optimized (thank you, Tobi ;-) RRD update
(open, lseek, write).

Przemek


--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi