[rrd-users] Re: I know this is repetitive... but, is there a way to add a new DS to

[Hamish Marson]

Przemyslaw Karwasiecki wrote:

>On Tue, 2002-06-25 at 06:08, Hamish Marson wrote:
>  
>
>>Alex van den Bogaerdt wrote:
>>
>>    
>>
>>>Francisco Rivas C wrote:
>>>
>>><snip long story>
>>>
>>>There currently is no way to add a DS without doing quite some work.
>>>I can see four ways to approach your problem.  I present them in the
>>>order which I think are best to worst:
>>>
>>>1: rethink your collecting/updating script.  I find it hard to believe
>>>  you cannot alter it so that in one pass it generates multiple rrdtool
>>>  update lines, one DS per RRD would be best.
>>>
>>>      
>>>
>>The problem with this is keeping the data in step. Eventually the 
>>updates take so long to process that the data is too skewed to be useful 
>>when you want it to mean something. If you have 200 DS's to be updated 
>>every 10 seconds, and it takes 100ms to update each one, then you are 
>>going to be falling behind... Having to open, seek & update even 32 
>>files (My current number of DS's in my mail rrd's) is going to take way 
>>more resources than just updating 32 columns in just one file.
>>
>>There is a limit of course when you should split instead of adding a DS 
>>to an existing rrd. But just blindly thinking you should just add rrd's 
>>instead of adding DS's to an existing rrd is just as bad.
>>    
>>
>
>I am either missing something or I need to disagree with you.
>I understand your comment related to performance, but I don't 
>understand "The problem with this is keeping the data in step"
>
>Basically, the process of data acquisition, 
>can be split into 2 steps:
>1. collecting data with SNMP,
>2. storing data into some archive - RRD in this case.
>  
>

Firstly, why do you assume that all data is cololected by SNMP? The bulk 
of mine is collected from a logfile (Postfix & amavisd logs)

>Ad 1.
>First, during data collection, you need to associate your data,
>with specific moment in time, when they have been collected.
>There are several ways to do this:
>
>Simplistic -- based on (assumption that there is no latency 
>             in agent/network:
>Gather all data in single or multiple SNMP packets, and timestamp
>them as they arrive.
>
>Realistic -- based on assumption that SNMP agent will not be
>             preempted, while processing single packet response:
>Gather all data in single or multiple SNMP packets, but add to the
>list of OIDs you are retrieving, sysUpTime.0 -- This will be your 
>timestamp.
>
>Ad 2.
>Second, once you have your data properly timestamped, there is 
>nothing which will prevent you from building RRD update string
>with proper timestamp -- hence actual time, when you perfrom 
>RRD update is irrelevant -- RRD will use their interpolation
>mechanism based on update time passed to it in update string,
>-- not the actual time when update operation is being performed.
>
>Any comments?
>

A couple.

1. I'm not & others aren't using snmp. There is far more data to be 
processed that isn't snmp than is.
2. Not everyone has hardware that's capable of updating 1300 files in 
just a second or so.
3. When you perform the rrd update is NOT irrelevent if the update time 
is short. If yuou update once an hour, you have plenty of time. Every 10 
seconds or even every minutes, and it gets a little more critical.

YMMV of course... Which is a point I made.

>
>Przemek
>PS -- As far as performance is concerned, I have a script which 
>is updating 1300 RRD files:
>
>[karwas at cflow nm]$ ls -l /disk/store0/RRD/*.rrd | wc
>   1309   11781  158471
>[karwas at cflow nm]$ 
>
>and it takes less then couple second to do this.
>
>I know that this is not a scientific performance measurement,
>but I would be more concerned and troubled with rebuilding RRDs
>online, (dump to XML --> modify --> compile/rebuild ), 
>

You add new DS's perhaps once a month or once a week at the most? No 
need for much optimisation there. It doesn't actually require an XMP 
dump to do either, if you know the binary format of the file...

>then with highly optimized (thank you, Tobi ;-) RRD update
>(open, lseek, write).
>  
>

As it should be since it's the most ofthen used task. remember though 
with lots of files, just the open alone requires a linear search through 
the directory structure... That's a fair wack of IO just there...


>Przemek
>
>
>--
>Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
>Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
>Archive     http://www.ee.ethz.ch/~slist/rrd-users
>WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi
>
>  
>


-- 

I don't suffer from Insanity... 	| Linux User #16396
	I enjoy every minute of it...	|
					|
http://www.travellingkiwi.com/		|




--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi