[rrd-developers] Re: How to get the most performance when using lots of RRD files

[Ian Holsman]

personally I always splitting out the meta-data from the variable  
data would help alot.

for example of the 100k of RRDs I have, most have exactly the same  
definition/RRA length etc.
so in this case my update program could load the definition areas  
once in memory, and then proceed to update
the RRD variable stuff.

ie.. split out the 3 parts of a RRD file into 3 files, and share some  
it.
my update program would then go from:

for i in *update do
  rrd_update filename value
end

to
read stat_head_t_file

for i in *update do
    rrd_update stat_head_t filename value

this would at least save on a read/seek
you could possibly also factor out the last update time so that  
multiple files all have the same value
which you update at the end.

you still end up with files, but might even end up with a single file  
per RRA and rra_def_t



On 18/08/2006, at 2:48 AM, Thorsten von Eicken wrote:

> Ole Bjørn Hessen wrote:
>>
>> I see a couple of ways out of this complexity problem:
>>
>> 1. Rewrite RRD-tool to separate the database from the graphing and  
>> wait
>>    for someone to come up with a much smarter database method ;-)
>>
> Yup, this would be a huge step forward for RRDtool! It would make it
> more into a toolkit and make it easier for others to contribute as  
> they
> could focus on a specific portion of the codebase/functionality. By
> separating the storage, we could experiment with storing individual  
> RRAs
> in a SQL database or in a BDB. I can imagine that the buffer  
> management
> and sequential log writes built into databases could significantly
> improve the performance, but then I can also imagine that they could
> screw it up in a major way :-).
>
> Thorsten
>
> --
> Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch? 
> subject=unsubscribe
> Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
> Archive     http://lists.ee.ethz.ch/rrd-developers
> WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi
>

--
Ian Holsman
Ian at Holsman.net
http://car-chatter.com/ where car fanatics meet


--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-developers
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi