[rrd-users] Re: Who has the most RRD files (or data sources)?

[Mark Plaksin]

Tarus Balog <tarus at opennms.org> writes:

> On Mar 15, 2006, at 3:20 PM, Mark Plaksin wrote:
>
>> Who has the most RRD files (or total data sources in RRDs)?
>
> This is from an ISP in Oregon:
>
> -bash-3.00$ find . -name "*rrd" | wc -l
>    413132

Wow!

> We have a couple of client that large.
>
>>   How frequently are they updated?
>
> Once every five minutes.
>
>> What do you use to update them?
>
> OpenNMS. Each file is 283K long containing ~24K rows, and they each  
> represent a single SNMP variable.

Can you explain this a bit more?  Do you mean there's only one RRD data
source per file?  I think that's what "single SNMP variable" means but I
can't reconcile that with each RRD having "~24k rows".

How long do you keep data before consolidating?  283k sounds very small for
an RRD file.

> We found that with so many small files it is easy to overload the disk
> I/O (and please, please don't use RAID-5), so if we miss writing one
> value we will store it until we get the opportunity to write it again.
> It takes almost as much time to write 2 or 3 values as it does one, so we
> get caught up.  Without such a queue, the smallest load on the filesystem
> (like the one I caused doing the find above) would result in the
> application getting farther and farther behind until it crashed. Now once
> steady state is reached after startup, you can almost set your watch by
> the update.

Hmm, what does it mean to "miss writing one"?  Do you have an interval and
a time-limit associated with writing data to RRD files?  And if you don't
write all the data on hand within the limit you leave it until the next
interval to write them?

> Hardware notes: this is a Sun v440 with 8GB of RAM connected via NFS  
> to a NetApp NAS. Without the NetApp it would be almost impossible to  
> handle the load.

Assuming that you mean one DS per RRD, do you have any idea why some people
say having more DSes per RRD gives better performance?  Are you just
avoiding that issue because your disks are so fast?

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi