[rrd-users] rrds on ext2 partition?

[nate]

Bernard Li wrote:
> In conjunction with the ongoing discussions here and in rrd-developers
> on "accelerating" rrdtool (either via using RAM/SSD as the underlying
> storage component) or having an accelerator daemon, I'd like to hear
> your thoughts on using ext2 as the underlying filesystem for storing
> the rrds (or any other filesystems).
>
> My current experience is with lots of rrd files on ext3 filesystem
> which causes high load and high I/O resulting in unusable system.
> When I put the rrds in an ext2 filesystem, load has dropped
> dramatically and it still has fairly high I/O, but at least the system
> is not locked up.

Any way you can reduce the amount of RRD files by consolidating data
points into fewer files? My main system has about 19 data points per
file. 6878 data points in 369 files updated once a minute(running cacti).
Some files have only 2 data points, others have as many as 32.

CPU I/O wait is less than 2% with a 4-disk SATA RAID 10 setup running
RRDTool 1.2.x. And it takes about 5 seconds to collect all of the
data and update all of the files. System is dual proc quad core with
16GB memory, though averages 14% cpu and 600MB memory usage. I migrated
my RRDs to a BlueArc NFS NAS box a few weeks ago, mainly for availability
purposes(can mount the volume on any system), I/O hasn't been an issue
before or after the migration. I don't know what sort of file system
BlueArc uses other than it's a custom cluster file system that is
ASIC accelerated.

A few years ago I had a half dozen systems with RRDTool 1.0.x typically
with 1 data point per file and even a RAID-10 of 14 10k RPM disks would
struggle mightily with 5000 RRD files(systems were almost unusable at
the command line the I/O wait was so bad).

Performance for me has gone up more than 10x with RRDTool 1.2.x and
massive consolidation of data points over 1.0.x.

nate