[rrd-developers] Suggestion for NAN-safe TREND operation

[Timo Stripf]

Hi Bernhard,

i have missing data not due performance problems 
or anything rrd related. It is simple the fact that the daemons are under
development and i've to restart them after i 
changed the code. Or a mysql query gets locked 
due to a big sql copy operation. I only noticed 
the trend behavior and thought i would look 
better if a small gap would not cause a large gap in the trend curve.

It sounds interesting that 1.3 will increase hdd 
performance since i use it for 8000 rrd files :)

Best regards
Timo Stripf



At 13:37 11.06.2007, you wrote:
>On Mon, Jun 11, 2007 at 12:45:36AM +0200, Tobias Oetiker wrote:
> >>written daemons. In my rrd data i have often short periods of missing
> >>data. In result the trend curve is missing over the complete sliding
> >>window. So i modified the TREND function to ignore NAN-number. I
>
> >Hi Timo,
> >
> >NAN safe TREND sounds interesting ...
>
>Note that this should happen much, much less frequent (or not at all)
>with current trunk, since rrdtool-1.2 is excercising the hard-disks way
>too much, as opposed to the upcoming 1.3 version.
>
>Since you seem to observe gaps in your graphs, i assume that you also
>suffer from very high load-average¹) due to waiting on blocking I/O, which
>a) renders your machine unuseable for interactive use
>b) is responsible for the fact that your data-collection and/or RRD
>    updates cannot finish in due time.
>
>The new file access method introduced in rrdtool-1.3 has several
>advantages over the FILE* (filp) based implementation that rrdtool-1.2
>uses:
>a) less copying.
>   The filp-based I/O had to do more than 3 (!) copies per RRD, whereas
>   the mmap-based I/O in rrdtool-1.3 does only 1 copy (in the kernel).
>   This greatly reduces the stress upon the memory-subsystem of your
>   host.
>b) less buffer-cache pollution.
>   As a consequence of a), we don't pollute the caches like the
>   filp-based file accessors did.
>c) less seeking
>   The filp-based I/O did several dozend (blocking!) seeks to update
>   the internal positions from where to read/write.
>   In the mmap-based implementation, we do not seek at all, thus greatly
>   reducing the time spent on-disk. We just calculate some integer
>   offsets into the file and use those for our read/write operations.
>d) less read/write ops
>   While the filp-based file-accessor approach did blocking reads/writes
>   (interleved with a dozend seeks) to read RRDs, then copy this data to
>   a malloc()ed area, eventually updated this now in-memory data only to
>   do a blocking write to propagate the update back to the file, the
>   mmap-based accessors operate directly on the 1 copy we created when
>   opening the RRD file.
>
>
>¹) where too high of course depends on the number of rrd's you have, but
>generally, everything above 1.0 which *blocks* will get you into trouble
>sooner or later.
> >
> >if you provide a patch for current trunk I will be glad to include
> >it for 1.3 ... make sure you patch docs as well.
>
>And while you're at it, give 1.3 a whirl and let us know if this makes
>the immediate need for your patch obsolete. That said, i can see how
>your proposal is still something very useful for machines that are way
>too small for the workload they are supposed to handle.
>
>cheers,