[rrd-users] RHEL RPMS of rrdtool corrupts timestamps in both 1.3.9 and 1.4.7

[Ryan Kubica]

Jo,

In regards to RedHat:

Open a case with them and tell them about the corruption ... maybe they'll patch rrdtool and release a 1.3.8-xyz :-)

In regards to rrdtool:

Compile the latest revision 1.4.7 http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.7.tar.gz and if it also corrupts report that to the list here.

<rant>

I don't feel it's rrdtool's place to compile binaries; it's far too simple to build a new binary and there are too many variances in distros ... additionally some binaries which come from operating system distros are such old revisions I'd never _ever_ use them.  For rrdtool.org to provide binaries isn't good because rrdtool is often compiled against other software on the system: perl, mod_perl, python(?), pango, cairo, etc.

On a RHEL6.1 distro:

[epicadm at epicdn002 ]$ yum list | grep rrdtool
rrdtool.i686                             1.3.8-6.el6                      base  

In checking when that revision was released:
 rrdtool-1.3.8.tar.gz 19-May-2009 15:47  1.0M
2009 ... there are numerous improvements/changes between 1.3 -> 1.4 all worth upgrading too.  The -6 in the revision means RedHat made additional patches on their own, lord knows what, one would need to check redhat errata. 

This can be said for nearly every software release on a distro; and not all are double-tripple-checked by distro providers either; MySQL is old, Perl is old, Apache, memcached ... nearly everything are all old to some extent or another.  Should they be stable? Yes ... but it's the distros responsibility to keep up-to-date; and many things are included for no other reason than convenience and being able to state 'we have that!'

RHEL's version of perl for years was so broken it couldn't even be used to compile rrdtool, net-snmp or mod_perl ... so I was compiling perl for a long time as well.


It's far too easy to build a new rrdtool binary and put it in a directory of your choosing, the shell script to do this automatically for me (directly from the rrdtool doc/rrdbuild.txt) is a handful of lines and creates a directory which I bundle in an RPM.  doc/rrdbuild.txt is so good in fact it provides fully working examples for building zlib, libpng, freetype, libxml2, cairo, pango, zlib ... in case the distro has old-revs or no-revs of those (RHEL 5/6 need none of this.)

To the extent of corruption, I've tested this as well and also can't reproduce.  I know of less than a handful of bugs in rrdtool that are minor annoyances; none of which are corruption or stability concerns.

The amusing end if you've made it this far ... since a distro will only fix what users report to them as broken; I doubt many use 'resize' in rrdtool ... it causes so much IO at scale that it's unlikely to be used very often; I for one nearly never use it and only provide a 'safe' app-based facility to curb the IO storm it would create when modifying thousands/millions of rrds.

The most I use resize is for directly after creating a Holt-Winters datafile and wanting to resize the FAILURES RRA; as this is quick (in-memory entirely) and on a case-by-case creation basis.

</rant>



Wishful thinking: RRDTool could stand to have a new datafile format supporting extent-based RRAs, the header tracking extent offsets into the datafile so users can tune extent-count for an RRA and the datafile just be grown by an extent equal to the row-size of the original and not resized(rewritten) ... which would resolve a number of scaling/long-term concerns.


-Ryan



________________________________
 From: Tobias Oetiker <tobi at oetiker.ch>
To: Jo Rhett <jrhett at netconsonance.com> 
Cc: rrd-users at lists.oetiker.ch 
Sent: Saturday, August 18, 2012 2:38 AM
Subject: Re: [rrd-users] RHEL RPMS of rrdtool corrupts timestamps in both 1.3.9 and 1.4.7
 
Hi Joe,

Yesterday Jo Rhett wrote:

> So I am curious. There is not much interest in the fact that the
> CentOS / RHEL rpms for rrdtool corrupt/destroy data when resize
> is used?  How many people have to slam into this issue on their
> own before someone will take action to get those RPMs updated?

there are no 'official' rrd rpms from me, so I have no say over how
and when the ones you are using are being updated ...

otoh, I would certainly like to make sure rrdtool does not have
bugs in the resize code ... I have tried to reproduce what you are
experiencing on my system (64bit Intel ubuntu 12.04).

* rrdtool compiled with mmap enabled
* rrd files even residing on an nfs mounted filesystem ...

to no avail ... it all works.

there were a bunch of rrd_resize bugs fixed for rrdtool 1.4.6
(r2192,r2169,r2166) though, so maybe for some odd reason, you are
actually running PRE 1.4.6 code and not 1.4.7

it would help a lot if you could try and reproduce the problem you
are seeing with a freshly compiled rrdtool (from original source).

there is even a rpm spec file included in the distro, so that you
should be able to build an rpm if you so desire ...

hth
tobi

>
> > On Jul 16, 2012, at 10:28 PM, Ryan Kubica wrote:
> >> I've never run into this issue before (and store a lot of data) and on 64bit hosts, but:
> >>
> >>     a) I always compile rrdtool not use stock OS rpm
> >
> > This isn't a common situation -- most people use RPMs, and that means that this is broken for most people.  Also we are using a package that bundles a binary image of rrdtool, another situation where many people won't or can't change it out.
> >
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-users mailing list
rrd-users at lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-users/attachments/20120818/fbd1c58d/attachment.htm