[rrd-users] Re: Deleting old data from an archive

[Tobias Oetiker]

Today Shack Toms wrote:

> I'd like to set all the saved values to undefined for a particular RRA
> within an RRD.   What's going on is that the data associated with the DS
> turns out to be invalid and I want to start supplying good data, and I don't
> want the old bad data to appear, mixed with the good.   There are lots of
> other DS's that have good data in them, and I don't want to lose that
> either.
>
> I thought that an easy way to do this would be to dump the database to xml,
> edit the xml, and then restore the database.   Then all I would need to
> write would be the xml filter to go in the middle, to change some values to
> NaN.  This works well on small databases, but on any database over a few
> tens of MB it seems to take a *very* long time to do the restore (even if I
> don't edit the xml, so don't blame it on that).   The dump goes reasonably
> fast.   A restore seems go go fairly fast for small databases, but slows
> down for large ones.   It looks like the rrd_restore code operates by trying
> to work on everything in memory and then writing out the file, so I'm
> wondering if my problem is excessive paging.   On my machine restore starts
> doing something like 25,000 page faults per second.  I notice that the
> rrd_dump code seems to read and dump the values incrementally, so it ought
> to (and seems to) have a lot better paging performance.

how about adding some memory to the machine ?

> Alternatively, I could try to change the behavior of restore to work
> incrementally through the data, the way dump does (and submit the results if
> it improved things).   Is there a FAQ on how to build/submit changes, so
> that I fit in with the way things are done?  Is there a reason why an
> incremental restore of the rows won't work/is a bad idea?   Am I going about
> this all wrong?

the reason for building the rrd in memmory is, that I only know all
the necessary information to write the rrd file after I have read
the whole xml file ... If you want to make the restore incremental,
you would have to add some padding into the database structure and
then fill in the missing bits when you are done converting the xml
... this is mainly concerning the size of the RRAs

>
> One more small point.   I wanted to make sure that the dump/restore
> operation worked, so I did a dump/restore/dump and compared the xml from the
> first dump with the xml from the second.  The only difference was that I
> noticed that the parameter "unknown_sec" changes to 0 for all the DS's,
> although looking at the code it is not immediately obvious to me why this
> is.   I don't see a lot of documentation for "unknown_sec", and I wondered
> how bad this would turn out to be and how best to keep the old, non-zero
> value.

Hmm interesting will add this to my todo list

as for contributing the rrdtool make sure you subscribe to the
rrd-developers list and propose your project there ... you will get
some feedback ...

there is a development version of rrdtool available from the beta
directory on the rrdtool website

cheers
tobi

> Thanks,
> Shack Toms
>
>
> -- Attached file removed by Listar and put at URL below --
> -- Type: application/ms-tnef
> -- Size: 2k (2844 bytes)
> -- URL : http://www.ee.ethz.ch/~slist/pantomime/73-winmail.dat
>
>
> --
> Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
> Archive     http://www.ee.ethz.ch/~slist/rrd-users
> WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi
>
>

-- 
 ______    __   _
/_  __/_  / /  (_) Oetiker, ETZ J97, ETH, 8092 Zurich, Switzerland
 / // _ \/ _ \/ / phoneto:+41(0)1-632-5286  faxto:+41(0)1-632-1517
/_/ \.__/_.__/_/ mailto:oetiker at ee.ethz.ch http://people.ee.ethz.ch/~oetiker


--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi