[rrd-users] Re: Migrating RRD data to RDBMS

[Marc Powell]

> -----Original Message-----
> From: rrd-users-bounce at list.ee.ethz.ch [mailto:rrd-users-
> bounce at list.ee.ethz.ch] On Behalf Of Ruttenberg, Tanya
> Sent: Wednesday, January 25, 2006 11:20 AM
> To: rrd-users at list.ee.ethz.ch
> Subject: [rrd-users] Migrating RRD data to RDBMS
> 
> We have a requirement to migrate data from RRD files to an RDBMS.
> 
> My current idea is to do a daily export from the rrdfiles, parse the
> export
> file, and update the RDBMS with the resulting numbers.
> 
> Is anyone out there doing this already?  Suggestions?

I'm not going to say that what we're doing is optimal and I am very
interested in other answers to this question but we're doing this for
approximately 10,000 rrd files every 30 minutes, sticking the data into
SQLServer. We've written a perl script that essentially does the
following--

- Walk the file system (File::Find) looking for .rrd files, ignoring
ones we don't care about
- based on the directory and file name, query the SQLServer for the last
timestamp for which we inserted non-NULL data
- use RRDs::fetch to retrieve entries starting from the last timestamp
in SQLServer until RRDs::last(rrdfile)-300 from the rrd file in question
- insert the data into SQLServer
- rinse, repeat

We run multiple insert scripts in parallel, each looking at different
parts of the file system. We keep data for ~600 days in the DB which now
has about 100 billion rows. We run every 30 minutes because the RRD
files are actually populated on other servers and are rsync'd over every
30 minutes. We could probably run on a much more frequent schedule if we
didn't move the files around (NFS maybe?). In the event of some
unforeseen failure of the insert process, we have plenty of other
opportunities during the first 50 hours to obtain the 5 minute intervals
we want from the first RRA. If we only ran once a day we'd have at most
1 other run to get that data without manual intervention.

Hope that helps.

--
Marc

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi