[rrd-users] Re: Disk space usage when using RRDTool

[PAUL WILLIAMSON]

Nothing increases.  The files are created at that size and do not grow.

I'm not surprised by the space consumed by one RRD.  If you have an 
RRA of 5 seconds for a period of 2 years (12614400) points, that's 
a whole lot of data.  I would think either a database or a text file 
would take up less space.  I store 1 year of 5 minute samples and 
my files are 3.5mb.  You are basically taking up 120 times that size.

3.5mb * 120 = 420mb.  Give or take some file space fudge factors, 
450mb is not a surprising figure at all.  Keep in mind that MRTG will "normalize"
the data if your polls are not at the exact time you specifiy, providing small error 
corrections for what you are measuring.  

In a previous life, I contemplated using MRTG to measure 1 second datapoints, 
and came to the conclusion that the error correcting MRTG did would have 
totally messed up the data.  But, I was not looking for error correction, 
I was looking for recording the data and the exact time I measured it.
If you want something similar, MRTG is not for you.  You are looking for 
a database (I would suggest MySQL or SQLite) or a flat text file.
Even then, a text file with that many entries might be kind of slow 
to open.  The advantage with MRTG, as long as you don't mind the 
error correcting/normalization, is that the files do not grow.  They stay 
the same size and the data is just filled in.

If you only have a hammer, every problem looks like a nail.

Paul

>>> "Eldor Rødseth (SystemSoft AS)" <er at systemsoft.no> 8/11/2004 3:18:48 PM >>>

Dear all,

I am looking into efficient ways of storing large amounts of data
for post-processing.

My application shall collect periodically reported data (historical)
from a number of "Units" in my system, and store such data to file(s).
There may be (n*100) such units in my system (max number of units
will probably be <= 1000). I have to assume that each unit will
store its data to one-, or more files.

Reported data items are typically 32-bit number(s). Each reported
data item group may consist of one-, or more 32-bit numbers.
Reporting intervals are typically 5-15 seconds.

Post-processing of the data will be done to enable presentation
by means of graphs and numerical output. The presentation by means
of graph(s) must be done by Java applets, hence I believe that
"rrdtool graph ..." will not be used by us.

I have done a simple test using "rrdtool create ...." and
"rrdtool update ..." and was surprised regarding the file size
produced by RRDTool. I experienced a file size of 450MB within
a very short period after a statement like this:

#ifdef RRDTOOL_TEST
  sprintf( str, "rrdtool create varyparam.rrd --start %lu \
                 --step 5 DS:VaryparamLong:GAUGE:60:U:U \
                 RRA:AVERAGE:0.5:1:12614400", curr_time );
  if (result = system( str ))
    fprintf( stderr, "exit code = %d, errno = %d\n", result, errno );
#endif

I am using a very large number of entries - actually one entry
per 5 second for the next 2 years!
However, this could be the number of records I need to store for
each unit mentioned above.

I guess my bottom-line question is: Would you, as experienced RRDTool users,
recommend the usage of RRDTool for such a scenario?
(An alternative is of course to use ssimple ASCII-files...)

Also - can someone explain to me the reason for the rapidly increasing
file size in my test above? Can this be avoided?

Thanks.

Regards, Eldor R.

------------------------
Eldor Rødseth (MD)
SystemSoft AS
http://www.systemsoft.no 
Cellular: +47 950 86888




--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe 
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help 
Archive     http://www.ee.ethz.ch/~slist/rrd-users 
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi 




--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi