<div dir="ltr">Maybe the file size growth is expected to be parabolic - i think every RRA is 'global' - it covers all the data sources. So when you're adding DS and RRA pair you're actually adding N RRAs where N is number of data sources. At least this is how i understood the concept of RRAs.<div>Best regards</div><div>Rafal</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 19, 2015 at 12:51 AM, Jared Henley <span dir="ltr"><<a href="mailto:jared.henley@pelena.com.au" target="_blank">jared.henley@pelena.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><div><div class="h5">
<br>
<br>
<div>On 16/10/15 17:19, Tobias Oetiker
wrote:<br>
</div>
<blockquote type="cite">
<pre>Hi Jared,
Today Jared Henley wrote:
</pre>
<blockquote type="cite">
<pre>Hi,
I've been working on a logging application that uses rrdtool. It works
brilliantly on my PC, not so good on the Raspberry Pi model B+. But before I
go there, there's something interesting I noticed while working on the PC.
I'm creating a database with the following command (in rrdpython)
rrdtool.create(filepath,
'--step', '1s',
'DS:progreset:GAUGE:1000s:0:1',
'RRA:MAX:0.1:5s:5000m',
<snip - there are 29 definitions all up, all the same>
This creates a 386MB file.
I didn't think too hard about it, until I did a graph to a CSV file. Of
course the CSV would be less space-efficient than the binary rrd file, right?
But the generated CSV turns out to be only 15MB?
rrdtool seems to use 64-bit integers for everything, so I figure the rrd file
above should use:
8 bytes per RRA entry * 29 RRAs + 1 timestamp * 60,000 locations in the
round-robin (5000 minutes / 5 seconds) = 14.4MB. I'm confused. Why is the rrd
file 30 times bigger than I'd expect? I did experiment with a step size of 5
seconds, but the created file was the same size. But I can live with largish
files.
</pre>
</blockquote>
<pre>when I create this database on my intel box it looks like this:
oetiker>./rrdtool create /tmp/demo.rrd --step 1s DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
oetiker>ls -l /tmp/demo.rrd
-rw-r--r-- 1 oetiker oep 480584 Oct 16 08:17 /tmp/demo.rrd
so something looks rather odd here ... are you using 1.5.4 on your
raspy ?
cheers
tobi
</pre>
</blockquote>
<br></div></div>
Hi Tobi,<br>
<br>
I can't actually create the database on the rpi - the process gets
killed. Results mentioned are on my AMD 64-bit box, as are all
tests below.<br>
<br>
I just ran your test:<br>
<br>
[08:32:39 pe@hope] $ rrdtool create /tmp/demo.rrd --step 1s
DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m<br>
0 ~<br>
[08:32:49 pe@hope] $ ls -l /tmp/demo.rrd <br>
-rw-rw-r-- 1 pe pe 480584 Oct 19 08:32 /tmp/demo.rrd<br>
<br>
Results are identical and obviously what I would expect.<br>
<br>
I'm having another issue with rrdpython, (but my mail to Hye-Shik
Chang is bouncing), so I just tried my database creation directly in
rrdtool with the following command (complete):<br>
<br>
[08:32:57 pe@hope] $ rrdtool create test.rrd \<br>
> --step 1s \<br>
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \<br>
> DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \<br>
> DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \<br>
> DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \<br>
> DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m<br>
0 ~<br>
[08:45:55 pe@hope] $ ls -l test.rrd <br>
-rw-rw-r-- 1 pe pe 403757864 Oct 19 08:45 test.rrd<br>
<br>
Obviously something funny is going on between the one datasource/RRA
test and the 29 datasource/RRA test, so I tried with just two
DS/RRAs:<br>
<br>
[08:47:32 pe@hope] $ rrdtool create test.rrd \<br>
> --step 1s \<br>
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \<br>
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m <br>
0 ~<br>
[08:47:52 pe@hope] $ ls -l test.rrd <br>
-rw-rw-r-- 1 pe pe 1921184 Oct 19 08:47 test.rrd<br>
<br>
This file also seems too large. 1921184/480584 = 3.997... so it
naively looks like the filesize is following a parabolic function
against DS/RRA pairs rather than linear. I then created the
following two files with almost no data, so I could look at them in
a hex editor easily.<br>
<br>
[08:55:59 pe@hope] $ rrdtool create test.rrd \<br>
> --step 1s \<br>
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s<br>
0 ~<br>
[08:56:01 pe@hope] $ ls -l test.rrd <br>
-rw-rw-r-- 1 pe pe 632 Oct 19 08:56 test.rrd<br>
0 ~<br>
[08:56:03 pe@hope] $ rrdtool create test2.rrd \<br>
> --step 1s \<br>
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s \<br>
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:30s<br>
0 ~<br>
[08:58:23 pe@hope] $ ls -l test2.rrd <br>
-rw-rw-r-- 1 pe pe 1376 Oct 19 08:58 test2.rrd<br>
<br>
I really don't understand the structure of an rrd file, but there's
a section at the end populated with "00 00 00 00 00 00 F8 FF"
multiple times over. I assume this is where the RRA data is stored,
and that the first section is the definitions and
most-recently-updated values. Working on that understanding, the
RRA data section of test.rrd is:<br>
<br>
<pre style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">0000:0240 | 00 00 00 00 00 00 F8 FF | ......øÿ
0000:0250 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0260 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0270 | 00 00 00 00 00 00 F8 FF | ......øÿ
</pre>
<pre style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </pre>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">(as
expected, 6 64-bit values)<br>
</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><br>
And the RRA data section of test2.rrd is:<br>
</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<pre>0000:04A0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04B0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04D0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04E0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04F0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0500 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0510 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0520 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0530 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0540 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ</pre>
(I would expect 12 64-bit values, there are 24 here.)<br>
<br>
I did some updates to these files. test.rrd made some sense to me.
Updates were 0.1, 0.2, 0.3, 0.4, 0.5 in 5 second intervals.<br>
<pre>0000:0240 | 9A 99 99 99 99 99 D9 3F | ......Ù?
0000:0250 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0260 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:0270 | 33 33 33 33 33 33 D3 3F | 333333Ó? </pre>
and again, with updates 0.7, 0.7, 0.8, 0.9, 1:<br>
<br>
<pre>0000:0240 | CD CC CC CC CC CC EC 3F | ÍÌÌÌÌÌì?
0000:0250 | 00 00 00 00 00 00 F0 3F 33 33 33 33 33 33 E3 3F | ......ð?333333ã?
0000:0260 | 66 66 66 66 66 66 E6 3F 66 66 66 66 66 66 E6 3F | ffffffæ?ffffffæ?
0000:0270 | 9A 99 99 99 99 99 E9 3F | ......é? </pre>
<br>
Updates to test2.rrd were 0.1:0.2, 0.3:0.4, 0.5:0.6, 0.7:0.9, 0.9:1
in 5 second intervals.<br>
<pre>0000:04A0 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
0000:04B0 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04D0 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:04E0 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | 333333Ó?......Ù?
0000:04F0 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0500 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:0510 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | 333333Ó?......Ù?
0000:0520 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0530 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
0000:0540 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ</pre>
Using the info from test.rrd's hex dump, it looks like the values in
test2.rrd are:<br>
<pre>0.7 0.9
0.9 1
UNKNOWN UNKNOWN
0.4 0.5
0.6 0.1
0.2 0.3
0.4 0.5
0.6 0.1
0.2 0.3
0.7 0.9
0.9 1
UNKNOWN UNKNOWN
</pre>
It seems like the file test2.rrd contains two sets of RRA data like
this:<br>
<br>
first-half data<br>
second-half data<br>
second-half data<br>
first-half data<span class="HOEnZb"><font color="#888888"><br>
<br>
Jared.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<p> </p>
<p> </p>
</font></span></div>
<br>_______________________________________________<br>
rrd-users mailing list<br>
<a href="mailto:rrd-users@lists.oetiker.ch">rrd-users@lists.oetiker.ch</a><br>
<a href="https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users" rel="noreferrer" target="_blank">https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users</a><br>
<br></blockquote></div><br></div>