[rrd-users] Questions about effective database parameters

[Alex van den Bogaerdt]

Any average that contains some unknown data is weird, after
all you cannot determine the average of 1,1,1,1,unknown,1,1,1.
The X-File Factor (xff) is the amount of weirdness that is allowed.
Higher number means more weirdness allowed. 0<=xff<1.

If you always want a value, even if you only got only one valid
PDP, make your xff as high as possible.  0.9999999999999999
Should RRDtool complain, then there are too many nines and
the number is rounded up to 1.0  => remove one or more 9

(side note: I think a value of 1 for xff should be allowed,
it would signify that you don't care how many PDPs have
data, as long as there is one, you'll take it. It may be not
mathematically correct, but that's the whole purpose of xff)

Plotting 3.5 hours is something I cannot explain from this data.
You say nothing ends up in the other RRAs, so the data can only
come from the 1st RRA which is 86400 seconds long. That is one
day. The other RRAs span more time.

Use rrdtool dump to verify your claim about no data (you mean:
all unknowns) ending up in the other RRAs. No need to post here,
just look for yourself.

And when plotting, try to set times "on a boundary", and have
an integral number of CDPs for each of your 500 pixel columns.
For instance start=end-24000 end=1293148800 to get 500 half
hours from the 2nd RRA.

HTH
Alex


>
> Hi again.
> I can't make it work. A rrd is setup as:
> rrdtool create -s 1 \
> DS:DS1:GAUGE:2:0:3600 \
> DS:DS1:GAUGE:2:0:3600 \
> RRA:AVERAGE:0:1:86400 \
> RRA:AVERAGE:0.02:48:3600 \
> RRA:AVERAGE:0.002:336:3600 \
> RRA:AVERAGE:0.001:720:3600 \
> RRA:MAX:0.02:48:3600 \
> RRA:MAX:0.002:336:3600 \
> RRA:MAX:0.001:720:3600
>
> Nothing gets stored in the rra's except in the 86400 one. And when
> generating an image, only time spans around 3.5 hours produces
> graphs with data but there is data anywhere during a day. Is this because
> RRDtool tries to plot data from the "wider" rra?
>
> Plot image width is 500. I'm using rrdcached with my own timestamps, and
> the journals are full of updates and writes.
> I have rrd with step 60 and they work fine. But all the 1s step are not
> working as intended.
>
> Here's a info from a file.
> filename = "INT-7055.rrd"
> rrd_version = "0003"
> step = 1
> last_update = 1293034861
> header_size = 2632
> ds[DS1].index = 0
> ds[DS1].type = "GAUGE"
> ds[DS1].minimal_heartbeat = 2
> ds[DS1].min = 0.0000000000e+00
> ds[DS1].max = 3.6000000000e+03
> ds[DS1].last_ds = "U"
> ds[DS1].value = NaN
> ds[DS1].unknown_sec = 0
> ds[DS2].index = 1
> ds[DS2].type = "GAUGE"
> ds[DS2].minimal_heartbeat = 2
> ds[DS2].min = 0.0000000000e+00
> ds[DS2].max = 3.6000000000e+03
> ds[DS2].last_ds = "0.0441428571428571"
> ds[DS2].value = 0.0000000000e+00
> ds[DS2].unknown_sec = 0
> rra[0].cf = "AVERAGE"
> rra[0].rows = 86400
> rra[0].cur_row = 28857
> rra[0].pdp_per_row = 1
> rra[0].xff = 0.0000000000e+00
> rra[0].cdp_prep[0].value = NaN
> rra[0].cdp_prep[0].unknown_datapoints = 0
> rra[0].cdp_prep[1].value = NaN
> rra[0].cdp_prep[1].unknown_datapoints = 0
> rra[1].cf = "AVERAGE"
> rra[1].rows = 3600
> rra[1].cur_row = 2285
> rra[1].pdp_per_row = 48
> rra[1].xff = 2.0000000000e-02
> rra[1].cdp_prep[0].value = 0.0000000000e+00
> rra[1].cdp_prep[0].unknown_datapoints = 13
> rra[1].cdp_prep[1].value = 8.8142857143e-02
> rra[1].cdp_prep[1].unknown_datapoints = 11
> rra[2].cf = "AVERAGE"
> rra[2].rows = 3600
> rra[2].cur_row = 1438
> rra[2].pdp_per_row = 336
> rra[2].xff = 2.0000000000e-03
> rra[2].cdp_prep[0].value = 0.0000000000e+00
> rra[2].cdp_prep[0].unknown_datapoints = 13
> rra[2].cdp_prep[1].value = 8.8142857143e-02
> rra[2].cdp_prep[1].unknown_datapoints = 11
> rra[3].cf = "AVERAGE"
> rra[3].rows = 3600
> rra[3].cur_row = 1343
> rra[3].pdp_per_row = 720
> rra[3].xff = 1.0000000000e-03
> rra[3].cdp_prep[0].value = 0.0000000000e+00
> rra[3].cdp_prep[0].unknown_datapoints = 541
> rra[3].cdp_prep[1].value = 2.8852595238e+00
> rra[3].cdp_prep[1].unknown_datapoints = 474
> rra[4].cf = "MAX"
> rra[4].rows = 3600
> rra[4].cur_row = 1784
> rra[4].pdp_per_row = 48
> rra[4].xff = 2.0000000000e-02
> rra[4].cdp_prep[0].value = -inf
> rra[4].cdp_prep[0].unknown_datapoints = 13
> rra[4].cdp_prep[1].value = 4.4142857143e-02
> rra[4].cdp_prep[1].unknown_datapoints = 11
> rra[5].cf = "MAX"
> rra[5].rows = 3600
> rra[5].cur_row = 2409
> rra[5].pdp_per_row = 336
> rra[5].xff = 2.0000000000e-03
> rra[5].cdp_prep[0].value = -inf
> rra[5].cdp_prep[0].unknown_datapoints = 13
> rra[5].cdp_prep[1].value = 4.4142857143e-02
> rra[5].cdp_prep[1].unknown_datapoints = 11
> rra[6].cf = "MAX"
> rra[6].rows = 3600
> rra[6].cur_row = 2751
> rra[6].pdp_per_row = 720
> rra[6].xff = 1.0000000000e-03
> rra[6].cdp_prep[0].value = -inf
> rra[6].cdp_prep[0].unknown_datapoints = 541
> rra[6].cdp_prep[1].value = 5.7166666667e-02
> rra[6].cdp_prep[1].unknown_datapoints = 474
>
>  I've been staring at all this for too long and can't come up with a
> solution.
> Merry Christmas
> Henrik
> From: henrikhegardt at hotmail.com
> To: s.shipway at auckland.ac.nz; rrd-users at lists.oetiker.ch
> Date: Wed, 15 Dec 2010 16:10:15 +0100
> Subject: Re: [rrd-users] Questions about effective database parameters
>
>
>
>
>
>
>
>
>
>
>
>
> Thank you for your reply!
> For some reason I was thinking of the xff the other way around. I'm using
> your ideas, and it looks promising.
> Regards Henrik
> From: s.shipway at auckland.ac.nz
> To: henrikhegardt at hotmail.com; rrd-users at lists.oetiker.ch
> Subject: RE: [rrd-users] Questions about effective database parameters
> Date: Wed, 15 Dec 2010 01:27:16 +0000
>
>
>
>
>
>
>
>
>
>
> RRA:AVERAGE:0.5:30:720
>
> RRA:AVERAGE:0.5:60:720
>
>
>
> You need to set the XFF (the fraction of the data which must be known for
> an RRA to store a value) to be much lower; in fact, you might even want to
> make it as near to zero as you can, maybe 1/(#dp)
>
>
>
> So, for the above, use:
>
>
>
> RRA:AVERAGE:0.03:30:720
>
> RRA:AVERAGE:0.01:60:720
>
>
>
> and even make it smaller for other RRAs.
>
>
>
> This is a rather clever way to get around the whole data normalisation
> thing; by making the step size 1s you avoid normalisation, but use a tiny
> XFF to make sure your 1-min RRA still stores the data as posted (and
> averages if you get multiple samples).
>
>
>
> Steve
>
>
>
>
>
> Steve Shipway
> University of Auckland ITS
> UNIX Systems Design Lead
> s.shipway at auckland.ac.nz
> Ph: +64 9 373 7599 ext 86487
>
>
>
>
>
> From: rrd-users-bounces+s.shipway=auckland.ac.nz at lists.oetiker.ch
> [rrd-users-bounces+s.shipway=auckland.ac.nz at lists.oetiker.ch] on behalf of
> Henrik Hegardt [henrikhegardt at hotmail.com]
>
> Sent: Tuesday, 14 December 2010 10:59 p.m.
>
> To: rrd-users at lists.oetiker.ch
>
> Subject: [rrd-users] Questions about effective database parameters
>
>
>
>
>
> Hello.
>
> I'm writing a program to monitor a server's activities. Its about response
> times for services on the server. There are quite many services and most
> of the written data are unknowns. The response times is quite crucial and
> would want to have as high resolution
>  as possible even though there are unknowns. If there's a response time I
> write it to the DB with 1 second interval though since there are several
> other services per rrd a update operation could look like 'update
> database.rrd 1292313983:U:U:U:U:U:U:U:0.012:U:U:U:U:U:U'.
>  I would like to store it for atleast 6 months and thus view it over that
> time. I'm not interested of if there was no value, but the value itself.
>
> I can't tell if there will be a lot of updates during a time interval or
> none at all which should result in some of the entries won't even get a
> 'U' value (most probably).
>
> I have worked with following
>
> rrdtool create -s 1 \
>
> DS:ds1:GAUGE:2:0:3600 \
>
> ...
>
> RRA:AVERAGE:0.5:30:720
>
> RRA:AVERAGE:0.5:60:720
>
> RRA:AVERAGE:0.5:420:720
>
> RRA:AVERAGE:0.5:1800:720
>
> RRA:MAX:0.5:30:720
>
> RRA:MAX:0.5:60:720
>
> RRA:MAX:0.5:420:720
>
> RRA:MAX:0.5:1800:720
>
>
>
> This didn't work, since I believe there a lot more 'U' values than 0.5 and
> the resolution isn't good enough and really doesn't make sense according
> to what I'm trying to sample.
>
> RRA:AVERAGE:0.5:1:86400 worked but then having a rra over 6 months
> (6*30*86400) will result in a big db and slow rendering of images. Any
> ideas?
>
>
>
> Forgive my English.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>