[rrd-users] to many "data agregation" transactions at the same time

[Kristoff Bonne]

Hi,


On 15-12-11 17:58, Simon Hobson wrote:
>> We have an annoying problem.
(...)
>> However, as this runs on crontab, we have now end up with a situation
>> where a lot of .rrd files (thousands) where created at the same time of
>> the day. The result is that the "data aggregation" process of all these
>> .rrd files start at roughly the same time; and this causes disk IO
>> congestion.
> I doubt it. RRD tool does not aggregate data based on arbitrary times 
> (eg file creation). Aggregation (consolidation/normalisation) is done 
> when data is entered, and all consolidated time periods are an 
> integer multiple of the period size since unix epoch (midnight, 1st 
> Jan 1970).
(...)

> Basically, every time you do an update, rrd tool will update all 
> consolidated time periods which have ended since the last update. 
> Thus if you run a script once a day at 1am, it will close the 24 hour 
> consolidation for the previous day regardless of what time of day the 
> file was created. It would also update and close 12 2 hour 
> consolidation periods for the previous day.

OK, I get it. I did notice all .rrd files had a timestamp for the 24h
stats at 2 AM CEST/1AM CET (midnight UTC).


Hmm. As most script run either somewhere between every 5 to 15 minutes;
all rrd files (over 10000) do the "24h consolidation" at the same time.
There is no solution for that.


Cheerio! Kr. Bonne.