[rrd-developers] Update ex post?

[Till Dörges]

Hi,

On 26.08.2010 11:37, Sebastian Harl wrote:
> On Wed, Aug 25, 2010 at 07:49:12PM +0200, Till Dörges wrote:
>> On 24.08.2010 12:44, Sebastian Harl wrote:

>>> What other real-world use-cases, other than removing spikes or undefined
>>> values, are out there? Those are the only ones I've encountered so far.
> 
>> For our particular scenario we can even limit the time window for which we'll accept
>> an update ex post.
>>
>> With that particular constraint the most favorite solution right now looks like a
>> combination of a normal RRDtool db and a MySQL db. The former works as always, except
>> that it doesn't store updates until t_now but until t_now - t_window. t_window is the
>> time window for which we'll accept updates ex post. All the data between t_now -
>> t_window and t_now ends up in MySQL. If any late updates arrive putting them in MySQL
>> shouldn't be a problem.
> […]
>> The downsides I can see is that an additional db adds extra complexity. And you have
>> a bit of effort to put into transferring the data.
> 
> That sounds like a rather special use-case to me but which adds quite
> some complexity and which does not seem (on a first glance) to solve the
> actual problem: modifying existing old CDPs, which would have to be done
> when transferring the SQL data back to the RRD.

We don't necessarily have to modify CDPs. The biggest problem for us are late
updates, i.e. data with a timestamp before the last actual update. Inserting these
late updates into an SQL db is easy. No math/averaging/...


> As a more general solution, it might make sense to provide a feature to
> replace a whole range of data in one go. That would allow to update some
> CDPs with rather exact data and leave interpolation (or whatever might
> be done) to one CDPs at each side of the specified time slot. Storing
> that data would then be left to the user but could be supported by
> RRDtool by providing means to read the data from several data-sources
> (SQL db, CSV text file, …).

If I understood your solution correctly that would solve our problem, too. And
certainly with less hassle.
Ideally one could give rrdupdate a timestamp from the past and (if feasible for the
current set of RRAs) it simply exchanges the CDPs.

We might have a few resources to spare, but have only experience using RRDtool - not
developing it. That means I don't have a "gut feeling" of how hard this would really be.

Regards -- Till
-- 
Besuchen Sie uns auf it-sa und PITS 2010:
      6. -  7.10.2010  http://www.public-it-security.de
     19. - 21.10.2010  http://www.it-sa.de

Dipl.-Inform. Till Dörges                  doerges at pre-sense.de
                                  Tel. +49 - 40 - 244 2407 - 14
                                  Fax  +49 - 40 - 244 2407 - 24
PRESENSE Technologies GmbH            Sachsenstr. 5, D-20097 HH
                                         USt-IdNr.: DE263765024
Geschäftsführer/Managing Directors       AG Hamburg, HRB 107844
Till Dörges           Jürgen Sander              Axel Theilmann