[rrd-developers] rrdcached dies when journal reaches 2G (fwd)

[kevin brintnall]

On Wed, Jul 15, 2009 at 08:42:27PM +0200, Tobias Oetiker wrote:
> Hi Kevin
> 
> any ideas on how to fix ? large file support or maybe split the
> journal ?

Interesting...

On systems that support large file support (but where it is not the
default), it's possible to use pre-processor macros to select which API to
use.  Typically this increases the size of 'off_t' data type, and aliases
the common function names (i.e. #define fopen fopen64).

For large file support, we need to consider the things that link against
librrd.  We can either require all dependents (transitively) to enable
large file support (bad) or we could remove any functions that use off_t
from the public librrd interface.  It looks like all such functions are
already deprecated.

In the end, it's probably better to just rotate the journals before 2^31
bytes.  That way, we can support systems that do not have large file
support.  There are some implicit assumptions during journal replay that
I'll have to take a look at.

Thoughts?

-- 
 kevin brintnall =~ /kbrint at rufus.net/


> cheers
> tobi
> 
> 
> -- 
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900
> 
> ---------- Forwarded message ----------
> From: RRDtool Trac <tobi+r at oetiker.ch>
> Reply-To: tobi+trac at oetiker.ch
> To: undisclosed-recipients:  ;
> Date: Wed, 15 Jul 2009 15:52:32 -0000
> X-Spam-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00
>     autolearn=unavailable version=3.2.4
> Subject: [RRDtool Trac] #234: rrdcached dies when journal reaches 2G
> 
> #234: rrdcached dies when journal reaches 2G
> -----------------------+----------------------------------------------------
>  Reporter:  human      |       Owner:  oetiker
>      Type:  defect     |      Status:  new
>  Priority:  major      |   Milestone:
> Component:  misc       |     Version:  1.4dev
>  Keywords:  rrdcached  |
> -----------------------+----------------------------------------------------
>  Hello,
> 
>  When running rrdcached with -j, -f 14400, and -w 7200, the journal file
>  reaches 2G which causes rrdcached to exit.   We chose these parameters to
>  cache two hours of updates before starting the write cycle for 400k rrds.
>  Trying to run with -f 7200 -w 7200 produced a warning that -f should be at
>  least 2x -w.
> 
>  Linux reports01 2.6.18-92.el5PAE #1 SMP Tue Apr 29 13:31:02 EDT 2008 i686
>  i686 i386 GNU/Linux
>  Red Hat Enterprise Linux Server release 5.2 (Tikanga)
>  RRDtool 1.3.99909060808  Copyright 1997-2008 by Tobias Oetiker
>  <tobi at oetiker.ch>
>                 Compiled Jul  1 2009 18:45:11
> 
> -- 
> Ticket URL: <http://oss.oetiker.ch/rrdtool-trac/ticket/234>
> RRDtool Trac <http://oss.oetiker.ch/rrdtool-trac/>
> Wiki, Issue Tracking, Source Browsing for RRDtool