[rrd-developers] Re: rrdtool readfile patch

[Tobias Oetiker]

Apr 23 Peter Speck wrote:
Peter,

your patch is in
tob

> Hi Tobi,
>
> readline in rrd_open.c reads the file in 8 KB blocks, and calls realloc
> for each block.
>
> realloc is very slow in Mac OS X for huge blocks, e.g. when restoring
> databases from huge xml files.
>
> This patch finds the size of the file, and starts out with malloc'ing
> the full size.
>
> --- ../rrdtool-2003-04-20/src/rrd_open.c	Fri Apr 11 21:43:44 2003
> +++ src/rrd_open.c	Wed Apr 23 08:25:23 2003
> @@ -185,6 +185,7 @@
>
>   int readfile(char *file_name, char **buffer, int skipfirst){
>       long writecnt=0,totalcnt = MEMBLK;
> +    long offset = 0;
>       FILE *input=NULL;
>       char c ;
>       if ((strcmp("-",file_name) == 0)) { input = stdin; }
> @@ -195,14 +196,22 @@
>         }
>       }
>       if (skipfirst){
> -      do { c = getc(input); } while (c != '\n' && ! feof(input));
> +      do { c = getc(input); offset++; } while (c != '\n' && !
> feof(input));
>       }
> -    if (((*buffer) = (char *) malloc((MEMBLK+4)*sizeof(char))) ==
> NULL) {
> +    if (strcmp("-",file_name)) {
> +      fseek(input, 0, SEEK_END);
> +      /* have extra space for detecting EOF without realloc */
> +      totalcnt = (ftell(input) + 1) / sizeof(char) - offset;
> +      if (totalcnt < MEMBLK)
> +	totalcnt = MEMBLK; /* sanitize */
> +      fseek(input, offset * sizeof(char), SEEK_SET);
> +    }
> +    if (((*buffer) = (char *) malloc((totalcnt+4) * sizeof(char))) ==
> NULL) {
>   	perror("Allocate Buffer:");
>   	exit(1);
>       };
>       do{
> -      writecnt += fread((*buffer)+writecnt, 1, MEMBLK * sizeof(char)
> ,input);
> +      writecnt += fread((*buffer)+writecnt, 1, (totalcnt - writecnt) *
> sizeof(char),input);
>         if (writecnt >= totalcnt){
>   	totalcnt += MEMBLK;
>   	if (((*buffer)=rrd_realloc((*buffer), (totalcnt+4) *
> sizeof(char)))==NULL){
>
> ----
>     - Peter Speck
>

-- 
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETZ J97, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/~oetiker   +41(0)1-632-5286

--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi