[rrd-developers] rrdcached performance with >200k nodes

kevin brintnall kbrint at rufus.net
Wed Jan 13 20:22:37 CET 2010


On Wed, Jan 13, 2010 at 11:41:44AM -0600, kevin brintnall wrote:
> On Wed, Jan 13, 2010 at 06:28:36PM +0100, Mirek Lau?? wrote:
> > Kevin,
> > 
> > attached is the callgrind annotate output with --tree=both.
> 
> Thanks, I'll review.

BTW Mirek, it looks like your rrdcached is without debugging symbols.  It
would be helpful to see the same callgrind output with rrdcached's
debugging symbols.

-- 
 kevin brintnall =~ /kbrint at rufus.net/


> > Is it possible to attach valgrind/callgrind to a running process.
> 
> I don't think so.  I think it does its work by intercepting normal calls
> with LD_PRELOAD.
> 
> > Running the rrdcached under
> > callgrind would take years to reach the point where it get stuck in the queue.
> 
> Understood..  Maybe there's a way to reproduce the problem on a smaller scale.
> 
> > I've also noticed very high number of context switches while the queue
> > is stuck:
> 
> How much higher compared to normal operation?
> 
> > last pid: 31196;  load averages:  6.35,  4.17,  3.57
> > 
> >                              up 50+20:06:20  18:26:44
> > 267 processes: 10 running, 239 sleeping, 18 zombie
> > CPU 0:  7.5% user,  0.0% nice, 91.4% system,  1.1% interrupt,  0.0% idle
> > CPU 1:  6.0% user,  0.0% nice, 92.9% system,  0.7% interrupt,  0.4% idle
> > CPU 2: 13.9% user,  0.0% nice, 85.4% system,  0.0% interrupt,  0.7% idle
> > CPU 3: 17.7% user,  0.0% nice, 82.0% system,  0.0% interrupt,  0.4% idle
> > Mem: 1578M Active, 13G Inact, 815M Wired, 577M Cache, 399M Buf, 185M Free
> > Swap: 8192M Total, 100K Used, 8192M Free
> > 
> >   PID USERNAME   VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
> > 96584 portax     5495   5367     99     13      0    112   1.67% rrdcached
> > 
> > Now, our rrdcached made it to 3k files in the queue after flushall
> > (220k) in about 20 minutes.
> > Now it is almost stuck, very slowly writing the queue - write rate
> > went from 200/sec to about 5/sec :-(
> 
> I'm surprised with how much CPU time is spent in "system"..  While you are
> still in this "slow trickle" mode, can you attach to the running process
> with ktrace and see what system calls it's making?
> 
> Also, what locale are you using?
> 
> -- 
>  kevin brintnall =~ /kbrint at rufus.net/
> 
> 
> > On Wed, Jan 13, 2010 at 5:33 PM, kevin brintnall <kbrint at rufus.net> wrote:
> > > On Wed, Jan 13, 2010 at 05:21:57PM +0100, Mirek Lau?? wrote:
> > >> there is about 224k nodes in the tree, after issuing FLUSHALL
> > >> it takes about 20 minuts (with -t 8) to write almost all nodes
> > >
> > > This sounds about right.
> > >
> > >> we're now stuck at approx 10k nodes in queue,
> > >> journal continues to write, updates are received at normal rate
> > >
> > > Try issuing the "QUEUE" command to the daemon when in this state. ??I just
> > > want to verify that you are seeing files continually added to the end of
> > > the queue, and being removed from the front.
> > >
> > >> ?? PID USERNAME ??THR PRI NICE ?? SIZE ?? ??RES STATE ??C ?? TIME ?? ??CPU COMMAND
> > >> 91510 portax ?? ?? 60 ??44 ?? ??0 ?? 133M 90052K select 3 ?? 0:00 102.10% rrdcached
> > >
> > > What does 'top -H' show for this pid? ??Is the CPU spread out evenly or
> > > dominated by one thread? ??Since it's so close to 100%, I'm guessing one
> > > thread. ??It would be nice to determine whether it's the thread that's
> > > handling your UPDATE (vs. the queue threads).
> > >
> > > Also, can you repeat the earlier 'callgrind' output, but with --tree=both ?
> > >
> > > --
> > > ??kevin brintnall =~ /kbrint at rufus.net/
> > >
> > 
> > -ml
> 
> > --------------------------------------------------------------------------------
> > Profile data file 'callgrind.out.72526' (creator: callgrind-3.5.0)
> > --------------------------------------------------------------------------------
> > I1 cache: 
> > D1 cache: 
> > L2 cache: 
> > Timerange: Basic block 0 - 15555346483
> > Trigger: Program termination
> > Profiled target:  /usr/local/bin/rrdcached -l unix:/tmp/rrdcached.sock -w 3600 -z 3600 -t 2 -f 7200 -m 64 -j /usr/local/portax/var/rrd/journal -p /tmp/rrdcached.pid (PID 72526, part 1)
> > Events recorded:  Ir
> > Events shown:     Ir
> > Event sort order: Ir
> > Thresholds:       99
> > Include dirs:     
> > User annotated:   
> > Auto-annotation:  off
> > 
> > --------------------------------------------------------------------------------
> >             Ir 
> > --------------------------------------------------------------------------------
> > 56,987,676,075  PROGRAM TOTALS
> > 
> > --------------------------------------------------------------------------------
> >             Ir  file:function
> > --------------------------------------------------------------------------------
> > 
> >     44,378,954  < ???:strcmp'2 (132864x) [/lib/libc.so.7]
> > 31,901,245,144  < ???:strcmp (92178524x) [/lib/libc.so.7]
> > 31,901,245,144  *  ???:strcmp'2 [/lib/libc.so.7]
> >     44,378,954  >   ???:strcmp'2 (132864x) [/lib/libc.so.7]
> > 
> >  8,027,935,482  < ???:0x00000000004044d0 (10449096x) [/usr/local/bin/rrdcached]
> >        163,873  < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >    643,631,298  < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
> >  7,011,268,017  *  ???:0x0000000000402ac0 [/usr/local/bin/rrdcached]
> > 
> >             23  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >          3,643  < ???:0x00000000000d6570 (77x) [/lib/libc.so.7]
> >    386,717,461  < ???:0x0000000000405d50 (1428158x) [/usr/local/bin/rrdcached]
> >            478  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >  4,918,759,802  < ???:fgets (4618515x) [/lib/libc.so.7]
> >  2,860,147,538  *  ???:memchr [/lib/libc.so.7]
> > 
> >            164  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >  1,536,016,818  < ???:0x00000000004044d0 (5224547x) [/usr/local/bin/rrdcached]
> >            294  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >        239,663  < ???:atoi (1462x) [/lib/libc.so.7]
> >  1,536,256,939  *  ???:strtol [/lib/libc.so.7]
> > 
> >    650,318,686  < ???:g_tree_replace (3882113x) [/usr/local/lib/libglib-2.0.so.0]
> >            154  < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> > 32,824,017,339  < ???:0x000000000005e6c0 (92187514x) [/usr/local/lib/libglib-2.0.so.0]
> >        593,674  < ???:setlocale (15623x) [/lib/libc.so.7]
> >  1,347,413,568  *  ???:strcmp [/lib/libc.so.7]
> > 31,901,245,144  >   ???:strcmp'2 (92178524x) [/lib/libc.so.7]
> > 
> > 34,013,769,458  < ???:g_tree_lookup (5455193x) [/usr/local/lib/libglib-2.0.so.0]
> >  1,189,752,119  *  ???:0x000000000005e6c0 [/usr/local/lib/libglib-2.0.so.0]
> > 32,824,017,339  >   ???:strcmp (92187514x) [/lib/libc.so.7]
> > 
> > 44,799,381,715  < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
> >    916,274,439  *  ???:0x00000000004054b0 [/usr/local/bin/rrdcached]
> >            524  >   ???:calloc (1x) [/lib/libc.so.7]
> >             88  >   ???:memset (2x) [/lib/libc.so.7]
> >            202  >   ???:readdir (4x) [/lib/libc.so.7]
> >        287,740  >   ???:0x00000000000034d0 (12x) [/libexec/ld-elf.so.1]
> >  2,891,857,874  >   ???:fgets (4519717x) [/lib/libc.so.7]
> >         98,364  >   ???:syslog (6x) [/lib/libc.so.7]
> >            616  >   ???:fclose (2x) [/lib/libc.so.7]
> >            947  >   ???:rrd_add_strdup (1x) [/usr/local/lib/librrd_th.so.5]
> >          8,407  >   ???:snprintf (5x) [/lib/libc.so.7]
> > 40,991,003,725  >   ???:0x0000000000404a10 (4519716x) [/usr/local/bin/rrdcached]
> >             20  >   ???:time (1x) [/lib/libc.so.7]
> >         33,287  >   ???:0x0000000000403930 (1x) [/usr/local/bin/rrdcached]
> >              5  >   ???:stat (1x) [/lib/libc.so.7]
> >             24  >   ???:rename (1x) [/lib/libc.so.7]
> >            181  >   ???:fopen (1x) [/lib/libc.so.7]
> >             43  >   ???:pthread_mutex_unlock (1x) [/lib/libthr.so.3]
> > 
> >    538,124,378  < ???:vsnprintf (727345x) [/lib/libc.so.7]
> >    765,331,067  < ???:vfprintf (713010x) [/lib/libc.so.7]
> >          6,917  < ???:snprintf (7x) [/lib/libc.so.7]
> >    357,841,714  < ???:sprintf (714114x) [/lib/libc.so.7]
> >    838,626,751  *  ???:0x00000000000d6570 [/lib/libc.so.7]
> >    197,342,955  >   ???:strlen (1426020x) [/lib/libc.so.7]
> >     18,554,936  >   ???:0x00000000000d56d0 (1439955x) [/lib/libc.so.7]
> >         21,083  >   ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >    789,153,788  >   ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
> >         17,448  >   ???:0x00000000000e2240 (28x) [/lib/libc.so.7]
> >     25,853,700  >   ???:localeconv (2154475x) [/lib/libc.so.7]
> >          3,643  >   ???:memchr (77x) [/lib/libc.so.7]
> > 
> >     94,618,746  < ???:0x0000000000403200 (724865x) [/usr/local/bin/rrdcached]
> >            132  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >     77,121,504  < ???:0x0000000000404a10 (1428176x) [/usr/local/bin/rrdcached]
> >    691,354,068  < ???:0x00000000004032d0 (5252093x) [/usr/local/bin/rrdcached]
> >    837,387,282  *  ???:strcasecmp [/lib/libc.so.7]
> > 
> >  1,819,672,633  < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
> >    750,265,924  *  ???:0x0000000000012d90 [/usr/local/lib/librrd_th.so.5]
> >    866,968,696  >   ???:rrd_write (15209978x) [/usr/local/lib/librrd_th.so.5]
> >     27,760,698  >   ???:rrd_test_error (263644x) [/usr/local/lib/librrd_th.so.5]
> >          1,023  >   ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >        141,504  >   ???:rpnstack_free (16001x) [/usr/local/lib/librrd_th.so.5]
> >        215,832  >   ???:0x00000000000034d0'2 (15x) [/libexec/ld-elf.so.1]
> >    304,445,060  >   ???:__error (15225602x) [/lib/libthr.so.3]
> >      2,709,912  >   ???:dst_conv (32004x) [/usr/local/lib/librrd_th.so.5]
> >      4,176,726  >   ???:strncpy (16001x) [/lib/libc.so.7]
> >        401,820  >   ???:free (13394x) [/lib/libc.so.7]
> >     10,546,716  >   ???:setlocale (15623x) [/lib/libc.so.7]
> >         22,120  >   ???:rrd_set_to_DINF (1589x) [/usr/local/lib/librrd_th.so.5]
> >      7,599,079  >   ???:rrd_notify_row (7599079x) [/usr/local/lib/librrd_th.so.5]
> >      1,756,830  >   ???:strtod (7811x) [/lib/libc.so.7]
> >      4,604,204  >   ???:rrd_diff (13201x) [/usr/local/lib/librrd_th.so.5]
> >      1,952,480  >   ???:strchr (13397x) [/lib/libc.so.7]
> >      6,949,956  >   ???:rrd_seek (151086x) [/usr/local/lib/librrd_th.so.5]
> >        396,340  >   ???:rrd_set_to_DNAN (35467x) [/usr/local/lib/librrd_th.so.5]
> >         90,796  >   ???:rpnstack_init (16001x) [/usr/local/lib/librrd_th.so.5]
> >      3,969,434  >   ???:cf_conv (69221x) [/usr/local/lib/librrd_th.so.5]
> >        226,200  >   ???:floor (8385x) [/lib/libm.so.5]
> >        262,648  >   ???:0x0000000000012d40 (17140x) [/usr/local/lib/librrd_th.so.5]
> > 
> > 48,762,508,375  < ???:0x0000000000404a10 (5224548x) [/usr/local/bin/rrdcached]
> >    650,488,788  *  ???:0x00000000004044d0 [/usr/local/bin/rrdcached]
> >  8,027,935,482  >   ???:0x0000000000402ac0 (10449096x) [/usr/local/bin/rrdcached]
> >     27,130,238  >   ???:memset (444758x) [/lib/libc.so.7]
> >  1,668,406,047  >   ???:rrd_add_strdup_chunk (5223925x) [/usr/local/lib/librrd_th.so.5]
> >  1,536,016,818  >   ???:strtol (5224547x) [/lib/libc.so.7]
> >        129,578  >   ???:0x00000000000034d0 (6x) [/libexec/ld-elf.so.1]
> >     99,266,412  >   ???:0x0000000000403730 (5224548x) [/usr/local/bin/rrdcached]
> >  1,029,779,151  >   ???:0x0000000000403ac0 (712421x) [/usr/local/bin/rrdcached]
> >    545,599,355  >   ???:g_tree_replace (222378x) [/usr/local/lib/libglib-2.0.so.0]
> > 35,420,195,423  >   ???:g_tree_lookup (5446926x) [/usr/local/lib/libglib-2.0.so.0]
> >    471,263,222  >   ???:strncpy (712421x) [/lib/libc.so.7]
> >     11,386,067  >   ???:0x00000000004042d0 (222379x) [/usr/local/bin/rrdcached]
> >     84,479,701  >   ???:strdup (222379x) [/lib/libc.so.7]
> >  1,244,348,322  >   ???:0x0000000000402cb0 (5224548x) [/usr/local/bin/rrdcached]
> >     50,510,043  >   ???:pthread_cond_init (222378x) [/lib/libthr.so.3]
> >     94,041,864  >   ???:0x0000000000403160 (5224548x) [/usr/local/bin/rrdcached]
> >     32,049,707  >   ???:malloc (222379x) [/lib/libc.so.7]
> >    565,570,923  >   ???:pthread_mutex_lock (10671475x) [/lib/libthr.so.3]
> >      1,111,895  >   ???:stat (222379x) [/lib/libc.so.7]
> >      1,111,895  >   ???:access (222379x) [/lib/libc.so.7]
> >    693,462,263  >   ???:pthread_mutex_unlock (10671474x) [/lib/libthr.so.3]
> > 
> >        239,084  < ???:pthread_attr_init (998x) [/lib/libthr.so.3]
> >        101,037  < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> >         28,930  < ???:0x000000000000e9a0 (59x) [/lib/libthr.so.3]
> >            107  < ???:g_malloc (1x) [/usr/local/lib/libglib-2.0.so.0]
> >          1,207  < ???:__opendir2 (2x) [/lib/libc.so.7]
> >          1,017  < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >        672,167  < ???:0x0000000000405d50 (998x) [/usr/local/bin/rrdcached]
> >      2,535,802  < ???:rrd_open (4374x) [/usr/local/lib/librrd_th.so.5]
> >     32,049,707  < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
> >         16,099  < ???:0x00000000000e23d0 (30x) [/lib/libc.so.7]
> >            527  < ???:rrd_new_context (1x) [/usr/local/lib/librrd_th.so.5]
> >      1,470,084  < ???:_rrd_update (2916x) [/usr/local/lib/librrd_th.so.5]
> >        514,667  < ???:0x00000000004059e0 (999x) [/usr/local/bin/rrdcached]
> >            573  < ???:dirname (1x) [/lib/libc.so.7]
> >            107  < ???:telldir (1x) [/lib/libc.so.7]
> >    707,922,677  < ???:strdup (5453496x) [/lib/libc.so.7]
> >    617,364,221  *  ???:malloc [/lib/libc.so.7]
> >     28,607,651  >   ???:0x000000000006e630 (99809x) [/lib/libc.so.7]
> >        113,886  >   ???:0x00000000000704c0 (999x) [/lib/libc.so.7]
> >     30,052,836  >   ???:__tls_get_addr (834801x) [/libexec/ld-elf.so.1]
> >            112  >   ???:pthread_mutex_lock (1x) [/lib/libthr.so.3]
> >         22,403  >   ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >         84,578  >   ???:0x000000000006faa0 (1x) [/lib/libc.so.7]
> >     35,853,076  >   ???:pthread_mutex_unlock (833792x) [/lib/libthr.so.3]
> >     30,853,918  >   ???:pthread_mutex_trylock (833858x) [/lib/libthr.so.3]
> >        481,682  >   ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
> > 
> >      2,254,189  < ???:calloc (52423x) [/lib/libc.so.7]
> >        250,475  < ???:posix_memalign (5825x) [/lib/libc.so.7]
> >     30,660,333  < ???:funlockfile (713031x) [/lib/libc.so.7]
> >             43  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >     91,965,245  < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
> >             43  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >        124,442  < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
> >      2,338,082  < ???:realloc (54374x) [/lib/libc.so.7]
> >        326,327  < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> >      1,034,279  < ???:free (24053x) [/lib/libc.so.7]
> >         43,516  < ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
> >         24,897  < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
> >         20,984  < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
> >      9,567,388  < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
> >             43  < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
> >     35,853,076  < ???:malloc (833792x) [/lib/libc.so.7]
> >      1,866,845  < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
> >    693,462,263  < ???:0x00000000004044d0 (10671474x) [/usr/local/bin/rrdcached]
> >             43  < ???:0x00000000004054b0 (1x) [/usr/local/bin/rrdcached]
> >        104,186  < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
> >             63  < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> >         41,108  < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
> >        190,217  < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
> >            854  < ???:vsyslog (18x) [/lib/libc.so.7]
> >    604,145,186  *  ???:pthread_mutex_unlock [/lib/libthr.so.3]
> >        148,960  >   ???:0x0000000000011d10 (9310x) [/lib/libthr.so.3]
> > 
> >            910  < ???:fputs (26x) [/lib/libc.so.7]
> >    197,342,955  < ???:0x00000000000d6570 (1426020x) [/lib/libc.so.7]
> >             94  < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >            127  < ???:dirname (2x) [/lib/libc.so.7]
> >            286  < ???:strstr (26x) [/lib/libc.so.7]
> >        171,864  < ???:setlocale (15624x) [/lib/libc.so.7]
> >    394,492,400  < ???:strdup (5453497x) [/lib/libc.so.7]
> >      1,045,860  < ???:rrd_diff (39605x) [/usr/local/lib/librrd_th.so.5]
> >    592,636,144  *  ???:strlen [/lib/libc.so.7]
> > 
> >         77,380  < ???:0x00000000000e3cd0 (15476x) [/lib/libc.so.7]
> >    714,959,918  < ???:rrd_write (15211914x) [/usr/local/lib/librrd_th.so.5]
> >            103  < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
> >     79,637,630  < ???:write (15927475x) [/lib/libthr.so.3]
> >          7,062  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >     33,700,754  < ???:0x0000000000402cb0 (715559x) [/usr/local/bin/rrdcached]
> >             47  < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> >    477,901,748  *  ???:write [/lib/libthr.so.3]
> >     79,637,365  >   ???:0x00000000000105c0 (15927473x) [/lib/libthr.so.3]
> >          7,020  >   ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >    191,129,712  >   ???:0x0000000000010ad0 (15927476x) [/lib/libthr.so.3]
> >            160  >   ???:__error (16x) [/lib/libthr.so.3]
> >     79,637,630  >   ???:write (15927475x) [/lib/libthr.so.3]
> > 
> >      4,176,726  < ???:0x0000000000012d90 (16001x) [/usr/local/lib/librrd_th.so.5]
> >            216  < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >    471,263,222  < ???:0x00000000004044d0 (712421x) [/usr/local/bin/rrdcached]
> >      2,330,094  < ???:0x0000000000402be0 (13257x) [/usr/local/bin/rrdcached]
> >          4,203  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >             37  < ???:dirname (1x) [/lib/libc.so.7]
> >          8,867  < ???:0x0000000000404e30 (2x) [/usr/local/bin/rrdcached]
> >    475,805,061  *  ???:strncpy [/lib/libc.so.7]
> > 
> >     24,956,869  < ???:flockfile (713031x) [/lib/libc.so.7]
> >          8,450  < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> >             35  < ???:0x0000000000407060 (1x) [/usr/local/bin/rrdcached]
> >            112  < ???:malloc (1x) [/lib/libc.so.7]
> >     74,855,316  < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
> >        102,016  < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
> >    565,570,923  < ???:0x00000000004044d0 (10671475x) [/usr/local/bin/rrdcached]
> >        265,615  < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> >        101,710  < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
> >             35  < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> >            980  < ???:vsyslog (17x) [/lib/libc.so.7]
> >         20,265  < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
> >         21,280  < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
> >        156,707  < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
> >      7,787,360  < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
> >    457,089,329  *  ???:pthread_mutex_lock [/lib/libthr.so.3]
> >         10,375  >   ???:0x000000000000ce70 (7x) [/lib/libthr.so.3]
> >        293,892  >   ???:0x0000000000011f50 (9163x) [/lib/libthr.so.3]
> > 
> >          2,288  < ???:fwrite (26x) [/lib/libc.so.7]
> >    530,685,403  < ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
> >          2,288  < ???:fputs (26x) [/lib/libc.so.7]
> >    423,052,821  *  ???:0x00000000000e1c00 [/lib/libc.so.7]
> >    158,460,479  >   ???:memcpy (6475088x) [/lib/libc.so.7]
> >      1,330,936  >   ???:0x00000000000e2860 (15476x) [/lib/libc.so.7]
> > 
> >            131  < ???:0x000000000000f270 (1x) [/lib/libthr.so.3]
> >            282  < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> >    246,508,222  < ???:fgets (4618515x) [/lib/libc.so.7]
> >        521,326  < ???:0x0000000000071020 (14539x) [/lib/libc.so.7]
> >             19  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >        150,698  < ???:0x00000000004059e0 (998x) [/usr/local/bin/rrdcached]
> >            679  < ???:0x00000000000c1d20 (26x) [/lib/libc.so.7]
> >    158,460,479  < ???:0x00000000000e1c00 (6475088x) [/lib/libc.so.7]
> >    112,592,690  < ???:strdup (5453496x) [/lib/libc.so.7]
> >    343,900,593  *  ???:memcpy [/lib/libc.so.7]
> > 
> >  2,891,857,874  < ???:0x00000000004054b0 (4519717x) [/usr/local/bin/rrdcached]
> >          1,230  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >    287,810,433  *  ???:fgets [/lib/libc.so.7]
> >    246,508,222  >   ???:memcpy (4618515x) [/lib/libc.so.7]
> >  4,918,759,802  >   ???:memchr (4618515x) [/lib/libc.so.7]
> >      6,294,192  >   ???:0x00000000000cde70 (99891x) [/lib/libc.so.7]
> > 
> >  9,794,932,044  < ???:0x0000000000405d50 (714088x) [/usr/local/bin/rrdcached]
> > 40,991,003,725  < ???:0x00000000004054b0 (4519716x) [/usr/local/bin/rrdcached]
> >    266,462,145  *  ???:0x0000000000404a10 [/usr/local/bin/rrdcached]
> >    643,631,298  >   ???:0x0000000000402ac0 (5233804x) [/usr/local/bin/rrdcached]
> >    106,850,750  >   ???:0x0000000000403200 (714088x) [/usr/local/bin/rrdcached]
> >    811,896,161  >   ???:0x00000000004032d0 (5233804x) [/usr/local/bin/rrdcached]
> >     77,121,504  >   ???:strcasecmp (1428176x) [/lib/libc.so.7]
> > 48,762,508,375  >   ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> >     67,182,858  >   ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> >     43,596,406  >   ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> >      1,782,130  >   ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >     30,611,310  >   ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
> > 
> >         17,220  < ???:close (1435x) [/lib/libthr.so.3]
> >      9,788,352  < ???:poll (815696x) [/lib/libthr.so.3]
> >          5,832  < ???:fcntl (486x) [/lib/libthr.so.3]
> >    191,129,712  < ???:write (15927476x) [/lib/libthr.so.3]
> >      8,621,064  < ???:read (718422x) [/lib/libthr.so.3]
> >          5,868  < ???:open (489x) [/lib/libthr.so.3]
> >         11,976  < ???:accept (998x) [/lib/libthr.so.3]
> >    209,580,024  *  ???:0x0000000000010ad0 [/lib/libthr.so.3]
> > 
> >          8,329  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >  1,668,406,047  < ???:0x00000000004044d0 (5223925x) [/usr/local/bin/rrdcached]
> >            395  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >          1,862  < ???:rrd_add_strdup (2x) [/usr/local/lib/librrd_th.so.5]
> >    208,957,160  *  ???:rrd_add_strdup_chunk [/usr/local/lib/librrd_th.so.5]
> >    206,042,383  >   ???:rrd_add_ptr_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
> >          7,685  >   ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >  1,253,409,405  >   ???:strdup (5223928x) [/lib/libc.so.7]
> > 
> >  1,244,348,322  < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> >        377,691  < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >      4,533,242  < ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
> >    182,150,140  *  ???:0x0000000000402cb0 [/usr/local/bin/rrdcached]
> >        174,291  >   ???:syslog (16x) [/lib/libc.so.7]
> >      2,055,311  >   ???:strchr (14729x) [/lib/libc.so.7]
> >     33,700,754  >   ???:write (715559x) [/lib/libthr.so.3]
> >    594,262,965  >   ???:vsnprintf (714086x) [/lib/libc.so.7]
> >     15,036,542  >   ???:free (714070x) [/lib/libc.so.7]
> >         67,482  >   ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
> >    422,021,707  >   ???:sprintf (714086x) [/lib/libc.so.7]
> > 
> >          7,864  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >    206,042,383  < ???:rrd_add_strdup_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
> >    169,961,388  *  ???:rrd_add_ptr_chunk [/usr/local/lib/librrd_th.so.5]
> >          7,820  >   ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >     36,081,039  >   ???:realloc (232968x) [/lib/libc.so.7]
> > 
> >             11  < ???:connect (1x) [/lib/libc.so.7]
> >            160  < ???:write (16x) [/lib/libthr.so.3]
> >             11  < ???:unlink (1x) [/lib/libc.so.7]
> >    304,445,060  < ???:0x0000000000012d90 (15225602x) [/usr/local/lib/librrd_th.so.5]
> >             27  < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
> >             11  < ???:0x000000000000c8b0 (1x) [/lib/libthr.so.3]
> >            200  < ???:stat (20x) [/lib/libc.so.7]
> >             22  < ???:rename (2x) [/lib/libc.so.7]
> >      1,093,680  < ???:setlocale (62496x) [/lib/libc.so.7]
> >             11  < ???:fork (1x) [/lib/libthr.so.3]
> >             99  < ???:0x00000000000e3aa0 (5x) [/lib/libc.so.7]
> >            269  < ???:vsyslog (26x) [/lib/libc.so.7]
> >    152,881,757  *  ???:__error [/lib/libthr.so.3]
> > 
> >    866,968,696  < ???:0x0000000000012d90 (15209978x) [/usr/local/lib/librrd_th.so.5]
> >          7,761  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >        193,116  < ???:_rrd_update (1936x) [/usr/local/lib/librrd_th.so.5]
> >    152,119,140  *  ???:rrd_write [/usr/local/lib/librrd_th.so.5]
> >          7,751  >   ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >    714,959,918  >   ???:write (15211914x) [/lib/libthr.so.3]
> > 
> >  1,253,409,405  < ???:rrd_add_strdup_chunk (5223928x) [/usr/local/lib/librrd_th.so.5]
> >         15,065  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >            214  < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
> >            682  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >     84,479,701  < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
> >      2,321,501  < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
> >            210  < ???:0x0000000000404c50 (1x) [/usr/local/bin/rrdcached]
> >        225,797  < ???:0x0000000000407270 (486x) [/usr/local/bin/rrdcached]
> >            558  < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
> >    125,430,431  *  ???:strdup [/lib/libc.so.7]
> >         22,458  >   ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> >    394,492,400  >   ???:strlen (5453497x) [/lib/libc.so.7]
> >    112,592,690  >   ???:memcpy (5453496x) [/lib/libc.so.7]
> >    707,922,677  >   ???:malloc (5453496x) [/lib/libc.so.7]
> > 
> >    811,896,161  < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
> >    120,542,093  *  ???:0x00000000004032d0 [/usr/local/bin/rrdcached]
> >    691,354,068  >   ???:strcasecmp (5252093x) [/lib/libc.so.7]
> > 
> >     99,266,412  < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> >          3,667  < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >     99,270,079  *  ???:0x0000000000403730 [/usr/local/bin/rrdcached]
> > 
> >     94,041,864  < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> >          3,474  < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >     94,045,338  *  ???:0x0000000000403160 [/usr/local/bin/rrdcached]
> > 
> >            541  < ???:0x000000000006dd20 (1x) [/lib/libc.so.7]
> >      8,229,036  < ???:calloc (222396x) [/lib/libc.so.7]
> >            245  < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> >         78,078  < ???:0x000000000000e9a0 (1001x) [/lib/libthr.so.3]
> >             44  < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
> >             45  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >     32,730,525  < ???:vsnprintf (727345x) [/lib/libc.so.7]
> >             18  < ???:0x00000000004059e0 (1x) [/usr/local/bin/rrdcached]
> >            117  < ???:0x000000000006f960 (3x) [/lib/libc.so.7]
> >         42,042  < ???:pthread_create (1001x) [/lib/libthr.so.3]
> >          1,320  < ???:0x0000000000404110 (30x) [/usr/local/bin/rrdcached]
> >          1,350  < ???:0x00000000000e2660 (30x) [/lib/libc.so.7]
> >            315  < ???:snprintf (7x) [/lib/libc.so.7]
> >     32,135,130  < ???:sprintf (714114x) [/lib/libc.so.7]
> >             88  < ???:0x00000000004054b0 (2x) [/usr/local/bin/rrdcached]
> >     27,130,238  < ???:0x00000000004044d0 (444758x) [/usr/local/bin/rrdcached]
> >     91,676,351  *  ???:memset [/lib/libc.so.7]
> > 
> >          7,175  < ???:close (1435x) [/lib/libthr.so.3]
> >      4,078,230  < ???:poll (815646x) [/lib/libthr.so.3]
> >          2,430  < ???:fcntl (486x) [/lib/libthr.so.3]
> >     79,637,365  < ???:write (15927473x) [/lib/libthr.so.3]
> >      3,592,110  < ???:read (718422x) [/lib/libthr.so.3]
> >          2,445  < ???:open (489x) [/lib/libthr.so.3]
> >          4,990  < ???:accept (998x) [/lib/libthr.so.3]
> >     87,324,745  *  ???:0x00000000000105c0 [/lib/libthr.so.3]
> > 
> >    545,599,355  < ???:0x00000000004044d0 (222378x) [/usr/local/bin/rrdcached]
> >         27,442  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >     81,093,490  *  ???:g_tree_replace [/usr/local/lib/libglib-2.0.so.0]
> >     51,719,493  >   ???:0x000000000005f0c0 (222379x) [/usr/local/lib/libglib-2.0.so.0]
> >      5,275,989  >   ???:0x000000000005e9d0 (120343x) [/usr/local/lib/libglib-2.0.so.0]
> >    650,318,686  >   ???:strcmp (3882113x) [/lib/libc.so.7]
> > 
> >  2,675,232,567  < ???:0x00000000000083b0 (52x) [/lib/libthr.so.3]
> >  7,670,744,511  < ???:0x00000000000083b0'2 (946x) [/lib/libthr.so.3]
> >     67,444,844  *  ???:0x0000000000405d50 [/usr/local/bin/rrdcached]
> >     38,237,750  >   ???:poll (813590x) [/lib/libthr.so.3]
> >         60,653  >   ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
> >        672,167  >   ???:malloc (998x) [/lib/libc.so.7]
> >  9,794,932,044  >   ???:0x0000000000404a10 (714088x) [/usr/local/bin/rrdcached]
> >        102,016  >   ???:pthread_mutex_lock (1946x) [/lib/libthr.so.3]
> >      9,282,910  >   ???:memmove (714070x) [/lib/libc.so.7]
> >     14,281,760  >   ???:time (714088x) [/lib/libc.so.7]
> >     33,605,893  >   ???:read (715019x) [/lib/libthr.so.3]
> >        612,062  >   ???:0x0000000000403850 (948x) [/usr/local/bin/rrdcached]
> >    386,717,461  >   ???:memchr (1428158x) [/lib/libc.so.7]
> >             90  >   ???:pthread_cond_broadcast (2x) [/lib/libthr.so.3]
> >        124,442  >   ???:pthread_mutex_unlock (1946x) [/lib/libthr.so.3]
> > 
> > 35,420,195,423  < ???:0x00000000004044d0 (5446926x) [/usr/local/bin/rrdcached]
> >     47,658,712  < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> >             23  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >      1,105,775  < ???:0x0000000000407270 (484x) [/usr/local/bin/rrdcached]
> >      1,174,749  < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> >     59,562,102  *  ???:g_tree_lookup [/usr/local/lib/libglib-2.0.so.0]
> > 34,013,769,458  >   ???:0x000000000005e6c0 (5455193x) [/usr/local/lib/libglib-2.0.so.0]
> > 
> >    789,153,788  < ???:0x00000000000d6570 (5020519x) [/lib/libc.so.7]
> >     50,205,190  *  ???:0x00000000000d6540 [/lib/libc.so.7]
> >    530,685,403  >   ???:0x00000000000e1c00 (5020519x) [/lib/libc.so.7]
> > 
> >      1,939,651  < ???:calloc (52423x) [/lib/libc.so.7]
> >             37  < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
> >        215,525  < ???:posix_memalign (5825x) [/lib/libc.so.7]
> >     30,853,918  < ???:malloc (833858x) [/lib/libc.so.7]
> >      1,606,429  < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
> >             37  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >      2,011,838  < ???:realloc (54374x) [/lib/libc.so.7]
> >        889,961  < ???:free (24053x) [/lib/libc.so.7]
> >         35,372  < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
> >         37,940  < ???:0x0000000000070e80 (1021x) [/lib/libc.so.7]
> >     37,053,112  *  ???:pthread_mutex_trylock [/lib/libthr.so.3]
> > 
> >            833  < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> >     11,503,249  < ???:0x0000000000402fd0 (13257x) [/usr/local/bin/rrdcached]
> >    594,262,965  < ???:0x0000000000402cb0 (714086x) [/usr/local/bin/rrdcached]
> >            416  < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> >     34,912,560  *  ???:vsnprintf [/lib/libc.so.7]
> >    538,124,378  >   ???:0x00000000000d6570 (727345x) [/lib/libc.so.7]
> >     32,730,525  >   ???:memset (727345x) [/lib/libc.so.7]
> > 
> >        521,028  < ???:0x0000000000071020 (14473x) [/lib/libc.so.7]
> >      1,887,300  < ???:calloc (52425x) [/lib/libc.so.7]
> >             36  < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> >        108,108  < ???:0x00000000000704c0 (2002x) [/lib/libc.so.7]
> >      1,957,464  < ???:realloc (54374x) [/lib/libc.so.7]
> >        209,700  < ???:posix_memalign (5825x) [/lib/libc.so.7]
> >     30,052,836  < ???:malloc (834801x) [/lib/libc.so.7]
> >     34,700,436  *  ???:__tls_get_addr [/libexec/ld-elf.so.1]
> > 
> 
> _______________________________________________
> rrd-developers mailing list
> rrd-developers at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers



More information about the rrd-developers mailing list