[rrd-developers] rrdcached performance with >200k nodes
kevin brintnall
kbrint at rufus.net
Wed Jan 13 18:41:44 CET 2010
On Wed, Jan 13, 2010 at 06:28:36PM +0100, Mirek Lau?? wrote:
> Kevin,
>
> attached is the callgrind annotate output with --tree=both.
Thanks, I'll review.
> Is it possible to attach valgrind/callgrind to a running process.
I don't think so. I think it does its work by intercepting normal calls
with LD_PRELOAD.
> Running the rrdcached under
> callgrind would take years to reach the point where it get stuck in the queue.
Understood.. Maybe there's a way to reproduce the problem on a smaller scale.
> I've also noticed very high number of context switches while the queue
> is stuck:
How much higher compared to normal operation?
> last pid: 31196; load averages: 6.35, 4.17, 3.57
>
> up 50+20:06:20 18:26:44
> 267 processes: 10 running, 239 sleeping, 18 zombie
> CPU 0: 7.5% user, 0.0% nice, 91.4% system, 1.1% interrupt, 0.0% idle
> CPU 1: 6.0% user, 0.0% nice, 92.9% system, 0.7% interrupt, 0.4% idle
> CPU 2: 13.9% user, 0.0% nice, 85.4% system, 0.0% interrupt, 0.7% idle
> CPU 3: 17.7% user, 0.0% nice, 82.0% system, 0.0% interrupt, 0.4% idle
> Mem: 1578M Active, 13G Inact, 815M Wired, 577M Cache, 399M Buf, 185M Free
> Swap: 8192M Total, 100K Used, 8192M Free
>
> PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
> 96584 portax 5495 5367 99 13 0 112 1.67% rrdcached
>
> Now, our rrdcached made it to 3k files in the queue after flushall
> (220k) in about 20 minutes.
> Now it is almost stuck, very slowly writing the queue - write rate
> went from 200/sec to about 5/sec :-(
I'm surprised with how much CPU time is spent in "system".. While you are
still in this "slow trickle" mode, can you attach to the running process
with ktrace and see what system calls it's making?
Also, what locale are you using?
--
kevin brintnall =~ /kbrint at rufus.net/
> On Wed, Jan 13, 2010 at 5:33 PM, kevin brintnall <kbrint at rufus.net> wrote:
> > On Wed, Jan 13, 2010 at 05:21:57PM +0100, Mirek Lau?? wrote:
> >> there is about 224k nodes in the tree, after issuing FLUSHALL
> >> it takes about 20 minuts (with -t 8) to write almost all nodes
> >
> > This sounds about right.
> >
> >> we're now stuck at approx 10k nodes in queue,
> >> journal continues to write, updates are received at normal rate
> >
> > Try issuing the "QUEUE" command to the daemon when in this state. ??I just
> > want to verify that you are seeing files continually added to the end of
> > the queue, and being removed from the front.
> >
> >> ?? PID USERNAME ??THR PRI NICE ?? SIZE ?? ??RES STATE ??C ?? TIME ?? ??CPU COMMAND
> >> 91510 portax ?? ?? 60 ??44 ?? ??0 ?? 133M 90052K select 3 ?? 0:00 102.10% rrdcached
> >
> > What does 'top -H' show for this pid? ??Is the CPU spread out evenly or
> > dominated by one thread? ??Since it's so close to 100%, I'm guessing one
> > thread. ??It would be nice to determine whether it's the thread that's
> > handling your UPDATE (vs. the queue threads).
> >
> > Also, can you repeat the earlier 'callgrind' output, but with --tree=both ?
> >
> > --
> > ??kevin brintnall =~ /kbrint at rufus.net/
> >
>
> -ml
> --------------------------------------------------------------------------------
> Profile data file 'callgrind.out.72526' (creator: callgrind-3.5.0)
> --------------------------------------------------------------------------------
> I1 cache:
> D1 cache:
> L2 cache:
> Timerange: Basic block 0 - 15555346483
> Trigger: Program termination
> Profiled target: /usr/local/bin/rrdcached -l unix:/tmp/rrdcached.sock -w 3600 -z 3600 -t 2 -f 7200 -m 64 -j /usr/local/portax/var/rrd/journal -p /tmp/rrdcached.pid (PID 72526, part 1)
> Events recorded: Ir
> Events shown: Ir
> Event sort order: Ir
> Thresholds: 99
> Include dirs:
> User annotated:
> Auto-annotation: off
>
> --------------------------------------------------------------------------------
> Ir
> --------------------------------------------------------------------------------
> 56,987,676,075 PROGRAM TOTALS
>
> --------------------------------------------------------------------------------
> Ir file:function
> --------------------------------------------------------------------------------
>
> 44,378,954 < ???:strcmp'2 (132864x) [/lib/libc.so.7]
> 31,901,245,144 < ???:strcmp (92178524x) [/lib/libc.so.7]
> 31,901,245,144 * ???:strcmp'2 [/lib/libc.so.7]
> 44,378,954 > ???:strcmp'2 (132864x) [/lib/libc.so.7]
>
> 8,027,935,482 < ???:0x00000000004044d0 (10449096x) [/usr/local/bin/rrdcached]
> 163,873 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 643,631,298 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
> 7,011,268,017 * ???:0x0000000000402ac0 [/usr/local/bin/rrdcached]
>
> 23 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 3,643 < ???:0x00000000000d6570 (77x) [/lib/libc.so.7]
> 386,717,461 < ???:0x0000000000405d50 (1428158x) [/usr/local/bin/rrdcached]
> 478 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 4,918,759,802 < ???:fgets (4618515x) [/lib/libc.so.7]
> 2,860,147,538 * ???:memchr [/lib/libc.so.7]
>
> 164 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 1,536,016,818 < ???:0x00000000004044d0 (5224547x) [/usr/local/bin/rrdcached]
> 294 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 239,663 < ???:atoi (1462x) [/lib/libc.so.7]
> 1,536,256,939 * ???:strtol [/lib/libc.so.7]
>
> 650,318,686 < ???:g_tree_replace (3882113x) [/usr/local/lib/libglib-2.0.so.0]
> 154 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 32,824,017,339 < ???:0x000000000005e6c0 (92187514x) [/usr/local/lib/libglib-2.0.so.0]
> 593,674 < ???:setlocale (15623x) [/lib/libc.so.7]
> 1,347,413,568 * ???:strcmp [/lib/libc.so.7]
> 31,901,245,144 > ???:strcmp'2 (92178524x) [/lib/libc.so.7]
>
> 34,013,769,458 < ???:g_tree_lookup (5455193x) [/usr/local/lib/libglib-2.0.so.0]
> 1,189,752,119 * ???:0x000000000005e6c0 [/usr/local/lib/libglib-2.0.so.0]
> 32,824,017,339 > ???:strcmp (92187514x) [/lib/libc.so.7]
>
> 44,799,381,715 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
> 916,274,439 * ???:0x00000000004054b0 [/usr/local/bin/rrdcached]
> 524 > ???:calloc (1x) [/lib/libc.so.7]
> 88 > ???:memset (2x) [/lib/libc.so.7]
> 202 > ???:readdir (4x) [/lib/libc.so.7]
> 287,740 > ???:0x00000000000034d0 (12x) [/libexec/ld-elf.so.1]
> 2,891,857,874 > ???:fgets (4519717x) [/lib/libc.so.7]
> 98,364 > ???:syslog (6x) [/lib/libc.so.7]
> 616 > ???:fclose (2x) [/lib/libc.so.7]
> 947 > ???:rrd_add_strdup (1x) [/usr/local/lib/librrd_th.so.5]
> 8,407 > ???:snprintf (5x) [/lib/libc.so.7]
> 40,991,003,725 > ???:0x0000000000404a10 (4519716x) [/usr/local/bin/rrdcached]
> 20 > ???:time (1x) [/lib/libc.so.7]
> 33,287 > ???:0x0000000000403930 (1x) [/usr/local/bin/rrdcached]
> 5 > ???:stat (1x) [/lib/libc.so.7]
> 24 > ???:rename (1x) [/lib/libc.so.7]
> 181 > ???:fopen (1x) [/lib/libc.so.7]
> 43 > ???:pthread_mutex_unlock (1x) [/lib/libthr.so.3]
>
> 538,124,378 < ???:vsnprintf (727345x) [/lib/libc.so.7]
> 765,331,067 < ???:vfprintf (713010x) [/lib/libc.so.7]
> 6,917 < ???:snprintf (7x) [/lib/libc.so.7]
> 357,841,714 < ???:sprintf (714114x) [/lib/libc.so.7]
> 838,626,751 * ???:0x00000000000d6570 [/lib/libc.so.7]
> 197,342,955 > ???:strlen (1426020x) [/lib/libc.so.7]
> 18,554,936 > ???:0x00000000000d56d0 (1439955x) [/lib/libc.so.7]
> 21,083 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 789,153,788 > ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
> 17,448 > ???:0x00000000000e2240 (28x) [/lib/libc.so.7]
> 25,853,700 > ???:localeconv (2154475x) [/lib/libc.so.7]
> 3,643 > ???:memchr (77x) [/lib/libc.so.7]
>
> 94,618,746 < ???:0x0000000000403200 (724865x) [/usr/local/bin/rrdcached]
> 132 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 77,121,504 < ???:0x0000000000404a10 (1428176x) [/usr/local/bin/rrdcached]
> 691,354,068 < ???:0x00000000004032d0 (5252093x) [/usr/local/bin/rrdcached]
> 837,387,282 * ???:strcasecmp [/lib/libc.so.7]
>
> 1,819,672,633 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
> 750,265,924 * ???:0x0000000000012d90 [/usr/local/lib/librrd_th.so.5]
> 866,968,696 > ???:rrd_write (15209978x) [/usr/local/lib/librrd_th.so.5]
> 27,760,698 > ???:rrd_test_error (263644x) [/usr/local/lib/librrd_th.so.5]
> 1,023 > ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 141,504 > ???:rpnstack_free (16001x) [/usr/local/lib/librrd_th.so.5]
> 215,832 > ???:0x00000000000034d0'2 (15x) [/libexec/ld-elf.so.1]
> 304,445,060 > ???:__error (15225602x) [/lib/libthr.so.3]
> 2,709,912 > ???:dst_conv (32004x) [/usr/local/lib/librrd_th.so.5]
> 4,176,726 > ???:strncpy (16001x) [/lib/libc.so.7]
> 401,820 > ???:free (13394x) [/lib/libc.so.7]
> 10,546,716 > ???:setlocale (15623x) [/lib/libc.so.7]
> 22,120 > ???:rrd_set_to_DINF (1589x) [/usr/local/lib/librrd_th.so.5]
> 7,599,079 > ???:rrd_notify_row (7599079x) [/usr/local/lib/librrd_th.so.5]
> 1,756,830 > ???:strtod (7811x) [/lib/libc.so.7]
> 4,604,204 > ???:rrd_diff (13201x) [/usr/local/lib/librrd_th.so.5]
> 1,952,480 > ???:strchr (13397x) [/lib/libc.so.7]
> 6,949,956 > ???:rrd_seek (151086x) [/usr/local/lib/librrd_th.so.5]
> 396,340 > ???:rrd_set_to_DNAN (35467x) [/usr/local/lib/librrd_th.so.5]
> 90,796 > ???:rpnstack_init (16001x) [/usr/local/lib/librrd_th.so.5]
> 3,969,434 > ???:cf_conv (69221x) [/usr/local/lib/librrd_th.so.5]
> 226,200 > ???:floor (8385x) [/lib/libm.so.5]
> 262,648 > ???:0x0000000000012d40 (17140x) [/usr/local/lib/librrd_th.so.5]
>
> 48,762,508,375 < ???:0x0000000000404a10 (5224548x) [/usr/local/bin/rrdcached]
> 650,488,788 * ???:0x00000000004044d0 [/usr/local/bin/rrdcached]
> 8,027,935,482 > ???:0x0000000000402ac0 (10449096x) [/usr/local/bin/rrdcached]
> 27,130,238 > ???:memset (444758x) [/lib/libc.so.7]
> 1,668,406,047 > ???:rrd_add_strdup_chunk (5223925x) [/usr/local/lib/librrd_th.so.5]
> 1,536,016,818 > ???:strtol (5224547x) [/lib/libc.so.7]
> 129,578 > ???:0x00000000000034d0 (6x) [/libexec/ld-elf.so.1]
> 99,266,412 > ???:0x0000000000403730 (5224548x) [/usr/local/bin/rrdcached]
> 1,029,779,151 > ???:0x0000000000403ac0 (712421x) [/usr/local/bin/rrdcached]
> 545,599,355 > ???:g_tree_replace (222378x) [/usr/local/lib/libglib-2.0.so.0]
> 35,420,195,423 > ???:g_tree_lookup (5446926x) [/usr/local/lib/libglib-2.0.so.0]
> 471,263,222 > ???:strncpy (712421x) [/lib/libc.so.7]
> 11,386,067 > ???:0x00000000004042d0 (222379x) [/usr/local/bin/rrdcached]
> 84,479,701 > ???:strdup (222379x) [/lib/libc.so.7]
> 1,244,348,322 > ???:0x0000000000402cb0 (5224548x) [/usr/local/bin/rrdcached]
> 50,510,043 > ???:pthread_cond_init (222378x) [/lib/libthr.so.3]
> 94,041,864 > ???:0x0000000000403160 (5224548x) [/usr/local/bin/rrdcached]
> 32,049,707 > ???:malloc (222379x) [/lib/libc.so.7]
> 565,570,923 > ???:pthread_mutex_lock (10671475x) [/lib/libthr.so.3]
> 1,111,895 > ???:stat (222379x) [/lib/libc.so.7]
> 1,111,895 > ???:access (222379x) [/lib/libc.so.7]
> 693,462,263 > ???:pthread_mutex_unlock (10671474x) [/lib/libthr.so.3]
>
> 239,084 < ???:pthread_attr_init (998x) [/lib/libthr.so.3]
> 101,037 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> 28,930 < ???:0x000000000000e9a0 (59x) [/lib/libthr.so.3]
> 107 < ???:g_malloc (1x) [/usr/local/lib/libglib-2.0.so.0]
> 1,207 < ???:__opendir2 (2x) [/lib/libc.so.7]
> 1,017 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 672,167 < ???:0x0000000000405d50 (998x) [/usr/local/bin/rrdcached]
> 2,535,802 < ???:rrd_open (4374x) [/usr/local/lib/librrd_th.so.5]
> 32,049,707 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
> 16,099 < ???:0x00000000000e23d0 (30x) [/lib/libc.so.7]
> 527 < ???:rrd_new_context (1x) [/usr/local/lib/librrd_th.so.5]
> 1,470,084 < ???:_rrd_update (2916x) [/usr/local/lib/librrd_th.so.5]
> 514,667 < ???:0x00000000004059e0 (999x) [/usr/local/bin/rrdcached]
> 573 < ???:dirname (1x) [/lib/libc.so.7]
> 107 < ???:telldir (1x) [/lib/libc.so.7]
> 707,922,677 < ???:strdup (5453496x) [/lib/libc.so.7]
> 617,364,221 * ???:malloc [/lib/libc.so.7]
> 28,607,651 > ???:0x000000000006e630 (99809x) [/lib/libc.so.7]
> 113,886 > ???:0x00000000000704c0 (999x) [/lib/libc.so.7]
> 30,052,836 > ???:__tls_get_addr (834801x) [/libexec/ld-elf.so.1]
> 112 > ???:pthread_mutex_lock (1x) [/lib/libthr.so.3]
> 22,403 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 84,578 > ???:0x000000000006faa0 (1x) [/lib/libc.so.7]
> 35,853,076 > ???:pthread_mutex_unlock (833792x) [/lib/libthr.so.3]
> 30,853,918 > ???:pthread_mutex_trylock (833858x) [/lib/libthr.so.3]
> 481,682 > ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
>
> 2,254,189 < ???:calloc (52423x) [/lib/libc.so.7]
> 250,475 < ???:posix_memalign (5825x) [/lib/libc.so.7]
> 30,660,333 < ???:funlockfile (713031x) [/lib/libc.so.7]
> 43 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 91,965,245 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
> 43 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 124,442 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
> 2,338,082 < ???:realloc (54374x) [/lib/libc.so.7]
> 326,327 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> 1,034,279 < ???:free (24053x) [/lib/libc.so.7]
> 43,516 < ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
> 24,897 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
> 20,984 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
> 9,567,388 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
> 43 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
> 35,853,076 < ???:malloc (833792x) [/lib/libc.so.7]
> 1,866,845 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
> 693,462,263 < ???:0x00000000004044d0 (10671474x) [/usr/local/bin/rrdcached]
> 43 < ???:0x00000000004054b0 (1x) [/usr/local/bin/rrdcached]
> 104,186 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
> 63 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> 41,108 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
> 190,217 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
> 854 < ???:vsyslog (18x) [/lib/libc.so.7]
> 604,145,186 * ???:pthread_mutex_unlock [/lib/libthr.so.3]
> 148,960 > ???:0x0000000000011d10 (9310x) [/lib/libthr.so.3]
>
> 910 < ???:fputs (26x) [/lib/libc.so.7]
> 197,342,955 < ???:0x00000000000d6570 (1426020x) [/lib/libc.so.7]
> 94 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 127 < ???:dirname (2x) [/lib/libc.so.7]
> 286 < ???:strstr (26x) [/lib/libc.so.7]
> 171,864 < ???:setlocale (15624x) [/lib/libc.so.7]
> 394,492,400 < ???:strdup (5453497x) [/lib/libc.so.7]
> 1,045,860 < ???:rrd_diff (39605x) [/usr/local/lib/librrd_th.so.5]
> 592,636,144 * ???:strlen [/lib/libc.so.7]
>
> 77,380 < ???:0x00000000000e3cd0 (15476x) [/lib/libc.so.7]
> 714,959,918 < ???:rrd_write (15211914x) [/usr/local/lib/librrd_th.so.5]
> 103 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
> 79,637,630 < ???:write (15927475x) [/lib/libthr.so.3]
> 7,062 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 33,700,754 < ???:0x0000000000402cb0 (715559x) [/usr/local/bin/rrdcached]
> 47 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> 477,901,748 * ???:write [/lib/libthr.so.3]
> 79,637,365 > ???:0x00000000000105c0 (15927473x) [/lib/libthr.so.3]
> 7,020 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 191,129,712 > ???:0x0000000000010ad0 (15927476x) [/lib/libthr.so.3]
> 160 > ???:__error (16x) [/lib/libthr.so.3]
> 79,637,630 > ???:write (15927475x) [/lib/libthr.so.3]
>
> 4,176,726 < ???:0x0000000000012d90 (16001x) [/usr/local/lib/librrd_th.so.5]
> 216 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 471,263,222 < ???:0x00000000004044d0 (712421x) [/usr/local/bin/rrdcached]
> 2,330,094 < ???:0x0000000000402be0 (13257x) [/usr/local/bin/rrdcached]
> 4,203 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 37 < ???:dirname (1x) [/lib/libc.so.7]
> 8,867 < ???:0x0000000000404e30 (2x) [/usr/local/bin/rrdcached]
> 475,805,061 * ???:strncpy [/lib/libc.so.7]
>
> 24,956,869 < ???:flockfile (713031x) [/lib/libc.so.7]
> 8,450 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> 35 < ???:0x0000000000407060 (1x) [/usr/local/bin/rrdcached]
> 112 < ???:malloc (1x) [/lib/libc.so.7]
> 74,855,316 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
> 102,016 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
> 565,570,923 < ???:0x00000000004044d0 (10671475x) [/usr/local/bin/rrdcached]
> 265,615 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> 101,710 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
> 35 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> 980 < ???:vsyslog (17x) [/lib/libc.so.7]
> 20,265 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
> 21,280 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
> 156,707 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
> 7,787,360 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
> 457,089,329 * ???:pthread_mutex_lock [/lib/libthr.so.3]
> 10,375 > ???:0x000000000000ce70 (7x) [/lib/libthr.so.3]
> 293,892 > ???:0x0000000000011f50 (9163x) [/lib/libthr.so.3]
>
> 2,288 < ???:fwrite (26x) [/lib/libc.so.7]
> 530,685,403 < ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
> 2,288 < ???:fputs (26x) [/lib/libc.so.7]
> 423,052,821 * ???:0x00000000000e1c00 [/lib/libc.so.7]
> 158,460,479 > ???:memcpy (6475088x) [/lib/libc.so.7]
> 1,330,936 > ???:0x00000000000e2860 (15476x) [/lib/libc.so.7]
>
> 131 < ???:0x000000000000f270 (1x) [/lib/libthr.so.3]
> 282 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> 246,508,222 < ???:fgets (4618515x) [/lib/libc.so.7]
> 521,326 < ???:0x0000000000071020 (14539x) [/lib/libc.so.7]
> 19 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 150,698 < ???:0x00000000004059e0 (998x) [/usr/local/bin/rrdcached]
> 679 < ???:0x00000000000c1d20 (26x) [/lib/libc.so.7]
> 158,460,479 < ???:0x00000000000e1c00 (6475088x) [/lib/libc.so.7]
> 112,592,690 < ???:strdup (5453496x) [/lib/libc.so.7]
> 343,900,593 * ???:memcpy [/lib/libc.so.7]
>
> 2,891,857,874 < ???:0x00000000004054b0 (4519717x) [/usr/local/bin/rrdcached]
> 1,230 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 287,810,433 * ???:fgets [/lib/libc.so.7]
> 246,508,222 > ???:memcpy (4618515x) [/lib/libc.so.7]
> 4,918,759,802 > ???:memchr (4618515x) [/lib/libc.so.7]
> 6,294,192 > ???:0x00000000000cde70 (99891x) [/lib/libc.so.7]
>
> 9,794,932,044 < ???:0x0000000000405d50 (714088x) [/usr/local/bin/rrdcached]
> 40,991,003,725 < ???:0x00000000004054b0 (4519716x) [/usr/local/bin/rrdcached]
> 266,462,145 * ???:0x0000000000404a10 [/usr/local/bin/rrdcached]
> 643,631,298 > ???:0x0000000000402ac0 (5233804x) [/usr/local/bin/rrdcached]
> 106,850,750 > ???:0x0000000000403200 (714088x) [/usr/local/bin/rrdcached]
> 811,896,161 > ???:0x00000000004032d0 (5233804x) [/usr/local/bin/rrdcached]
> 77,121,504 > ???:strcasecmp (1428176x) [/lib/libc.so.7]
> 48,762,508,375 > ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> 67,182,858 > ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> 43,596,406 > ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> 1,782,130 > ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 30,611,310 > ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
>
> 17,220 < ???:close (1435x) [/lib/libthr.so.3]
> 9,788,352 < ???:poll (815696x) [/lib/libthr.so.3]
> 5,832 < ???:fcntl (486x) [/lib/libthr.so.3]
> 191,129,712 < ???:write (15927476x) [/lib/libthr.so.3]
> 8,621,064 < ???:read (718422x) [/lib/libthr.so.3]
> 5,868 < ???:open (489x) [/lib/libthr.so.3]
> 11,976 < ???:accept (998x) [/lib/libthr.so.3]
> 209,580,024 * ???:0x0000000000010ad0 [/lib/libthr.so.3]
>
> 8,329 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 1,668,406,047 < ???:0x00000000004044d0 (5223925x) [/usr/local/bin/rrdcached]
> 395 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 1,862 < ???:rrd_add_strdup (2x) [/usr/local/lib/librrd_th.so.5]
> 208,957,160 * ???:rrd_add_strdup_chunk [/usr/local/lib/librrd_th.so.5]
> 206,042,383 > ???:rrd_add_ptr_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
> 7,685 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 1,253,409,405 > ???:strdup (5223928x) [/lib/libc.so.7]
>
> 1,244,348,322 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> 377,691 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 4,533,242 < ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
> 182,150,140 * ???:0x0000000000402cb0 [/usr/local/bin/rrdcached]
> 174,291 > ???:syslog (16x) [/lib/libc.so.7]
> 2,055,311 > ???:strchr (14729x) [/lib/libc.so.7]
> 33,700,754 > ???:write (715559x) [/lib/libthr.so.3]
> 594,262,965 > ???:vsnprintf (714086x) [/lib/libc.so.7]
> 15,036,542 > ???:free (714070x) [/lib/libc.so.7]
> 67,482 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
> 422,021,707 > ???:sprintf (714086x) [/lib/libc.so.7]
>
> 7,864 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 206,042,383 < ???:rrd_add_strdup_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
> 169,961,388 * ???:rrd_add_ptr_chunk [/usr/local/lib/librrd_th.so.5]
> 7,820 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 36,081,039 > ???:realloc (232968x) [/lib/libc.so.7]
>
> 11 < ???:connect (1x) [/lib/libc.so.7]
> 160 < ???:write (16x) [/lib/libthr.so.3]
> 11 < ???:unlink (1x) [/lib/libc.so.7]
> 304,445,060 < ???:0x0000000000012d90 (15225602x) [/usr/local/lib/librrd_th.so.5]
> 27 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
> 11 < ???:0x000000000000c8b0 (1x) [/lib/libthr.so.3]
> 200 < ???:stat (20x) [/lib/libc.so.7]
> 22 < ???:rename (2x) [/lib/libc.so.7]
> 1,093,680 < ???:setlocale (62496x) [/lib/libc.so.7]
> 11 < ???:fork (1x) [/lib/libthr.so.3]
> 99 < ???:0x00000000000e3aa0 (5x) [/lib/libc.so.7]
> 269 < ???:vsyslog (26x) [/lib/libc.so.7]
> 152,881,757 * ???:__error [/lib/libthr.so.3]
>
> 866,968,696 < ???:0x0000000000012d90 (15209978x) [/usr/local/lib/librrd_th.so.5]
> 7,761 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 193,116 < ???:_rrd_update (1936x) [/usr/local/lib/librrd_th.so.5]
> 152,119,140 * ???:rrd_write [/usr/local/lib/librrd_th.so.5]
> 7,751 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 714,959,918 > ???:write (15211914x) [/lib/libthr.so.3]
>
> 1,253,409,405 < ???:rrd_add_strdup_chunk (5223928x) [/usr/local/lib/librrd_th.so.5]
> 15,065 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 214 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
> 682 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 84,479,701 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
> 2,321,501 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
> 210 < ???:0x0000000000404c50 (1x) [/usr/local/bin/rrdcached]
> 225,797 < ???:0x0000000000407270 (486x) [/usr/local/bin/rrdcached]
> 558 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
> 125,430,431 * ???:strdup [/lib/libc.so.7]
> 22,458 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
> 394,492,400 > ???:strlen (5453497x) [/lib/libc.so.7]
> 112,592,690 > ???:memcpy (5453496x) [/lib/libc.so.7]
> 707,922,677 > ???:malloc (5453496x) [/lib/libc.so.7]
>
> 811,896,161 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
> 120,542,093 * ???:0x00000000004032d0 [/usr/local/bin/rrdcached]
> 691,354,068 > ???:strcasecmp (5252093x) [/lib/libc.so.7]
>
> 99,266,412 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> 3,667 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 99,270,079 * ???:0x0000000000403730 [/usr/local/bin/rrdcached]
>
> 94,041,864 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
> 3,474 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 94,045,338 * ???:0x0000000000403160 [/usr/local/bin/rrdcached]
>
> 541 < ???:0x000000000006dd20 (1x) [/lib/libc.so.7]
> 8,229,036 < ???:calloc (222396x) [/lib/libc.so.7]
> 245 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
> 78,078 < ???:0x000000000000e9a0 (1001x) [/lib/libthr.so.3]
> 44 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
> 45 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 32,730,525 < ???:vsnprintf (727345x) [/lib/libc.so.7]
> 18 < ???:0x00000000004059e0 (1x) [/usr/local/bin/rrdcached]
> 117 < ???:0x000000000006f960 (3x) [/lib/libc.so.7]
> 42,042 < ???:pthread_create (1001x) [/lib/libthr.so.3]
> 1,320 < ???:0x0000000000404110 (30x) [/usr/local/bin/rrdcached]
> 1,350 < ???:0x00000000000e2660 (30x) [/lib/libc.so.7]
> 315 < ???:snprintf (7x) [/lib/libc.so.7]
> 32,135,130 < ???:sprintf (714114x) [/lib/libc.so.7]
> 88 < ???:0x00000000004054b0 (2x) [/usr/local/bin/rrdcached]
> 27,130,238 < ???:0x00000000004044d0 (444758x) [/usr/local/bin/rrdcached]
> 91,676,351 * ???:memset [/lib/libc.so.7]
>
> 7,175 < ???:close (1435x) [/lib/libthr.so.3]
> 4,078,230 < ???:poll (815646x) [/lib/libthr.so.3]
> 2,430 < ???:fcntl (486x) [/lib/libthr.so.3]
> 79,637,365 < ???:write (15927473x) [/lib/libthr.so.3]
> 3,592,110 < ???:read (718422x) [/lib/libthr.so.3]
> 2,445 < ???:open (489x) [/lib/libthr.so.3]
> 4,990 < ???:accept (998x) [/lib/libthr.so.3]
> 87,324,745 * ???:0x00000000000105c0 [/lib/libthr.so.3]
>
> 545,599,355 < ???:0x00000000004044d0 (222378x) [/usr/local/bin/rrdcached]
> 27,442 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 81,093,490 * ???:g_tree_replace [/usr/local/lib/libglib-2.0.so.0]
> 51,719,493 > ???:0x000000000005f0c0 (222379x) [/usr/local/lib/libglib-2.0.so.0]
> 5,275,989 > ???:0x000000000005e9d0 (120343x) [/usr/local/lib/libglib-2.0.so.0]
> 650,318,686 > ???:strcmp (3882113x) [/lib/libc.so.7]
>
> 2,675,232,567 < ???:0x00000000000083b0 (52x) [/lib/libthr.so.3]
> 7,670,744,511 < ???:0x00000000000083b0'2 (946x) [/lib/libthr.so.3]
> 67,444,844 * ???:0x0000000000405d50 [/usr/local/bin/rrdcached]
> 38,237,750 > ???:poll (813590x) [/lib/libthr.so.3]
> 60,653 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
> 672,167 > ???:malloc (998x) [/lib/libc.so.7]
> 9,794,932,044 > ???:0x0000000000404a10 (714088x) [/usr/local/bin/rrdcached]
> 102,016 > ???:pthread_mutex_lock (1946x) [/lib/libthr.so.3]
> 9,282,910 > ???:memmove (714070x) [/lib/libc.so.7]
> 14,281,760 > ???:time (714088x) [/lib/libc.so.7]
> 33,605,893 > ???:read (715019x) [/lib/libthr.so.3]
> 612,062 > ???:0x0000000000403850 (948x) [/usr/local/bin/rrdcached]
> 386,717,461 > ???:memchr (1428158x) [/lib/libc.so.7]
> 90 > ???:pthread_cond_broadcast (2x) [/lib/libthr.so.3]
> 124,442 > ???:pthread_mutex_unlock (1946x) [/lib/libthr.so.3]
>
> 35,420,195,423 < ???:0x00000000004044d0 (5446926x) [/usr/local/bin/rrdcached]
> 47,658,712 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
> 23 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 1,105,775 < ???:0x0000000000407270 (484x) [/usr/local/bin/rrdcached]
> 1,174,749 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
> 59,562,102 * ???:g_tree_lookup [/usr/local/lib/libglib-2.0.so.0]
> 34,013,769,458 > ???:0x000000000005e6c0 (5455193x) [/usr/local/lib/libglib-2.0.so.0]
>
> 789,153,788 < ???:0x00000000000d6570 (5020519x) [/lib/libc.so.7]
> 50,205,190 * ???:0x00000000000d6540 [/lib/libc.so.7]
> 530,685,403 > ???:0x00000000000e1c00 (5020519x) [/lib/libc.so.7]
>
> 1,939,651 < ???:calloc (52423x) [/lib/libc.so.7]
> 37 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
> 215,525 < ???:posix_memalign (5825x) [/lib/libc.so.7]
> 30,853,918 < ???:malloc (833858x) [/lib/libc.so.7]
> 1,606,429 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
> 37 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 2,011,838 < ???:realloc (54374x) [/lib/libc.so.7]
> 889,961 < ???:free (24053x) [/lib/libc.so.7]
> 35,372 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
> 37,940 < ???:0x0000000000070e80 (1021x) [/lib/libc.so.7]
> 37,053,112 * ???:pthread_mutex_trylock [/lib/libthr.so.3]
>
> 833 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
> 11,503,249 < ???:0x0000000000402fd0 (13257x) [/usr/local/bin/rrdcached]
> 594,262,965 < ???:0x0000000000402cb0 (714086x) [/usr/local/bin/rrdcached]
> 416 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
> 34,912,560 * ???:vsnprintf [/lib/libc.so.7]
> 538,124,378 > ???:0x00000000000d6570 (727345x) [/lib/libc.so.7]
> 32,730,525 > ???:memset (727345x) [/lib/libc.so.7]
>
> 521,028 < ???:0x0000000000071020 (14473x) [/lib/libc.so.7]
> 1,887,300 < ???:calloc (52425x) [/lib/libc.so.7]
> 36 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
> 108,108 < ???:0x00000000000704c0 (2002x) [/lib/libc.so.7]
> 1,957,464 < ???:realloc (54374x) [/lib/libc.so.7]
> 209,700 < ???:posix_memalign (5825x) [/lib/libc.so.7]
> 30,052,836 < ???:malloc (834801x) [/lib/libc.so.7]
> 34,700,436 * ???:__tls_get_addr [/libexec/ld-elf.so.1]
>
More information about the rrd-developers
mailing list