[rrd-developers] rrdcached performance with >200k nodes
Mirek Lauš
mirek.laus at gmail.com
Wed Jan 13 20:53:25 CET 2010
Well, now it is getting a little bit more interesting.
The rrdcached has stabilized itself at 155M RSS, 30%WCPU (accorriding to top)
and is able to flush files as they timeout ...
QueueLength: 0
UpdatesReceived: 9851647
FlushesReceived: 2842
UpdatesWritten: 672531
DataSetsWritten: 7711165
TreeNodesNumber: 230373
TreeDepth: 22
JournalBytes: 700189287
JournalRotate: 0
I will try to compile unstriped port of rrdtool.
-ml
On Wed, Jan 13, 2010 at 8:22 PM, kevin brintnall <kbrint at rufus.net> wrote:
> On Wed, Jan 13, 2010 at 11:41:44AM -0600, kevin brintnall wrote:
>> On Wed, Jan 13, 2010 at 06:28:36PM +0100, Mirek Lau?? wrote:
>> > Kevin,
>> >
>> > attached is the callgrind annotate output with --tree=both.
>>
>> Thanks, I'll review.
>
> BTW Mirek, it looks like your rrdcached is without debugging symbols. It
> would be helpful to see the same callgrind output with rrdcached's
> debugging symbols.
>
> --
> kevin brintnall =~ /kbrint at rufus.net/
>
>
>> > Is it possible to attach valgrind/callgrind to a running process.
>>
>> I don't think so. I think it does its work by intercepting normal calls
>> with LD_PRELOAD.
>>
>> > Running the rrdcached under
>> > callgrind would take years to reach the point where it get stuck in the queue.
>>
>> Understood.. Maybe there's a way to reproduce the problem on a smaller scale.
>>
>> > I've also noticed very high number of context switches while the queue
>> > is stuck:
>>
>> How much higher compared to normal operation?
>>
>> > last pid: 31196; load averages: 6.35, 4.17, 3.57
>> >
>> > up 50+20:06:20 18:26:44
>> > 267 processes: 10 running, 239 sleeping, 18 zombie
>> > CPU 0: 7.5% user, 0.0% nice, 91.4% system, 1.1% interrupt, 0.0% idle
>> > CPU 1: 6.0% user, 0.0% nice, 92.9% system, 0.7% interrupt, 0.4% idle
>> > CPU 2: 13.9% user, 0.0% nice, 85.4% system, 0.0% interrupt, 0.7% idle
>> > CPU 3: 17.7% user, 0.0% nice, 82.0% system, 0.0% interrupt, 0.4% idle
>> > Mem: 1578M Active, 13G Inact, 815M Wired, 577M Cache, 399M Buf, 185M Free
>> > Swap: 8192M Total, 100K Used, 8192M Free
>> >
>> > PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
>> > 96584 portax 5495 5367 99 13 0 112 1.67% rrdcached
>> >
>> > Now, our rrdcached made it to 3k files in the queue after flushall
>> > (220k) in about 20 minutes.
>> > Now it is almost stuck, very slowly writing the queue - write rate
>> > went from 200/sec to about 5/sec :-(
>>
>> I'm surprised with how much CPU time is spent in "system".. While you are
>> still in this "slow trickle" mode, can you attach to the running process
>> with ktrace and see what system calls it's making?
>>
>> Also, what locale are you using?
>>
>> --
>> kevin brintnall =~ /kbrint at rufus.net/
>>
>>
>> > On Wed, Jan 13, 2010 at 5:33 PM, kevin brintnall <kbrint at rufus.net> wrote:
>> > > On Wed, Jan 13, 2010 at 05:21:57PM +0100, Mirek Lau?? wrote:
>> > >> there is about 224k nodes in the tree, after issuing FLUSHALL
>> > >> it takes about 20 minuts (with -t 8) to write almost all nodes
>> > >
>> > > This sounds about right.
>> > >
>> > >> we're now stuck at approx 10k nodes in queue,
>> > >> journal continues to write, updates are received at normal rate
>> > >
>> > > Try issuing the "QUEUE" command to the daemon when in this state. ??I just
>> > > want to verify that you are seeing files continually added to the end of
>> > > the queue, and being removed from the front.
>> > >
>> > >> ?? PID USERNAME ??THR PRI NICE ?? SIZE ?? ??RES STATE ??C ?? TIME ?? ??CPU COMMAND
>> > >> 91510 portax ?? ?? 60 ??44 ?? ??0 ?? 133M 90052K select 3 ?? 0:00 102.10% rrdcached
>> > >
>> > > What does 'top -H' show for this pid? ??Is the CPU spread out evenly or
>> > > dominated by one thread? ??Since it's so close to 100%, I'm guessing one
>> > > thread. ??It would be nice to determine whether it's the thread that's
>> > > handling your UPDATE (vs. the queue threads).
>> > >
>> > > Also, can you repeat the earlier 'callgrind' output, but with --tree=both ?
>> > >
>> > > --
>> > > ??kevin brintnall =~ /kbrint at rufus.net/
>> > >
>> >
>> > -ml
>>
>> > --------------------------------------------------------------------------------
>> > Profile data file 'callgrind.out.72526' (creator: callgrind-3.5.0)
>> > --------------------------------------------------------------------------------
>> > I1 cache:
>> > D1 cache:
>> > L2 cache:
>> > Timerange: Basic block 0 - 15555346483
>> > Trigger: Program termination
>> > Profiled target: /usr/local/bin/rrdcached -l unix:/tmp/rrdcached.sock -w 3600 -z 3600 -t 2 -f 7200 -m 64 -j /usr/local/portax/var/rrd/journal -p /tmp/rrdcached.pid (PID 72526, part 1)
>> > Events recorded: Ir
>> > Events shown: Ir
>> > Event sort order: Ir
>> > Thresholds: 99
>> > Include dirs:
>> > User annotated:
>> > Auto-annotation: off
>> >
>> > --------------------------------------------------------------------------------
>> > Ir
>> > --------------------------------------------------------------------------------
>> > 56,987,676,075 PROGRAM TOTALS
>> >
>> > --------------------------------------------------------------------------------
>> > Ir file:function
>> > --------------------------------------------------------------------------------
>> >
>> > 44,378,954 < ???:strcmp'2 (132864x) [/lib/libc.so.7]
>> > 31,901,245,144 < ???:strcmp (92178524x) [/lib/libc.so.7]
>> > 31,901,245,144 * ???:strcmp'2 [/lib/libc.so.7]
>> > 44,378,954 > ???:strcmp'2 (132864x) [/lib/libc.so.7]
>> >
>> > 8,027,935,482 < ???:0x00000000004044d0 (10449096x) [/usr/local/bin/rrdcached]
>> > 163,873 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 643,631,298 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
>> > 7,011,268,017 * ???:0x0000000000402ac0 [/usr/local/bin/rrdcached]
>> >
>> > 23 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 3,643 < ???:0x00000000000d6570 (77x) [/lib/libc.so.7]
>> > 386,717,461 < ???:0x0000000000405d50 (1428158x) [/usr/local/bin/rrdcached]
>> > 478 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 4,918,759,802 < ???:fgets (4618515x) [/lib/libc.so.7]
>> > 2,860,147,538 * ???:memchr [/lib/libc.so.7]
>> >
>> > 164 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 1,536,016,818 < ???:0x00000000004044d0 (5224547x) [/usr/local/bin/rrdcached]
>> > 294 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 239,663 < ???:atoi (1462x) [/lib/libc.so.7]
>> > 1,536,256,939 * ???:strtol [/lib/libc.so.7]
>> >
>> > 650,318,686 < ???:g_tree_replace (3882113x) [/usr/local/lib/libglib-2.0.so.0]
>> > 154 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 32,824,017,339 < ???:0x000000000005e6c0 (92187514x) [/usr/local/lib/libglib-2.0.so.0]
>> > 593,674 < ???:setlocale (15623x) [/lib/libc.so.7]
>> > 1,347,413,568 * ???:strcmp [/lib/libc.so.7]
>> > 31,901,245,144 > ???:strcmp'2 (92178524x) [/lib/libc.so.7]
>> >
>> > 34,013,769,458 < ???:g_tree_lookup (5455193x) [/usr/local/lib/libglib-2.0.so.0]
>> > 1,189,752,119 * ???:0x000000000005e6c0 [/usr/local/lib/libglib-2.0.so.0]
>> > 32,824,017,339 > ???:strcmp (92187514x) [/lib/libc.so.7]
>> >
>> > 44,799,381,715 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
>> > 916,274,439 * ???:0x00000000004054b0 [/usr/local/bin/rrdcached]
>> > 524 > ???:calloc (1x) [/lib/libc.so.7]
>> > 88 > ???:memset (2x) [/lib/libc.so.7]
>> > 202 > ???:readdir (4x) [/lib/libc.so.7]
>> > 287,740 > ???:0x00000000000034d0 (12x) [/libexec/ld-elf.so.1]
>> > 2,891,857,874 > ???:fgets (4519717x) [/lib/libc.so.7]
>> > 98,364 > ???:syslog (6x) [/lib/libc.so.7]
>> > 616 > ???:fclose (2x) [/lib/libc.so.7]
>> > 947 > ???:rrd_add_strdup (1x) [/usr/local/lib/librrd_th.so.5]
>> > 8,407 > ???:snprintf (5x) [/lib/libc.so.7]
>> > 40,991,003,725 > ???:0x0000000000404a10 (4519716x) [/usr/local/bin/rrdcached]
>> > 20 > ???:time (1x) [/lib/libc.so.7]
>> > 33,287 > ???:0x0000000000403930 (1x) [/usr/local/bin/rrdcached]
>> > 5 > ???:stat (1x) [/lib/libc.so.7]
>> > 24 > ???:rename (1x) [/lib/libc.so.7]
>> > 181 > ???:fopen (1x) [/lib/libc.so.7]
>> > 43 > ???:pthread_mutex_unlock (1x) [/lib/libthr.so.3]
>> >
>> > 538,124,378 < ???:vsnprintf (727345x) [/lib/libc.so.7]
>> > 765,331,067 < ???:vfprintf (713010x) [/lib/libc.so.7]
>> > 6,917 < ???:snprintf (7x) [/lib/libc.so.7]
>> > 357,841,714 < ???:sprintf (714114x) [/lib/libc.so.7]
>> > 838,626,751 * ???:0x00000000000d6570 [/lib/libc.so.7]
>> > 197,342,955 > ???:strlen (1426020x) [/lib/libc.so.7]
>> > 18,554,936 > ???:0x00000000000d56d0 (1439955x) [/lib/libc.so.7]
>> > 21,083 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 789,153,788 > ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
>> > 17,448 > ???:0x00000000000e2240 (28x) [/lib/libc.so.7]
>> > 25,853,700 > ???:localeconv (2154475x) [/lib/libc.so.7]
>> > 3,643 > ???:memchr (77x) [/lib/libc.so.7]
>> >
>> > 94,618,746 < ???:0x0000000000403200 (724865x) [/usr/local/bin/rrdcached]
>> > 132 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 77,121,504 < ???:0x0000000000404a10 (1428176x) [/usr/local/bin/rrdcached]
>> > 691,354,068 < ???:0x00000000004032d0 (5252093x) [/usr/local/bin/rrdcached]
>> > 837,387,282 * ???:strcasecmp [/lib/libc.so.7]
>> >
>> > 1,819,672,633 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
>> > 750,265,924 * ???:0x0000000000012d90 [/usr/local/lib/librrd_th.so.5]
>> > 866,968,696 > ???:rrd_write (15209978x) [/usr/local/lib/librrd_th.so.5]
>> > 27,760,698 > ???:rrd_test_error (263644x) [/usr/local/lib/librrd_th.so.5]
>> > 1,023 > ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 141,504 > ???:rpnstack_free (16001x) [/usr/local/lib/librrd_th.so.5]
>> > 215,832 > ???:0x00000000000034d0'2 (15x) [/libexec/ld-elf.so.1]
>> > 304,445,060 > ???:__error (15225602x) [/lib/libthr.so.3]
>> > 2,709,912 > ???:dst_conv (32004x) [/usr/local/lib/librrd_th.so.5]
>> > 4,176,726 > ???:strncpy (16001x) [/lib/libc.so.7]
>> > 401,820 > ???:free (13394x) [/lib/libc.so.7]
>> > 10,546,716 > ???:setlocale (15623x) [/lib/libc.so.7]
>> > 22,120 > ???:rrd_set_to_DINF (1589x) [/usr/local/lib/librrd_th.so.5]
>> > 7,599,079 > ???:rrd_notify_row (7599079x) [/usr/local/lib/librrd_th.so.5]
>> > 1,756,830 > ???:strtod (7811x) [/lib/libc.so.7]
>> > 4,604,204 > ???:rrd_diff (13201x) [/usr/local/lib/librrd_th.so.5]
>> > 1,952,480 > ???:strchr (13397x) [/lib/libc.so.7]
>> > 6,949,956 > ???:rrd_seek (151086x) [/usr/local/lib/librrd_th.so.5]
>> > 396,340 > ???:rrd_set_to_DNAN (35467x) [/usr/local/lib/librrd_th.so.5]
>> > 90,796 > ???:rpnstack_init (16001x) [/usr/local/lib/librrd_th.so.5]
>> > 3,969,434 > ???:cf_conv (69221x) [/usr/local/lib/librrd_th.so.5]
>> > 226,200 > ???:floor (8385x) [/lib/libm.so.5]
>> > 262,648 > ???:0x0000000000012d40 (17140x) [/usr/local/lib/librrd_th.so.5]
>> >
>> > 48,762,508,375 < ???:0x0000000000404a10 (5224548x) [/usr/local/bin/rrdcached]
>> > 650,488,788 * ???:0x00000000004044d0 [/usr/local/bin/rrdcached]
>> > 8,027,935,482 > ???:0x0000000000402ac0 (10449096x) [/usr/local/bin/rrdcached]
>> > 27,130,238 > ???:memset (444758x) [/lib/libc.so.7]
>> > 1,668,406,047 > ???:rrd_add_strdup_chunk (5223925x) [/usr/local/lib/librrd_th.so.5]
>> > 1,536,016,818 > ???:strtol (5224547x) [/lib/libc.so.7]
>> > 129,578 > ???:0x00000000000034d0 (6x) [/libexec/ld-elf.so.1]
>> > 99,266,412 > ???:0x0000000000403730 (5224548x) [/usr/local/bin/rrdcached]
>> > 1,029,779,151 > ???:0x0000000000403ac0 (712421x) [/usr/local/bin/rrdcached]
>> > 545,599,355 > ???:g_tree_replace (222378x) [/usr/local/lib/libglib-2.0.so.0]
>> > 35,420,195,423 > ???:g_tree_lookup (5446926x) [/usr/local/lib/libglib-2.0.so.0]
>> > 471,263,222 > ???:strncpy (712421x) [/lib/libc.so.7]
>> > 11,386,067 > ???:0x00000000004042d0 (222379x) [/usr/local/bin/rrdcached]
>> > 84,479,701 > ???:strdup (222379x) [/lib/libc.so.7]
>> > 1,244,348,322 > ???:0x0000000000402cb0 (5224548x) [/usr/local/bin/rrdcached]
>> > 50,510,043 > ???:pthread_cond_init (222378x) [/lib/libthr.so.3]
>> > 94,041,864 > ???:0x0000000000403160 (5224548x) [/usr/local/bin/rrdcached]
>> > 32,049,707 > ???:malloc (222379x) [/lib/libc.so.7]
>> > 565,570,923 > ???:pthread_mutex_lock (10671475x) [/lib/libthr.so.3]
>> > 1,111,895 > ???:stat (222379x) [/lib/libc.so.7]
>> > 1,111,895 > ???:access (222379x) [/lib/libc.so.7]
>> > 693,462,263 > ???:pthread_mutex_unlock (10671474x) [/lib/libthr.so.3]
>> >
>> > 239,084 < ???:pthread_attr_init (998x) [/lib/libthr.so.3]
>> > 101,037 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
>> > 28,930 < ???:0x000000000000e9a0 (59x) [/lib/libthr.so.3]
>> > 107 < ???:g_malloc (1x) [/usr/local/lib/libglib-2.0.so.0]
>> > 1,207 < ???:__opendir2 (2x) [/lib/libc.so.7]
>> > 1,017 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 672,167 < ???:0x0000000000405d50 (998x) [/usr/local/bin/rrdcached]
>> > 2,535,802 < ???:rrd_open (4374x) [/usr/local/lib/librrd_th.so.5]
>> > 32,049,707 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
>> > 16,099 < ???:0x00000000000e23d0 (30x) [/lib/libc.so.7]
>> > 527 < ???:rrd_new_context (1x) [/usr/local/lib/librrd_th.so.5]
>> > 1,470,084 < ???:_rrd_update (2916x) [/usr/local/lib/librrd_th.so.5]
>> > 514,667 < ???:0x00000000004059e0 (999x) [/usr/local/bin/rrdcached]
>> > 573 < ???:dirname (1x) [/lib/libc.so.7]
>> > 107 < ???:telldir (1x) [/lib/libc.so.7]
>> > 707,922,677 < ???:strdup (5453496x) [/lib/libc.so.7]
>> > 617,364,221 * ???:malloc [/lib/libc.so.7]
>> > 28,607,651 > ???:0x000000000006e630 (99809x) [/lib/libc.so.7]
>> > 113,886 > ???:0x00000000000704c0 (999x) [/lib/libc.so.7]
>> > 30,052,836 > ???:__tls_get_addr (834801x) [/libexec/ld-elf.so.1]
>> > 112 > ???:pthread_mutex_lock (1x) [/lib/libthr.so.3]
>> > 22,403 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 84,578 > ???:0x000000000006faa0 (1x) [/lib/libc.so.7]
>> > 35,853,076 > ???:pthread_mutex_unlock (833792x) [/lib/libthr.so.3]
>> > 30,853,918 > ???:pthread_mutex_trylock (833858x) [/lib/libthr.so.3]
>> > 481,682 > ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
>> >
>> > 2,254,189 < ???:calloc (52423x) [/lib/libc.so.7]
>> > 250,475 < ???:posix_memalign (5825x) [/lib/libc.so.7]
>> > 30,660,333 < ???:funlockfile (713031x) [/lib/libc.so.7]
>> > 43 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 91,965,245 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
>> > 43 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 124,442 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
>> > 2,338,082 < ???:realloc (54374x) [/lib/libc.so.7]
>> > 326,327 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
>> > 1,034,279 < ???:free (24053x) [/lib/libc.so.7]
>> > 43,516 < ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
>> > 24,897 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
>> > 20,984 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
>> > 9,567,388 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
>> > 43 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
>> > 35,853,076 < ???:malloc (833792x) [/lib/libc.so.7]
>> > 1,866,845 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
>> > 693,462,263 < ???:0x00000000004044d0 (10671474x) [/usr/local/bin/rrdcached]
>> > 43 < ???:0x00000000004054b0 (1x) [/usr/local/bin/rrdcached]
>> > 104,186 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
>> > 63 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
>> > 41,108 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
>> > 190,217 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
>> > 854 < ???:vsyslog (18x) [/lib/libc.so.7]
>> > 604,145,186 * ???:pthread_mutex_unlock [/lib/libthr.so.3]
>> > 148,960 > ???:0x0000000000011d10 (9310x) [/lib/libthr.so.3]
>> >
>> > 910 < ???:fputs (26x) [/lib/libc.so.7]
>> > 197,342,955 < ???:0x00000000000d6570 (1426020x) [/lib/libc.so.7]
>> > 94 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 127 < ???:dirname (2x) [/lib/libc.so.7]
>> > 286 < ???:strstr (26x) [/lib/libc.so.7]
>> > 171,864 < ???:setlocale (15624x) [/lib/libc.so.7]
>> > 394,492,400 < ???:strdup (5453497x) [/lib/libc.so.7]
>> > 1,045,860 < ???:rrd_diff (39605x) [/usr/local/lib/librrd_th.so.5]
>> > 592,636,144 * ???:strlen [/lib/libc.so.7]
>> >
>> > 77,380 < ???:0x00000000000e3cd0 (15476x) [/lib/libc.so.7]
>> > 714,959,918 < ???:rrd_write (15211914x) [/usr/local/lib/librrd_th.so.5]
>> > 103 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
>> > 79,637,630 < ???:write (15927475x) [/lib/libthr.so.3]
>> > 7,062 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 33,700,754 < ???:0x0000000000402cb0 (715559x) [/usr/local/bin/rrdcached]
>> > 47 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
>> > 477,901,748 * ???:write [/lib/libthr.so.3]
>> > 79,637,365 > ???:0x00000000000105c0 (15927473x) [/lib/libthr.so.3]
>> > 7,020 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 191,129,712 > ???:0x0000000000010ad0 (15927476x) [/lib/libthr.so.3]
>> > 160 > ???:__error (16x) [/lib/libthr.so.3]
>> > 79,637,630 > ???:write (15927475x) [/lib/libthr.so.3]
>> >
>> > 4,176,726 < ???:0x0000000000012d90 (16001x) [/usr/local/lib/librrd_th.so.5]
>> > 216 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 471,263,222 < ???:0x00000000004044d0 (712421x) [/usr/local/bin/rrdcached]
>> > 2,330,094 < ???:0x0000000000402be0 (13257x) [/usr/local/bin/rrdcached]
>> > 4,203 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 37 < ???:dirname (1x) [/lib/libc.so.7]
>> > 8,867 < ???:0x0000000000404e30 (2x) [/usr/local/bin/rrdcached]
>> > 475,805,061 * ???:strncpy [/lib/libc.so.7]
>> >
>> > 24,956,869 < ???:flockfile (713031x) [/lib/libc.so.7]
>> > 8,450 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
>> > 35 < ???:0x0000000000407060 (1x) [/usr/local/bin/rrdcached]
>> > 112 < ???:malloc (1x) [/lib/libc.so.7]
>> > 74,855,316 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
>> > 102,016 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
>> > 565,570,923 < ???:0x00000000004044d0 (10671475x) [/usr/local/bin/rrdcached]
>> > 265,615 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
>> > 101,710 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
>> > 35 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
>> > 980 < ???:vsyslog (17x) [/lib/libc.so.7]
>> > 20,265 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
>> > 21,280 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
>> > 156,707 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
>> > 7,787,360 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
>> > 457,089,329 * ???:pthread_mutex_lock [/lib/libthr.so.3]
>> > 10,375 > ???:0x000000000000ce70 (7x) [/lib/libthr.so.3]
>> > 293,892 > ???:0x0000000000011f50 (9163x) [/lib/libthr.so.3]
>> >
>> > 2,288 < ???:fwrite (26x) [/lib/libc.so.7]
>> > 530,685,403 < ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
>> > 2,288 < ???:fputs (26x) [/lib/libc.so.7]
>> > 423,052,821 * ???:0x00000000000e1c00 [/lib/libc.so.7]
>> > 158,460,479 > ???:memcpy (6475088x) [/lib/libc.so.7]
>> > 1,330,936 > ???:0x00000000000e2860 (15476x) [/lib/libc.so.7]
>> >
>> > 131 < ???:0x000000000000f270 (1x) [/lib/libthr.so.3]
>> > 282 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
>> > 246,508,222 < ???:fgets (4618515x) [/lib/libc.so.7]
>> > 521,326 < ???:0x0000000000071020 (14539x) [/lib/libc.so.7]
>> > 19 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 150,698 < ???:0x00000000004059e0 (998x) [/usr/local/bin/rrdcached]
>> > 679 < ???:0x00000000000c1d20 (26x) [/lib/libc.so.7]
>> > 158,460,479 < ???:0x00000000000e1c00 (6475088x) [/lib/libc.so.7]
>> > 112,592,690 < ???:strdup (5453496x) [/lib/libc.so.7]
>> > 343,900,593 * ???:memcpy [/lib/libc.so.7]
>> >
>> > 2,891,857,874 < ???:0x00000000004054b0 (4519717x) [/usr/local/bin/rrdcached]
>> > 1,230 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 287,810,433 * ???:fgets [/lib/libc.so.7]
>> > 246,508,222 > ???:memcpy (4618515x) [/lib/libc.so.7]
>> > 4,918,759,802 > ???:memchr (4618515x) [/lib/libc.so.7]
>> > 6,294,192 > ???:0x00000000000cde70 (99891x) [/lib/libc.so.7]
>> >
>> > 9,794,932,044 < ???:0x0000000000405d50 (714088x) [/usr/local/bin/rrdcached]
>> > 40,991,003,725 < ???:0x00000000004054b0 (4519716x) [/usr/local/bin/rrdcached]
>> > 266,462,145 * ???:0x0000000000404a10 [/usr/local/bin/rrdcached]
>> > 643,631,298 > ???:0x0000000000402ac0 (5233804x) [/usr/local/bin/rrdcached]
>> > 106,850,750 > ???:0x0000000000403200 (714088x) [/usr/local/bin/rrdcached]
>> > 811,896,161 > ???:0x00000000004032d0 (5233804x) [/usr/local/bin/rrdcached]
>> > 77,121,504 > ???:strcasecmp (1428176x) [/lib/libc.so.7]
>> > 48,762,508,375 > ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
>> > 67,182,858 > ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
>> > 43,596,406 > ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
>> > 1,782,130 > ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 30,611,310 > ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
>> >
>> > 17,220 < ???:close (1435x) [/lib/libthr.so.3]
>> > 9,788,352 < ???:poll (815696x) [/lib/libthr.so.3]
>> > 5,832 < ???:fcntl (486x) [/lib/libthr.so.3]
>> > 191,129,712 < ???:write (15927476x) [/lib/libthr.so.3]
>> > 8,621,064 < ???:read (718422x) [/lib/libthr.so.3]
>> > 5,868 < ???:open (489x) [/lib/libthr.so.3]
>> > 11,976 < ???:accept (998x) [/lib/libthr.so.3]
>> > 209,580,024 * ???:0x0000000000010ad0 [/lib/libthr.so.3]
>> >
>> > 8,329 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 1,668,406,047 < ???:0x00000000004044d0 (5223925x) [/usr/local/bin/rrdcached]
>> > 395 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 1,862 < ???:rrd_add_strdup (2x) [/usr/local/lib/librrd_th.so.5]
>> > 208,957,160 * ???:rrd_add_strdup_chunk [/usr/local/lib/librrd_th.so.5]
>> > 206,042,383 > ???:rrd_add_ptr_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
>> > 7,685 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 1,253,409,405 > ???:strdup (5223928x) [/lib/libc.so.7]
>> >
>> > 1,244,348,322 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
>> > 377,691 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 4,533,242 < ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
>> > 182,150,140 * ???:0x0000000000402cb0 [/usr/local/bin/rrdcached]
>> > 174,291 > ???:syslog (16x) [/lib/libc.so.7]
>> > 2,055,311 > ???:strchr (14729x) [/lib/libc.so.7]
>> > 33,700,754 > ???:write (715559x) [/lib/libthr.so.3]
>> > 594,262,965 > ???:vsnprintf (714086x) [/lib/libc.so.7]
>> > 15,036,542 > ???:free (714070x) [/lib/libc.so.7]
>> > 67,482 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
>> > 422,021,707 > ???:sprintf (714086x) [/lib/libc.so.7]
>> >
>> > 7,864 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 206,042,383 < ???:rrd_add_strdup_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
>> > 169,961,388 * ???:rrd_add_ptr_chunk [/usr/local/lib/librrd_th.so.5]
>> > 7,820 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 36,081,039 > ???:realloc (232968x) [/lib/libc.so.7]
>> >
>> > 11 < ???:connect (1x) [/lib/libc.so.7]
>> > 160 < ???:write (16x) [/lib/libthr.so.3]
>> > 11 < ???:unlink (1x) [/lib/libc.so.7]
>> > 304,445,060 < ???:0x0000000000012d90 (15225602x) [/usr/local/lib/librrd_th.so.5]
>> > 27 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
>> > 11 < ???:0x000000000000c8b0 (1x) [/lib/libthr.so.3]
>> > 200 < ???:stat (20x) [/lib/libc.so.7]
>> > 22 < ???:rename (2x) [/lib/libc.so.7]
>> > 1,093,680 < ???:setlocale (62496x) [/lib/libc.so.7]
>> > 11 < ???:fork (1x) [/lib/libthr.so.3]
>> > 99 < ???:0x00000000000e3aa0 (5x) [/lib/libc.so.7]
>> > 269 < ???:vsyslog (26x) [/lib/libc.so.7]
>> > 152,881,757 * ???:__error [/lib/libthr.so.3]
>> >
>> > 866,968,696 < ???:0x0000000000012d90 (15209978x) [/usr/local/lib/librrd_th.so.5]
>> > 7,761 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 193,116 < ???:_rrd_update (1936x) [/usr/local/lib/librrd_th.so.5]
>> > 152,119,140 * ???:rrd_write [/usr/local/lib/librrd_th.so.5]
>> > 7,751 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 714,959,918 > ???:write (15211914x) [/lib/libthr.so.3]
>> >
>> > 1,253,409,405 < ???:rrd_add_strdup_chunk (5223928x) [/usr/local/lib/librrd_th.so.5]
>> > 15,065 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 214 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
>> > 682 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 84,479,701 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
>> > 2,321,501 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
>> > 210 < ???:0x0000000000404c50 (1x) [/usr/local/bin/rrdcached]
>> > 225,797 < ???:0x0000000000407270 (486x) [/usr/local/bin/rrdcached]
>> > 558 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
>> > 125,430,431 * ???:strdup [/lib/libc.so.7]
>> > 22,458 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
>> > 394,492,400 > ???:strlen (5453497x) [/lib/libc.so.7]
>> > 112,592,690 > ???:memcpy (5453496x) [/lib/libc.so.7]
>> > 707,922,677 > ???:malloc (5453496x) [/lib/libc.so.7]
>> >
>> > 811,896,161 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
>> > 120,542,093 * ???:0x00000000004032d0 [/usr/local/bin/rrdcached]
>> > 691,354,068 > ???:strcasecmp (5252093x) [/lib/libc.so.7]
>> >
>> > 99,266,412 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
>> > 3,667 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 99,270,079 * ???:0x0000000000403730 [/usr/local/bin/rrdcached]
>> >
>> > 94,041,864 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
>> > 3,474 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 94,045,338 * ???:0x0000000000403160 [/usr/local/bin/rrdcached]
>> >
>> > 541 < ???:0x000000000006dd20 (1x) [/lib/libc.so.7]
>> > 8,229,036 < ???:calloc (222396x) [/lib/libc.so.7]
>> > 245 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
>> > 78,078 < ???:0x000000000000e9a0 (1001x) [/lib/libthr.so.3]
>> > 44 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
>> > 45 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 32,730,525 < ???:vsnprintf (727345x) [/lib/libc.so.7]
>> > 18 < ???:0x00000000004059e0 (1x) [/usr/local/bin/rrdcached]
>> > 117 < ???:0x000000000006f960 (3x) [/lib/libc.so.7]
>> > 42,042 < ???:pthread_create (1001x) [/lib/libthr.so.3]
>> > 1,320 < ???:0x0000000000404110 (30x) [/usr/local/bin/rrdcached]
>> > 1,350 < ???:0x00000000000e2660 (30x) [/lib/libc.so.7]
>> > 315 < ???:snprintf (7x) [/lib/libc.so.7]
>> > 32,135,130 < ???:sprintf (714114x) [/lib/libc.so.7]
>> > 88 < ???:0x00000000004054b0 (2x) [/usr/local/bin/rrdcached]
>> > 27,130,238 < ???:0x00000000004044d0 (444758x) [/usr/local/bin/rrdcached]
>> > 91,676,351 * ???:memset [/lib/libc.so.7]
>> >
>> > 7,175 < ???:close (1435x) [/lib/libthr.so.3]
>> > 4,078,230 < ???:poll (815646x) [/lib/libthr.so.3]
>> > 2,430 < ???:fcntl (486x) [/lib/libthr.so.3]
>> > 79,637,365 < ???:write (15927473x) [/lib/libthr.so.3]
>> > 3,592,110 < ???:read (718422x) [/lib/libthr.so.3]
>> > 2,445 < ???:open (489x) [/lib/libthr.so.3]
>> > 4,990 < ???:accept (998x) [/lib/libthr.so.3]
>> > 87,324,745 * ???:0x00000000000105c0 [/lib/libthr.so.3]
>> >
>> > 545,599,355 < ???:0x00000000004044d0 (222378x) [/usr/local/bin/rrdcached]
>> > 27,442 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 81,093,490 * ???:g_tree_replace [/usr/local/lib/libglib-2.0.so.0]
>> > 51,719,493 > ???:0x000000000005f0c0 (222379x) [/usr/local/lib/libglib-2.0.so.0]
>> > 5,275,989 > ???:0x000000000005e9d0 (120343x) [/usr/local/lib/libglib-2.0.so.0]
>> > 650,318,686 > ???:strcmp (3882113x) [/lib/libc.so.7]
>> >
>> > 2,675,232,567 < ???:0x00000000000083b0 (52x) [/lib/libthr.so.3]
>> > 7,670,744,511 < ???:0x00000000000083b0'2 (946x) [/lib/libthr.so.3]
>> > 67,444,844 * ???:0x0000000000405d50 [/usr/local/bin/rrdcached]
>> > 38,237,750 > ???:poll (813590x) [/lib/libthr.so.3]
>> > 60,653 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
>> > 672,167 > ???:malloc (998x) [/lib/libc.so.7]
>> > 9,794,932,044 > ???:0x0000000000404a10 (714088x) [/usr/local/bin/rrdcached]
>> > 102,016 > ???:pthread_mutex_lock (1946x) [/lib/libthr.so.3]
>> > 9,282,910 > ???:memmove (714070x) [/lib/libc.so.7]
>> > 14,281,760 > ???:time (714088x) [/lib/libc.so.7]
>> > 33,605,893 > ???:read (715019x) [/lib/libthr.so.3]
>> > 612,062 > ???:0x0000000000403850 (948x) [/usr/local/bin/rrdcached]
>> > 386,717,461 > ???:memchr (1428158x) [/lib/libc.so.7]
>> > 90 > ???:pthread_cond_broadcast (2x) [/lib/libthr.so.3]
>> > 124,442 > ???:pthread_mutex_unlock (1946x) [/lib/libthr.so.3]
>> >
>> > 35,420,195,423 < ???:0x00000000004044d0 (5446926x) [/usr/local/bin/rrdcached]
>> > 47,658,712 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
>> > 23 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 1,105,775 < ???:0x0000000000407270 (484x) [/usr/local/bin/rrdcached]
>> > 1,174,749 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
>> > 59,562,102 * ???:g_tree_lookup [/usr/local/lib/libglib-2.0.so.0]
>> > 34,013,769,458 > ???:0x000000000005e6c0 (5455193x) [/usr/local/lib/libglib-2.0.so.0]
>> >
>> > 789,153,788 < ???:0x00000000000d6570 (5020519x) [/lib/libc.so.7]
>> > 50,205,190 * ???:0x00000000000d6540 [/lib/libc.so.7]
>> > 530,685,403 > ???:0x00000000000e1c00 (5020519x) [/lib/libc.so.7]
>> >
>> > 1,939,651 < ???:calloc (52423x) [/lib/libc.so.7]
>> > 37 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
>> > 215,525 < ???:posix_memalign (5825x) [/lib/libc.so.7]
>> > 30,853,918 < ???:malloc (833858x) [/lib/libc.so.7]
>> > 1,606,429 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
>> > 37 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 2,011,838 < ???:realloc (54374x) [/lib/libc.so.7]
>> > 889,961 < ???:free (24053x) [/lib/libc.so.7]
>> > 35,372 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
>> > 37,940 < ???:0x0000000000070e80 (1021x) [/lib/libc.so.7]
>> > 37,053,112 * ???:pthread_mutex_trylock [/lib/libthr.so.3]
>> >
>> > 833 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
>> > 11,503,249 < ???:0x0000000000402fd0 (13257x) [/usr/local/bin/rrdcached]
>> > 594,262,965 < ???:0x0000000000402cb0 (714086x) [/usr/local/bin/rrdcached]
>> > 416 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
>> > 34,912,560 * ???:vsnprintf [/lib/libc.so.7]
>> > 538,124,378 > ???:0x00000000000d6570 (727345x) [/lib/libc.so.7]
>> > 32,730,525 > ???:memset (727345x) [/lib/libc.so.7]
>> >
>> > 521,028 < ???:0x0000000000071020 (14473x) [/lib/libc.so.7]
>> > 1,887,300 < ???:calloc (52425x) [/lib/libc.so.7]
>> > 36 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
>> > 108,108 < ???:0x00000000000704c0 (2002x) [/lib/libc.so.7]
>> > 1,957,464 < ???:realloc (54374x) [/lib/libc.so.7]
>> > 209,700 < ???:posix_memalign (5825x) [/lib/libc.so.7]
>> > 30,052,836 < ???:malloc (834801x) [/lib/libc.so.7]
>> > 34,700,436 * ???:__tls_get_addr [/libexec/ld-elf.so.1]
>> >
>>
>> _______________________________________________
>> rrd-developers mailing list
>> rrd-developers at lists.oetiker.ch
>> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
>
More information about the rrd-developers
mailing list