[rrd-developers] rrdcached performance with >200k nodes
Mirek Lauš
mirek.laus at gmail.com
Wed Jan 13 18:28:36 CET 2010
Kevin,
attached is the callgrind annotate output with --tree=both.
Is it possible to attach valgrind/callgrind to a running process.
Running the rrdcached under
callgrind would take years to reach the point where it get stuck in the queue.
I've also noticed very high number of context switches while the queue is stuck:
last pid: 31196; load averages: 6.35, 4.17, 3.57
up 50+20:06:20 18:26:44
267 processes: 10 running, 239 sleeping, 18 zombie
CPU 0: 7.5% user, 0.0% nice, 91.4% system, 1.1% interrupt, 0.0% idle
CPU 1: 6.0% user, 0.0% nice, 92.9% system, 0.7% interrupt, 0.4% idle
CPU 2: 13.9% user, 0.0% nice, 85.4% system, 0.0% interrupt, 0.7% idle
CPU 3: 17.7% user, 0.0% nice, 82.0% system, 0.0% interrupt, 0.4% idle
Mem: 1578M Active, 13G Inact, 815M Wired, 577M Cache, 399M Buf, 185M Free
Swap: 8192M Total, 100K Used, 8192M Free
PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
96584 portax 5495 5367 99 13 0 112 1.67% rrdcached
Now, our rrdcached made it to 3k files in the queue after flushall
(220k) in about 20 minutes.
Now it is almost stuck, very slowly writing the queue - write rate
went from 200/sec to about 5/sec :-(
On Wed, Jan 13, 2010 at 5:33 PM, kevin brintnall <kbrint at rufus.net> wrote:
> On Wed, Jan 13, 2010 at 05:21:57PM +0100, Mirek Lau?? wrote:
>> there is about 224k nodes in the tree, after issuing FLUSHALL
>> it takes about 20 minuts (with -t 8) to write almost all nodes
>
> This sounds about right.
>
>> we're now stuck at approx 10k nodes in queue,
>> journal continues to write, updates are received at normal rate
>
> Try issuing the "QUEUE" command to the daemon when in this state. I just
> want to verify that you are seeing files continually added to the end of
> the queue, and being removed from the front.
>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND
>> 91510 portax 60 44 0 133M 90052K select 3 0:00 102.10% rrdcached
>
> What does 'top -H' show for this pid? Is the CPU spread out evenly or
> dominated by one thread? Since it's so close to 100%, I'm guessing one
> thread. It would be nice to determine whether it's the thread that's
> handling your UPDATE (vs. the queue threads).
>
> Also, can you repeat the earlier 'callgrind' output, but with --tree=both ?
>
> --
> kevin brintnall =~ /kbrint at rufus.net/
>
-ml
-------------- next part --------------
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.72526' (creator: callgrind-3.5.0)
--------------------------------------------------------------------------------
I1 cache:
D1 cache:
L2 cache:
Timerange: Basic block 0 - 15555346483
Trigger: Program termination
Profiled target: /usr/local/bin/rrdcached -l unix:/tmp/rrdcached.sock -w 3600 -z 3600 -t 2 -f 7200 -m 64 -j /usr/local/portax/var/rrd/journal -p /tmp/rrdcached.pid (PID 72526, part 1)
Events recorded: Ir
Events shown: Ir
Event sort order: Ir
Thresholds: 99
Include dirs:
User annotated:
Auto-annotation: off
--------------------------------------------------------------------------------
Ir
--------------------------------------------------------------------------------
56,987,676,075 PROGRAM TOTALS
--------------------------------------------------------------------------------
Ir file:function
--------------------------------------------------------------------------------
44,378,954 < ???:strcmp'2 (132864x) [/lib/libc.so.7]
31,901,245,144 < ???:strcmp (92178524x) [/lib/libc.so.7]
31,901,245,144 * ???:strcmp'2 [/lib/libc.so.7]
44,378,954 > ???:strcmp'2 (132864x) [/lib/libc.so.7]
8,027,935,482 < ???:0x00000000004044d0 (10449096x) [/usr/local/bin/rrdcached]
163,873 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
643,631,298 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
7,011,268,017 * ???:0x0000000000402ac0 [/usr/local/bin/rrdcached]
23 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
3,643 < ???:0x00000000000d6570 (77x) [/lib/libc.so.7]
386,717,461 < ???:0x0000000000405d50 (1428158x) [/usr/local/bin/rrdcached]
478 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
4,918,759,802 < ???:fgets (4618515x) [/lib/libc.so.7]
2,860,147,538 * ???:memchr [/lib/libc.so.7]
164 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
1,536,016,818 < ???:0x00000000004044d0 (5224547x) [/usr/local/bin/rrdcached]
294 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
239,663 < ???:atoi (1462x) [/lib/libc.so.7]
1,536,256,939 * ???:strtol [/lib/libc.so.7]
650,318,686 < ???:g_tree_replace (3882113x) [/usr/local/lib/libglib-2.0.so.0]
154 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
32,824,017,339 < ???:0x000000000005e6c0 (92187514x) [/usr/local/lib/libglib-2.0.so.0]
593,674 < ???:setlocale (15623x) [/lib/libc.so.7]
1,347,413,568 * ???:strcmp [/lib/libc.so.7]
31,901,245,144 > ???:strcmp'2 (92178524x) [/lib/libc.so.7]
34,013,769,458 < ???:g_tree_lookup (5455193x) [/usr/local/lib/libglib-2.0.so.0]
1,189,752,119 * ???:0x000000000005e6c0 [/usr/local/lib/libglib-2.0.so.0]
32,824,017,339 > ???:strcmp (92187514x) [/lib/libc.so.7]
44,799,381,715 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
916,274,439 * ???:0x00000000004054b0 [/usr/local/bin/rrdcached]
524 > ???:calloc (1x) [/lib/libc.so.7]
88 > ???:memset (2x) [/lib/libc.so.7]
202 > ???:readdir (4x) [/lib/libc.so.7]
287,740 > ???:0x00000000000034d0 (12x) [/libexec/ld-elf.so.1]
2,891,857,874 > ???:fgets (4519717x) [/lib/libc.so.7]
98,364 > ???:syslog (6x) [/lib/libc.so.7]
616 > ???:fclose (2x) [/lib/libc.so.7]
947 > ???:rrd_add_strdup (1x) [/usr/local/lib/librrd_th.so.5]
8,407 > ???:snprintf (5x) [/lib/libc.so.7]
40,991,003,725 > ???:0x0000000000404a10 (4519716x) [/usr/local/bin/rrdcached]
20 > ???:time (1x) [/lib/libc.so.7]
33,287 > ???:0x0000000000403930 (1x) [/usr/local/bin/rrdcached]
5 > ???:stat (1x) [/lib/libc.so.7]
24 > ???:rename (1x) [/lib/libc.so.7]
181 > ???:fopen (1x) [/lib/libc.so.7]
43 > ???:pthread_mutex_unlock (1x) [/lib/libthr.so.3]
538,124,378 < ???:vsnprintf (727345x) [/lib/libc.so.7]
765,331,067 < ???:vfprintf (713010x) [/lib/libc.so.7]
6,917 < ???:snprintf (7x) [/lib/libc.so.7]
357,841,714 < ???:sprintf (714114x) [/lib/libc.so.7]
838,626,751 * ???:0x00000000000d6570 [/lib/libc.so.7]
197,342,955 > ???:strlen (1426020x) [/lib/libc.so.7]
18,554,936 > ???:0x00000000000d56d0 (1439955x) [/lib/libc.so.7]
21,083 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
789,153,788 > ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
17,448 > ???:0x00000000000e2240 (28x) [/lib/libc.so.7]
25,853,700 > ???:localeconv (2154475x) [/lib/libc.so.7]
3,643 > ???:memchr (77x) [/lib/libc.so.7]
94,618,746 < ???:0x0000000000403200 (724865x) [/usr/local/bin/rrdcached]
132 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
77,121,504 < ???:0x0000000000404a10 (1428176x) [/usr/local/bin/rrdcached]
691,354,068 < ???:0x00000000004032d0 (5252093x) [/usr/local/bin/rrdcached]
837,387,282 * ???:strcasecmp [/lib/libc.so.7]
1,819,672,633 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
750,265,924 * ???:0x0000000000012d90 [/usr/local/lib/librrd_th.so.5]
866,968,696 > ???:rrd_write (15209978x) [/usr/local/lib/librrd_th.so.5]
27,760,698 > ???:rrd_test_error (263644x) [/usr/local/lib/librrd_th.so.5]
1,023 > ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
141,504 > ???:rpnstack_free (16001x) [/usr/local/lib/librrd_th.so.5]
215,832 > ???:0x00000000000034d0'2 (15x) [/libexec/ld-elf.so.1]
304,445,060 > ???:__error (15225602x) [/lib/libthr.so.3]
2,709,912 > ???:dst_conv (32004x) [/usr/local/lib/librrd_th.so.5]
4,176,726 > ???:strncpy (16001x) [/lib/libc.so.7]
401,820 > ???:free (13394x) [/lib/libc.so.7]
10,546,716 > ???:setlocale (15623x) [/lib/libc.so.7]
22,120 > ???:rrd_set_to_DINF (1589x) [/usr/local/lib/librrd_th.so.5]
7,599,079 > ???:rrd_notify_row (7599079x) [/usr/local/lib/librrd_th.so.5]
1,756,830 > ???:strtod (7811x) [/lib/libc.so.7]
4,604,204 > ???:rrd_diff (13201x) [/usr/local/lib/librrd_th.so.5]
1,952,480 > ???:strchr (13397x) [/lib/libc.so.7]
6,949,956 > ???:rrd_seek (151086x) [/usr/local/lib/librrd_th.so.5]
396,340 > ???:rrd_set_to_DNAN (35467x) [/usr/local/lib/librrd_th.so.5]
90,796 > ???:rpnstack_init (16001x) [/usr/local/lib/librrd_th.so.5]
3,969,434 > ???:cf_conv (69221x) [/usr/local/lib/librrd_th.so.5]
226,200 > ???:floor (8385x) [/lib/libm.so.5]
262,648 > ???:0x0000000000012d40 (17140x) [/usr/local/lib/librrd_th.so.5]
48,762,508,375 < ???:0x0000000000404a10 (5224548x) [/usr/local/bin/rrdcached]
650,488,788 * ???:0x00000000004044d0 [/usr/local/bin/rrdcached]
8,027,935,482 > ???:0x0000000000402ac0 (10449096x) [/usr/local/bin/rrdcached]
27,130,238 > ???:memset (444758x) [/lib/libc.so.7]
1,668,406,047 > ???:rrd_add_strdup_chunk (5223925x) [/usr/local/lib/librrd_th.so.5]
1,536,016,818 > ???:strtol (5224547x) [/lib/libc.so.7]
129,578 > ???:0x00000000000034d0 (6x) [/libexec/ld-elf.so.1]
99,266,412 > ???:0x0000000000403730 (5224548x) [/usr/local/bin/rrdcached]
1,029,779,151 > ???:0x0000000000403ac0 (712421x) [/usr/local/bin/rrdcached]
545,599,355 > ???:g_tree_replace (222378x) [/usr/local/lib/libglib-2.0.so.0]
35,420,195,423 > ???:g_tree_lookup (5446926x) [/usr/local/lib/libglib-2.0.so.0]
471,263,222 > ???:strncpy (712421x) [/lib/libc.so.7]
11,386,067 > ???:0x00000000004042d0 (222379x) [/usr/local/bin/rrdcached]
84,479,701 > ???:strdup (222379x) [/lib/libc.so.7]
1,244,348,322 > ???:0x0000000000402cb0 (5224548x) [/usr/local/bin/rrdcached]
50,510,043 > ???:pthread_cond_init (222378x) [/lib/libthr.so.3]
94,041,864 > ???:0x0000000000403160 (5224548x) [/usr/local/bin/rrdcached]
32,049,707 > ???:malloc (222379x) [/lib/libc.so.7]
565,570,923 > ???:pthread_mutex_lock (10671475x) [/lib/libthr.so.3]
1,111,895 > ???:stat (222379x) [/lib/libc.so.7]
1,111,895 > ???:access (222379x) [/lib/libc.so.7]
693,462,263 > ???:pthread_mutex_unlock (10671474x) [/lib/libthr.so.3]
239,084 < ???:pthread_attr_init (998x) [/lib/libthr.so.3]
101,037 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
28,930 < ???:0x000000000000e9a0 (59x) [/lib/libthr.so.3]
107 < ???:g_malloc (1x) [/usr/local/lib/libglib-2.0.so.0]
1,207 < ???:__opendir2 (2x) [/lib/libc.so.7]
1,017 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
672,167 < ???:0x0000000000405d50 (998x) [/usr/local/bin/rrdcached]
2,535,802 < ???:rrd_open (4374x) [/usr/local/lib/librrd_th.so.5]
32,049,707 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
16,099 < ???:0x00000000000e23d0 (30x) [/lib/libc.so.7]
527 < ???:rrd_new_context (1x) [/usr/local/lib/librrd_th.so.5]
1,470,084 < ???:_rrd_update (2916x) [/usr/local/lib/librrd_th.so.5]
514,667 < ???:0x00000000004059e0 (999x) [/usr/local/bin/rrdcached]
573 < ???:dirname (1x) [/lib/libc.so.7]
107 < ???:telldir (1x) [/lib/libc.so.7]
707,922,677 < ???:strdup (5453496x) [/lib/libc.so.7]
617,364,221 * ???:malloc [/lib/libc.so.7]
28,607,651 > ???:0x000000000006e630 (99809x) [/lib/libc.so.7]
113,886 > ???:0x00000000000704c0 (999x) [/lib/libc.so.7]
30,052,836 > ???:__tls_get_addr (834801x) [/libexec/ld-elf.so.1]
112 > ???:pthread_mutex_lock (1x) [/lib/libthr.so.3]
22,403 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
84,578 > ???:0x000000000006faa0 (1x) [/lib/libc.so.7]
35,853,076 > ???:pthread_mutex_unlock (833792x) [/lib/libthr.so.3]
30,853,918 > ???:pthread_mutex_trylock (833858x) [/lib/libthr.so.3]
481,682 > ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
2,254,189 < ???:calloc (52423x) [/lib/libc.so.7]
250,475 < ???:posix_memalign (5825x) [/lib/libc.so.7]
30,660,333 < ???:funlockfile (713031x) [/lib/libc.so.7]
43 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
91,965,245 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
43 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
124,442 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
2,338,082 < ???:realloc (54374x) [/lib/libc.so.7]
326,327 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
1,034,279 < ???:free (24053x) [/lib/libc.so.7]
43,516 < ???:0x0000000000070e80 (1012x) [/lib/libc.so.7]
24,897 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
20,984 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
9,567,388 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
43 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
35,853,076 < ???:malloc (833792x) [/lib/libc.so.7]
1,866,845 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
693,462,263 < ???:0x00000000004044d0 (10671474x) [/usr/local/bin/rrdcached]
43 < ???:0x00000000004054b0 (1x) [/usr/local/bin/rrdcached]
104,186 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
63 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
41,108 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
190,217 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
854 < ???:vsyslog (18x) [/lib/libc.so.7]
604,145,186 * ???:pthread_mutex_unlock [/lib/libthr.so.3]
148,960 > ???:0x0000000000011d10 (9310x) [/lib/libthr.so.3]
910 < ???:fputs (26x) [/lib/libc.so.7]
197,342,955 < ???:0x00000000000d6570 (1426020x) [/lib/libc.so.7]
94 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
127 < ???:dirname (2x) [/lib/libc.so.7]
286 < ???:strstr (26x) [/lib/libc.so.7]
171,864 < ???:setlocale (15624x) [/lib/libc.so.7]
394,492,400 < ???:strdup (5453497x) [/lib/libc.so.7]
1,045,860 < ???:rrd_diff (39605x) [/usr/local/lib/librrd_th.so.5]
592,636,144 * ???:strlen [/lib/libc.so.7]
77,380 < ???:0x00000000000e3cd0 (15476x) [/lib/libc.so.7]
714,959,918 < ???:rrd_write (15211914x) [/usr/local/lib/librrd_th.so.5]
103 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
79,637,630 < ???:write (15927475x) [/lib/libthr.so.3]
7,062 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
33,700,754 < ???:0x0000000000402cb0 (715559x) [/usr/local/bin/rrdcached]
47 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
477,901,748 * ???:write [/lib/libthr.so.3]
79,637,365 > ???:0x00000000000105c0 (15927473x) [/lib/libthr.so.3]
7,020 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
191,129,712 > ???:0x0000000000010ad0 (15927476x) [/lib/libthr.so.3]
160 > ???:__error (16x) [/lib/libthr.so.3]
79,637,630 > ???:write (15927475x) [/lib/libthr.so.3]
4,176,726 < ???:0x0000000000012d90 (16001x) [/usr/local/lib/librrd_th.so.5]
216 < ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
471,263,222 < ???:0x00000000004044d0 (712421x) [/usr/local/bin/rrdcached]
2,330,094 < ???:0x0000000000402be0 (13257x) [/usr/local/bin/rrdcached]
4,203 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
37 < ???:dirname (1x) [/lib/libc.so.7]
8,867 < ???:0x0000000000404e30 (2x) [/usr/local/bin/rrdcached]
475,805,061 * ???:strncpy [/lib/libc.so.7]
24,956,869 < ???:flockfile (713031x) [/lib/libc.so.7]
8,450 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
35 < ???:0x0000000000407060 (1x) [/usr/local/bin/rrdcached]
112 < ???:malloc (1x) [/lib/libc.so.7]
74,855,316 < ???:0x0000000000403ac0 (1425810x) [/usr/local/bin/rrdcached]
102,016 < ???:0x0000000000405d50 (1946x) [/usr/local/bin/rrdcached]
565,570,923 < ???:0x00000000004044d0 (10671475x) [/usr/local/bin/rrdcached]
265,615 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
101,710 < ???:0x0000000000407270 (1454x) [/usr/local/bin/rrdcached]
35 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
980 < ???:vsyslog (17x) [/lib/libc.so.7]
20,265 < ???:0x0000000000404110 (386x) [/usr/local/bin/rrdcached]
21,280 < ???:0x00000000004034b0 (488x) [/usr/local/bin/rrdcached]
156,707 < ???:0x0000000000403580 (2946x) [/usr/local/bin/rrdcached]
7,787,360 < ???:0x0000000000403f20 (222496x) [/usr/local/bin/rrdcached]
457,089,329 * ???:pthread_mutex_lock [/lib/libthr.so.3]
10,375 > ???:0x000000000000ce70 (7x) [/lib/libthr.so.3]
293,892 > ???:0x0000000000011f50 (9163x) [/lib/libthr.so.3]
2,288 < ???:fwrite (26x) [/lib/libc.so.7]
530,685,403 < ???:0x00000000000d6540 (5020519x) [/lib/libc.so.7]
2,288 < ???:fputs (26x) [/lib/libc.so.7]
423,052,821 * ???:0x00000000000e1c00 [/lib/libc.so.7]
158,460,479 > ???:memcpy (6475088x) [/lib/libc.so.7]
1,330,936 > ???:0x00000000000e2860 (15476x) [/lib/libc.so.7]
131 < ???:0x000000000000f270 (1x) [/lib/libthr.so.3]
282 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
246,508,222 < ???:fgets (4618515x) [/lib/libc.so.7]
521,326 < ???:0x0000000000071020 (14539x) [/lib/libc.so.7]
19 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
150,698 < ???:0x00000000004059e0 (998x) [/usr/local/bin/rrdcached]
679 < ???:0x00000000000c1d20 (26x) [/lib/libc.so.7]
158,460,479 < ???:0x00000000000e1c00 (6475088x) [/lib/libc.so.7]
112,592,690 < ???:strdup (5453496x) [/lib/libc.so.7]
343,900,593 * ???:memcpy [/lib/libc.so.7]
2,891,857,874 < ???:0x00000000004054b0 (4519717x) [/usr/local/bin/rrdcached]
1,230 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
287,810,433 * ???:fgets [/lib/libc.so.7]
246,508,222 > ???:memcpy (4618515x) [/lib/libc.so.7]
4,918,759,802 > ???:memchr (4618515x) [/lib/libc.so.7]
6,294,192 > ???:0x00000000000cde70 (99891x) [/lib/libc.so.7]
9,794,932,044 < ???:0x0000000000405d50 (714088x) [/usr/local/bin/rrdcached]
40,991,003,725 < ???:0x00000000004054b0 (4519716x) [/usr/local/bin/rrdcached]
266,462,145 * ???:0x0000000000404a10 [/usr/local/bin/rrdcached]
643,631,298 > ???:0x0000000000402ac0 (5233804x) [/usr/local/bin/rrdcached]
106,850,750 > ???:0x0000000000403200 (714088x) [/usr/local/bin/rrdcached]
811,896,161 > ???:0x00000000004032d0 (5233804x) [/usr/local/bin/rrdcached]
77,121,504 > ???:strcasecmp (1428176x) [/lib/libc.so.7]
48,762,508,375 > ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
67,182,858 > ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
43,596,406 > ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
1,782,130 > ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
30,611,310 > ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
17,220 < ???:close (1435x) [/lib/libthr.so.3]
9,788,352 < ???:poll (815696x) [/lib/libthr.so.3]
5,832 < ???:fcntl (486x) [/lib/libthr.so.3]
191,129,712 < ???:write (15927476x) [/lib/libthr.so.3]
8,621,064 < ???:read (718422x) [/lib/libthr.so.3]
5,868 < ???:open (489x) [/lib/libthr.so.3]
11,976 < ???:accept (998x) [/lib/libthr.so.3]
209,580,024 * ???:0x0000000000010ad0 [/lib/libthr.so.3]
8,329 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
1,668,406,047 < ???:0x00000000004044d0 (5223925x) [/usr/local/bin/rrdcached]
395 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
1,862 < ???:rrd_add_strdup (2x) [/usr/local/lib/librrd_th.so.5]
208,957,160 * ???:rrd_add_strdup_chunk [/usr/local/lib/librrd_th.so.5]
206,042,383 > ???:rrd_add_ptr_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
7,685 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
1,253,409,405 > ???:strdup (5223928x) [/lib/libc.so.7]
1,244,348,322 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
377,691 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
4,533,242 < ???:0x0000000000403580 (1473x) [/usr/local/bin/rrdcached]
182,150,140 * ???:0x0000000000402cb0 [/usr/local/bin/rrdcached]
174,291 > ???:syslog (16x) [/lib/libc.so.7]
2,055,311 > ???:strchr (14729x) [/lib/libc.so.7]
33,700,754 > ???:write (715559x) [/lib/libthr.so.3]
594,262,965 > ???:vsnprintf (714086x) [/lib/libc.so.7]
15,036,542 > ???:free (714070x) [/lib/libc.so.7]
67,482 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
422,021,707 > ???:sprintf (714086x) [/lib/libc.so.7]
7,864 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
206,042,383 < ???:rrd_add_strdup_chunk (5223929x) [/usr/local/lib/librrd_th.so.5]
169,961,388 * ???:rrd_add_ptr_chunk [/usr/local/lib/librrd_th.so.5]
7,820 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
36,081,039 > ???:realloc (232968x) [/lib/libc.so.7]
11 < ???:connect (1x) [/lib/libc.so.7]
160 < ???:write (16x) [/lib/libthr.so.3]
11 < ???:unlink (1x) [/lib/libc.so.7]
304,445,060 < ???:0x0000000000012d90 (15225602x) [/usr/local/lib/librrd_th.so.5]
27 < ???:0x00000000000034d0'2 (3x) [/libexec/ld-elf.so.1]
11 < ???:0x000000000000c8b0 (1x) [/lib/libthr.so.3]
200 < ???:stat (20x) [/lib/libc.so.7]
22 < ???:rename (2x) [/lib/libc.so.7]
1,093,680 < ???:setlocale (62496x) [/lib/libc.so.7]
11 < ???:fork (1x) [/lib/libthr.so.3]
99 < ???:0x00000000000e3aa0 (5x) [/lib/libc.so.7]
269 < ???:vsyslog (26x) [/lib/libc.so.7]
152,881,757 * ???:__error [/lib/libthr.so.3]
866,968,696 < ???:0x0000000000012d90 (15209978x) [/usr/local/lib/librrd_th.so.5]
7,761 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
193,116 < ???:_rrd_update (1936x) [/usr/local/lib/librrd_th.so.5]
152,119,140 * ???:rrd_write [/usr/local/lib/librrd_th.so.5]
7,751 > ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
714,959,918 > ???:write (15211914x) [/lib/libthr.so.3]
1,253,409,405 < ???:rrd_add_strdup_chunk (5223928x) [/usr/local/lib/librrd_th.so.5]
15,065 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
214 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
682 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
84,479,701 < ???:0x00000000004044d0 (222379x) [/usr/local/bin/rrdcached]
2,321,501 < ???:_rrd_update (6699x) [/usr/local/lib/librrd_th.so.5]
210 < ???:0x0000000000404c50 (1x) [/usr/local/bin/rrdcached]
225,797 < ???:0x0000000000407270 (486x) [/usr/local/bin/rrdcached]
558 < ???:0x00000000004061d0 (1x) [/usr/local/bin/rrdcached]
125,430,431 * ???:strdup [/lib/libc.so.7]
22,458 > ???:0x00000000000034d0'2 (2x) [/libexec/ld-elf.so.1]
394,492,400 > ???:strlen (5453497x) [/lib/libc.so.7]
112,592,690 > ???:memcpy (5453496x) [/lib/libc.so.7]
707,922,677 > ???:malloc (5453496x) [/lib/libc.so.7]
811,896,161 < ???:0x0000000000404a10 (5233804x) [/usr/local/bin/rrdcached]
120,542,093 * ???:0x00000000004032d0 [/usr/local/bin/rrdcached]
691,354,068 > ???:strcasecmp (5252093x) [/lib/libc.so.7]
99,266,412 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
3,667 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
99,270,079 * ???:0x0000000000403730 [/usr/local/bin/rrdcached]
94,041,864 < ???:0x00000000004044d0 (5224548x) [/usr/local/bin/rrdcached]
3,474 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
94,045,338 * ???:0x0000000000403160 [/usr/local/bin/rrdcached]
541 < ???:0x000000000006dd20 (1x) [/lib/libc.so.7]
8,229,036 < ???:calloc (222396x) [/lib/libc.so.7]
245 < ???:0x00000000000034d0 (2x) [/libexec/ld-elf.so.1]
78,078 < ???:0x000000000000e9a0 (1001x) [/lib/libthr.so.3]
44 < ???:0x0000000000404e30 (1x) [/usr/local/bin/rrdcached]
45 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
32,730,525 < ???:vsnprintf (727345x) [/lib/libc.so.7]
18 < ???:0x00000000004059e0 (1x) [/usr/local/bin/rrdcached]
117 < ???:0x000000000006f960 (3x) [/lib/libc.so.7]
42,042 < ???:pthread_create (1001x) [/lib/libthr.so.3]
1,320 < ???:0x0000000000404110 (30x) [/usr/local/bin/rrdcached]
1,350 < ???:0x00000000000e2660 (30x) [/lib/libc.so.7]
315 < ???:snprintf (7x) [/lib/libc.so.7]
32,135,130 < ???:sprintf (714114x) [/lib/libc.so.7]
88 < ???:0x00000000004054b0 (2x) [/usr/local/bin/rrdcached]
27,130,238 < ???:0x00000000004044d0 (444758x) [/usr/local/bin/rrdcached]
91,676,351 * ???:memset [/lib/libc.so.7]
7,175 < ???:close (1435x) [/lib/libthr.so.3]
4,078,230 < ???:poll (815646x) [/lib/libthr.so.3]
2,430 < ???:fcntl (486x) [/lib/libthr.so.3]
79,637,365 < ???:write (15927473x) [/lib/libthr.so.3]
3,592,110 < ???:read (718422x) [/lib/libthr.so.3]
2,445 < ???:open (489x) [/lib/libthr.so.3]
4,990 < ???:accept (998x) [/lib/libthr.so.3]
87,324,745 * ???:0x00000000000105c0 [/lib/libthr.so.3]
545,599,355 < ???:0x00000000004044d0 (222378x) [/usr/local/bin/rrdcached]
27,442 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
81,093,490 * ???:g_tree_replace [/usr/local/lib/libglib-2.0.so.0]
51,719,493 > ???:0x000000000005f0c0 (222379x) [/usr/local/lib/libglib-2.0.so.0]
5,275,989 > ???:0x000000000005e9d0 (120343x) [/usr/local/lib/libglib-2.0.so.0]
650,318,686 > ???:strcmp (3882113x) [/lib/libc.so.7]
2,675,232,567 < ???:0x00000000000083b0 (52x) [/lib/libthr.so.3]
7,670,744,511 < ???:0x00000000000083b0'2 (946x) [/lib/libthr.so.3]
67,444,844 * ???:0x0000000000405d50 [/usr/local/bin/rrdcached]
38,237,750 > ???:poll (813590x) [/lib/libthr.so.3]
60,653 > ???:0x00000000000034d0 (4x) [/libexec/ld-elf.so.1]
672,167 > ???:malloc (998x) [/lib/libc.so.7]
9,794,932,044 > ???:0x0000000000404a10 (714088x) [/usr/local/bin/rrdcached]
102,016 > ???:pthread_mutex_lock (1946x) [/lib/libthr.so.3]
9,282,910 > ???:memmove (714070x) [/lib/libc.so.7]
14,281,760 > ???:time (714088x) [/lib/libc.so.7]
33,605,893 > ???:read (715019x) [/lib/libthr.so.3]
612,062 > ???:0x0000000000403850 (948x) [/usr/local/bin/rrdcached]
386,717,461 > ???:memchr (1428158x) [/lib/libc.so.7]
90 > ???:pthread_cond_broadcast (2x) [/lib/libthr.so.3]
124,442 > ???:pthread_mutex_unlock (1946x) [/lib/libthr.so.3]
35,420,195,423 < ???:0x00000000004044d0 (5446926x) [/usr/local/bin/rrdcached]
47,658,712 < ???:0x0000000000404340 (7589x) [/usr/local/bin/rrdcached]
23 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
1,105,775 < ???:0x0000000000407270 (484x) [/usr/local/bin/rrdcached]
1,174,749 < ???:0x0000000000404110 (193x) [/usr/local/bin/rrdcached]
59,562,102 * ???:g_tree_lookup [/usr/local/lib/libglib-2.0.so.0]
34,013,769,458 > ???:0x000000000005e6c0 (5455193x) [/usr/local/lib/libglib-2.0.so.0]
789,153,788 < ???:0x00000000000d6570 (5020519x) [/lib/libc.so.7]
50,205,190 * ???:0x00000000000d6540 [/lib/libc.so.7]
530,685,403 > ???:0x00000000000e1c00 (5020519x) [/lib/libc.so.7]
1,939,651 < ???:calloc (52423x) [/lib/libc.so.7]
37 < ???:0x00000000000704c0 (1x) [/lib/libc.so.7]
215,525 < ???:posix_memalign (5825x) [/lib/libc.so.7]
30,853,918 < ???:malloc (833858x) [/lib/libc.so.7]
1,606,429 < ???:0x0000000000071020 (28944x) [/lib/libc.so.7]
37 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
2,011,838 < ???:realloc (54374x) [/lib/libc.so.7]
889,961 < ???:free (24053x) [/lib/libc.so.7]
35,372 < ???:0x0000000000070a20 (956x) [/lib/libc.so.7]
37,940 < ???:0x0000000000070e80 (1021x) [/lib/libc.so.7]
37,053,112 * ???:pthread_mutex_trylock [/lib/libthr.so.3]
833 < ???:0x00000000000034d0 (1x) [/libexec/ld-elf.so.1]
11,503,249 < ???:0x0000000000402fd0 (13257x) [/usr/local/bin/rrdcached]
594,262,965 < ???:0x0000000000402cb0 (714086x) [/usr/local/bin/rrdcached]
416 < ???:0x0000000000403ed0 (1x) [/usr/local/bin/rrdcached]
34,912,560 * ???:vsnprintf [/lib/libc.so.7]
538,124,378 > ???:0x00000000000d6570 (727345x) [/lib/libc.so.7]
32,730,525 > ???:memset (727345x) [/lib/libc.so.7]
521,028 < ???:0x0000000000071020 (14473x) [/lib/libc.so.7]
1,887,300 < ???:calloc (52425x) [/lib/libc.so.7]
36 < ???:0x00000000000034d0'2 (1x) [/libexec/ld-elf.so.1]
108,108 < ???:0x00000000000704c0 (2002x) [/lib/libc.so.7]
1,957,464 < ???:realloc (54374x) [/lib/libc.so.7]
209,700 < ???:posix_memalign (5825x) [/lib/libc.so.7]
30,052,836 < ???:malloc (834801x) [/lib/libc.so.7]
34,700,436 * ???:__tls_get_addr [/libexec/ld-elf.so.1]
More information about the rrd-developers
mailing list