[rrd-developers] performance under 1.3 beta

[Richard A Steenbergen]

So, I think you guys might want to do some performance tests on a wider 
range of platforms before 1.3 is released. I just tried running a few 
comparison tests between 1.3 (with various settings) and some older 1.2 
rrdupdate code, and got significantly worse performance on the newer 
"better" code.

System tested is a dual quad-proc core 2 xeon (8xCPU SMP) writing to 6xSAS 
15krpm drives doing raid 10, running FreeBSD 7 (scheduled to be released 
shortly), writing updates to 6330 RRD files (tracking 8 counters with 12 
RRA's each) polled from 138 routers.

Each line is a 1 second increment showing KB/transactions, transactions 
per second, and percent cpu used in user, nice'd user, system, interrupt, 
and idle. The first 5-6 seconds of every run is the SNMP poller gathering 
up all the data, then after that you see the rrdupdates starting.

First test is against a very old 1.2rc2 rrdupdate build, with mmap 
enabled. You can see that the system completes the rrdupdates mostly 
asynchronously via mmap, and then flushes the pages to disk all at once. 

      tty           mfid0             cpu
 tin tout  KB/t tps  MB/s  us ni sy in id
   0   44  2.00   1  0.00  56  0 16  0 28
   0   43  0.00   0  0.00  82  0 16  1  1
   0   43  0.00   0  0.00  67  0 32  0  1
   0   43  0.00   0  0.00  38  0 61  1  1
   0   43 16.00  69  1.08  25  0 70  0  5
   0   43  0.00   0  0.00  21  0 66  0 12
   0   43 15.99 1948 30.41  15  0 71  1 12
   0   44  0.00   0  0.00  12  0 82  0  7
   0   43  0.00   0  0.00  11  0 87  0  3
   0   43  0.00   0  0.00   8  0 89  0  3
   0   43  0.00   0  0.00   9  0 85  0  6
   0   43  0.00   0  0.00   8  0 89  0  3
   0  128  0.00   0  0.00   4  0 91  0  5
   0   43  0.00   0  0.00   2  0 90  0  8
   0   43 16.00 249  3.89   9  0 88  0  4
   0   43 16.00 274  4.29   2  0 81  0 17
   0   43 16.00 300  4.69   5  0 43  0 52
   0   43 16.00 208  3.25   0  0  0  0 100

Here is the same test but with rrdupdate from the latest svn trunk of 1.3 
beta, using stock (well almost stock) settings. I had to fix a few things 
to make it compile, which (excluding the blatant violations of C code 
which wouldn't compile anywhere :P) consisted primarily of changing 
fdatasync() to fsync() since fdatasync() does not exist on FreeBSD 
(clearly this would impact performance somewhat). You can see a massive 
increase in the number of transactions, and that we're flushing a huge 
amount of data to disk for a significantly longer period of time. The only 
plus seems to be less CPU use, but this is probably just because we're 
blockingso much during the fsync() that its keeping concurrency down and 
preventing so many context switches and disk seeks (this is a test of 
around 80 or so concurrent rrdupdate processes, not serialized writes).

      tty           mfid0             cpu
 tin tout  KB/t tps  MB/s  us ni sy in id
   0   44  0.00   0  0.00  32  0 16  0 52
   0   43 45.83 499 22.33  82  0 18  0  0
   0   43 45.32 1492 66.05  77  0 22  0  1
   0   44 44.87 1880 82.39  51  0 38  0 12
   0   44 38.37 2012 75.39  33  0 37  1 30
   0   44 38.55 1746 65.72  16  0 11  0 73
   0   44 36.62 871 31.16  10  0  5  0 86
   0   43 38.17 1532 57.09   6  0  8  0 87
   0   44 41.11 971 38.99   3  0  4  1 93
   0   43 43.18 1184 49.94   3  0  6  0 92
   0   44 39.53 1098 42.38   3  0  2  0 95
   0  129 41.63 1269 51.61   5  0  8  0 88
   0   44 38.40 1199 44.96   1  0  2  1 96
   0   44 39.12 1203 45.95   1  0 11  0 88
   0   44 40.75 1190 47.36   1  0  7  0 92
   0   44 38.73 1150 43.48   1  0  6  0 93
   0   44 41.78 1199 48.92   1  0  4  1 95
   0   44 36.51 1142 40.70   0  0  4  0 96
   0   44 42.21 1149 47.39   2  0  9  0 89
   0   44 39.41 1186 45.65   1  0  2  0 97
   0   44 42.38 1197 49.54   1  0  4  0 95
   0   44 37.67 1136 41.77   2  0  2  0 96
   0   44 40.85 1113 44.39   0  0  7  0 93
   0   44 40.05 1123 43.91   0  0  6  0 94
   0   44 40.94 1108 44.29   1  0  3  0 96
   0   44 28.76 1244 34.93   0  0  5  0 95
   0   44 18.92 837 15.45   0  0  4  0 96
   0   43 50.97 1019 50.71   1  0  6  1 92
   0   44 50.53 1194 58.92   4  0 10  0 87
   0   44 49.21 1132 54.38   2  0  9  0 89
   0   44 32.68 1187 37.89   1  0  6  1 92
   0  129 45.49 1056 46.93   0  0  6  0 94
   0   44 35.56 972 33.77   1  0  3  0 96
   0   43 43.87 890 38.13   1  0  5  0 94
   0   43 37.39 773 28.23   0  0  2  1 97
   0   43 43.83 842 36.02   0  0  4  0 96
   0   43 34.81 599 20.36   0  0  1  0 99

Same as above, but with the fsync() removed. Significantly improved, but 
still flushing many more pages to disk compared to the old 1.2 build.

      tty           mfid0             cpu
 tin tout  KB/t tps  MB/s  us ni sy in id
   0   44 16.00  12  0.19  60  0 34  0  7
   0   43 16.00 145  2.26  85  0 15  0  0
   0   43 16.00 253  3.96  63  0 36  0  1
   0   43 19.89 1207 23.45  26  0 70  0  4
   0   44 16.00 266  4.16  21  0 67  0 12
   0   43 16.00 273  4.26  23  0 67  0 10
   0   43 19.03 618 11.48  16  0 79  0  5
   0   43 19.72 876 16.87  10  0 83  0  8
   0   43 19.95 962 18.74   4  0 88  0  9
   0   43 20.07 806 15.80  10  0 78  0 12
   0   43 20.18 845 16.65   6  0 84  1 10
   0   43 20.27 820 16.23   6  0 86  0  7
   0   43 20.38 877 17.45   5  0 84  0 11
   0   43 20.45 895 17.86   4  0 81  0 15
   0   43 17.50 486  8.31   0  0 81  0 19
   0   43 21.03 848 17.40   2  0 15  0 83
   0   43 22.15 317  6.85   0  0  1  0 99

Haven't had free time to tinker with turning off some of the prodigious 
madvise()'s or anything else, but clearly on some platforms this is not 
working out so well.

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)