[rrd-developers] striping - initial observations

[Daniel.Pocock at barclayscapital.com]

I've configured a stripe file, each stripe is 32 bytes, and there are
32768 stripes.

The data section of the file is aligned with the filesystem block size.
The RRA pointers are aligned when the RRDs are created.

The data file on disk is accessed with mmap().  memcpy() is used to move
data on and off the stripes.

Approximately 26000 RRDs, 2MB each, updated every 10 seconds by Ganglia.

rrdcached is also in use:
  /usr/bin/rrdcached -w 300 -z 300 -p /var/run/rrdcached/rrdcached.pid
-l /var/run/rrdcached/rrdcached.sock

To trick rrdcached and gmetad into believing that real RRD files are
created (until the API is extended to handle these virtual RRDs), I
create a dummy RRD with size 0 in the UNIX filesystem each time
rrd_create is called.  All access to the RRD file is done through the
striping code.

Here are my initial observations (see iostat output below):

- tcpdump reveals that Ganglia is querying the hosts at the correct
interval

- IO levels are reduced dramatically - from over 5000 IO/s to an average
of 615 IO/s over a 15 minute sample period

- IO block sizes increase from the minimum size - instead of sending
many small writes to SAN, the writes appear to be merged, due to the
fact that the RRAs are aligned and each disk block contains stripes
relating to the same time interval (avgrq-sz = 14.94 instead of 8.0
before striping)

- Using striping without rrdcached, the average IO request size becomes
even bigger: from 8 sectors up to about 50 sectors.  I'm investigating
why rrdcached reduces this figure.

This is still very much a work in progress, but the results seem to
suggest that IO bottlenecks are much less likely to occr when rrdcached
and striping are combined.

Regards,

Daniel


iostat -k 900 -x
----------------

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.38    0.00    2.22    3.32    0.00   89.08

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
[...]
sda               0.00   545.47 13.54 615.54    54.16  4644.13    14.94
10.30   16.38   0.24  14.80
sdb               0.00   543.99 15.04 615.47    60.16  4637.97    14.90
10.16   16.11   0.25  15.62
dm-5              0.00     0.00 28.58 2320.53   114.32  9282.10     8.00
38.88   16.55   0.11  26.07
dm-6              0.00     0.00 28.58 2320.53   114.32  9282.10     8.00
38.88   16.55   0.11  26.07

_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________