[rrd-developers] Use of madvise / msync kills performance for me

[Bernhard Fischer]

On Wed, Jul 02, 2008 at 03:16:19AM -0700, Marcus Reid wrote:
>On Mon, Jun 30, 2008 at 09:41:13AM +0200, Bernhard Fischer wrote:
>> On Fri, Jun 27, 2008 at 01:41:59PM -0700, Marcus Reid wrote:
>> >On Fri, Jun 27, 2008 at 01:13:17PM +0200, Bernhard Fischer wrote:
>> >> On Fri, Jun 27, 2008 at 02:58:15AM -0700, Marcus Reid wrote:
>> >> >
>> >> >Matt Dillon provided some good information on this subject that I would
>> >> >like to pass on.  First, a little background..  I discovered that the
>> >> >long slow msync() calls only happen on files over a certain size.  The
>> >> >file that's slowing things down is 1161mB long, and msync() calls to a
>> >> >file that's 940mB long are fast.  That's probably a kernel problem that
>> >> >could be worth looking into.
>> >> 
>> >> eh, that's quite big, an order of magnitudes bigger than my files, at
>> >> least.
>> >> 
>> >> >
>> >> >This is probably an edge case -- I'm updating an rrd file that's over
>> >> >a gig in size and I don't know how common that is.
>> >
>> >Well, this is starting to feel like beating on a dead horse, but
>> >I've been doing some more testing and msync() makes things really
>> >slow for smaller files as well.
>> >
>> >I think the main difference between the platforms is that under
>> >FreeBSD, msync() even with MS_ASYNC doesn't return until the data
>> >is on the physical disk, whereas with Linux it's just scheduled to
>> >make it there at some point.  I'm not sure that this behavior will
>> >change due to the OpenGroup spec that Matt mentioned, but I will
>> >bring the issue up again.
>> 
>> SUS requires that an MS_ASYNC and MS_SYNC is available. If your ASYNC
>> impl is broken (i.e. behaves as if it was SYNC) then you should really
>> fix your VM :)
>> 
>> Of course we (userspace) can avoid calling msync on __FreeBSD__, but
>> that's not exactly the proper thing to do, imo.
>
>Hi Again,
>
>Just writing to say that I agree with you, and that I understand that
>avoiding cluttering up your codebase with a bunch of compatibility
>stuff is a good thing.

Well from my POV it is not really about "cluttering" up rrdtool's source
but more to avoid potential trouble on FreeBSD due to the missing msync.
>
>> >I'm just going to file a PR and try and get the msync() call
>> >patched out in the ports collection.
>
>The patch that I submitted that takes out the msync() call got
>accepted, and things are good for the time being.  I haven't made
>another issue of the root of the problem yet -- though it does seem
>to me that a big difference in msync() behavior could impact a lot
>of things.

Indeed. You (as in somebody of the FreeBSD folks who is familiar with
the VM) should audit and benchmark MS_SYNC vs. MS_ASYNC msync()s, as
already stated above.
>
>As an aside, though I have been using FreeBSD everywhere I can since
>2001 or so, I took offense when you referred to it as "your VM."  It's
>easy to polarise peoples opinions when you define a common enemy, but
>I think we're all working toward the same thing.

ouch. I didn't mean to sound offensive, please accept my appologies.
I was merely refering to the FreeBSD VM as a VM that i'm not really
familiar with (i glanced over it some 7 or 8 years ago) but that you use
and may know way better than i do :)
>
>Thanks for your time,

friendly,
Bernhard