[rrd-developers] Use of madvise / msync kills performance for me

[Marcus Reid]

On Wed, Jul 02, 2008 at 12:53:34PM +0200, Bernhard Fischer wrote:
> On Wed, Jul 02, 2008 at 03:16:19AM -0700, Marcus Reid wrote:
> >On Mon, Jun 30, 2008 at 09:41:13AM +0200, Bernhard Fischer wrote:
> >> On Fri, Jun 27, 2008 at 01:41:59PM -0700, Marcus Reid wrote:
> >> >On Fri, Jun 27, 2008 at 01:13:17PM +0200, Bernhard Fischer wrote:
> >> >> On Fri, Jun 27, 2008 at 02:58:15AM -0700, Marcus Reid wrote:
> >> >> >
> >> >> >Matt Dillon provided some good information on this subject that I would
> >> >> >like to pass on.  First, a little background..  I discovered that the
> >> >> >long slow msync() calls only happen on files over a certain size.  The
> >> >> >file that's slowing things down is 1161mB long, and msync() calls to a
> >> >> >file that's 940mB long are fast.  That's probably a kernel problem that
> >> >> >could be worth looking into.
> >> >> 
> >> >> eh, that's quite big, an order of magnitudes bigger than my files, at
> >> >> least.
> >> >> 
> >> >> >
> >> >> >This is probably an edge case -- I'm updating an rrd file that's over
> >> >> >a gig in size and I don't know how common that is.
> >> >
> >> >Well, this is starting to feel like beating on a dead horse, but
> >> >I've been doing some more testing and msync() makes things really
> >> >slow for smaller files as well.
> >> >
> >> >I think the main difference between the platforms is that under
> >> >FreeBSD, msync() even with MS_ASYNC doesn't return until the data
> >> >is on the physical disk, whereas with Linux it's just scheduled to
> >> >make it there at some point.  I'm not sure that this behavior will
> >> >change due to the OpenGroup spec that Matt mentioned, but I will
> >> >bring the issue up again.
> >> 
> >> SUS requires that an MS_ASYNC and MS_SYNC is available. If your ASYNC
> >> impl is broken (i.e. behaves as if it was SYNC) then you should really
> >> fix your VM :)
> >> 
> >> Of course we (userspace) can avoid calling msync on __FreeBSD__, but
> >> that's not exactly the proper thing to do, imo.
> >
> >Hi Again,
> >
> >Just writing to say that I agree with you, and that I understand that
> >avoiding cluttering up your codebase with a bunch of compatibility
> >stuff is a good thing.
> 
> Well from my POV it is not really about "cluttering" up rrdtool's source
> but more to avoid potential trouble on FreeBSD due to the missing msync.
> >
> >> >I'm just going to file a PR and try and get the msync() call
> >> >patched out in the ports collection.
> >
> >The patch that I submitted that takes out the msync() call got
> >accepted, and things are good for the time being.  I haven't made
> >another issue of the root of the problem yet -- though it does seem
> >to me that a big difference in msync() behavior could impact a lot
> >of things.
> 
> Indeed. You (as in somebody of the FreeBSD folks who is familiar with
> the VM) should audit and benchmark MS_SYNC vs. MS_ASYNC msync()s, as
> already stated above.
> >
> >As an aside, though I have been using FreeBSD everywhere I can since
> >2001 or so, I took offense when you referred to it as "your VM."  It's
> >easy to polarise peoples opinions when you define a common enemy, but
> >I think we're all working toward the same thing.
> 
> ouch. I didn't mean to sound offensive, please accept my appologies.
> I was merely refering to the FreeBSD VM as a VM that i'm not really
> familiar with (i glanced over it some 7 or 8 years ago) but that you use
> and may know way better than i do :)

Hi Bernhard,

Thanks for your quick response.  Apparently I'm far too easily offended :)

I'm in the middle of composing an email to freebsd-stable that brings
up the msync() issue.  Hopefully there will be a clean and simple
solution.  If there is some closure on the issue I will be sure to let
you guys know.

Sincerely,

Marcus