[smokeping-users] Smokeping FastCGI Woes

[Simon Liang]

Hey,

They're actually on two separate blade servers. Smokeping poller is on a
server by itself, and the web is also on a server by itself. There is
absolutely nothing on them except Smokeping.

Could it be that over a period of time, smokeping_cgi is run in which it's
try to rebuild everything again?

I used to run everything all on one server, so when the config gets changed
I see the server lag a bit (assuming it's preparing everything again) and
then will come back. However this is shouldn't be the case, as the config
is only rsync'd to the web server when a restart on the poller is executed.

Cheers,
Simon


On Sat, Oct 19, 2013 at 12:55 AM, <ITheodoridis at bankofgreece.gr> wrote:

> Is there anything else running on the same physical server? I mean
> except those two VMs you are talking about. Could it be there is a
> resource issue there? Have you tried giving it another core and see how
> it goes with performance? I have had similar problems with VMs running
> classic MRTG (without RRD) and things got a lot better when I added
> another core on those VMs.
> I 'm definitely interested to see what answer you will get on the
> subject in the end.
>
> -----Original Message-----
> From: Simon Liang [mailto:simonjai at gmail.com]
> Sent: Friday, October 18, 2013 2:04 AM
> To: smokeping-users at lists.oetiker.ch
> Subject: [smokeping-users] Smokeping FastCGI Woes
>
> Hi there,
>
> I've been running an early 2.6 version of Smokeping and recently decided
> to upgrade to the latest 2.6.9. The polling side of Smokeping is rock
> solid, however it's quite the opposite with the change to FastCGI. I
> thought this would be a good place to start and hope someone can help
> me.
>
> Just some background of my current setup. I have separated the Polling
> (forked with FPing) and web service side of Smokeping into two VMs (to
> reduce the load the server). The config files are rsynced from the
> poller to the web and the RRD files are shared via NFS.
>
> Poller VM - 8 vCPU, 16GB RAM
> Web VM - 4 vCPU, 16GB RAM
>
> I understand when the web side of Smokeping starts (after a config
> change), it does a lot of things under the hood. To address this issue,
> on restarts I create an IP table rule on the web server so only the
> poller can access it via port 80, I then use curl to trigger the first
> (and only) session. Once it's completed I remove the IP table rule so it
> becomes accessible. This works quite work and everything works fine.
>
> But after a random period of time, the load on the web server spikes up
> quite high, memory starts swapping and the server becomes almost
> unresponsive.
>
> This is a screenshot of the web server under load (before I upgraded to
> 16GB RAM)
> https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_load.JPG
>
> The spikes in the graph are when the load on the server just randomly
> spikes up and I'm forced to restart Smokeping manually.
> https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_stats.JPG
>
> I can assure you there are no cron jobs running which may be loading up
> the server. On peak hour traffic I have maybe 100 requests per minute.
>
> Here is my Apache (fcgid) config: http://pastebin.com/QU6XRcFg
>
> If you need any more information please let me know.
>
> Cheers,
> Simon
>
>
>
> ==============================================================================================================
> Με την επιφύλαξη παντός νομίμου δικαιώματός μας. Κάθε μήνυμα ηλεκτρονικού
> ταχυδρομείου από την Τράπεζα της Ελλάδος (ΤτΕ) αποστέλλεται καλόπιστα αλλά
> δεν τη δεσμεύει ούτε ερμηνεύεται ως να συνιστούσε ή επηρέαζε συμβατική ή
> άλλη δέσμευση της ΤτΕ.
>
> Το μήνυμα ηλεκτρονικού ταχυδρομείου προορίζεται προς αποκλειστική χρήση
> του προσώπου, του οποίου η διεύθυνση αναγράφεται στην επικεφαλίδα του
> μηνύματος. Ο αποστολέας και η ΤτΕ δεν αναλαμβάνουν καμμία ευθύνη για
> ανακριβειες, παραβίαση της ακεραιότητας, απώλεια ή καθυστερημένη διαβίβαση
> του μηνύματος, για αστοχία, διακοπή ή υποβάθμιση της υπηρεσίας ή του
> μηνύματος καθώς και για κάθε εξ αυτού του λόγου απώλεια ή ζημία καθόλη την
> υπό του νόμου προβλεπόμενη έκταση.
>
> Εάν λάβατε κατά λάθος το παρόν μήνυμα ηλεκτρονικού ταχυδρομείου,
> παρακαλούμε να ειδοποιήσετε αμέσως μέσω ηλεκτρονικού ταχυδρομείου τον
> αποστολέα και να διαγράψετε το μήνυμα. Οποιαδήποτε ανακοίνωση, διάδοση ή
> χρήση μέρους ή ολόκληρου του μηνύματος χωρίς άδεια απαγορεύεται αυστηρά και
> μπορεί να επιφέρει ποινική και αστική ευθύνη.
>
>
> Any e-mail message from the Bank of Greece (BoG) is sent in good faith but
> shall neither be binding nor construed as constituting or affecting a
> contractual arrangement or other commitment by the BoG.
>
> The e-mail is intended for the exclusive use of the person whose e-mail
> address appears in caption as recipient. The sender and the BoG decline
> liability for inaccuracy, breach of integrity, loss or delayed delivery of
> the message, for any failure in, interruption to or degradation of either
> the service or the message, as well as for any loss or damage sustained
> thereof to the fullest extent provided by law.
>
> If this e-mail was not intended for you, please notify the sender
> immediately via e-mail and delete it at once. Any unauthorized disclosure,
> dissemination or use, either in whole or in part is strictly prohibited and
> may give rise to both criminal and civil liability. All rights reserved.
>
> ==============================================================================================================
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/smokeping-users/attachments/20131021/88ae8f3e/attachment.htm