[smokeping-users] Smokeping FastCGI Woes

[Simon Liang]

Hey,

I have discovered that during this high load period, the web server is
actually doing all the rebuild and preparation that it does when
Smokeping is first started.

The poller was not started, so what actually causes this? Does
Smokeping do some kind of auto refresh?

Cheers,
Simon

On Mon, Oct 21, 2013 at 4:01 PM, Simon Liang <simonjai at gmail.com> wrote:
> Hey,
>
> They're actually on two separate blade servers. Smokeping poller is on a
> server by itself, and the web is also on a server by itself. There is
> absolutely nothing on them except Smokeping.
>
> Could it be that over a period of time, smokeping_cgi is run in which it's
> try to rebuild everything again?
>
> I used to run everything all on one server, so when the config gets changed
> I see the server lag a bit (assuming it's preparing everything again) and
> then will come back. However this is shouldn't be the case, as the config is
> only rsync'd to the web server when a restart on the poller is executed.
>
> Cheers,
> Simon
>
>
> On Sat, Oct 19, 2013 at 12:55 AM, <ITheodoridis at bankofgreece.gr> wrote:
>>
>> Is there anything else running on the same physical server? I mean
>> except those two VMs you are talking about. Could it be there is a
>> resource issue there? Have you tried giving it another core and see how
>> it goes with performance? I have had similar problems with VMs running
>> classic MRTG (without RRD) and things got a lot better when I added
>> another core on those VMs.
>> I 'm definitely interested to see what answer you will get on the
>> subject in the end.
>>
>> -----Original Message-----
>> From: Simon Liang [mailto:simonjai at gmail.com]
>> Sent: Friday, October 18, 2013 2:04 AM
>> To: smokeping-users at lists.oetiker.ch
>> Subject: [smokeping-users] Smokeping FastCGI Woes
>>
>> Hi there,
>>
>> I've been running an early 2.6 version of Smokeping and recently decided
>> to upgrade to the latest 2.6.9. The polling side of Smokeping is rock
>> solid, however it's quite the opposite with the change to FastCGI. I
>> thought this would be a good place to start and hope someone can help
>> me.
>>
>> Just some background of my current setup. I have separated the Polling
>> (forked with FPing) and web service side of Smokeping into two VMs (to
>> reduce the load the server). The config files are rsynced from the
>> poller to the web and the RRD files are shared via NFS.
>>
>> Poller VM - 8 vCPU, 16GB RAM
>> Web VM - 4 vCPU, 16GB RAM
>>
>> I understand when the web side of Smokeping starts (after a config
>> change), it does a lot of things under the hood. To address this issue,
>> on restarts I create an IP table rule on the web server so only the
>> poller can access it via port 80, I then use curl to trigger the first
>> (and only) session. Once it's completed I remove the IP table rule so it
>> becomes accessible. This works quite work and everything works fine.
>>
>> But after a random period of time, the load on the web server spikes up
>> quite high, memory starts swapping and the server becomes almost
>> unresponsive.
>>
>> This is a screenshot of the web server under load (before I upgraded to
>> 16GB RAM)
>> https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_load.JPG
>>
>> The spikes in the graph are when the load on the server just randomly
>> spikes up and I'm forced to restart Smokeping manually.
>> https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_stats.JPG
>>
>> I can assure you there are no cron jobs running which may be loading up
>> the server. On peak hour traffic I have maybe 100 requests per minute.
>>
>> Here is my Apache (fcgid) config: http://pastebin.com/QU6XRcFg
>>
>> If you need any more information please let me know.
>>
>> Cheers,
>> Simon
>>
>>
>>
>> ==============================================================================================================
>> Με την επιφύλαξη παντός νομίμου δικαιώματός μας. Κάθε μήνυμα ηλεκτρονικού
>> ταχυδρομείου από την Τράπεζα της Ελλάδος (ΤτΕ) αποστέλλεται καλόπιστα αλλά
>> δεν τη δεσμεύει ούτε ερμηνεύεται ως να συνιστούσε ή επηρέαζε συμβατική ή
>> άλλη δέσμευση της ΤτΕ.
>>
>> Το μήνυμα ηλεκτρονικού ταχυδρομείου προορίζεται προς αποκλειστική χρήση
>> του προσώπου, του οποίου η διεύθυνση αναγράφεται στην επικεφαλίδα του
>> μηνύματος. Ο αποστολέας και η ΤτΕ δεν αναλαμβάνουν καμμία ευθύνη για
>> ανακριβειες, παραβίαση της ακεραιότητας, απώλεια ή καθυστερημένη διαβίβαση
>> του μηνύματος, για αστοχία, διακοπή ή υποβάθμιση της υπηρεσίας ή του
>> μηνύματος καθώς και για κάθε εξ αυτού του λόγου απώλεια ή ζημία καθόλη την
>> υπό του νόμου προβλεπόμενη έκταση.
>>
>> Εάν λάβατε κατά λάθος το παρόν μήνυμα ηλεκτρονικού ταχυδρομείου,
>> παρακαλούμε να ειδοποιήσετε αμέσως μέσω ηλεκτρονικού ταχυδρομείου τον
>> αποστολέα και να διαγράψετε το μήνυμα. Οποιαδήποτε ανακοίνωση, διάδοση ή
>> χρήση μέρους ή ολόκληρου του μηνύματος χωρίς άδεια απαγορεύεται αυστηρά και
>> μπορεί να επιφέρει ποινική και αστική ευθύνη.
>>
>>
>> Any e-mail message from the Bank of Greece (BoG) is sent in good faith but
>> shall neither be binding nor construed as constituting or affecting a
>> contractual arrangement or other commitment by the BoG.
>>
>> The e-mail is intended for the exclusive use of the person whose e-mail
>> address appears in caption as recipient. The sender and the BoG decline
>> liability for inaccuracy, breach of integrity, loss or delayed delivery of
>> the message, for any failure in, interruption to or degradation of either
>> the service or the message, as well as for any loss or damage sustained
>> thereof to the fullest extent provided by law.
>>
>> If this e-mail was not intended for you, please notify the sender
>> immediately via e-mail and delete it at once. Any unauthorized disclosure,
>> dissemination or use, either in whole or in part is strictly prohibited and
>> may give rise to both criminal and civil liability. All rights reserved.
>>
>> ==============================================================================================================
>>
>>
>>
>