WPMU and a spiking apache server

[wjdennen]

Hi folks,

WPMU 2.9.1.1
Apache 2.2 (prefork)
PHP 5.2.12

We recently went live with a WPMU server. There are only ~20 blogs in it now, and admin usage is pretty low. There are many (thousands) of static files being served by this setup too.

In the two days since launch, we have had the server load spike incredibly high a few times. So much so that it's hard to log in.

The spikes appear to be somewhat random. CPU load will be low (less than 1) for a long time and then suddenly it'll spike. In other words, it's not a gradual climb, it's truly a spike.

We've been using WP Super Cache, which seems to work well. I disabled it for a while, just trying to eliminate potential causes of the spikes. Server performed fine, except that it eventually spiked again. So, I re-enabled WP Super Cache as it's not likely the cause.

Server memory does not seem to be an issue. The server is a 1.5GB RAM (3GB burstable) VPS running CentOS. (I've run on a similarly sized VPS for a few years, with no problems like this... however that one is running apache 1.3).

Apache processes are somewhat large... 12-24MB each.

It does not seem to be a mysql problem either. That is, my munin graphs do not show anything weird related to mysql.

There does seem to be an apache problem. The munin graphs show spikes in "apache volume" and "apache processes." MaxClients is set to 150.

The really weird thing is how random the spikes seem to be.

I've bumped up apache's logging. But, there hasn't been a recent spike since then, so I am still waiting.

Has anyone seen anything similar? Got any tips?

I'm willing to change the version of apache to 1.3, or use 2.2 with worker instead of prefork.

Thanks, much appreciated.

[Ryan]

What plugins are you running?

I've had similar problems due to a auto-posting plugin I used once. It kept searching for posts all at the same time, then maxed out the memory limit of the WP admin on loading as it was trying to process too many brand new posts all at the same time.

[wjdennen]

Thanks. I'm not using any autoposting plugin, but I am using various plugins on various blogs. I've written a few of my own, too, and I guess I'd suspect them first! I'll take a look at disabling some.

With WP Super Cache, though, my guess is that plugins would have less of an impact on performance?

[Ryan]

With WP Super Cache, though, my guess is that plugins would have less of an impact on performance?

Less, yes. But my auto-posting plugin problem was on a site running WP Super Cache. The problem wasn't improved by using a caching plugin since the site still needed to go hunting for the posts from various RSS feeds across the web.

My client needed a way to parse RSS feeds as posts. So logically I turned to the scum of the internet for the best way to do this and found the most popular splogging tool to Splogs generally just automate everything, whereas my client needed to manually modify the posts, add credits, remove advertising and various other tasks, plus they didn't want all of the posts showing up, hence we put them into a moderation queue. The problem being, that we were using it with post moderation. Every post was placed into moderation, which at a few thousand posts, sent the WP admin into overload whenever it was loaded. It didn't just kill the admin panel, but killed the front-end too. I'm guessing it was probably slowing things down in spikes too, we just didn't notice until it got to the point that WP ran out of memory :( Bumping up the WP memory limit fixed it temporarily thankfully, but that took at least 15 mins to figure out how to fix it, and then an hour or so to figure out why it happened. Clearly this isn't your exact problem here, I'm just mentioning it in case you potentially have a similar problem going on.

[andrea_r]

I was just going to suggest looking at any feed plugins. They spike servers like crazy.

What other plugins do you have running? Cuz we're basically in guess-and-eliminate round here.

edit: remember, your caching usually does not apply for logged-in users.

[wjdennen]

Thanks!

I'll eliminate some plugins and will get back. It happened again overnight, when noone was logged into the admin, which is odd. Generally, the number of logged in users is always low -- we have only a dozen users total. This is not a "public sign-me-up" wpmu instance!

When it happens, apache processes spike, but mysql does not seem to.

Very frustrating. :( Thanks for your help.

[wjdennen]

At the risk of changing the subject, i noticed that wp-cron.php does not seem to be being called. That is, it does not show up in any of my server logs.

After googling it, I found that I should check /etc/hosts -- but this looks good:

127.0.0.1 localhost

from: http://yoast.com/wp-cron-issues/

I don't mean to suggest that this is related to the spiking problems. But, has anyone seen this before? I can't find any requests for wp-cron.php anywhere in my logs.

(Also, I noticed that old, expired cache WP Super Cache files were hanging around... which made me look to see if wp-cron.php was being called... )

[andrea_r]

I've seen the odd plugin just sort of build and build till the server tanks too. And I've seen servers tank with just 20 users on the site. Yep, crazy plugin there too.