Digg, The Server Killer And How To Rescue Wordpress From It

I love Wordpress and I’ll continue to recommend it to anybody over another blog platform. On top of a great product, you also have a HUGE library of available plug-ins available which allow you to extend Wordpress into almost any situation. That said, that same extensibility also beefs up the code-base of Wordpress. This leads to server load.

I have talked about high server load with Wordpress before. I got the problem under control at the time, but it had gotten progressively worse over time. PCMECH had been dealing with some serious server load issues. With enough incoming traffic, the server would literally use up all it’s swap file space. When that happens, Apache literally just gives up and restarts. While that happens, the site is inaccessible.

Last week, to compound this problem, one of our stories on PCMECH made it onto the homepage of Digg. As expected, we got a HUGE influx of traffic.

The server could not keep up. Apache was dying and restarting every couple minutes – literally. For this reason, the incoming traffic logs did show an increase in traffic, but clearly not the huge effect we would usually see from the Digg homepage. All this because it was literally KILLING the server.

Short Term “Fix”

I clearly couldn’t have PCMech offline for several hours, and that’s what was happening. So, the first thing I did was create a static version of the post which was featured on Digg. The URL to the article is:

http://www.pcmech.com/article/ubuntu-810-coming-soon/

So, I could create a static version by taking the HTML output of that article, saving it as “index.htm”, and placing it in the “/article/ubuntu-810-coming-soon/” directory. Obviously, that directory does not exist, so I created it.

The server was croaking so badly because of Digg that I couldn’t even access the article. So, I went to Google, searched for the article, and pulled the output out of the Google Cache. I took the HTML from the cache, saved it, and placed it on my server.

This action did help with my server load, but the incoming traffic was still so high that my server was still croaking. Not as badly, but to the point that it made surfing PCMech a very frustrating experience.

The Long Term Fix

Ironically, at the time this happened, I was already in the process of investigating caching for Wordpress. I hadn’t implemented it, however, because I was looking for a way to maintain dynamic elements on page.

We have a premium membership program on PCMech and, when a paid member is logged in, it welcomes them by name, grants access to premium articles, and hides all the advertising. My concern was that the cache would break this.

Come to find out, on real-world trials, that WP-Cache handles it quite well. Apparently, it detects the user’s cookie and caches uniquely for them. All anonymous users would see the same cached copy of the page. This setup meant that WP-Cache could operate, yet when a member logs in, they still see their own, proper version of the site.

On discovering this, I immediately got the WP-Cache up and running and the results were what I was looking for. Server load dropped off considerably and the speed of page loading has increased dramatically.

Lessons Learned

Due to the unfortunate timing of this Digg bomb, I was in the unique position of being a blogger who actually wanted to get OFF the Digg homepage. Anybody else would have been cheering.

Here is what I learned out of this experience:

1. Don’t sit on a problem like this. I was gambling that the server would hold up while I investigated options, and I was wrong.
2. WP-Cache WILL properly handle unique page arrangements on a per-user basis. The plug-in is a lot smarter than simply a single static cache file which everybody sees.
3. DO install WP-Cache or WP-Super-Cache to your Wordpress installation – even if you’re not seeing that much traffic yet. It is easier to get set up before a problem occurs. If I can tap out a dedicated server, you could certainly piss off your web host if you’re on a shared plan and get a post into Digg.

You Might Also Like:

1. Wordpress Borks My Server? Dealing with High Server Load
2. Wordpress and Curing High Server Load
3. The Digg Effect
4. Digg Effect - In Living Color
5. Yahoo Slurp Spider Drives Forum Server Load Through the Roof!

If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.