Slashdot Effect on Holiday Lights

For those that have not heard of the Slashdot Effect, it occurs when your web site is mentioned on the front page of a story on which is very popular since it is the "news for nerds" site. Because thousands of web surfers will click on your site within an hour of it being posted, it really stresses your web server (and network connections) and often results in your web site becoming unresponsive ... or a "smoldering pile of rubble" due due to the slashdot effect. Note that any heavily trafficked site can generate lotsa inbound traffic to your web server, so technically the "slashdot effect" is a generic term referring to this phenomena of a large increase in "flash" traffic, often times resulting in a (basically) Distributed Denial of Service.

The Slashdot thundering herd has made several "visits" to my web site - in 2002 & 2003 for my christmas lights and christmas webcam and then in 2004 for my halloween decorations and halloween webcam. My ISP popped a 40-amp circuit breaker in 2002 (really did happen!), but you can read my detailed writeups of the:

While numerous folks have written about the Slashdot Effect, this was a little different in that not only was it a test of "digital" stuff like the web server, ISP bandwidth, Perl/CGI code, but also "analog" stuff as the code has interfaces to various sensors and the webcam itself, plus you are turning a lotta lights ON & OFF - it certainly provided one heck of a light show for the neighbors! ;-)

Christmas/2004 Update: Now using mod_perl ... shoulda done this a long time ago - ApacheBench testing shows CGI is now capable of 20 requests/second versus 4 using cgi_exec ... so this rocks ... and even uncovered a coding boo-boo where I forgot to close a lock file in a certain case - but I'm sure there are few others that will pop-up in "stress" testing. And once I disabled KeepAlive in httpd.conf, the 2.4 GHz XEON (with a Gbyte of RAM) held up pretty darn decently to not one, but two waves of (inter)national media attention as the christmas lights webcam hoax was revealed.

Halloween/2005 Update: I ante'd up the big bucks for a 2nd server to handle the images. I also finally switched over to Apache2 and have most everything running under mod_perl which just ROCKS. So while the web site has always been pretty responsive, I'm now really well prepared for any possible heavy traffic. And with KeepAlive OFF in httpd.conf, I bet these two 3.2 GHz Pentiums with a GByte of RAM running Linux/Apache (connected at 100 Mbit/sec) will even be able to handle the "slashdot effect" - it will be interesting if the /. crowd decides to pay a visit ... ;-)

Christmas/2005 Update: In 2002, there was a single 1 Ghz Pentium box with 128 Mbytes of RAM on a T-1 connection. For 2005, there are FIVE web servers handling the load. They all run Linux/Apache (mod_perl rocks!) and are 3 GHz or so Pentium boxes with a Gbyte of memory - all 100Mbps connected. I.e. there is half a Gigabit of bandwidth available - BRING IT ON SLASHDOT! They chicken'ed out. Two servers load balance the christmas movies since those are huge. The third server handles only ChristmasCam #1 and #2. The fourth server handles ChristmasCam #3 and a few misc. images like those buttons. And the fifth server handles the rest of I.e. I could add one more server to fully load balance this real-time application, but that would be it.