"Slashdot Effect" on 2005 Christmas Lights Webcam
Shortcuts to stories and
stats table and lessons learned.
Intro and Technical Notes
Once again, there are christmas lights at Komar.Org - what a surprise!
There was a ton of traffic in 2004 so I figured I'd do a similar inbound traffic
analysis for 2005 since it's fairly easy to do.
The most noteable difference this year is that unlike the previous 3 years when
it was a simulation, it's a 100% real for 2005.
3 webcams (3 more than previous years!) viewing 26,000 Christmas lights,
(and 3 inflatables - Santa, Frosty, and ... Frankenstein)
all controllable via the Internet using X10 technology ... and
for Celiac Disease Research ... an auto-immune disease that my kids have.
For those that don't know, Slashdot is the incredibly popular
"News for Nerds" site and can bring a thundering herd of
surfers to your site, all in a very short amount of time.
I've been on Slashdot 5 times for my holidays lights so I've had
a little bit of experience with this. The term can
generically apply to other sites that generate a lotta traffic which can
result in a DDoS - Distributed Denial of Service. Slashdot paid me a visit
in 2002, 2003, and 2004 (3 times), but not in 2005 - ironic since it was
for real this year..
While numerous folks have written about the Slashdot Effect, this is a little
different in that not only is it a test of "digital" stuff like the
web server, ISP bandwidth, Perl/CGI code, but also "analog" stuff as the
the 3 ChristmasCams are FTP'ing up images and on/off commands are being
sent to the X10 controls at the house. And oh yeah, you are
turning a lotta lights ON & OFF - it certainly provided one heck of a
light show for the neighbors! ;-)
The webcam page is about 50 KBytes of HTML/CSS that is very
infrequently refreshed. There is the infamous
Komar-Santa Cursor that is 12 KBytes along with a couple of
of the images meta-refreshes every 60 second and is
about 15KBytes along with 5 GIF buttons that total less
than another KByte. When you are viewing ALL of the webcams,
each of the 352x240 small images is 10-15 KBytes ... they are actually
resized on the fly (using the GD library) from the incoming 704x480 JPEG's
that the webcams FTP over and are about 40-50 Kbytes. The images
are meta-refreshed every 5 seconds.
So while there is a lot going on, the file sizes are relative small,
so raw bandwidth is probably not the limiting factor. My guess is
either I'd run out of RAM or hit an Apache connection limit on
the Linux box - neither has come close yet, so I don't know and I
have not done benchmarking in this area.
In 2002, there was a single 1 Ghz Pentium box with 128 Mbytes of RAM
on a T-1 connection.
For 2005, there are FIVE web servers handling the load. They all
run Linux/Apache (mod_perl rocks!) and are 3 GHz or so Pentium boxes
with a Gbyte of memory - all 100Mbps connected.
I.e. there is half a Gigabit of bandwidth available - BRING
IT ON SLASHDOT! As noted above, Slashdot didn't show up -
maybe there were afraid?!? ;-)
Two servers load balance
the christmas movies since those are huge.
The third server handles only ChristmasCam #1 and #2. The fourth
server handles ChristmasCam #3 and a few misc. images like those buttons.
And the fifth server handles the rest of komar.org. I.e. I could add
one more server to fully load balance this real-time application,
but that would be it.
Special thanx to Dominic at Vidahost who
graciousely volunteered to help out. Dominic is a director of
Thermal Degree, a UK-based web
design and hosting company setup by two friends in early 2004.
Vidahost, launched just over one year
ago, is their hosting branch which offers shared and reseller hosting.
Dominic also handled some mirroring of
a collection of video footage from the tsunami of December 2004 which attracted thousands of visitors and nearly 1Gbit/sec of traffic.
And he is also a member of the World Wind community at
Misc. Sites that mention the Christmas Webcam
Needless to say, the christmas lights webcam is talked about on
countless blogs and web sites ... but I picked a few noteable/interesting
one to show here.
For the first 15 days of December and there
were 18,255,169 hits on on Christmas related stuff at komar.org
scattered among the 5 servers supporting this effort;
i.e. it's averaging a little over a million hits/day.
- Philipp Lenssen's excellent Google Blogoscoped was an early site to mention
the christmas lights webcam by posting at 1033 on Monday, November 28th.
Philipp's site didn't send a large quantity of surfers ... but his site is
very high quality and one I've always enjoyed reading it. He does some
great analytics, so I thought he'd appreciate seeing his numbers.
- DIGG popped it on their front page at Midnight on
the morning of Friday, December 2nd. This is the so-called "next Slashdot"
and needless to say generated a ton of traffic. I was actually just about to
go to sleep when BAMMMM ... the web server was hammered. The 26,000 christmas
lights were actually off (usually 10:00PM) ... but I figured what the heck,
so I turned 'em back on and left 'em on/controllable until 8:00AM.
The first inbound was at 1407 on Thursday, December 1st ... December 1st
and there were a 199 inbounds until 2358 (and 43 seconds) ... and which point
it hit the main page and a BIG uptick in traffic as can be seen in the numbers
below which start from that time. It stayed on the front page for over 12 hours.
- MajorGeeks.Com probably doesn't require much explanation. They added to their "Way Off Base" area
on their main page on Monday, December 5th at 0714. Nice guys and
great support forum - go check it out!
- The Washington Post story when online a little after midnight on Tuesday, December 6th. No surprise that not a lot of
inbounds until people woke up in the morning. Certainly many more people read
the story in print (ran on page D4) that is implied in the numbers below.
An interesting number would be how many times was the story read online at
washingtonpost.com, because then you could calcuate the percentage that
clicked through. I doubt they would give me that number.
- The Rocky Mountain News does a
definitive writeup and went online shortly after midnight on Saturday,
December 10th. Main article is very thorough with an accompanying
hilarious piece - funny reading - highly recommended. Again, no way
to track paper readership, but it ran on the front page. My guess is
that this link will continue to send in a trickle of people for a
few weeks since it is well-written and definitive ... "content is king"
and people read/link to good stuff.
- The ever-so-funny Fark.Com greenlights to the front page a writeup at 1631 (and 33 seconds) on Monday, December 12th. While 257 TotalFark'ers had swung by before
then from the subscriber only section, there's a HUGE uptick once it hits
the main Fark.com page.
- DIGG popped up again when a story about
the time-lapse video of the DIGG'ers turning the 26,000 christmas lights
(see above) hit the front page on Tuesday, December 20th at
0357 (and 39 seconds) and stayed there for about 10 hours.
Number of inbounds per site after being posted
The Apache Web Server generates copious logging data and one can use the
raw data to "track" traffic from sites that link to you based on what is
called the referrer (which spammers sometimes spoof for referrer log spamming) - I actually use
this to do a customized body alert="message" from certain sites, and many
folks appreciated that personal touch.
It's actually quite easy to see when these sites posted about the Christmas
stuff because there is a very noticeable uptick in traffic - all times are
Mountain Standard Time (GMT-7) and the numbers below are definately
LOW because some browsers block the referrer. In date/time order, we have:
|Site - Time Posted
||Site - Time Posted
|BlogScoped - 28/1033
||BlogScoped - 28/1033
|DIGG - 1/2358
||DIGG - 1/2358
|MajorGeeks - 5/0714
||MajorGeeks - 5/0714
|WashPost - 6/0045
||WashPost - 6/0045
|Rocky - 10/0152
||Rocky - 10/0152
|FARK - 12/1631
||FARK - 12/1631
|DIGG - 20/0357
||DIGG - 20/0357
|Site - Time Posted
||Site - Time Posted
Slashdot Effect Analysis - Misc. Interesting Tidbits/Ideas/Lessons Learned - SO FAR!
The 2005 optimization of the X10, Webcam, and mod_perl code is working well:
While the web servers aren't exactly monster machines, they are so far
handling the load well. For instance, despite 10,000 DIGG'ers the morning of
December 2nd, it never skipped a beat ... and mod_perl just rocks.
And I subsequently brought online that second ChristmasCam server.
Is DIGG the "next" Slashdot?
16,000+ inbounds in a day is nothing to sneeze at ... but at least
compared to last year, Slashdot (and Fark) was five times that.
For the coders out there - be sure to think about and handle race conditions - they DO happen!
If you have never dealt with race conditions, The Slashdot Effect
is certainly a good test of your code. A classic (but flawed)
approach to file locking is checking to see if a file exists, and if not,
creating it ... but that is not an atomic operation - i.e. it's two different
steps seperated by a small amount of time - so you better do it right!
BTW, I use Perl Cookbook Recipe 7.11 to handle file locking and incrementing
of a couple of counters - this has worked flawlessly ... although it did
burp in previous years when Slashdot hit.
Is Alek ready for Christmas/2005?
Wellllll ... you are never truly "ready" for Slashdot ... but the
application is fairly tight and spread over 3 web servers - you could
basically only add one more for load balancing this real-time application.
So that is a cumulative 9.6 Ghz of
computing power and 300 Mbps of bandwidth - holy mackeral! And all
of the video's are redirected to two other servers (aggregrate of
200 Mbps) ... so that a lotta bits and bytes!
I never got around to implementing Apache proxying which would
certainly help ... and my Perl code could certainly be better ...
but the error logs are clean.
Want more info and/or have a suggestion/idea for me?
The Christmas FAQ has some
more info as does my responses when folks
Email Santa. And if
those don't answer your questions and/or if you have an idea (or just
want to send me an atta-boy), then
send me an Email.
Go back to the main slashdot effect analysis page.