Slashdot Effect on 2004 Christmas Lights Webcam and Hoax Revealed

I once again put out my christmas lights in 2004 which seem to be fairly popular as numerous sites have picked up on. I actually think I might stand a chance against the Slashdot horde this time as I have converted to using mod_perl on the web server and have improved/optimized the Analog interfaces - see what's new for Christmas/2004 - Update: /. kicked my a** when the article first hit and I had to disable the webcam for about 90 minutes, but it was up after that! BTW, a /. reader recommended setting KeepAlive=no and this appears to make a BIG difference for me - the web server held up just fine to higher sustained load a few days later from the AP story that went National ... so thank you Slashdot! ;-)

For those that don't know, Slashdot is the incredibly popular "News for Nerds" site and can bring a thundering herd of surfers to your site, all in a very short amount of time. I've written about the slashdot effect before - I did writeups for Christmas 2002 & 2003 and Halloween 2004 ... but this can generically apply to other sites that generate a lotta traffic which can result in a DDoS - Distributed Denial of Service.

The Web Server (a 2.4 GHz Xeon with a GByte of RAM running pretty standard Linux with the Apache web server) generates copious logging data and one can use the raw data to "track" traffic from sites that link to you based on what is called the referrer (which spammers sometimes spoof for referrer log spamming) - I actually use this to do a customized body alert="message" from certain sites, so some folks were amazed I "knew" about it so soon! ;-) It's actually quite easy to see when these sites posted about the Christmas stuff because there is a very noticeable uptick in traffic - all times are Mountain Standard Time (GMT-7) and the numbers below are definately LOW because some browsers block the referrer. In date/time order, we have:

• MajorGeeks.Com is a web site that probably doesn't need much explanation! ;-) They put it on their main page under "Off Base Postings" at 2017 on Wednesday, December 1st, 2004 and there was also some discussion in their Forums. It rolled off a day or two later, but they periodically put it back up. Doesn't generate flash traffic, but it is a pretty decent steady stream, plus I appreciated having my site recognized by the MajorGeeks guys ... as did my wife who got quite the chuckle out of it and said it was quite fitting for me! ;-)
• USA TODAY listed it as one of their "Hot Sites" in their Web Guide. This was posted at 0806 on Thursday, December 2nd, 2004 and they provided a direct link with an explanation - they liked my "Hulkariffic Halloween Decorations" - their term - love it!. It appears this "rolled off" of the main "Hot Sites" page and into the December 2nd archives just before noon the next day. There is no opportunity for web surfers to comment on the posting that I'm aware of. I never heard if it made the Print Edition, but that would be hard to track (since no referrer), although I assume it would generate a TON of traffic - note that The Hulk enjoys reading USA Today under Christmas Lights! ;-)
• WeirdLinks.Com is a blog/news like site that probably doesn't need much explanation. Looks like they post a story or so a day, so it slowly moves down the page. They linked to my site at 0855 on Friday, December 3rd, 2004 with the comment: "Turn This Guy's Christmas Lights on and Off - Alek wants you to mess around with his Christmas lights from your computer. The image is of Alek's gaudy housey." HEY, "guady" is your opinion - I think they are ... well, maybe just a little bit guady! ;-)
• Honorable mention goes to Kim Komando which picked it her Kool Site on Saturday, December 4th, 2004 and notified her (claimed) 3.5 million Email subscribers and 8 million radio listeners. While there was no where close to that amount of traffic surge, she did send a noticeable uptick. The self-proclaimed America's Digital Goddess was NOT happy when the hoax broke and thinks it would have been easier to do for real; so we'll see what Kim's Christmas Lights look like in 2005! ;-)
• GorrilaMask.Net is a blog-like page (with links to mature content) that mentioned on their front page that you can "Control this guy's Christmas lights from your computer" along with a handful of other links. This was posted at 1333 on Monday, December 6th, 2004 and appears to be generating a stream of traffic. There is some sort of forum at MindLessBullSh*t.Com, but I didn't browse around there to see if things were discussed, although there was traffic from the main page there.
• Ernie's House of WhoopAss is blog-like page (with links to mature content) from (surprise!) Ernie and in an article posted at 1520 (and 01 seconds!) on Tuesday, December 7th, 2004 he writes "See The Hulk and other cool sh*t on Komar's controllable Christmas webcam!" There are "pay-for" forums, so I don't know if there was discussion about this at his site. Ernie posts a new story about once a day (or every other day) ... so this slowly gets pushed down to the bottom. While the traffic isn't as "flashy" as Slashdot or FARK, he must have one heck of a readership because web surfers just keep coming from his site to the Christmas Webcam - and this wasn't even a "boobies" link! ;-) As can be seen from the stats, traffic just keeps coming, so Ernie has some serious staying power!
• The Inquirer is a Britsh based news/review/etc. site. Wil Harris wrote a funny halloween article titled "Lights chap is one pumpkin short of a party" and he sub-titled the Christmas one "Bonkers house gets even bonkier." It hit the web at 0312 (and 23 seconds!) on Decemember 11th, 2004 which would be mid-morning in Jolly ol' England. They post 5-10 stories/day, so this slowly moves down the page and gets less interest. Their forum requires a login and I didn't look there, so I don't know if it was discussed.
• Heise Online is a German News/Review site (similar in style (and apparently readership!) as Slashdot) that posted an article about LED lights (here's an English translation) at 1032 (and 36 seconds!) on Saturday, December 11th, 2004. Ironically, the article was mostly about LED's, with the webcam link buried near the end, which makes me wonder what the traffic would have been like if the article had specifically called on the Christmas Lights Webcam. Similiar to the Inquirer, new stories are periodically posted, so this moves downward. Comments are allowed, but mostly seemed to be about the LED's. While this data is not presented/included below, another German site Stern.De has sent over 4,000 surfers my way starting at 0939 on Wednesday, December 15th, 2004.
• Slashdot is the "News for Nerds" sites that has short articles with links for more info. Their front page article was titled "Alek's Christmas Lights Webcam is Back" and had multiple links to various URL's. I suspect that many /. readers simply click on every link listed, so the inbound referral count is probably quite a bit higher than if a single link had been listed ... however, the data below is low because I had to restart the web server several time in an attempt to tune things for the traffic - i.e. so far, Slashdot is the only site to bring komar.org to its knees ... but it was never rebooted! ;-) Slashdot posted at 1749 (and 3 seconds!) on Sunday, December 12th, 2004. Slashdot posts a dozen or so stories to the main page, so this story slowly scrolled down. There was 150+ comments in the article (still being commented upon) including a couple from me. Note that "controversial" postings may generate more comments so this suggests that /. traffic can certainly be higher for those sites. I also had a running play-by-play. I thought Heise generated a fair amount of traffic, but as the numbers clearly show below, Slashdot is still the king of a DDoS ... although Fark isn't that far behind as noted below!
• AP/Associated Press/Traditional Media The local paper did a story on my Christmas lights/webcam which got sent out nationally! This one is kinda hard to track since (as typical of the traditional press), there aren't links in the article itself, but a sidebar with a point to my home page. So for lack of better methods, I'm going to pick a starting time when the first hit came in from news.yahoo.com at 1715 on Tuesday, December 14th, 2004 and simply count the numbers of hits on my main page from ALL sites ... since this story was replicated on dozens (maybe hundreds?) of various media sites ... although some of those sites were a bit more savvy and added a proper link to the webcam In addition, people came in through the search engines - a HUGE surge in searches for "Alek Komarnitsky" As expected, this is not quite as "flashy" (such as Slashdot), but builds as a surging tidal wave over time! I also show below the inbound traffic ONLY from news.yahoo.com.
• Aunty Spam's Net Patrol wrote a funny story about the Christmas Lights that hit at 0503 on Monday, December 13th, 2004 and generated a steady stream of traffic - always listen to your Aunty! ;-) They also joined the "Put Alek on National TV campaign" with a post the afternoon of Monday, December 20th.
• 9News is a local Denver TV Station. They did a story on the Christmas Lights for the 10:00 News on Tuesday, December 14th, 2004. At the end of the show, the anchor commented that if they put a link to my site at www.9news.com, that it would probably crash my server. Not even close (but bring it on baby! ;-) ... but I'll give 'em credit in that they have an decent readership as seen by the numbers below. And they did generate a bit of flash traffic (my guess is timed when the story runs) since while it first was deep-linked on their web site at the above URL at 1818, they put it in their "extra" section (and probably mentioned on the air) at 2217, so the data below is shown from there. It was just a reprint of the AP article and included a link to my home page instead of the Christmas Webcam. I did suggest to the Web Producer that they include a link to the webcam since then I could (based on the referrer) show 9News viwers a picture of their reporter stepping on my lights with her high heels ... but they declined! ;-)
• I got an Email that "Norway's largest paper" ran the story and just for grins, I took the top-3 .no domains and analyzed that traffic. Inbounds started at 1336 on Wednesday, December 15th from AftenPosten.No followed by Sol.No at 1503 on the same day, and then StartSiden.No showed up at 0519 on Friday, December 17th. Combined data from all three sites is presented below.
• The Seattle PI (my old hometown) put the story up at 1401 on Wednesday, Decemeber 15th, 2004. They were a little more savvy that most of the media outlets in that they added a link to the Christmas Webcam rather than just my main page ... so I give 'em bonus points for that!
• Index.HU is some sort of Hungary news (?) website. I have no idea what is being said in the brief article about my Christmas Lights, but it must be a fairly popular web site as can be seen in the numbers below which started up at 0901 on Saturday, December 18th, 2004.
• MBnet.FI is some sort of Finland news (?) website. Again, I have no idea what is being said in the brief article about my Christmas Lights, but it must be a fairly popular web site as can be seen in the numbers below which started up at 1209 on Sunday, December 19th, 2004
• David Pogue from the NYTimes wrote a little writeup in his technology blog - (registration required link) that described how my site has become a "mini-lesson in surviving a Slashdotting" ... but since this didn't run on the front page of the NY Times, I was able to handle the load from the article! ;-)
• The Canada.Com Web Junkie wrote a short little blurb that popped up at 2218 on Wednesday, December 22nd, 2004. Not a whole lotta traffic, but I like our neighbors to the North (did several Canadian radio interviews), so I thought I'd toss 'em in here.
• UserFriendly.Org picked the Christmas Webcam as their LOTD (Link of the Day) at Thursday, December 24th, 2004. This is the home of the ever-funny JD and his comic strip ... has quite the readership ... and highly recommended for a daily visit. As implied, the links stays up for a day, and then rolls into the archives, so most of the traffic is in the first 24 hours
  
  
  Needless to say, once the story broke that the christmas lights webcam was a hoax there was quite a bit of interest. I knew this was coming (but didn't know exactly when) and had moved aside the images/videos to other servers ... so it's a little hard to compare traffic numbers, but it appears to have been a bit less that the media wave two weeks earlier. My guess is that earlier, many people were more compelled to come actually look at the site, whereas now, they simply read the story at various media outlets and didn't go from there ... but my impression is that this story was MUCH more well read as it was not only main page of all the major web sites, but was actually the #1 or #2 most popular story on CNN.Com and WSJ.Com (based on their click-thru data) during the day. While I knew some of the stories would hammer me personally, I was disappointed at some of the inaccuracies in many of the press reports but hey, this is the press! ;-) Fortunately, the Email response has been overwhelmingly positive.
  
  
• ABC-7 Denver Channel started generating traffic at 1419 on Monday, December 27th, 2004. This is kinda interesting, because the article is time-stamped as posted at 1402 (and subsequently updated). Being the nice guy that I am, I had said something was up beforehand which was a mistake. In their haste to get the story up, they made some errors and read more about this here. One thing cool is they ran an online poll asking if viewers were "mad to find out that the Christmas lights were faked" ... and the percentages were fairly constant over days of voting and thousands of votes - over 84% of people said "No, because it's funny" whereas less than 16% said "Yes, because he duped me" ... not too shabby considering the article was fairly negative against me.
• The Wall Street Journal article went live at 1425 on Monday, December 27th, 2004 based on inbound traffic, although it's dated 1423. Interestingly enough, there were three inbounds from the same URL with parameters at the end 11 minutes BEFORE that time - those IP addresses resolve to sb2insgproxy3.dowjones.com, 130.muh43.nycm.n54ny31ur.dsl.att.net, and 66.100.13.4 which doesn't resolve, but appears to belong to an ISP named SAVVIS outa New York. I.e. some folks (my guess is internal to the WSJ) were looking at the story early before it went public; probably link checking and catogorization based on what those parameters looked like to me. I wonder if the the WSJ knows information is bleeding out? BTW, it looks like most of the WSJ traffic comes from various proxies at dowjones.com, so if you want to know if the WSJ is looking at your web pages, this is certainly a way ... and they may want to consider some sort of anonymous proxy or off-site ISP to "cloak" the poking around that they do so webmasters don't see 'em in their web servers logs. Ditto for organizations such as McGraw Hill, New York Times, and others. BTW, I could say a little bit more, but I like the WSJ (have subscribed for over 20 years), so I don't want to give too many clues how people can tell if a story is coming their way! ;-)
• I again hit the main page of Slashdot which posted the story at 1618 on Monday, December 27th, 2004. The "good news" (from a non-flash traffic point of view) is the article link was to the ABC-7 story - recall they were the ones who took me up in the helicopter ... which was probably the "bad news" for me since their article was especially negative toward me, plus it said the Wall Street Journal uncovered it when in fact I approached the WSJ. While there was a lotta "atta-boys, good job Alek" comments, there were also a number of negative comments, so I wonder if it would have been different if they had gone to my hoax page rather than ABC-7's. One poster said it all when he provided a link to my site and said: "Read what Alex himself has to say about the hoax. Pretty interesting to hear it from the man himself instead a news agency ticked off at him for fooling them." Regardless, this reduced the flash traffic on my server since it wasn't in the story itself, but linked throughout the story ... so these numbers aren't as high as before. My guess is this never showed up early in the subscriber-only "mysterious future" since there was no real increase in inbound slashdot traffic before 1618 - but at 1618 and 38 seconds, there was a very noticeable increase in traffic from the OLD slashdot article, so people were double clicking to get to the site.
• FARK posted at 1822 and pounded the site with a direct link. Fark can be a bit risque, but they kept it pretty clean, and the commentary was pretty funny - I chimed in a few times. Fark's claim to fame is stories with headlines that the media would love to write, but would be too politically incorrect to do so - very funny. My favorite comment was by bornyesterday who wrote "Hulkster will forever be a revered name in Fark" ;-) One interesting thing is that this Fark posting had a single link to my web site and generated 33,732 referrals in 24 hours from 30,962 unique IP addresses. For comparison purposes, the December 12th Slashdot posting, before the hoax, which had multiple links to various places on my web site generated 90,607 referrals in 24 hours from 55,168 unique IP addresses (unfortunately, the December 27th posting didn't link to my hoax page, so no data available there). So Fark isn't that far away from Slashdot in terms of generating traffic! Take that with a grain of salt - Slashdot's posting was on a Sunday, whereas Fark's was on a Tuesday (although later in the day and also between Xmas and New Years) ... and hard to say what the readership feels is more "interesting" ... but Fark has a lotta eyeballs looking at it. For comparison purposes, take a look at the Halloween/2004 traffic analysis where Fark traffic was less than a quarter of that from Slashdot.
• Colleen Slevin's AP correction showed up on Yahoo News at 2057 on Monday, December 27th, 2004. She was quite annoyed about the whole thing when we talked, and while she emphasized some negative stuff, I'm surprised she didn't slant it more, although she got some "good" insinuations in there. One major gaff was the quote from Paul McLellan who said I was "unethical" - the next morning, Paul posted a correction that explained "my words were miscontrued to mean something entirely different" but it appears he did not push Colleen for a correction to her correction - now that would been funny!
• FOXNews had an article starting at 2328 on Monday, December 27th, 2004. I noticed the next day it was on their front page, but I don't know when that happened ... but inbound traffic was around one/second in some of the morning hours - this is a well read news site.
• USA Today writes a correction that comes out at 0218 on Tuesday, December 28th, 2004. The USA Today folks are not only up late, they have a good sense of humor - funny writeup.
• Wil Harris from the Inquirer writes a followup to his previous article that hits at 0336 on Wednesday, December 29th, 2004. He's not too happy about the whole thing.
• David Pogue from the NYTimes did a follow up post at 0615 on Thursday, December 30th, 2004. Here is the registration required link and he expressed his displeasure with the hoax - ironically, his bio says magic has been a hobby of his since his childhood - isn't that a hoax? While I disagree with what he wrote - read more in my media updates - I do agree that the media is "so embarrassed to have fallen for the hoax" that I suspect this colored his writing.

Number of inbounds per site after being posted

Some General "Slashdot Effect" Commentary

It should be noted that the "digital" stuff has extensive logging (as did the the webcam daemon) but "analog" stuff does not. So I don't "know" if something goes wrong (for instance, sometimes the images just come back "bad") and you have to infer problems/breakdowns due to heavy load indirectly. Plus real-world analog stuff just isn't that fast, nor as "reliable", so I had coded things intentionally with a 1-second throttle with the hope that the analog side would not get overloaded - as of 12/8, I haven't popped a circuit breaker yet, but I did have one inline light fuse pop and take out 6 strands of 900 lights (yea, I know, only connect 3 strands togather) ... but hey, when you have 17,000 lights, it probably wasn't even noticed! ;-)

The webcam page is about 20 KBytes of HTML/CSS and the webcam image varies between 10-20 KBytes ... so from a bandwidth point of view, it's a pretty lightweight page - hey, I designed it so even dial-up'ers can use it! Pre-Christmas/2004, the several thousand lines of Perl code (plus Benchmark, CGI::Carp, Fcntl, IPfree, Socket, and Time::HiRes modules) had to be slurped in each time since it ran as a CGI_EXEC. With mod_perl, it does this once ... and some misc. ApacheBench testing showed the web server is capable of 5 times as many connections under load. My code run fairly quickly - it uses signals to communicate with the "webcam daemon", so there's a sleep loop where it waits, but that isn't CPU intensive. I also have a flag to disable real-time DNS lookups and I may consider that under extremely heavy load, as those can take some time. There's a memory leak in the webcam daemon, but since this is throttled once/second, it takes most of the night to grow to a GByte in process size, and I can just restart that.

Slashdot Effect Analysis - Misc. Interesting Tidbits/Ideas/Lessons Learned - SO FAR!