 |
How Not To Deal With A Digg |
 |
In two months The Seminal has had two front page stories on digg.com. Our server has been down twice. Fool me once, shame on you, fool me twice, shame on me right? Not quite.
Our first front page story came from this article. When I got online at 10 am our website was already crawling. Soon it was pretty much dead. Our statistics from that day tell the story:
Almost 8,000 unique visitors total. The graph peaks at 886 visitors at 10 am. After that the server starts going downhill, only letting in half that number three hours later.
This was to be expected of course. We had run this site off of a $7/month shared hosting account from iPowerWeb. We didn't expect a front page story so quickly after launching our site and no shared host can be expected to handle a front page story, especially without any optimization on our end.
So, our server ran slowly throughout the day, and eventually when things calmed down we were able to approve and respond to comments and pick up the pieces.
Getting Smart
After our first server crash we decided to get smart and get better hosting. We upgraded to iPowerWeb's $49/month Virtual Private Server package as recommended by their tech support. After migrating our site, we spent two weeks tweaking Wordpress, our theme and CSS files, our images, and our MySQL implementation to deliver faster and more reliable service. We implemented the MySQL cache, PHP cache, and started using the plugin WP-Cache to help ease server load. All in all we reduced the weight of this site by 1/2! Combined with the new expensive hosting package we thought we were ready.
We Were Wrong
Our second front page story came from this article on Wednesday. Once again the diggs caught up with us overnight and when I logged on at 10 am our site was totally dead. I immediately got on the phone with iPowerWeb's tech support to figure out what was going on. Surely our VPS system hadn't been brought down by too much traffic?
In my first conversation I explained the situation. I said, "We have a front page story on digg.com right now, I think that's why our web site isn't responding." I didn't get the sense that the person on the end of the line understood what I was talking about, but I was told that our expensive hosting plan had in fact been brought to its knees by too much traffic. They reset the server on the phone, ensuring that this would bring the site back online. No such luck.
Tech support was flummoxed. Resetting the server should have re-awakened the site they said. They "upgraded" the issue to Level 2 support and ensured me I would receive an email when the problem was rooted out. Keep in mind that at this point our story was burning up the front page on digg and we were losing thousands of visitors an hour!
The Response
I heard back from Level 2 tech support at 1:30 PM, almost four hours after I had first called. Here's what they said:
Dear Customer,
Thank you for contacting IPOWER Technical Support.
We have checked the issue and currently we can't find any fault with your VPS. The issue you reported appears to be resolved at this time. I have tested your account and everything is working properly. This may have been caused by a temporary server issue that has already been resolved. Now all the three domains found working fine at our end. If this is not the case, please reply back so we can investigate this further.
Regards,
[name removed to protect the guilty]
Technical Support Team
IPOWER, Inc.
Well, a quick check told me that the site was still down and nothing was getting through. I have no idea where they came up with this bullshit but they certainly weren't right. None of the three domains hosted in the VPS were loading.
I wrote back telling them they were in fact dead wrong. Our website was still down and although our front page story was still receiving diggs, nobody could get through.
I called tech support again when I didn't hear back after an hour. I explained my story and frustration a second time to another representative and got a similar response. Everything seemed to be working fine (other than the fact the website was down) and they had no idea what was going on. My issue was upgraded to Level 3 tech support (oooo, Level 3…special!) and I was assured I would receive an email back when the issue was resolved. I asked if there was any way I could talk to someone who knew what was going on via phone (I still wasn't sure the people I was talking to understood the issue or even knew what digg was), but I was told email was the only way I would receive a response.
And so I waited…and waited…and waited…
I called a third time at 7 pm. The site was still down and our front page article had fallen out of the "Top 10 in All Topics" list it had been on all day. I talked with a third support representative who reset our server again. Finally, our website was back online. I asked what went wrong. He responded that he didn't know.
Aftermath
Once our statistics were updated we were left with this pathetic chart:
Not only had the server been completely dead at 1:30 when I received an email back from tech support saying everything was fine, but the VPS hosting account, which cost a full seven times more than our crappy shared hosting account, let in NOT EVEN HALF the number of visitors. It actually performed WORSE!
The article got a total of almost 3000 diggs. We only received 3,000 visitors. We missed out on a ton of traffic, possibly as much as 10,000 visitors!
On top of that, I NEVER GOT AN EXPLANATION about why our server died. Nothing.
Wow.
Resolution
We quickly searched for another host. The unresponsive tech support, the 12 HOURS of downtime, and the lack of an explanation forced the move. We switched that night to Survive Digg (aff), a service offered by Brius Web Hosting. They guarantee 100% uptime when you have a front page story on digg.com and 99.9% uptime otherwise.
So far, I have found their service exemplary. One tech stayed up with me until 2 am helping me move my site, troubleshooting MySQL issues, and waiting for our DNS to propagate. They checked in the next morning and the day after that to make sure we were doing all right. The service costs only $35/month ($10 less than we were paying at iPowerWeb) and it is a month-to-month contract which we can break at anytime.
But, can they handle the traffic? I talked to Ipp over at Tutorial Ninjas about their experience with Survive Digg (aff). They said they recently had two front page stories at the same time and the servers survived without a hitch. Of course, we'd love to test this service ourselves. So, if you liked our story of server woe please do us a favor, digg it, and help put Brius Hosting to the test.
If you are looking for good shared hosting try 1and1.com I've used them for years. My site tweekerville.com gets an average of 20,000 unique visits per day and doesn't have any issues.
Looks like you better get your money back and look into a dedicated server.
you made it….
Down for a few hours and back up now!
Bring on the comments Diggers.
Ironically your site is having db connection issues now. I've had to refresh a while to get it to show up like this now. You would have been better off with Media Temple in the first place.
Looks like you guys were down even on the new hosting site… Can anything handle a front page digg? =O
If 8,000 hits in a day shut your site down, you weren't hosted with a bad company, but a completely incompetent one. I received nearly 100,000 hits over a two day period from a recent Digg which eventually brought our virtually hosted site to a crawl and have now switched to a robust dedicated server which should more than handle the traffic.
Working just fine now.
So, why was the site down?
I thought the server would be able to survive the digg effect?
Your site is working great for me. Good idea moving to a different host.
My site used to be hosted on a host called Surpass Hosting. I ran into a similar issue as you did with your site so I moved. I'm now been with Media Temple for a year and couldn't be happier. They've been able to withstand being Dugg and have offered great service.
I hope Survive Digg works out for you.
Shared hosting CAN be Digg proof. I host with Media Temple; $10 /month (ss) package. My WordPress blog hit the frontpage Monday and the site didn't even slow down, with 20,000 hits in three days. While you can't still get the (ss) package, their (gs) package for $20/month is even better. If you're looking at trying out a new host, definatly give Media Temple a try.
Dugg. Bring on the server load!!
Well, it appears to be find now. Your wish has been granted, its on the homepage!
Sites still up and loading nice and snappy, as far as i can tell.. although you might want to slap up a bigger "dig this" button to encourage diggs and see if you can blast out your server…
Well, looks like we're back up. I'm hoping for an explanation from our web host shortly…
Well, the verdict is in.
I noticed this story on the front page 15 minutes after it hit. 30 seconds later Survive Digg's tech support was working on the issue. They understood what was going on and tweaked our Apache and MySQL personally to get The Seminal back online. Total downtime was less than two hours.
Obviously, this wasn't the 100% uptime we were hoping for, but I have been continually impressed by the support we've received. We are sticking with Survive Digg as a web host. I'm hoping next time we'll receive stellar support as well as stellar uptime.
So, my above recommendation for the service stands. I've never received this kind of personal attention from a hosting company. You'd do yourself a favor by checking out Survive Digg.
Since we're digging your story, would you mind posting an update showing the stats from this time around, too?
No problem! Google Analytics stats are behind by hours (they have only loaded up till 5 pm EST so far), but check back tomorrow. I will post with the stats, what went wrong, and how it was resolved.
BTW…our stats show 2,845 unique visitors for 4 pm (all in one hour)…obviously not complete, but that's what we're showing so far.
There is a free content distribution service called Coral CDN (coralcdn.org), that you can use by appending ".nyud.net:8080″ to the end of a domain in a URL. They cache the page on their nearest server and distribute it to servers worldwide, so your server is only accessed to cache it.
Funny I've had as much as 17,000 visitors in a day with nary a hickup at kattare.com. I did write my own software which does a lot of caching, but it's a shared host plus apache forwards all the traffic to my jetty server. I'm not worrying until I get about 50,000 or so.
I've used 13 hosts over the last decade, each more disappointing than the last. Host ratings are bullsheep. Host lists are also bull. People recommending hosts are full of it or have no idea what they're talking about.
I'd estimate that out of thousands of hosts in the USA, it's unlikely in the extreme that ANY of them are actually able to provide 99% uptime for an entire month, let alone a whole year.
There is certainly no evidence of any such host yet brought to my attention either at Digg or Fark, and both sites are notorious for bringing down hosted accounts.
At this point I'll disbelieve anyone who baldly states that they are the best. Point me to a consumer survey that isn't funded by the host. Point me to a University Study. Otherwise, they're lying.
I won't even believe you.
D
Grats, your site is still up, for now atleast!
Good luck.
-Digg
Shared web hosts can't make any money if you use more than a fraction of the allotted bandwidth, here's the math to prove it. Still, most shared hosts can't even deliver a reasonable fraction of what you're paying for.
That article may not make it clear, but not all shared hosts are bad. I've heard good things about both Media Temple and 1and1 (where I host) and it looks like Brius is doing well based on this article staying up.
The reason a shared hosting server could survive longer than your VPS was because there were more resources for it to use. Most likely, the shared server was a 2 cpu server (AMD's or Intel), with anywhere from 2-4GB of RAM. Compare that to your partial share of cpu and memory. The shared server was able to stay up longer (though it would have affected every customer) than the VPS. You also have to compare the nic on the shared and VPS.
You should add a digg tab on your site. That draws us diggers like flies.
That's strange. If you made to the frontpage of Digg then you should be getting much more visitors/pageviews than that.
Should have been 10 times more!
I've had nothing but frustration with ipower. They are one of the crappiest shared hosts out there. I finally switched, and couldn't be happier. No ssh, no control, poor control panel, and they threw a hissy fit when I tried to transfer the domain name away from them.
bah. good riddance.
As I said, Google Analytics stats are not realtime. I only have stats from that one hour. The others will come in sometime tonight and I'll post the full numbers tomorrow.
nice advertorial
I don't know about Media Temple (ss) package, however I have a site hosted with Media Temple (gs) Grid Server package. Their MySql database is down almost daily anywhere between a few minutes to a few hours. There goes their marketing gimmick that their grid server technology is Digg Proof. It might be true if you site runs without using mysql, otherwise stay away from Media Temple Grid Server.
Oh yeah, do a search for media temple on digg. There should be several posts from digg users just to test digg effect on media temple.
Here you go again
write a htaccess to redirect users from digg (or other sites that cause large swarms) and redirect them to coral cache. Also you could try an hourly cron job that checks to see if usage as spiked on a particular article and then generate a static html page automatically instead of having to query the db all the time. (it just doesnt make sense) Why? For the comments?
KieranMullen
try dreamhost…several duggs and no problems. They are also cheap, awesome customer support.
ipowerweb sucks:
http://jamiemart.in/index.php/2007/03/25/what-hosting-companies-have-a -sense-of-humor-an-experiment/
Duh, use Media Temple!
Interesting that you'd say that because I wouldn't have been surprised if the author wasn't talking about 1&1 from the shoddy service and responses I get from their staff.
I'll be taking a look at this company he's with - especially as they're doing well during this current front page digg status.
^^
Rob: As you probably know the site was down for about 2 hours. The support was exemplary though and it has been up since then and loading pretty quickly. Jason will post stats later today. Until I see evidence of a better plan for $35/month I would recommend survive digg to anyone.
Wow very interesting read, I can't believe the poor service you received with your previous company.
Hope this new one works out
I used to work at IPOWERWEB for a while. Tech Support. I have great respect for the two or three guys in the LA office guys, but really the company sucks BIG TIME. I would not recommend them to anyone as the amount of downtime they can have is massive.
If survive digg went down for a minute I'd run a mile away.
Clearly they're not providing what they're promising. The idea behind it is that you get 100% uptime through a digg and therefore don't lose traffic/revenue. They failed you, end of.
Also the reviews on their homepage are testimonials about the IDEA, not to their service.
If you really want to test the server, help push another story onto the front page… It is almost there already.
http://digg.com/political_opinion/Impeachment_History_101
My site used to be hosted on a host called Surpass Hosting. I ran into a similar issue as you did with your site so I moved.
thanks for it…. Regards…
If you stayed with the same hosting company you are dumb.
Should have moved it in the begining to a big compnay.
Stupid is as stupid does.
No pity here
Well, the traffic is still rolling in and the site is fine, so I'm pretty satisfied. I'll get you all an update with more stats tonight.
Funny that someone from a hosting company leaves a silly comment like that. Also funny that said hosting company advertises deals on its front page that are expired.
Honestly though, if anyone not affiliated with a hosting company knows what is up, please post up here and tell us what you think.
Uhmmm it really seems to works fine now. I think I'm gonna move to survive digg
Has anyone had a front-page digg story using godaddy?
not me…from what I've heard though I wouldn't trust em:
http://www.digg.com/hardware/Can_Godaddy_Hosting_Survive_the_Digg_Effe ct
Check out the comments…
I would say Survive-Digg DID NOT WORK: When your most recent DIGG came out fresh, I attempted to read your web site 3 or 4 TIMES in 3 HOURS, but EVERY SINGLE TIME I got nowhere. Now a few days later your web site is up. I recommend 1and1.com, there I've heard of a couple of people whose Wordpress Blog survived DIGG with NO special tweaking.
@Hermann:
Yes and no. We were down for 2 hours right after the digg but we were back up with the story on the front page. Now the site is working fine and the traffic is still coming in. Check back later tonight for a full post about what happened.
That said, I'm still recommending Survive Digg.
What servers does Digg run on? For all the traffic that your server gets hit with, Digg has to be getting so much more than that.
hosted with ip[ower for years. had many problems. looking away this year.
they give you tons of stuff, but drop password files, lose account information and require you to prove you purchased etc.
SurviveDigg is running a basic cPanel setup with Apache 1.3.x. By no stretch of imagination can they handle more traffic on their servers than anyone else running cPanel right out of the box. There's only so much "optimization" can do, and with their obsolete (old dual xeon) hardware, I don't think they're ready for this market sector. Cheers, V
Thanks for the opinion gsv…check out http://www.theseminal.com/2007/04/02/server-stories-follow-up-what-hap pened/ for a follow up on what happened. I think Survive Digg's setup ran pretty wel…
I really wouldn't recommend dream host like some people here. I use them as a file dump but nothing else. php as cgi sucks..
Wow, must've been quite hectic for you guys.
My question is…was iPOWER lying to you?
Not lying, per se, they were just incompetent. They never did figure out what was going on or give a good explanation. So we left for SurviveDigg. We've been much happier (and stable) ever since.
I really feel this article because I've been going through absolute nightmares with my web hosting. It seems like something is always going wrong… and what you describe with your "upgrade" is what I'm scared of with "upgrading" with these hosts… That I'll upgrade and there will be no real difference. I've also had the same experience where I will tell tech support something is wrong and they'll just give me some total BS in return. I've never had tech support at any web host actually admit that they messed something up. It's extremely frustrating. Anyways I've bookmarked your suggestion for "surviving digg" and I may be checking it out in the future if I have any problems. Cool article.