Wah, I don't want to give up that feature...
Popularity and performance
Friday, May 19, 2006 by Draginol | Discussion: Websites
Anyone who's been around Stardock's various websites knows the pattern. We find a new way to make things faster or upgrade some piece of hardware to make the sites run smoother and the traffic instantly jumps which brings back the problems for the regulars.
The #1 performance hogging feature on all of Stardock's sites (whether it be JoeUser or WinCustomize or GalCiv2.com or TotalGaming.net) is the same thing -- the referrals. That is, all items actually look at where a visitor is coming from and then adds to a massive table in a massive database on where that referral came from and what the URL was to get back to it.
When you have, literally, hundreds of thousands of people coming to the site every day (which is thousands per hour) it's a significant strain. There's no OFF button for it, it would be a lot of changes in a lot of places in code.
But people feel the pain of that database being hit because it's tied to the article database -- commenting timing out being one of the most common.
We have a new server for the database that is a real monster. It has 12 gigabytes of memory, 4 dual core CPUs, you get the picture. But it will take some weeks to migrate things to it. The same person doing the migration could be asked to go and turn off the referrals which would make things nicer but that would delay the migration which will make everything faster. Hence the dilemma.
I am tending to just let it suffer so that the migration goes quicker and refrain from writing articles like this one (that is why today was so slow) until the migration is done.
Reply #3 Saturday, May 20, 2006 7:44 AM
1. The tab bar at the top of the site will show that I'm logged it but sometimes I still get the ads, smartwords, and am unable to complete tasks as a member (such as posting a comment to this article).
2. Quite often various libraries (most commonly Screenshots) are completely missing from the Gallery page. The page seems to render fine otherwise, just some of the librarys are left off.
Thanks again!
Reply #5 Saturday, May 20, 2006 11:17 AM
The DB you're putting in is huge and should handle a ton of transactions per second, but it looks like it's still a single point of failure. Just tossing out some ideas on spreading the load out and reducing the reliance on the one box.
Reply #6 Saturday, May 20, 2006 4:53 PM
| any thoughts to clustering a series of smaller database servers and load balance connections out to them? How about a cache engine on the front end that people hit, and the cache engine goes back in and refreshes every minute or two? This could reduce load when viewing the main pages of any of the sites, tossing back a static page that links into dynamic data. The subpages such as forums and individual blogs could remain dynamic. There's also the possibility of farming off the collection of referral data to a secondary database that syncs into the main one periodically. These measures reduce the amount of "immediate change" users could possibly see, but it would spread the load a bit and make the site a little less reactive to spikes in traffic. The DB you're putting in is huge and should handle a ton of transactions per second, but it looks like it's still a single point of failure. Just tossing out some ideas on spreading the load out and reducing the reliance on the one box. |
We're doing much of all of the above already. If we didn't, things would have long since grounded to a total halt.
The gotcha is in the places where we didn't design for clustering early on and so as traffic increases you eventually run into some issue that would have seemed minor in the design but shows up when you have millions of hits a day.
In the long term, absolutely, we need to cluster things that aren't clustered and break things up into more pieces. But it is a very painful, slow process to do that. Just migrating a database to another machine can take many days.
Reply #7 Saturday, May 20, 2006 5:13 PM
I look forward to reading and interacting on JU every day.
Thanks for your efforts in trying to make it (and your whole company) better.
Reply #8 Saturday, May 20, 2006 5:15 PM
Reply #9 Friday, May 26, 2006 12:45 AM
Reply #10 Friday, May 26, 2006 8:24 AM
| what does this say about Stardock's code in general? |
I'm sure if it says anything, it says that it's holding up pretty well considering it was probably written before these issues were even comprehended. If you're trying to infer that the code on this site is sub-standard, I think you need to re-evaluate your position. Yes, the site has some problems, yes it can be annoying/frustrating at times, but I think if you worked on a site that got even 1/10th the traffic that this site gets you'd appreciate the technical issues that go into coding for it a bit more. Oh, and I doubt you'll be seeing the code any time soon. If you need to learn ASP.net buy a book, or go to the numerous community sites out there for .NET development.
Brad,
The info is appreciated and I've gotta say, the site has been pretty snappy for me recently so whatever you're doing keep it up!! It really is appreciated!
Reply #11 Friday, May 26, 2006 10:29 AM
(Disclaimer: Just in case no one gets it I was only joking)
Reply #13 Monday, May 29, 2006 6:00 PM
| If you're trying to infer that the code on this site is sub-standard |
Well, it obviously must be substandard, as all good code is written in C.
| Yes, the site has some problems, yes it can be annoying/frustrating at times |
FRUSTRATING? The twelve times I've tried to visit, I might have had a problem! What if I did? Don't you think that would be worse than annoying? I NEED the information on this site, my entire day revolves around it!
| I think if you worked on a site that got even 1/10th the traffic that this site gets you'd appreciate the technical issues that go into coding for it a bit more. |
Huh? I don't get it? Don't you just slap on another processor to halve the workload?
1 processor: work
2 processor: work / 2
3 processor: work / 4
And so on...
| Oh, and I doubt you'll be seeing the code any time soon. If you need to learn ASP.net buy a book, or go to the numerous community sites out there for .NET development. |
Wouldn't that require a purchase? I mean, if StarDock has nothing to hide, why not a full disclosure policy? Or is the code that bad?
~A/PM
P.S. FLAME: You don't seem to understand tongue-in-cheek humor, so in the hope of appealing to the lowest common denominator, this one is blatant sarcasm.
Reply #14 Monday, May 29, 2006 7:06 PM
| You don't seem to understand tongue-in-cheek humor, |
That would be 'wry humour'....
Often it will only appear funny to the author.
To the rest, the necessity for its explanation belies its worth....
Reply #15 Tuesday, May 30, 2006 9:48 AM
| You don't seem to understand tongue-in-cheek humor |
Of course, labeling your previous post as humour implies it has even the slightest element of humour for those of us who aren't sitting with you at your desk as you type it to pick up on. Which your statement had not an inkling of humour to it.
| so in the hope of appealing to the lowest common denominator |
You're already posting the comment, so why attempt to appeal to yourself. Oh, and your comment speaks volumes about your character and maturity but then again I guess we shouldn't expect too much.
| That would be 'wry humour' |
More like 'Dry humour'!
Please login to comment and/or vote for this skin.
Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:
- Richer content, access to many features that are disabled for guests like commenting on the forums and downloading skins.
- Access to a great community, with a massive database of many, many areas of interest.
- Access to contests & subscription offers like exclusive emails.
- It's simple, and FREE!
Reply #1 Friday, May 19, 2006 10:28 PM