UESP-crashes

[Bear]

It happened again at the same time as my last post in this thread and is happening again right now. The server is getting worse.

[Rohugh]

Half an hour ago, I could not get on this site - it kept timing out.

Now I have just tried and it was instant. It does seem to be intermittent.

[Most Honored Listener]

ok I am going to notify the admins and see if we can get this fixed

[Most Honored Listener]

ok this is what Nephele posted

You may have noticed some of the talk (e.g., Template Talk:NPC Summary) about the fact that editing certain templates basically shuts down the server for 5-20 minutes while the server updates all of the pages influenced by that template. It seems that there is a setting that would prevent this problem, namely:

$wgJobRunRate = 0.01

See Job queue. From what I can tell this would tell the server only do about 1 job for every 100 apache requests (whereas by default this parameter is set to 1, i.e. 1 job for every apache request). This means that it will take longer for all of the pages to get updated when a template (or any other transcluded page) gets updated. But I think it's a necessary tradeoff right now, especially since everybody is terrified to make any changes to certain templates because the effects are so diastrous. Since it seems like you're fiddling with settings right now, would it be possible to try implementing this one, too? Thanks --NepheleTalk 12:14, 1 August 2007 (EDT)

most of this probably doesn't make any sense unless you are a wiki member and advanced editor but it pretty much is stating that they are making one minor adjustment which effects content on thousands of the pages this creates a server overload because all of these pages have to reload. if Daveh can have the time to do any of these things then he should be able to fix this problem quite simply but or those who are wondering this is Not a bandwidth issue.

[Most Honored Listener]

this is Daveh's reply and followup

Yah...I've been wondering why the CPU usage was so high and the site sluggish the past week. I made the requested change...the server will be slow for a few hours as it rebuilds all the cached pages again. -- Daveh 13:50, 1 August 2007 (EDT)

Thanks! I'm curious to see what effect that will have next time one of those pages needs to be edited. It's somewhat bizarre that the job queue length had been stuck at 25 all morning (and it's been a high a few other times in the past few days); now that the job queue is getting processed more slowly, the length has dropped to 1. Any way, I'll try to remember to check the queue length occasionally and make sure that a huge backlog isn't starting to build up because of this tweak. --NepheleTalk 16:00, 1 August 2007 (EDT)

This is actually the first time I've ever seen a non-zero job queue size. Depending on what they mean by 'request' even a 0.01 would mean a job would run every 10 seconds or (based on 600,000 daily page requests). I also did a real server reboot (the whole machine, not just the web server) for the first time in half-year. Not sure if that made a difference too but will keep a closer eye on the server in the next week or so. -- Daveh 16:34, 1 August 2007 (EDT)

[Nephele]

The issue fixed by the $wgJobRunRate setting is only a small part of the issue. Most of the site access issues mentioned in this thread have nothing to do with updating templates.

The big picture problem is just that the current server is pretty much at its capacity. Every day there are multiple hour periods when the server is getting more requests than it can handle. On a good day, the server just gets sluggish during those periods. But if there are any additional problems that have come up, then the site becomes nearly inaccessible. Sometimes there's a rogue spider that's crawling the site and downloading every page it can; sometimes there are broken server connections and the server can only handle 30 connections at once instead of 42 at a time. There just isn't any spare capacity to cope with problems, so issues that normally wouldn't be a big deal instead end up pushing the server past its limits.

These problems affect the entire site: the wiki is just as inaccessible as the forums when the server gets overloaded. Daveh's been tweaking settings to keep things functioning in the short term. In the long term, the entire server is going to have to be replaced with something more powerful. But that's something that can't be done overnight.

[Anfauglith126]

Today I couldnt enter the wiki or the forum (maybe it was my computer but I dony know so I posted here...)

[Bear]

yes it's still happening on the rare occasion but doesn't seem to last long though.

[Daveh]

There was a small 15 minute hiccup at 17:15 EST yesterday due to a SYN flood (too few to be called an attack but enough to slow things down a bit). Other than that there hasn't been any noticeable issues recently.

As it has been mentioned the current server is pretty much running at its maximum capacity, especially at peak hours and on weekends. There are a few tweaks left to try, mainly switching from Apache to Lighttpd for everything, but I don't believe that will get us that much benefit.

The server is actually the smallest dedicated server available from the hosting company and it has performed admirably...up to 1 million daily pages (3.5 million requests) serving some 25GB of data.

I'm looking into a server upgrade but want to get a good plan in place for migrating before it actually happens. Without proper planning UESP could easily be down for a few days. My goal is for almost zero apparent downtime with a minimal lock-down (read-only) period.

[ace62394]

the forum didn't crash or anything, but for some reason, i had to log in (i have auto-login or whatever) and it said that there are no new posts, yet there were

[Infamous]

Its dropped once for the pass 3 days

[Most Honored Listener]

Great now the Wiki is not letting anyone post.

[ragged-gothic]

How big is the pipeline to the site — i.e. how many gigabytes up/down per unit time does the server’s ISP allow?

What is the speed and memory capacity on the actual computer(s) that act as servers for the site?

[Daveh]

• Pipe -- 10 Mbps, easily upgradable to 100 Mbps. We've never used more than 5 Mbps and are currently averaging about 3 Mbps with a 95% utilization of 4 Mbps.
• Bandwidth -- Monthly bandwidth is currently 1500 Gb (total, up and down) with anything exceeding that charged on a per Gb rate. Currently we're only using about 1000 Gb a month and we can upgrade this easily as well.
• Speed -- Server is running only a P4 2.4 GHz and is currently the limiting factor in how many concurrent connections we can serve.
• Memory -- Currently running 2GB of RAM. My idea is to install Memcached in the near future to better use the memory and reduce the CPU load which should hopefully increase the server's capabilities noticeably.

I learned a lot from the experience of switching servers this time and if we have to upgrade servers again I can do so pretty quickly. Unfortunately, any future upgrades will up the monthly cost of the server a significant amount unless I'm lucky enough to find a good deal. We're currently in a 'sweet spot' in terms of server power/cost.