Page 1 of 1
Current Outage
Posted: Mon Aug 03, 2009 9:06 pm
by rpeh
The current outage is caused by the hard drive on the database server that powers UESP being replaced.
The idea was that a new disk would be added to replace a failing drive, but unfortunately the drive failed totally before that necessary data could be copied off. That means Daveh has had to reinstall everything pretty much from scratch and restore the site backups.
We will lose everything added to UESP since Sunday morning's backup (~4am EDT Sunday to 9am EDT Monday)
There will probably be no more information than this. Daveh is working on the reinstallation and nobody else can help him with that. The site will come back up as soon as he is confident that it will stay up.
rpeh
UESP Site Admin
Re: Current Outage
Posted: Mon Aug 03, 2009 9:11 pm
by rpeh
Okay, the fact I could even post that basically proved that it's all back up and running again. At least you know why everything was down now
Re: Current Outage
Posted: Mon Aug 03, 2009 9:18 pm
by Codex
Posting seems to be working fine (
), but there are still problems with the PM system. I can't access the inbox, outbox or the saved messages
Edit: And I got an SQL error when I tried to post this message the first time.
Re: Current Outage
Posted: Mon Aug 03, 2009 9:19 pm
by Caz
Some of the forums are still down too. It's more random... semi-dead, as I said.
Re: Current Outage
Posted: Mon Aug 03, 2009 10:36 pm
by Nephele
Yes, I can confirm that there are still SQL errors popping up, and I know at least one problem that's responsible. I'll do what I can to get them fixed, but it might take an hour.
Re: Current Outage
Posted: Mon Aug 03, 2009 10:39 pm
by Codex
Thank you all
Re: Current Outage
Posted: Mon Aug 03, 2009 11:09 pm
by Nephele
OK, one configuration problem has been fixed, and that appears to have fixed the errors I'd been getting. Let me know if there are still other errors popping up.
Re: Current Outage
Posted: Mon Aug 03, 2009 11:37 pm
by Caz
Ok now everything seems to have calmed down, I hope things can get back to normal.
First of all a big thank you to the admins for taking care of everything.
Now we have lost a whole bunch of posts, pms and at least one account due to this outage. So, if anyone posted something important that is now missing, please post your question or answer again. Our humble apologies to the unfortunate user who is now deleted, feel welcome to register again!
Any more problems or questions can be directed towards the mods. Thanks for your understanding!
- The UESP Mod team
Re: Current Outage
Posted: Tue Aug 18, 2009 9:33 pm
by H8Ball
Hi, guys,
Are we still suffering from the same "Current Outage"? It's always the first "stickied" thread I see on the forum and I keep re-reading it only to find no new info!
I can confirm that I'm still seeing occasional SQL errors from my end. Also, there seems to be a problem with the post editor, as I frequently have to click twice on Preview or Submit to get them to work. First click fails with a "submitted form is invalid" error in red. Second click always succeeds even though I've made no changes whatsoever to the post content.
Re: Current Outage
Posted: Tue Aug 18, 2009 9:52 pm
by Cactus
I had the most bizarre problem yesterday I've ever seen.
Oh look, a link. Click?
Re: Current Outage
Posted: Wed Aug 19, 2009 5:51 am
by Alilatias
That happens to me once in a while, and not just on this site. I even had it happen once over at the official Bethsoft forums. Nothing to get worried about, as far as I know.
I'd chalk it up to the computer loading the forum template in a rather strange manner/incorrectly, but I don't know much about technical stuff like that.
Re: Current Outage
Posted: Wed Aug 19, 2009 5:35 pm
by H8Ball
I can also confirm that, from here in the UK, UESP seems to go through a daily pattern of "accessibility" problems. These range from slow-downs to sporadic problems such as timing out for hours at a time or reporting occasional SQL errors. I've sometimes had to abandon attempts to access UESP and wait until the following morning to try again. Problems have been much less frequent since the UESP forum software upgrade but they still happen now and again.
I'm in the UK and I've noticed over several months that, after accounting for time zone differences between UK and US, the frequency of problems I see with UESP seems to tie in neatly with daily periods of peak US commercial Internet activity.
The UK has 1 time zone (Greenwich Mean Time = GMT), whereas the US has 4 time zones covering a 3 hour period ranging from Eastern Time = EDT (GMT-5) to Pacific Time = PDT (GMT-8). Events such as start-work, finish-work, etc, take 3 hours to propagate across US time zones. UESP slow-downs occur here in the UK from around 1pm to 4am = 13 hours, which corresponds to the US being commercially active from around 8am to 6pm = 10 hours + 3 hour propagation = 13 hours! I've also noticed that the sporadic UESP problems often occur during the UK period 4pm to 10pm = 6 hours, which corresponds to a US lunch break range of 11am to 2pm = 3 hours + 3 hour propagation = 6 hours! Maybe due to prolific surfing or on-line gaming during US lunch breaks?
Maybe UESP's web site hosting company treats UESP as a low priority destination via a gateway router? If that router favours high priority destinations (i.e. commercial traffic), it will have a tendency during VERY busy periods to drop packets to low priority destinations. This would neatly explain ALL the problems: Slow-down due to multiple retries before success; Timeout due to failure after n retries; SQL error due to truncated/incomplete SQL request (bad SQL command syntax).
Re: Current Outage
Posted: Wed Aug 19, 2009 5:38 pm
by videogmer314
That's happened to me on other sites before, but not this one, strangely enough.
Re: Current Outage
Posted: Wed Aug 19, 2009 6:22 pm
by Kestral
I say it's a reason with PhpBB3 not loading the styles correctly, but this only happens once every not-so often. So, it shouldn't be something to worry, although, this clearly is an examination from my copy of PhpBB3 3.0.4 on my computer.
Re: Current Outage
Posted: Wed Aug 19, 2009 9:27 pm
by rpeh
h8ball wrote:I can also confirm that, from here in the UK, UESP seems to go through a daily pattern of "accessibility" problems.
There's a known problem with the backup at around 9am UK time (BST at the moment). Other than that, I've not seen any of the problems you describe. Please can you provide more information?
rpeh
Re: Current Outage
Posted: Fri Aug 21, 2009 2:39 am
by H8Ball
@rpeh
Heh. You're right! Sorry, I should have said BST (British Summer Time), not GMT. In Winter, UK is on GMT and Montreal is on EST = GMT
-5. In Summer with DST (Daylight Savings Time) of +1 hour applied, UK is on BST = GMT+1 and Montreal is on EDT = GMT
-5+1. With or without the +1, the time difference is STILL -5 hours.
Anyway, on to problems. They don't occur every day but, when they do, the timing seems to tie in neatly with the pattern of typical UK/US daily Internet activities. Take a look at the table below, in which I've postulated a typical day (start work 8-9am, lunchtimes ranging from 11am-2pm, finish work 5-6pm, and leisure from 6pm-midnight). I've weighted the typical Internet activities and spread them across UK/US time zones. I've then added up the weights for each hour and the results reproduce almost EXACTLY the timing pattern I'm seeing!
Code: Select all
Internet Activity Weights: <blank>=Minimal, 1=Work, 2=Lunchtime Surfing, 4=Evening Surfing/Gaming
AM PM AM
TimeZone Offset 08-09-10-11-12-01-02-03-04-05-06-07-08-09-10-11-12-01-02-03-04-05-06-07-
--------------------------------------------------------------------------------------------
UK 0 1 1 1 2 2 2 1 1 1 1 4 4 4 4 4 4
US Eastern -5 1 1 1 2 2 2 1 1 1 1 4 4 4 4 4 4
US Central -6 1 1 1 2 2 2 1 1 1 1 4 4 4 4 4 4
US Mountain -7 1 1 1 2 2 2 1 1 1 1 4 4 4 4 4 4
US Pacific -8 1 1 1 2 2 2 1 1 1 1 4 4 4 4 4 4
--------------------------------------------------------------------------------------------
Weight Sum 1 1 1 2 2 3 3 4 6 7 11 11 10 9 8 11 10 13 16 16 16 12 8 4
As you can see, US activity overlaps the UK's from 1pm to 8am (19 hours) with US lunchtime/evening peak periods having greatest effect during UK times 6pm to 9pm (3 hours) and 11pm to 6am (7 hours) respectively. I've noticed that, when UESP timeouts DO occur, they typically start around 6pm and 11pm UK time. And, funnily enough, Weight Sum suddenly increases at those times, indicating high activity which I'm assuming is mainly from (on-line) Gaming!
Timeout: Page loading status stays at 0% progress until Firefox times out and indicates "unable to contact web server". Yesterday after 6pm, for example, UESP suddenly started acting up, taking up to 20 seconds for the web server to respond. Still within my 30 second Firefox timeout, though. Tonight, I haven't seen delays longer than about 10 seconds.
SQL Error: Message appears at top of UESP web pages. Unfortunately, it's not helpful in pinpointing the problem because "error" could mean anything, i.e. SQL Server timeout, SQL command syntax error, etc.
Three days ago, I upgraded from Firefox v3.0.x to v3.5.2 and browsing is now significantly faster, so maybe it's good riddance to timeouts? Then again, it may be too early to tell as yet.
Hope this helps.
Re: Current Outage
Posted: Fri Aug 21, 2009 8:42 am
by Cactus
I'll admit, I've never paid much attention to times or patterns, but I've gotten timeout errors and "Server didn't respond" errors several times. Sometimes I have to just leave and come back later because the site won't load. This has always occurred for me though on UESP and the forums, since I first joined. Only the SQL errors, which I rarely see, are new. I did get a SQL error yesterday when I tried to add a custom field to user profiles, however. Every time.
Re: Current Outage
Posted: Fri Aug 21, 2009 10:30 pm
by H8Ball
I spotted something of interest tonight. Take a look at this TraceRoute result from my home in the UK to the UESP Web Server in Montreal, Canada:-
Code: Select all
Bounce Time (ms)
Hop Pkt1 Pkt2 Pkt3 Router IP Address Location
---------------------------------------------------------------------------------------------------------------
1 <1 <1 <1 <My ADSL2 Router> x.x.x.x Newcastle-upon-Tyne area, UK
2 60 42 74 cr0.new.uk.easynet.net 87.87.248.209 Easynet Public Gateway
3 40 41 40 ip-87-87-137-65.easynet.co.uk 87.87.137.65 Easynet
4 57 58 59 bu1.er10.enslo.ov.easynet.net 83.231.199.162 Easynet OUT
5 163 207 220 po-1.r00.londen01.uk.bb.gin.ntt.net 83.231.199.161 London, UK
6 201 228 * xe-4-2.r00.londen03.uk.bb.gin.ntt.net 129.250.2.237 London, UK
7 58 54 55 if-2-1.icore1.LDN-London.as6453.net 195.219.83.21 London, UK
8 56 54 55 195.219.243.1 195.219.243.1 UK Atlantic Gateway
9 141 129 139 195.219.243.30 195.219.243.30 US Atlantic Gateway
10 129 129 128 if-4-0-0-1262.mcore4.NYY-NewYork.as6453.net 209.58.62.2 New York, US
11 136 136 136 if-1-0.mcore3.MTT-Montreal.as6453.net 216.6.114.17 Montreal, CA
12 141 143 143 Vlan19.icore1.MTT-Montreal.as6453.net 216.6.114.14 Montreal, CA
13 137 136 137 te6-2.cl-core04.teleglobe.mtl.iweb.com 206.82.135.30 iWeb = UESP hosting?
14 136 136 136 ip-72-55-174-245.static.privatedns.com 72.55.174.245 UESP Web Server
I've re-run the above trace multiple times today from 4pm (UK time) onwards and the route to UESP has remained EXACTLY the same every time, always via the two NTT routers in London (Hops 5-6) despite them being frequently bottlenecked (high Bounce Times) and even timing out (* symbol). If these each only have ONE neighbouring router, NO alternate routes are possible, so the bottleneck can't be bypassed. This has the knock-on effect of UESP slow-downs and, if they're occasionally "gridlocked" for minutes at a time, UESP timeouts.
So UESP is not to blame. Instead, I can blame NTT (Nippon Telegraph & Telephone Corporation, apparently). Any chance of installing parallel routers to ease the load, NTT guys?
