Current Outage

[rpeh]

The current outage is caused by the hard drive on the database server that powers UESP being replaced.

The idea was that a new disk would be added to replace a failing drive, but unfortunately the drive failed totally before that necessary data could be copied off. That means Daveh has had to reinstall everything pretty much from scratch and restore the site backups.

We will lose everything added to UESP since Sunday morning's backup (~4am EDT Sunday to 9am EDT Monday)

There will probably be no more information than this. Daveh is working on the reinstallation and nobody else can help him with that. The site will come back up as soon as he is confident that it will stay up.

rpeh
UESP Site Admin

[rpeh]

Okay, the fact I could even post that basically proved that it's all back up and running again. At least you know why everything was down now

[Codex]

Posting seems to be working fine ( ), but there are still problems with the PM system. I can't access the inbox, outbox or the saved messages

Edit: And I got an SQL error when I tried to post this message the first time.

[Caz]

Some of the forums are still down too. It's more random... semi-dead, as I said.

[Nephele]

Yes, I can confirm that there are still SQL errors popping up, and I know at least one problem that's responsible. I'll do what I can to get them fixed, but it might take an hour.

[Codex]

Thank you all

[Nephele]

OK, one configuration problem has been fixed, and that appears to have fixed the errors I'd been getting. Let me know if there are still other errors popping up.

[Caz]

Ok now everything seems to have calmed down, I hope things can get back to normal.

First of all a big thank you to the admins for taking care of everything.

Now we have lost a whole bunch of posts, pms and at least one account due to this outage. So, if anyone posted something important that is now missing, please post your question or answer again. Our humble apologies to the unfortunate user who is now deleted, feel welcome to register again!

Any more problems or questions can be directed towards the mods. Thanks for your understanding!

- The UESP Mod team

[H8Ball]

Hi, guys,

Are we still suffering from the same "Current Outage"? It's always the first "stickied" thread I see on the forum and I keep re-reading it only to find no new info!

I can confirm that I'm still seeing occasional SQL errors from my end. Also, there seems to be a problem with the post editor, as I frequently have to click twice on Preview or Submit to get them to work. First click fails with a "submitted form is invalid" error in red. Second click always succeeds even though I've made no changes whatsoever to the post content.

[Cactus]

I had the most bizarre problem yesterday I've ever seen.

Oh look, a link. Click?

[Alilatias]

That happens to me once in a while, and not just on this site. I even had it happen once over at the official Bethsoft forums. Nothing to get worried about, as far as I know.

I'd chalk it up to the computer loading the forum template in a rather strange manner/incorrectly, but I don't know much about technical stuff like that.

[H8Ball]

I can also confirm that, from here in the UK, UESP seems to go through a daily pattern of "accessibility" problems. These range from slow-downs to sporadic problems such as timing out for hours at a time or reporting occasional SQL errors. I've sometimes had to abandon attempts to access UESP and wait until the following morning to try again. Problems have been much less frequent since the UESP forum software upgrade but they still happen now and again.

I'm in the UK and I've noticed over several months that, after accounting for time zone differences between UK and US, the frequency of problems I see with UESP seems to tie in neatly with daily periods of peak US commercial Internet activity.

The UK has 1 time zone (Greenwich Mean Time = GMT), whereas the US has 4 time zones covering a 3 hour period ranging from Eastern Time = EDT (GMT-5) to Pacific Time = PDT (GMT-8). Events such as start-work, finish-work, etc, take 3 hours to propagate across US time zones. UESP slow-downs occur here in the UK from around 1pm to 4am = 13 hours, which corresponds to the US being commercially active from around 8am to 6pm = 10 hours + 3 hour propagation = 13 hours! I've also noticed that the sporadic UESP problems often occur during the UK period 4pm to 10pm = 6 hours, which corresponds to a US lunch break range of 11am to 2pm = 3 hours + 3 hour propagation = 6 hours! Maybe due to prolific surfing or on-line gaming during US lunch breaks?

Maybe UESP's web site hosting company treats UESP as a low priority destination via a gateway router? If that router favours high priority destinations (i.e. commercial traffic), it will have a tendency during VERY busy periods to drop packets to low priority destinations. This would neatly explain ALL the problems: Slow-down due to multiple retries before success; Timeout due to failure after n retries; SQL error due to truncated/incomplete SQL request (bad SQL command syntax).

[videogmer314]

I had the most bizarre problem yesterday I've ever seen.

Oh look, a link. Click?

That's happened to me on other sites before, but not this one, strangely enough.

[Kestral]

I had the most bizarre problem yesterday I've ever seen.

Oh look, a link. Click?

I say it's a reason with PhpBB3 not loading the styles correctly, but this only happens once every not-so often. So, it shouldn't be something to worry, although, this clearly is an examination from my copy of PhpBB3 3.0.4 on my computer.

[rpeh]

I can also confirm that, from here in the UK, UESP seems to go through a daily pattern of "accessibility" problems.

There's a known problem with the backup at around 9am UK time (BST at the moment). Other than that, I've not seen any of the problems you describe. Please can you provide more information?

rpeh

[H8Ball]

@rpeh

Heh. You're right! Sorry, I should have said BST (British Summer Time), not GMT. In Winter, UK is on GMT and Montreal is on EST = GMT-5. In Summer with DST (Daylight Savings Time) of +1 hour applied, UK is on BST = GMT+1 and Montreal is on EDT = GMT-5+1. With or without the +1, the time difference is STILL -5 hours.

Anyway, on to problems. They don't occur every day but, when they do, the timing seems to tie in neatly with the pattern of typical UK/US daily Internet activities. Take a look at the table below, in which I've postulated a typical day (start work 8-9am, lunchtimes ranging from 11am-2pm, finish work 5-6pm, and leisure from 6pm-midnight). I've weighted the typical Internet activities and spread them across UK/US time zones. I've then added up the weights for each hour and the results reproduce almost EXACTLY the timing pattern I'm seeing!

Code: Select all

Internet Activity Weights: <blank>=Minimal, 1=Work, 2=Lunchtime Surfing, 4=Evening Surfing/Gaming

                    AM          PM                                  AM
TimeZone    Offset  08-09-10-11-12-01-02-03-04-05-06-07-08-09-10-11-12-01-02-03-04-05-06-07-
--------------------------------------------------------------------------------------------
UK             0    1  1  1  2  2  2  1  1  1  1  4  4  4  4  4  4
US Eastern    -5                   1  1  1  2  2  2  1  1  1  1  4  4  4  4  4  4
US Central    -6                      1  1  1  2  2  2  1  1  1  1  4  4  4  4  4  4
US Mountain   -7                         1  1  1  2  2  2  1  1  1  1  4  4  4  4  4  4
US Pacific    -8                            1  1  1  2  2  2  1  1  1  1  4  4  4  4  4  4
--------------------------------------------------------------------------------------------
Weight Sum          1  1  1  2  2  3  3  4  6  7 11 11 10  9  8 11 10 13 16 16 16 12  8  4

As you can see, US activity overlaps the UK's from 1pm to 8am (19 hours) with US lunchtime/evening peak periods having greatest effect during UK times 6pm to 9pm (3 hours) and 11pm to 6am (7 hours) respectively. I've noticed that, when UESP timeouts DO occur, they typically start around 6pm and 11pm UK time. And, funnily enough, Weight Sum suddenly increases at those times, indicating high activity which I'm assuming is mainly from (on-line) Gaming!

Timeout: Page loading status stays at 0% progress until Firefox times out and indicates "unable to contact web server". Yesterday after 6pm, for example, UESP suddenly started acting up, taking up to 20 seconds for the web server to respond. Still within my 30 second Firefox timeout, though. Tonight, I haven't seen delays longer than about 10 seconds.

SQL Error: Message appears at top of UESP web pages. Unfortunately, it's not helpful in pinpointing the problem because "error" could mean anything, i.e. SQL Server timeout, SQL command syntax error, etc.

Three days ago, I upgraded from Firefox v3.0.x to v3.5.2 and browsing is now significantly faster, so maybe it's good riddance to timeouts? Then again, it may be too early to tell as yet.

Hope this helps.

[Cactus]

I'll admit, I've never paid much attention to times or patterns, but I've gotten timeout errors and "Server didn't respond" errors several times. Sometimes I have to just leave and come back later because the site won't load. This has always occurred for me though on UESP and the forums, since I first joined. Only the SQL errors, which I rarely see, are new. I did get a SQL error yesterday when I tried to add a custom field to user profiles, however. Every time.

[H8Ball]

I spotted something of interest tonight. Take a look at this TraceRoute result from my home in the UK to the UESP Web Server in Montreal, Canada:-

Code: Select all

     Bounce Time (ms)
Hop  Pkt1  Pkt2  Pkt3  Router                                       IP Address      Location
---------------------------------------------------------------------------------------------------------------
1     <1    <1    <1   <My ADSL2 Router>                            x.x.x.x         Newcastle-upon-Tyne area, UK
2     60    42    74   cr0.new.uk.easynet.net                       87.87.248.209   Easynet Public Gateway
3     40    41    40   ip-87-87-137-65.easynet.co.uk                87.87.137.65    Easynet
4     57    58    59   bu1.er10.enslo.ov.easynet.net                83.231.199.162  Easynet OUT
5    163   207   220   po-1.r00.londen01.uk.bb.gin.ntt.net          83.231.199.161  London, UK
6    201   228     *   xe-4-2.r00.londen03.uk.bb.gin.ntt.net        129.250.2.237   London, UK
7     58    54    55   if-2-1.icore1.LDN-London.as6453.net          195.219.83.21   London, UK
8     56    54    55   195.219.243.1                                195.219.243.1   UK Atlantic Gateway
9    141   129   139   195.219.243.30                               195.219.243.30  US Atlantic Gateway
10   129   129   128   if-4-0-0-1262.mcore4.NYY-NewYork.as6453.net  209.58.62.2     New York, US
11   136   136   136   if-1-0.mcore3.MTT-Montreal.as6453.net        216.6.114.17    Montreal, CA
12   141   143   143   Vlan19.icore1.MTT-Montreal.as6453.net        216.6.114.14    Montreal, CA
13   137   136   137   te6-2.cl-core04.teleglobe.mtl.iweb.com       206.82.135.30   iWeb = UESP hosting?
14   136   136   136   ip-72-55-174-245.static.privatedns.com       72.55.174.245   UESP Web Server

I've re-run the above trace multiple times today from 4pm (UK time) onwards and the route to UESP has remained EXACTLY the same every time, always via the two NTT routers in London (Hops 5-6) despite them being frequently bottlenecked (high Bounce Times) and even timing out (* symbol). If these each only have ONE neighbouring router, NO alternate routes are possible, so the bottleneck can't be bypassed. This has the knock-on effect of UESP slow-downs and, if they're occasionally "gridlocked" for minutes at a time, UESP timeouts.

So UESP is not to blame. Instead, I can blame NTT (Nippon Telegraph & Telephone Corporation, apparently). Any chance of installing parallel routers to ease the load, NTT guys?