Sometime after 2 p.m., we noticed the site was running slow. At a little before 4:30, I opened a ticket with our hosting service. That is pretty much the precise moment the site went completely offline.
Servers going down is part of the business and I wasn't too worried. They go down for a bit, get a swift kick from a tech, and start running again. After a couple of hours, I thought it really unusual that it wasn't up and running again but still expected we would be back online soon.
When we weren't back up after five hours, I was pretty unhappy and stopped dealing with support by email and got them on the phone. They got two senior engineers on the phone with me and I got my IT guy on the phone.
The four of us were on the phone together for more than four hours. We can't find the problem. There are no errors in any of the logs. Individually, every component of the server is handling requests as expected.
We did notice that pages with a single story load immediately. Pages with just a few stories load slow but they load.
So we've cut The Batavian's home page back from 20 stories to just 10. The fact that you're seeing this post shows that, yes, you can once again get to the home page. But it does have only 10 stories on it. It's not possible, it seems, to use the "next" button at the bottom of the page to get past that point.
One of our most popular sections is obituaries and that section is loading just fine.
We're all going to bed and will work on this issue more in the morning.
UPDATE 9:30 a.m.: The issue appears to be resolved. It turns out there was an old, no-longer-used connection to facebook.com, that was timing out. It's been disabled and everything seems back to expected performance.
After working in the computer
After working in the computer maintenance field since 1970s, and solving many tricky problems, I can say with confidence that you may never solve the reason the server slowed down and stop servicing requests. These are extremely complicated machines with the slightess deviation from the normal can bring it down to it's knees. All you can do is hope that it is a non recurrenting problem as all the logs showed nothing. This is why they run in an air conditioned room with clean and conditioned electrical power.
Live by the server, die by
Live by the server, die by the servier.
I'm glad you figured it out,
I'm glad you figured it out, Howard.
I thought it was something I did when I logged in yesterday.
When I went to log out, my tablet just "froze. But, it was only frozen on your website.
Everything else seemed to work fine.
At 3:30 A.M., I still couldn't log off, so I figured, "Oh well. I'm a permanent captive of thebatavian."
(I haven't tried logging out yet, so that may still be true).
I don't feel bad, though. For some reason, my tablet hasn't been able to log ONTO the Batavia Daily News website for almost 4 months now, so you are my SOLE news connection to the Genesee area.
#AintTechnologyFrustrating