Today was incredibly busy. Just getting through my email and associated tasks, given interruptions and meetings, ended up taking me until 3:30pm. I remember a couple of years ago, George observed me answering a question, while helping someone while on the way to do something, with two other people standing in line. He asked me later, "Just how deep is your stack?"
I got an email from UMassOnline today -- evidently for three hours last night they were UMassOffline. The email was a studied portrait in providing a lot of detail without actually saying anything.
The outage was due to a combination of factors. We have developed an infrastructure designed to maximize uptime in the face of all manner of disruptions. It involves four multiple processor high capacity redundant servers, load-balancers, a storage area network for data storage and restoration, uninterruptable power supplies, and direct connection to a high-bandwidth statewide network. Unfortunately, the combination of disruptions that occurred last night was not one that we had foreseen. We had a series of server failures, in part related to the application, in part related to the hardware themselves.
Uh-huh. Server failures related to the application and the hardware. Yep. Now I really understand what happened.
We've been pretty lucky with our servers. I've had minutes of downtime during the school year before. It happens. Usually it's my fault. Last fall, T and I restored a file into /tmp and changed the permissions on /tmp, which caused a bunch of stuff to fail, including PHPwiki -- it took me about 20 minutes to figure out what the problem was and reset the permissions.
This fall I'm getting involved with a course development project that has selected UMassOnline's Prometheus as their delivery platform. I will be glad for an excuse to actually try out Prometheus. My initial impressions, when I looked at it a year and a half ago, were that it is a very limiting environment. For that reason, I've never actually tried to use it for anything. I want to go into it with an open mind, however, and document just how difficult (or easy) it actually is to accomplish the tasks that I want to do in a course.