Earlier Downtime

Downtime

As you likely noticed, JKhub was down for a period of approximately 20 hours beginning around 5:00 PM EST yesterday and ending around 1:00 PM EST today.

What happened

I attempted to update the forum's software to a new minor version which contained bug fixes and some new features. Before I began the update, I contacted Caelum for instructions on how to properly carry out the update. I also downloaded backups just in case sometime went wrong. I performed the update yesterday afternoon following the various instructions. After the update was finished, I noticed that the layout of the site was completely broken.

I turned the forums offline to avoid confusion among visitors because of the broken layout. I also sent messages to both Caelum and the JKHub contact email notifying them what went wrong.

What happened was, I automatically merged JKHub's skin with the default skin so JKHub's would be updated with any new changes the software update brought. However, after everything broke and contacting Caelum, I realized that I should have manually merged them, not automatically merged them.

I attempted to restore from a backup, however because my upload speed is so slow, it was going to take around 9 hours or so before it would upload. Once Caelum realized this, he decided that he would restore from the backup after he got some sleep. Caelum ended up updating the forum's software himself.

How this will be prevented in the future

I will find out which willing staff member has the fastest upload and download speeds, so in the event a backup needs to be restored, it can be done so quickly. I will also be more careful with software updates, which includes notifying both the staff and community a few days in advance, deciding if an update is truly necessary, and carefully reviewing the necessary steps to take in updating the software.

I am very sorry about this incident, and should have planned more carefully and in advance for the update, which should have included a realization that my upload speeds aren't fast enough to restore a backup.