Jump to content

Earlier Downtime


Fighter

Recommended Posts

Posted Image

Downtime
As you likely noticed, JKhub was down for a period of approximately 20 hours beginning around 5:00 PM EST yesterday and ending around 1:00 PM EST today.

What happened
I attempted to update the forum's software to a new minor version which contained bug fixes and some new features. Before I began the update, I contacted Caelum for instructions on how to properly carry out the update. I also downloaded backups just in case sometime went wrong. I performed the update yesterday afternoon following the various instructions. After the update was finished, I noticed that the layout of the site was completely broken.

I turned the forums offline to avoid confusion among visitors because of the broken layout. I also sent messages to both Caelum and the JKHub contact email notifying them what went wrong.

What happened was, I automatically merged JKHub's skin with the default skin so JKHub's would be updated with any new changes the software update brought. However, after everything broke and contacting Caelum, I realized that I should have manually merged them, not automatically merged them.

I attempted to restore from a backup, however because my upload speed is so slow, it was going to take around 9 hours or so before it would upload. Once Caelum realized this, he decided that he would restore from the backup after he got some sleep. Caelum ended up updating the forum's software himself.

How this will be prevented in the future
I will find out which willing staff member has the fastest upload and download speeds, so in the event a backup needs to be restored, it can be done so quickly. I will also be more careful with software updates, which includes notifying both the staff and community a few days in advance, deciding if an update is truly necessary, and carefully reviewing the necessary steps to take in updating the software.

I am very sorry about this incident, and should have planned more carefully and in advance for the update, which should have included a realization that my upload speeds aren't fast enough to restore a backup.

Click here to view the article
Milamber, Szico VII, spior and 2 others like this
Link to comment

That does not include file uploads -- that's forum software, wiki software, etc. The file uploads are stored on Dropbox, not JKH itself. The file uploads total around 36 GB.

 

 

I thought it would be bigger tbh.

 

It's getting there gradually...I am the lucky bastard that gets to maintain an local backup of that monster, as I have the largest amount of space + bandwidth of the current staff....clocking in at roughly 4TB and on average a 30MBbit connection.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...