Syko's JKfiles Mirror

[Syko]

http://mrwonko.de/jk3files

[Mysterious Stranger]

You aren't the first to be paranoid lol. I fell for one of their April Fools' trick once that they were closing and downloaded half...

[Raz0r]

You aren't the first to be paranoid lol. I fell for one of their April Fools' trick once that they were closing and downloaded half...

That wasn't an April Fools trick. Ziff Davis Media went bankrupt, FileFront was eventually acquired by Break Media

[Mysterious Stranger]

well, in the end it still stayed up lol.

[Futuza]

So are you going to grab all the mods at one point as well?

[Syko]

I am actually thinking about that. All of the mods total about 55GB if I remember correctly and I could easily store that on one of my computers. The only issue is that there are about 9500 mods on JKfiles and there is no way I'm going to manually download each one of those. I'm sure a script could be written to do that somehow, except I would have no idea how to write one.

 

Plus, I don't know where I could upload 50>GB of data for free. My Box account now has a max of 50GB and I wouldn't want to use all of that up anyway. I do have a thing where if I invite other people to Box though, they get 50 GB of data for free as well. Maybe I could do that and upload all of the files there.

 

Anyone else want 50GB of free storage on Box.com?

[eezstreet]

This isn't going to back up the files using that backup, its just going to back up the pages linking to them.

also, you should ignore all files by Ful'CruM.

[Ryojin]

including all reviews, screenshots, polls, POTDs, etc.

You think that stuff is worth saving...? O.o

 

Seriously, I wrote quite a few of the reviews myself and I still don't think they are really worth saving. <.<

[Szico VII]

The files are what are really worth saving, although it'd be nice to have a record of all my old reviews there.

 

But yes I think all the above is worth saving, definitely.

[CaptainChar]

at the time I had 50% of the site on my HDD and dvd backups already, since I grabbed most of the mods to begin with, I have some that were even removed due to some "issues"

[Syko]

I certainly agree that the mods are of greater importance, but again, I'm not going to manually download 9500 files. Maybe three to four different people including myself can download certain types of mods and upload them. One guy could download all the skins/models for example, then one guy all the maps, etc. and then we could all upload them to the same source. Of course, some script may be able to be used but I have no idea how to create one. We could organize them into folders, etc.

 

You think that stuff is worth saving...? O.o

Seriously, I wrote quite a few of the reviews myself and I still don't think they are really worth saving. <.<

 

Well, in my opinion JK3Files itself represents a large part of the community, or what remains of the community. Losing it would be like loosing everything the JK modding culture had/has. I enjoyed browsing through the site as much as I did playing with the mods (and making them for that matter). 

[eezstreet]

Ask @@spior to code a bot that scours the site for downloads.

In theory, a bot for this wouldn't be complex, it would just be tedious and hard to stress test, especially with all that's going on. My advice for now would be to DL all files that are after the site breakiness.

[spior]

Ask @@spior to code a bot that scours the site for downloads.

In theory, a bot for this wouldn't be complex, it would just be tedious and hard to stress test, especially with all that's going on. My advice for now would be to DL all files that are after the site breakiness.

I'M ON IT!

Just as soon as you go into more detail

[eezstreet]

You'd want to write a bot that:

1. Detects if you're on the JK2Files main server (the one that lists Luke - Splinter of the Mind's Eye as one of the newer files)

2. Scours each category and downloads each file.

[Syko]

We still need to figure out hosting though. Perhaps it could be uploaded to JKHub? I could create a mirror on another Box account. I don't think that should be the main download source though.

[Omicron]

I would like to help with this, but I don't have much space left on my hardrive and I only have 10GB of bandwidth for each month :/

 

But I'm thinking of getting a 2TB hardrive anyway, and I could download files from one my school's computers on some days after school when I have to crash time for about an hour and a half.

 

Either way, I think this is a brilliant idea!

[therfiles]

While it would be awesome to have an archive of the files, wouldn't it technically violate the modder's terms of use? Some didn't give permission for their work to be re-distributed...

[Futuza]

While it would be awesome to have an archive of the files, wouldn't it technically violate the modder's terms of use? Some didn't give permission for their work to be re-distributed...

...maybe but...it might be worth ignoring that for the purpose of this backup.  Since at this point...we wouldn't technically be violating the terms of use yet, since this backup won't be distributed...yet.

[Fighter]

Perhaps it could be uploaded to JKHub?

I don't think we could do that unless the permission of every single author was given to upload their files to JKH.

[Syko]

Maybe instead of distributing the entire mod database, spior could distribute his bot script so we could all download the mod database ourselves. I don't think we'd need authors' permissions because the mods themselves would not be publicly distributed.

[eezstreet]

IMO, just grab the good/remotely interesting ones. No need wasting time over someone's very poor quality Reborn reskin.

[Syko]

Don't know if anyone already knew about this, but from this link, if you select Sort by A-Z it gives you the list of all mods in alphabetical order on the JK3 Files site.

 

For JK2: 

http://www.gamefront.com/files/listing/gamingfiles/Jedi_Knight_II

 

For JKA: 

http://www.gamefront.com/files/listing/gamingfiles/Jedi_Knight_III

 

I don't know if that will make bot coding any easier. 

[eezstreet]

Should make @@spior's bot (if he's making one, I dunno if he took my suggestion seriously) run a lot more efficiently.

[spior]

Should make @@spior's bot (if he's making one, I dunno if he took my suggestion seriously) run a lot more efficiently.

I might have a go at it, but you'd need to get into more details (IRC). Like, download where, download how, find out what to download how, etc.

[eezstreet]

Hmmk. In a nutshell, it would just download everything off the site via some HTTP transfers. Preferably done on a loop, each iteration being a file. Brownie points for making it multithreaded. The best way I can think of doing it multithreaded involves chucking all the downloads into a queue, and then assigning each thread an index in the queue.

 

You'd need to parse the JKFiles page for the div tag (or...shudder...table tag) that contains the download. It follows a fairly generic setup, and what you're looking for is the page link to Filefront. From there, it sends...something to the client to invoke a download prompt on the browser. You'd want to intercept this and make it chuck all the .zips into the queue.

 

The main thing we need to do before the big pull takes place is put in some kind of blacklist of files, to prevent files we don't want from being downloaded. But before the big download takes place, you'll want to try it out on one or two files first anyway.

 

That's the basic gist of it anyway.