Jump to content

Syko's JKfiles Mirror


Recommended Posts

I am actually thinking about that. All of the mods total about 55GB if I remember correctly and I could easily store that on one of my computers. The only issue is that there are about 9500 mods on JKfiles and there is no way I'm going to manually download each one of those. I'm sure a script could be written to do that somehow, except I would have no idea how to write one.

 

Plus, I don't know where I could upload 50>GB of data for free. My Box account now has a max of 50GB and I wouldn't want to use all of that up anyway. I do have a thing where if I invite other people to Box though, they get 50 GB of data for free as well. Maybe I could do that and upload all of the files there.

 

Anyone else want 50GB of free storage on Box.com?

yeyo JK likes this
Link to comment

I certainly agree that the mods are of greater importance, but again, I'm not going to manually download 9500 files. Maybe three to four different people including myself can download certain types of mods and upload them. One guy could download all the skins/models for example, then one guy all the maps, etc. and then we could all upload them to the same source. Of course, some script may be able to be used but I have no idea how to create one. We could organize them into folders, etc.

 

You think that stuff is worth saving...? O.o

Seriously, I wrote quite a few of the reviews myself and I still don't think they are really worth saving. <.<

 

Well, in my opinion JK3Files itself represents a large part of the community, or what remains of the community. Losing it would be like loosing everything the JK modding culture had/has. I enjoyed browsing through the site as much as I did playing with the mods (and making them for that matter). 

Link to comment

Ask @@spior to code a bot that scours the site for downloads.

In theory, a bot for this wouldn't be complex, it would just be tedious and hard to stress test, especially with all that's going on. My advice for now would be to DL all files that are after the site breakiness.

Link to comment

Ask @@spior to code a bot that scours the site for downloads.

In theory, a bot for this wouldn't be complex, it would just be tedious and hard to stress test, especially with all that's going on. My advice for now would be to DL all files that are after the site breakiness.

I'M ON IT!

Just as soon as you go into more detail :P

Link to comment

We still need to figure out hosting though. Perhaps it could be uploaded to JKHub? I could create a mirror on another Box account. I don't think that should be the main download source though.

Link to comment

I would like to help with this, but I don't have much space left on my hardrive and I only have 10GB of bandwidth for each month :/

 

But I'm thinking of getting a 2TB hardrive anyway, and I could download files from one my school's computers on some days after school when I have to crash time for about an hour and a half.

 

Either way, I think this is a brilliant idea!

Link to comment

While it would be awesome to have an archive of the files, wouldn't it technically violate the modder's terms of use? Some didn't give permission for their work to be re-distributed...

...maybe but...it might be worth ignoring that for the purpose of this backup.  Since at this point...we wouldn't technically be violating the terms of use yet, since this backup won't be distributed...yet.

JKG Developer

Link to comment

Maybe instead of distributing the entire mod database, spior could distribute his bot script so we could all download the mod database ourselves. I don't think we'd need authors' permissions because the mods themselves would not be publicly distributed.

Link to comment

Don't know if anyone already knew about this, but from this link, if you select Sort by A-Z it gives you the list of all mods in alphabetical order on the JK3 Files site.

 

For JK2: 

http://www.gamefront.com/files/listing/gamingfiles/Jedi_Knight_II

 

For JKA: 

http://www.gamefront.com/files/listing/gamingfiles/Jedi_Knight_III

 

I don't know if that will make bot coding any easier. 

Szico VII and eezstreet like this
Link to comment

Should make @@spior's bot (if he's making one, I dunno if he took my suggestion seriously) run a lot more efficiently.

I might have a go at it, but you'd need to get into more details (IRC). Like, download where, download how, find out what to download how, etc.

Link to comment

Hmmk. In a nutshell, it would just download everything off the site via some HTTP transfers. Preferably done on a loop, each iteration being a file. Brownie points for making it multithreaded. The best way I can think of doing it multithreaded involves chucking all the downloads into a queue, and then assigning each thread an index in the queue.

 

You'd need to parse the JKFiles page for the div tag (or...shudder...table tag) that contains the download. It follows a fairly generic setup, and what you're looking for is the page link to Filefront. From there, it sends...something to the client to invoke a download prompt on the browser. You'd want to intercept this and make it chuck all the .zips into the queue.

 

The main thing we need to do before the big pull takes place is put in some kind of blacklist of files, to prevent files we don't want from being downloaded. But before the big download takes place, you'll want to try it out on one or two files first anyway.

 

That's the basic gist of it anyway.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...