Syko's JKfiles Mirror

[spior]

Sounds easy enough. Except the part where there's a timer in FF's download page. I can probably just get the link directly (I hope).

[eezstreet]

I figured that would be the tricky part. You could bypass that and go for the direct download link on the page though by waiting for the correct amount of ms.

[spior]

I figured that would be the tricky part. You could bypass that and go for the direct download link on the page though by waiting for the correct amount of ms.

This isn't how it works

Know all those popup windows on sites like Facebook? They're not real buttons either, they're just hidden and when you click them they show up.

It's all a lie, man, IT'S ALL A LIE!

My point is, the link is probably in the page source.

[eezstreet]

This isn't how it works

Know all those popup windows on sites like Facebook? They're not real buttons either, they're just hidden and when you click them they show up.

It's all a lie, man, IT'S ALL A LIE!

My point is, the link is probably in the page source.

It is. As it turns out, you can just avoid the countdown, that's what I'm getting at here. It says "If it does not, click here". The "here" in the link can be extracted with no real issue.

[spior]

It is. As it turns out, you can just avoid the countdown, that's what I'm getting at here. It says "If it does not, click here". The "here" in the link can be extracted with no real issue.

Exactly my point - this isn't how it works. Bots have no sense of time, they can't just 'wait'.

The 'click here' button though works fine, ya.

[Syko]

I added a mirror to the site download on the first post. I'm going to add one or two more eventually. 

 

EDIT: Everything that was there before is now a mirror now. The main download link is going to be the whole thing in one file on Dropbox.

[Syko]

Erm...any progress on that bot? It's been 5 days now. 

 

I'm getting a 3 TB external hard drive at the end of the month. A complete JK3Files mirror will be the perfect thing to put on it.

[spior]

Erm...any progress on that bot? It's been 5 days now. 

 

I'm getting a 3 TB external hard drive at the end of the month. A complete JK3Files mirror will be the perfect thing to put on it.

Been kinda busy with school and all. I guess I can start now.

Also @poke me next time.

[Syko]

So after rebrowsing the JK3files mirror I had created, I found that I actually only had about 80% of all the pages. I took the liberty to redownload the entire website again and this time 99.9% of the pages are saved. I'm going to reupload everything on Box again as a zip file. I'm also going to see about uploading the HTML files themselves, so you can browse the site mirror in your browser directly.

 

The new download when extracted will total 4.9 GB (as opposed to 3.53 or whatever the previous one was)

[spior]

So uh, I have some progress going. Right now I can get the file PAGE links from all categories' first page (but not sub-categories).

Should I continue with this? @@Syko @@eezstreet ?

EDIT: Also another thing - every time you run this, it'll download all the files all over again. I suppose I can try and figure out some sort of check to see if the file already exists, but yea, you tell me. Is this gonna be a 1-runner or what?

[eezstreet]

I dunno, ask Syko.

[Syko]

Sorry, was gone all day today. Just saw your message now.

 

It should be OK if it downloads all of the files over again. It should be a "one-runner" like you said. 

[spior]

Sorry, was gone all day today. Just saw your message now.

 

It should be OK if it downloads all of the files over again. It should be a "one-runner" like you said. 

Okay then.

[Syko]

@@spior Is the program/bot in a usable state yet? If so, I'd like to test it a bit. Also, are the mods organized by folder or are they all thrown into one folder? It would be cool if they could be added to work with the site mirror so if you click on

 

 

It actually 'downloads' the file directly right there (effectively copying the file from the user's local mirror folder to their Downloads folder). It might be better that way because the user could read the review and screenshots right there instead of having to find the corresponding file in some folder they downloaded from the bot. I'm not sure how that would be possible though. Your bot would have to scan through the mirror's entire folder and edit the HTML files, editing the links and also would have to move the previously downloaded mods into their corresponding folders in the mirror folder.

[eezstreet]

The bot just scours the site and downloads things. It isn't interactive at all.

[spior]

I can just make the downloads folder user-settable.

No, not done yet. I have school and I don't feel like programming after school much, sorry.

Good news is today was the last day before vacation so I have whole week to work on it.

For now it gets all the links from all the categories and sub-categories so it shouldn't take /too/ long to finish it.

[spior]

I'd just like to point out 2 things

1) If things don't decide to break, this will probably be done very soon

2) the guy who made gamefront should kill himself. 4 (I REPEAT - FOUR) REDIRECTS JUST TO DOWNLOAD 1 FILE? FUCK YOU

[Syko]

Glad to see some progress! In several years the JK community will thank all of us for doing this. 

 

Also, If you don't mind, after the bot is finished I'd like to have a look at the source code (unless you want to release the source anyway.) I think it would be a good learning experience for me in programming.

[spior]

Glad to see some progress! In several years the JK community will thank all of us for doing this. 

 

Also, If you don't mind, after the bot is finished I'd like to have a look at the source code (unless you want to release the source anyway.) I think it would be a good learning experience for me in programming.

Oh lol.

Yeah I do plan on making it open source. I need to practice my git and this seems like a good opportunity.

Have fun understanding anything though.

[CrimsonStrife]

Oh lol.

Yeah I do plan on making it open source. I need to practice my git and this seems like a good opportunity.

Have fun understanding anything though.

I get the feeling someone doesn't comment their code lol

[spior]

I get the feeling someone doesn't comment their code lol

Oh I do.

But I just like to jam as much as code as I can in one place.

 

 

 

while(t = strstr(t++, "<a class=\"size11\" href=\"/file/"))

{

char temp[MAX_FILE_LEN] = { 0 };

t+=strlen("<a class=\"size11\" href=\"");

strncpy(temp, t, strstr(t, "\"")-t);

if(*num_file == 0 || strncmp(temp, files[(*num_file)-1], strlen(files[(*num_file)-1])) != 0)

strcpy(files[(*num_file)++], temp);

tt = t;

}

 

 

[spior]

1) If things don't decide to break, this will probably be done very soon

 

things broke. I keep getting redirected in circles. Might take a while to understand why this is happening since I don't know any of the GF staff.

 

Okay I think it's done. Gonna do some small tests and link it to you if all goes well @@Syko

Actually hold on. I wanna make this look pretty

[Syko]

@@spior is it a command line program or does it have a GUI?

[spior]

@@spior is it a command line program or does it have a GUI?

command line. I'm not 1337 enough to make proper GUIs yet.

[Syko]

Are you going to have other people test it first or are you just going to release it? 

 

P.S. PM me a copy of the program. I need to fill up that 2TB drive space I have!