Vicarious Visions GOB Tools (Editing the console version of the game!)

[eezstreet]

I've been working on tools to work with the Xbox version of the game, notably dealing with the GOB archive format that is present instead of the .pk3 format.

Here's a small preview of the different tools and what they do:

• gobextractor.exe: Extracts the contents of a VV .gob file, provided the gob and gfc file. Prints all of the info about the files extracted as well, including compressed size, file name, and date modified.
• gobconvert.exe: Converts a ZIP/PK3 archive into a gob and gfc archive of the same name.

   

Different games use different codec groups. jediCodec for instance supports zlib and null codecs, but other games may require different codecs groups with different codecs, like TAR.

Here's a list of currently supported codec groups:

• jediCodec: Supports STAR WARS: Jedi Knight: Jedi Academy on Xbox, STAR WARS: Jedi Knight 2: Jedi Outcast on Xbox, and STAR WARS: Jedi Knight 2: Jedi Outcast on Gamecube.

Since jediCodec is the only one supported, all of the tools assume jediCodec. If anyone knows of other games that use this format (Crash Bandicoot?) then there might be new codecs needed.

 

https://jkhub.org/files/file/3234-%7B%3F%7D/

[Teancum]

Over the Hedge, Doom 3, and Crash Team Racing (I think) all use GOB. It's awesome that you're doing this. Here's hoping to be able to port content-based mods.

[minilogoguy18]

When this works can someone extract the demo movie that plays when the controller is left idle at the menu on JA? It's a very neat little gameplay demo.

[eezstreet]

It seems like each GOB file might have its own magic number. Anyone want to crack open DOOM3 etc and see what the GOB file's first 4 bytes are?

[eezstreet]

I got it to print out a list of all of the files in the JA Xbox version. I don't have access to a JK2 Xbox version, and the Gamecube JK2 version has some odd quirks (the file count is absurdly high, so there's something obfuscating the actual read functions that are going on..and i don't have a clue how the gamecube version actually handles this stuff) which are making it hard to work with.

 

Here's the metadata that got printed out:

https://www.dropbox.com/s/cv0pzelvgwxt1j1/ja_xbox.txt?dl=0

[eezstreet]

Released first version: https://jkhub.org/files/file/3234-unofficial-vicarious-visions-gob-tools/

 

• gobextract: a tool for extracting content from .gob/gfc files.

[eezstreet]

Dude. Holy fuck.

 

Wrote gobconvert which converts a PK3 directly into a GOB and GFC, and now I can understand why they decided to use this format. As a baseline, I'll use hdassets0.pk3 from JK2:Enhanced.

• No compression: 23,261,184 bytes (0% compression ratio)
• PK3/Minizip compression: 18,108,416 bytes (22.15% compression ratio)
• GOB file (jediCodec, bitmask 1): 1,249,280 bytes (94.6% compression ratio)
• RAR ("best" quality): 528,679 bytes (97.72% compression ratio)

GOB is pretty damn good, suprisingly.

 

Next version will be coming out soon hopefully. I just need to finish up some stuff and it'll be good to go.

[Teancum]

Nice! Really glad you're taking the time to do this.

[Lancelot]

Wait, what? I can't believe that this is actually happening. That is awesome!

 

This has so much potential. Just think of all the possibilities we could have with that.

If it will be possible to replace many of the standard models with higher quality versions, like the Stormtroopers, that would be cool.

[Teancum]

Dude. Holy fuck.

 

Wrote gobconvert which converts a PK3 directly into a GOB and GFC, and now I can understand why they decided to use this format. As a baseline, I'll use hdassets0.pk3 from JK2:Enhanced.

• No compression: 23,261,184 bytes (0% compression ratio)
• PK3/Minizip compression: 18,108,416 bytes (22.15% compression ratio)
• GOB file (jediCodec, bitmask 1): 1,249,280 bytes (94.6% compression ratio)
• RAR ("best" quality): 528,679 bytes (97.72% compression ratio)

GOB is pretty damn good, suprisingly.

 

Next version will be coming out soon hopefully. I just need to finish up some stuff and it'll be good to go.

GOB support for OpenJK confirmed?

[eezstreet]

GOB support for OpenJK confirmed?

It's a proprietary format so most likely not.

[eezstreet]

v2 of the GOB tools is out now, it has gobconvert in it. I think that's about all that people should need to do console modding. Eat your hearts out etc

[Teancum]

It's a proprietary format so most likely not.

Just a joke of course.

 

This is perfect, though, and I thank you for that. I can extract all my assets, rebuild it as assets.pk3, add mods to the pk3, then convert to gob and test. Couldn't ask for more.

[eezstreet]

Just a joke of course.

 

This is perfect, though, and I thank you for that. I can extract all my assets, rebuild it as assets.pk3, add mods to the pk3, then convert to gob and test. Couldn't ask for more.

Keep in mind, you'll need to make some modifications that are specific to the console you're working with. So as a result, you'll need to edit a _console_dir_list_ file in models/players (although I'm not sure this is actually used by the game) and you'll need to make sure your textures are the correct format for the console - DDS for xbox and TDF for gamecube. For maps you'll need to break the BSP into its component lumps (the code for it exists in the xbox code).

[Teancum]

Yeah, I have BSPthing compiled to split the BSP. It works, but errors out after the last split. Not sure if there's extra data at the end or of there's an actual error. I'll have to check. So specifically Xbox textures need to be DXT3 DDS, max of 256x256 (small, I know). In other games loadscreens can be 512x512, but I'll have to test that as well. Sounds are banked separately, so if people are looking for the ability to dump them into a pk3-->gob, no can do (they live outside the gob in their own archive). Tools might already exist if it's a proprietary Xbox format, but I'll have to look into it more.

 

I did try using an Xbox .glm on PC, but it crashed, so I'm unsure if PC .glms will work on Xbox. I'll see what Modviewer shows (I think that's what it's called -- the model viewing app?). I'm wondering if the skeleton wasn't further simplified or something. Vicarious Visions optimized several other formats for other ports they did, like Doom 3. If nothing else code comparison for Ghoul2 between OpenJK and the Xbox code should provide some answers, but I'll try the simplest solutions first. I think it's only fitting to try the Jedi Academy Bonus Map Pack first.

 

So speaking of maps, I'm curious -- will a Jedi Outcast MP map play in Jedi Academy without any modifications? I feel like I remember issues with bots, but it's been AGES since I've messed with it. There are a few mod maps from JO I want to play.

[eezstreet]

Yeah, I have BSPthing compiled to split the BSP. It works, but errors out after the last split. Not sure if there's extra data at the end or of there's an actual error. I'll have to check. So specifically Xbox textures need to be DXT3 DDS, max of 256x256 (small, I know). In other games loadscreens can be 512x512, but I'll have to test that as well. Sounds are banked separately, so if people are looking for the ability to dump them into a pk3-->gob, no can do (they live outside the gob in their own archive). Tools might already exist if it's a proprietary Xbox format, but I'll have to look into it more.

 

I did try using an Xbox .glm on PC, but it crashed, so I'm unsure if PC .glms will work on Xbox. I'll see what Modviewer shows (I think that's what it's called -- the model viewing app?). I'm wondering if the skeleton wasn't further simplified or something. Vicarious Visions optimized several other formats for other ports they did, like Doom 3. If nothing else code comparison for Ghoul2 between OpenJK and the Xbox code should provide some answers, but I'll try the simplest solutions first. I think it's only fitting to try the Jedi Academy Bonus Map Pack first.

 

So speaking of maps, I'm curious -- will a Jedi Outcast MP map play in Jedi Academy without any modifications? I feel like I remember issues with bots, but it's been AGES since I've messed with it. There are a few mod maps from JO I want to play.

Yep, JK2 MP maps will work in Jedi Academy as long as the textures and shaders exist for it. SP maps, not so much. You can see my efforts in reverse engineering the JA SP game prior to the source release to load the JK2 SP maps with the JK2:Uncut mod:

https://jkhub.org/files/file/1018-%7B%3F%7D/

https://jkhub.org/files/file/1024-%7B%3F%7D/

[Archangel35757]

try importing the xbox GLM/GLA with Noesis or Blender if ModView fails you... but you'll need to point the xbox glm to use its own xbox gla in modview... like you said they could have modified some bones-- email them to me and I can try as well. I doubt you will... but if you find any root.xsi files in the xbox stuff then send me that too.

[Teancum]

So first run of an updated assets.gob froze the console. I got to the boot screen, but it never fully loaded the game. In looking the updated archive is roughly 1/5 of the original archive, and that's with adding a few things. Are there compression level settings in the code? Perhaps it compressed too much. I'll test with a simple extraction/recompress --> no new files.

 

***EDIT***

 

So using a clean copy of the files produces the same result. The file controller (.gfc) matches the original in bytes, but the game doesn't seem to like the .gob archive itself. My first guess is compression level. For comparison, the original file sits at exactly 222,932,992 bytes, while the new, recompressed version is 52,457,472 bytes.

[eezstreet]

I used simply zlib's compress() function. It must not play nice with that for some reason.

Try using gobconvett and then gobextract to verify all of the files got added.

[Teancum]

Hmm, that's not it. Both the original extraction and the re-extraction have the same number of files/folders taking up the same number of bytes. I verified this against using QuickBMS and the Vicarious Visions script. All three results were the same number of files/folders as well as the same byte size after extraction.
 

***EDIT***

 

At least you know the executable works correctly. I wonder if its a version-specific thing with zlib. Are you using the original one in the codebase or one of the updated versions?

[eezstreet]

zlib's compression algorithm doesn't change across versions, it's the same every time, otherwise you'd get issues with extraction for every application that uses zlib (and there are a lot).

I tried it with all 9 compression levels and it never produced anything that had the same size as the original archive, always either smaller (with compression) or larger (without compression).

I ran `gobextract <> <> -noextract -echodata -debug` to see if there was any differences between the two files and they both validated correctly, although the converted assets had backslashes and a leading dot in front of them, so for example:

 

Mine:

botfiles/alora.jkb

Theirs:

.\botfiles\alora.jkb

That alone couldn't cause freezing though, because the filesystem is smart enough to handle the paths correctly (at least this is the case on the pc version)

 

EDIT: Oh yeah, nope. That would definitely cause a problem. The filename is hashed and compared to other files in the file control table...that was my bad

I'll make some modifications and do a third revision, with a couple of planned features:

• gobconvert: ability to specify how compressed you want it to be with -level=X (scale of 0 to 9, 1 default, 0 for totally uncompressed)
• gobextract: drag and drop support, output argument optional (defaults to ./out/)

EDIT #2: In the meantime @@Teancum, could you test this for me? It should have the correct paths:

https://www.dropbox.com/s/aozovacngesuyv5/gobconvert_test.zip?dl=0

[Teancum]

Yep. I'll hop on that tonight when I get home.

[Teancum]

So I took the updated build for a spin. It still locked up the console. I don't know if it's the compression level or what, so I'll try out your version with compression settings when it's up and see what we have.

[eezstreet]

So I took the updated build for a spin. It still locked up the console. I don't know if it's the compression level or what, so I'll try out your version with compression settings when it's up and see what we have.

Are you running it on a physical device or are you using an emulator (XDK?)

[Teancum]

I'm running on a 360 through backwards compatability