I present a challenge.

[Cerez]

@@DT85, @@Psyk0Sith, @@Xycaleth, @@eezstreet, @@Cerez... and anyone else-- what features would you envision/expect to see in a facial expression/lip-sync GUI? I'm open to suggestions while I'm conceptualizing the GUI.

 

I can't really visualise how a GUI will work with scripting cutscenes, but the slider solution you've mentioned earlier sounds like a pretty advanced way to go -- anything more than that would be too much detail, I think. Just having the basic 6 emotions expressed is already an enormous improvement on the default JKA engine. Add to that eye movement, body animation, and lip-sync, and you've got one lively character. 

 

The fixed eye position has been a major detriment in the original game to realism, and the overall lack of expression in the character's faces.

 

Visual breathing would be a nice addition to the idle animation, but I'm not sure if the Q3 engine would support something like that well without extensive expansion work.

 

The pupil animation is a very fine touch. I'm not sure if it's worth all the work, but I'm sure it would add a lot of depth to drama with extreme close-ups on the eyes.

 

The thing is, the Q3 engine does maintain a certain break from realism because of its old-school design, like the floating character walk, rough body animations, and other features. Therefore, my recommendation is actually to not spend too much time on very fine details, but to fill the void where there is really space for improvement -- like the added eye movement, improved body animations, and expanded facial expressions. If we can achieve a decent and believable level of detail in these areas, a feeling that the characters are alive will be created. The visual experience of the whole game world needs to be taken into account when considering how much detail is really worth it -- how much of it really makes a difference to the player. The player is in a suspension of disbelief based on the world they see in the game, and the way the game stylises real life.

 

Maybe that's not what you want to hear, but I'm trying always to stay practical. I think these facial animations will be an amazing expansion to the core engine. But how much detail do we really need? That's something we need to consider carefully, I think -- especially because it impacts performance.

[Archangel35757]

@@Cerez -- cinematic cutscenes will benefit the most... during actual gameplay I don't see these new facial expression features being utilized... except to perhaps put facial expressions to taunts.

[Cerez]

And when in fixed 3rd person view, yes. And by adding more depth to cinematic cutscenes, we are creating a more lively world for the game overall. But the extent to which we add detailed animation needs to be balanced with the rest of the visuals in the game, I think. Unless we are upgrading the whole look of the game. So to an extent things like detailed facial wrinkles on all characters, detailed pupil animations, etc. may be an overkill. 

 

If the animation/detail stands out too much from the rest of the game, then that doesn't help to make it more believable. 

 

Edit: Not to mention that the more detail in the shapable face, the more work (the longer it takes) for the scripter/animator to script the character expressions for each cutscene for the game.

 

Don't get me wrong, I admire the ambitious effort of putting as much detail in the facial animations as possible; I'm just not sure if for the game we need so much detail in the face. It may be better to simplify a little for the sake of blending the new animations into the game's existing world (style).

[Archangel35757]

@@Cerez, one of the goals of this JK Enhanced effort was to improve the visuals-- the new Rend2 renderer makes this possible now across the board... environments, assets, characters, etc.

[Cerez]

Oh, nevermind what I said, then. If the graphics and detail improvement is consistent across all aspects of the game world, then the suspension of disbelief is maintained, but with better (more detailed and realistic) visuals.

[eezstreet]

How far have things gotten with this? I had a brief thought about lip sync files earlier. I was thinking that there could be .lip files (formatted with the generic COM_Parse, same as shaders, vehicles, etc use) which could be used.

 

From what I recall, I was never going to use the modified skeleton in actual gameplay, just in cinematics. Ingame, Kyle can just use the floppy fish sticks talking method that's already ingame. Jan on the other hand, I wanted her to be as reactive and interesting as Elizabeth from BioShock: Infinite. So, Jan would probably be the only exception.

 

I have a vague interest in getting back at this while I have some downtime. My meds are making me have fairly severe brain fog however.

[Archangel35757]

How far have things gotten with this? I had a brief thought about lip sync files earlier. I was thinking that there could be .lip files (formatted with the generic COM_Parse, same as shaders, vehicles, etc use) which could be used.

 

From what I recall, I was never going to use the modified skeleton in actual gameplay, just in cinematics. Ingame, Kyle can just use the floppy fish sticks talking method that's already ingame. Jan on the other hand, I wanted her to be as reactive and interesting as Elizabeth from BioShock: Infinite. So, Jan would probably be the only exception.

 

I have a vague interest in getting back at this while I have some downtime. My meds are making me have fairly severe brain fog however.

 

So far, I've been working on finishing the Hi-Def mesh so that I can get to a weighted mesh and start building the 3ds Max Facial-Rig/Lip-Sync GUI. This GUI will use PRAAT (www.praat.org) for generating the phonemes from a provided sound file... I've written a function to read the PRAAT utf8 text file results for phoneme parsing. The 3ds Max GUI will provide a button to export the resulting phonemes & visemes into the file formats you requested. So in regards to the GUI and it's functionality-- it's still in the conceptual phase.

 

I could probably also export the facial bones into the other game format that uses that face editor (...for those not using 3ds Max).

 

My plan was to use as many of the legacy face bones as possible-- and only add new ones where needed.

 

EDIT: But if a model is weighted to the new face.GLA then it would be needed in game-- regardless if you have it do expressions or lip-sync during actual gameplay.

[Cerez]

I'm not sure if this is redundant, but what if there were two possible stages for the facial animation: one utilising the Raven facial bones (simplified animation for base JKA), and one using the new face.gla (advanced animation for JKE). Then we could have backwards support, and fall back on the Raven bones if the new face.gla bones are not present in the model.

 

I think it may be worth to invest time in backwards compatibility (for the sake of preserving everyone's work through all these years, and the sake of the adoption rate of the new animation system).

 

This means that new character models, optimised for the new animation system, will be more detailed in their facial animation, but that older models (whose authors are no longer active in the community) will still benefit from the new implementation (as much as possible with Raven's facial bones).

[Archangel35757]

@@Cerez-- this wouldn't break compatibility with legacy models (assuming their face was even weighted)... because the Raven facial bones are still there in the _humanoid.gla file.

[Cerez]

No, I meant backwards compatibility for the new facial animations -- if that's something you'd be interested in. Meaning that when designing the motion of the bones for the new facial expressions, the set of Raven default bones are used/interpreted as the key bones, and the rest as refinement bones.

 

That way the old models that have the mouth weighted will also make use of the new animations. (For example, a more stretched, natural smile instead of the "V" shaped goof.)

[eezstreet]

You will have to generate visemes for the Raven skeleton, otherwise the process is the same. The skeleton alteration isn't strictly speaking necessary to have the new lip sync system, but it sure does make it a lot less ugly.

[Archangel35757]

@eezstreetnew Jan Ors model almost finished... based on Angela Harry:

[Archangel35757]

future updates on my Jan Ors character model will be posted in this thread: