I present a challenge.

[Archangel35757]

@-- I don't think one would see any visual difference between the /(h)w/ and /w/ phonemes... like in the word pairs:

 

whine / wine

which / witch

etc.

 

They'll both use the same viseme pose. With my facial rig my goal is to make facial animations (lip-sync and expressions) easier and intuitive with the GUI I have planned.

 

With the approach @@eezstreet layed out in this challenge... the new lip_sync.gla will be appended on top of the _humanoid.gla using the head bone.

[Cerez]

@-- I don't think one would see any visual difference between the /(h)w/ and /w/ phonemes... like in the word pairs:

 

whine / wine

which / witch

etc.

 

I agree. I think only when/win, whurry/worry, wheel/will are critical here.

 

They'll both use the same viseme pose. With my facial rig my goal is to make facial animations (lip-sync and expressions) easier and intuitive with the GUI I have planned.

 

With the approach @@eezstreet layed out in this challenge... the new lip_sync.gla will be appended on top of the _humanoid.gla using the head bone.

 

This sounds like like a fascinating project. I wish good luck to the both of you and I look forward to what you can create. If you need any help with testing, etc., count me in for additional support.

 

Will this project be freely available for others to integrate in their mods/projects once finished? I would be interested in seriously utilising it in an up-coming new game/mod I have planned, and in most of my cinematics work.

[Archangel35757]

@-- my full 3ds Max character rig will be released for others to use... on the coding side of things, @@eezstreet would need to answer that.

[Tempust85]

Regarding the needed facial bones, I'd suggest taking a look at what Blizzard have done for their Orc:

 

 

As you can see, there's a decent amount of bones everywhere to give full control over the Orc's expressions. My point of showing this is to make sure that with your facial bones you have every area covered to give you 100% over expressions.

[Archangel35757]

Yes... it will have all the necessary bones to drive fully-expressive facial expressions-- the 6 to 8 basic expressions... as well as be able to make custom facial animations for up-close cinematics. It will have a pose library where you can store custom facial poses.

 

As for the lip-synch'ing... the visemes will cover 44+ English phonemes... I also plan to build expandability for adding additional language viseme sets (others will have to make foreign language visemes that can be added to library of languages). I plan to use PRAAT for the sound analysis/phoneme identification-- already created the code to read in the phoneme list. PRAAT supports scores of languages.

[Tempust85]

Here's something I just found from SoF2's GLA:

 

 

Interesting.

[Archangel35757]

Yeah I saw that sometime back... when I loaded up the SoF2 GLA in ModView. It seems pretty crude... there are about 44 phonemes in the English language (varies by region and dialect) and they can be represented by about 20 visemes (...away from my PC right now so I can't recall the exact number) that I intend to create in the face rig. I think eye blinking should be treated separately from mouth visemes-- I think eye blinking should be in the facial expressions group (FACE_BLINK, FACE_WINK_R, FACE_WINK_L, etc.)-- or, create a separate "EYES_" group.

 

One thing I was thinking of was to be able to have the pupils dilate for close-up facial animations (they would be fixed/set to normal during gameplay-- but the feature would be there for close-up cinematics)... the eyes are what bring's a character to life in my opinion.

[Tempust85]

That would work easily if carcass could process scale animation data. May have to code in an eye texture swap from normal to dilated.

[Archangel35757]

That would work easily if carcass could process scale animation data. May have to code in an eye texture swap from normal to dilated.

Not sure if carcass would like scaling bones (even if linear)... guess we could do a test

 

My idea was to make the Iris inner radius as 8 sided and use 8 helpers at the vertices to contract or expand the Iris revealing the curved pupil underneath.

 

I have this prototyped... and have made a test animation in 3ds Max.

[Tempust85]

Yeah that would work but wouldn't it make the overall bone count too high?

[eezstreet]

That would work easily if carcass could process scale animation data. May have to code in an eye texture swap from normal to dilated.

Better idea - use an animmap and have the pupil dilation animate, but control the animation manually (JKG uses animmaps for the weapon indicators)

[Archangel35757]

Better idea - use an animmap and have the pupil dilation animate, but control the animation manually (JKG uses animmaps for the weapon indicators)

Edit: I prefer that the animator have direct control of pupil dilation in the skeleton rig and simply have a default eye pupil dilation animation in the _humanoid.gla. Then in cutscene GLA's, they can quickly animate the eyes (not breaking their workflow) however they choose-- and not force them to learn Icarus scripting to deal with eye details externally in the game engine.

 

I'll post an animation of my prototype setup.

[Tempust85]

Not possible unless GLA's can store scale animation, and my tests have revealed they don't.

[Archangel35757]

Not possible unless GLA's can store scale animation, and my tests have revealed they don't.

I wasn't intending to use scale animation... my method will work.

[Tempust85]

I have no doubt, but wouldn't it use too many bones just to do something a shader could do? I haven't a clue if JKA gets bogged down by a lot of bones or not.

[Archangel35757]

Not sure... we'd just have to try it out in-game... but they wouldn't be used during gameplay-- the pupils would be set to a normal position. I will animate an eyeball both ways-- one with bones and another by texture (in 3ds max only).

[eezstreet]

Animmaps, people. Animmaps. Animate the pupil using a shader and then manually control the pupil size using a SET_ value in ICARUS.

[Archangel35757]

Animmaps, people. Animmaps. Animate the pupil using a shader and then manually control the pupil size using a SET_ value in ICARUS.

Yes, that is certainly one way to do it... and I will create an example of that method in 3ds Max by animating a texture map.

 

And I also just thought of a way to do this by using only one (ah, ah, ah...) bone. (Did anyone catch my Sesame St. ref to "The Count"?)

 

I'll post example videos of each method.

[Archangel35757]

OK... I'm finally getting back to this while also trying to finish the 3ds Max dotXSI 1.x & 3.x file importer Maxscript (almost done-- struggling with skin weights!).  Anyways... for the facial rig you'll be able to control each area of the face to make unique expressions (could be used for custom facial.gla file for a cutscene cinematic); additionally to this overarching individual control, I plan to have six sliders that will allow you to pose the face quickly for the six universal facial expressions:  Joy, Sadness, Anger, Fear, Surprise, Disgust.  Within the 3ds Max Facial Rig GUI you'll also have the ability to create and store your own unique facial poses-- to be able to insert them anywhere along the animation timeline.  As for the six universal expression sliders, below is the progression of the emotional expression from a neutral face:

 

 

I'm still in the facial rig planning stages-- I intend to architect it so that the six expression sliders above can be combined/blended in an additive way-- e.g., you could have some amount of Disgust and Anger for a unique look... not every combination may result in believable expressions though.  I also hope to incorporate wrinkle-maps handled by shaders to go along with the facial expression (likely handled as animevents in the new facial.gla -- will have to work those details out with @@eezstreet, @@Xycaleth, and @@DT85).

 

...and I'll be working on the Jan Ors head mesh this weekend.  Any comments on the progression above?

[Archangel35757]

Ok... now that I've released the dotXSI importer maxscript, I'm brushing the dust off of this project and placing it at the top of my list. 

 

...so many neat 3ds Max projects-- so little time. 

[NumberWan]

It's been a while since I visited this thread. The last thing I remember was about the lips, and facial expressions as well. 

It is true, that both these and the body movement are in need of improvement.

 

In Deus Ex, which is far older than JA, the characters never looked static - one could notice them breathing for instance. Surely JA presented default animations, when an NPC would stand, doing nothing and look around occasionally. But it looks quite dull.

 

So what's the story now? Is there any chance of improving the animation, changing the skeleton and other stuff?

[Archangel35757]

It's been a while since I visited this thread. The last thing I remember was about the lips, and facial expressions as well. 

It is true, that both these and the body movement are in need of improvement.

 

In Deus Ex, which is far older than JA, the characters never looked static - one could notice them breathing for instance. Surely JA presented default animations, when an NPC would stand, doing nothing and look around occasionally. But it looks quite dull.

 

So what's the story now? Is there any chance of improving the animation, changing the skeleton and other stuff?

 

Anything is possible now... I've gotten back to working on the Jan Ors head model and also laying out the concept for the Facial Animation/Lip-Sync GUI for 3ds Max.  When @@Xycaleth returns to working on the rend2 renderer, I've asked him to support wrinkle maps (normal maps that accompany certain facial expressions and blend with the base mesh normal map ). 

 

For example:  http://www.chrisoat.com/papers/Oat-Wrinkles%28Siggraph07%29.pdf

[Archangel35757]

I've widened the bridge of the nose, and did some general tweaking on the nose... I calling the nose finished.  Moving on to the jaw area...

[Archangel35757]

@@DT85, @@Psyk0Sith, @@Xycaleth, @@eezstreet, @@Cerez... and anyone else-- what features would you envision/expect to see in a facial expression/lip-sync GUI? I'm open to suggestions while I'm conceptualizing the GUI.

[Psyk0Sith]

The simpler the better. It's already too much work with the current crappy rig haha, yeah i'm lazy.