Facial Animation Pipeline Optimisation with Blendshape Importance Ordering and Perceptual Metrics

Abstract

The quality of realistic virtual human faces has increased in recent years with the availability of new technologies such as photogrammetry systems to scan actors and reproduce their shape and texture digitally with minute detail, improved motion capture to allow for the recording of intricate facial movements, and new game engines which allow for the real-time rendering of complex structures and materials such as the human eye. All of these combined have led to an intensely high level of quality in virtual characters, but the added complexity of these technologies has increased the amount of time, money, and effort needed to create them, with a significant increase in the computational power and memory required due to the high level of detail.

The goal of this thesis is to review the facial animation pipeline, identify areas to optimise, and create new methods and algorithms to assist in the creation of high-quality, realistic facial animation. The current industry standard for the high quality facial animation is blendshape animation. This method requires a large amount of geometry to describe the different possible poses of the face, known as blendshapes.

We first optimise the rig creation stage, the stage in which blendshapes are created either by scanning an actor, or through a process called blendshape transfer, which transfers existing blendshape expressions from a template character to create the required shapes. The blendshapes created through blendshape transfer can be further improved by providing training example expressions of the target character. We conduct a perceptual experiment on the impact of this training on blendshape transfer, and then propose a novel algorithm to suggest training examples using blendshape importance ordering, i.e. the ordering of blendshapes such that the most and least important blendshapes can be easily identified for a given task, as an integral part of the example creation process, and an unexplored area that could be applied to various stages of the pipeline and benefit from further investigation.

We go on to apply this idea of blendshape ordering to the animation stage. We implement a GPU animation method with level of detail optimisation through blendshape reduction, using the idea of blendshape importance to identify which blendshapes are least important and can be removed.

We conclude with a perceptual experiment to further investigate the numeric metrics used throughout this thesis and their relation to human perception of faces. These results provide a perceptual basis for blendshape ordering which can be used to optimise the facial animation pipeline in methods such as blendshape reduction for level of detail optimisation and training example creation using blendshape importance ordering, such as those which we explore in this thesis.