Show simple item record

dc.contributor.authorMc Donnell, Rachel
dc.date.accessioned2024-02-21T06:40:26Z
dc.date.available2024-02-21T06:40:26Z
dc.date.issued2022
dc.date.submitted2022en
dc.identifier.citationBigioi, D. and Jordan, H. and Jain, R. and Mcdonnell, R. and Corcoran, P., Pose-Aware Speech Driven Facial Landmark Animation Pipeline for Automated Dubbing, IEEE Access, 10, 2022, 133357-133369en
dc.identifier.otherY
dc.description.abstractA novel neural pipeline allowing one to generate pose aware 3D animated facial landmarks synchronised to a target speech signal is proposed for the task of automatic dubbing. The goal is to automatically synchronize a target actors’ lips and facial motion to an unseen speech sequence, while maintaining the quality of the original performance. Given a 3D facial key point sequence extracted from any reference video, and a target audio clip, the neural pipeline learns how to generate head pose aware, identity aware landmarks and outputs accurate 3D lip motion directly at the inference stage. These generated landmarks can be used to render a photo-realistic video via an additional image to image conversion stage. In this paper, a novel data augmentation technique is introduced that increases the size of the training dataset from N audio/visual pairs up to NxN unique pairs for the task of automatic dubbing. The trained inference pipeline employs a LSTM-based network that takes Mel-coefficients as input from an unseen speech sequence, combined with head pose, and identity parameters extracted from a reference video to generate a new set of pose aware 3D landmarks that are synchronized with the unseen speech.en
dc.format.extent133357-133369en
dc.language.isoenen
dc.relation.ispartofseriesIEEE Access;
dc.relation.ispartofseries10;
dc.rightsYen
dc.subjectspeechen
dc.subjectnovel neural pipelineen
dc.subjectautomatic dubbingen
dc.subject.lcshspeechen
dc.subject.lcshnovel neural pipelineen
dc.subject.lcshautomatic dubbingen
dc.titlePose-Aware Speech Driven Facial Landmark Animation Pipeline for Automated Dubbingen
dc.typeJournal Articleen
dc.type.supercollectionscholarly_publicationsen
dc.type.supercollectionrefereed_publicationsen
dc.identifier.peoplefinderurlhttp://people.tcd.ie/ramcdonn
dc.identifier.rssinternalid251191
dc.identifier.doihttp://dx.doi.org/10.1109/ACCESS.2022.3231137
dc.rights.ecaccessrightsopenAccess
dc.identifier.orcid_id0000-0002-1957-2506
dc.identifier.urihttp://hdl.handle.net/2262/105583


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record