Self-supervised Light Field View Synthesis Using Cycle Consistency
Citation:
Chen, Y., Alain, M., Smolic, A., "Self-supervised Light Field View Synthesis Using Cycle Consistency," 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP), Tampere, Finland, 2020, pp. 1-6Download Item:
Abstract:
High angular resolution is advantageous for practical applications of light fields. In order to enhance the angular resolution of light fields, view synthesis methods can be utilized to generate dense intermediate views from sparse light field input. Most successful view synthesis methods are learning-based approaches which require a large amount of training data paired with ground truth. However, collecting such large datasets for light fields is challenging compared to natural images or videos. To tackle this problem, we propose a self-supervised light field view synthesis framework with cycle consistency. The proposed method aims to transfer prior knowledge learned from high quality natural video datasets to the light field view synthesis task, which reduces the need for labeled light field data. A cycle consistency constraint is used to build bidirectional mapping enforcing the generated views to be consistent with the input views. Derived from this key concept, two loss functions, cycle loss and reconstruction loss, are used to fine-tune the pre-trained model of a state-of-the-art video interpolation method. The proposed method is evaluated on various datasets to validate its robustness, and results show it not only achieves competitive performance compared to supervised fine-tuning, but also outperforms state-of-the-art light field view synthesis methods, especially when generating multiple intermediate views. Besides, our generic light field view synthesis framework can be adopted to any pre-trained model for advanced video interpolation.
Sponsor
Grant Number
Science Foundation Ireland (SFI)
15/RP/2776
Author's Homepage:
http://people.tcd.ie/smolica
Author: Smolic, Aljosa
Sponsor:
Science Foundation Ireland (SFI)Other Titles:
IEEE 22st International Workshop on Multimedia Signal Processing (MMSP 2020), 2020.Type of material:
Conference PaperCollections
Availability:
Full text availableKeywords:
Light Fields View Synthesis, Video Interpolation, Cycle Consistency, Self-supervised Fine-tuningSubject (TCD):
Digital Engagement , Computer Education/Literacy , Information technology in education , Multimedia & CreativityDOI:
10.1109/MMSP48831.2020.9287105Metadata
Show full item recordLicences: