An Alternative Representation of Video via Feature Extraction (RAAVE)

Abstract

This thesis proposes a novel approach called RAAVE to transform video from a linear stream of content into an interactive multimedia document and thereby enhance the exploration potential of video content. Exploring the content of a video is typically inefficient due to the linear streamed nature of its media and the lack of interactivity i.e. video content is defined as a sequence of consecutive images with or without a parallel audio component. While researchers have proposed many approaches to enhance the exploration experience of video content; the general view of video content is still the same i.e. a continuous stream of images. It is the contention of this thesis that such a constrained view on video is limiting its potential value as a content source. For this reason, the presented thesis explores the idea of viewing video as a diverse content source, opening new opportunities and applications to explore and consume video content. RAAVE transforms a video by representing its content as an automatically curated multimedia document. It does so by utilizing a template driven representation engine. Multimodal features are extracted automatically from the different modalities of video content, along with their timestamp, and stored in a repository. Upon receiving a content request, the representation engine utilizes a template collection to represent the content of a video in an appropriate configuration. By configuration it is meant that the presence and granularity of certain features are used to compose a representation of the source video. A video can have multiple multimodal representations. By automatically curating the content, the proposed approach allows users to not only configure the content in terms of the amount of detail, but also in terms of choice in the combination of different modalities. A modular framework and algorithm for the representation engine and template collection is described. The framework design is influenced by the state of the art and user studies conducted to streamline the design. The representation engine-based approach is evaluated by developing a prototype system grounded on the design of the proposed approach, allowing users to perform different content exploration within a video, tasks. The evaluation demonstrated the benefits of the proposed approach in terms of enhancing the user?s exploration experience with video content.