Computer Science 5th Year Masters Thesis Presentation August 17, 2022 3:00pm — 4:00pm Location: Virtual Presentation - ET - Remote Access - Zoom Speaker: JIAXIN (KELLY) SHI , Masters StudentComputer Science DepartmentCarnegie Mellon University A Self-Supervised Study of Multimodal Interactions in Opinion Videos Our experience of the world is inherently multimodal. Analyzing human multimodal language is an increasingly popular area of research that often focuses on sentimental analysis and emotion recognition, where three main modalities are present: language, acoustic, and vision. The advancements in deep learning rely heavily on the abundance of data available for the model to learn rich patterns. Due to the heavy labor required to annotate large-scale data, it is beneficial to explore what we could achieve from self-supervised learning methods. In this work, we propose a self-supervised task to study the cross-modal interactions present in the multimodal language datasets (with language, acoustic and visual modalities). Through our proposed self-supervised task, we can study the information overlap between modalities and if bimodal multiplicative interactions are present. A secondary advantage of our proposed self-supervised task is that it can also be used in downstream tasks where one of the modalities is missing. Our approach builds on the intuition that observed modalities may be able to generalize information about the missing modality. For example, people may be able to imagine the voice of a speaker when watching muted videos. In summary, this thesis is a self-supervised study on multimodal interactions in opinionated videos. Our work investigates how much information overlap exists between different modalities, quantifies the amount of cross-modal interactions, and evaluates how much information can be learned from a missing modality given other available modalities. Thesis Committee: Louis-Philippe Morency (Chair) Robert E. Frederking Additional InformationZoom Participation. See announcement. For More Information: tracyf@cs.cmu.edu Add event to Google Add event to iCal
