TrUMAn: Trope Understanding in Movies and Animations.
2021
Understanding and comprehending video content is crucial for many real-world
applications such as search and recommendation systems. While recent progress
of deep learning has boosted performance on various tasks using visual cues,
deep cognition to reason intentions, motivation, or causality remains
challenging. Existing datasets that aim to examine video reasoning capability
focus on visual signals such as actions, objects, relations, or could be
answered utilizing text bias. Observing this, we propose a novel task, along
with a new dataset: Trope Understanding in Movies and Animations (TrUMAn),
intending to evaluate and develop learning systems beyond visual signals.
Tropes are frequently used storytelling devices for creative works. By coping
with the trope understanding task and enabling the deep cognition skills of
machines, we are optimistic that data mining applications and algorithms could
be taken to the next level. To tackle the challenging TrUMAn dataset, we
present a Trope Understanding and Storytelling (TrUSt) with a new Conceptual
Storyteller module, which guides the video encoder by performing video
storytelling on a latent space. The generated story embedding is then fed into
the trope understanding model to provide further signals. Experimental results
demonstrate that state-of-the-art learning systems on existing tasks reach only
12.01% of accuracy with raw input signals. Also, even in the oracle case with
human-annotated descriptions, BERT contextual embedding achieves at most 28% of
accuracy. Our proposed TrUSt boosts the model performance and reaches 13.94%
performance. We also provide detailed analysis to pave the way for future
research. TrUMAn is publicly available
at:https://www.cmlab.csie.ntu.edu.tw/project/trope
Keywords:
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
23
References
0
Citations
NaN
KQI