☰
In this (visual) talk I will show a retrospective of my work in multimodal video annotations for performing arts documentation since 2005. It will mainly focus on the affordance of idiosyncratic video annotations as indispensable metadata when digitally documenting performing arts materials or visible intangible heritage in general.
Annotating and tagging directly over video in real-time have started (Cabral et al. 2011) to be mostly a personal practice used by those involved in the footage or analytical processes of video documentation for dance, but they have not yet been perceived as complementary metadata by Cultural Heritage archivists.
Based on previous experience with the TKB and BlackBox projects, we have continued to research on multimodal annotation practices as part of the creative process in contemporary performing arts. Within TKB, the innovation was to accept individual tagging as indexation cues and means of interaction amongst the artists who were self-curating their works, therefore generating dynamic and personal “archives of processes”. Under subsequent EU-funded projects, we developed a friendly video annotator in real-time to work as a digital notebook, replacing paper notes. “MotionNotes” is now freely available in the web3 and is being used by choreographers, ethnographers, sportspeople, and educators in general.
Throughout the talk, examples will be given of different genres of annotations we have developed and used in different projects, namely: manual video annotations with ELAN; animated infographics (Fernandes and Jürgens 2017); real-time annotations over 2D video (Rodrigues et al. 2019); 3D point-cloud renderings (Kuffner, Ribeiro, and Fernandes 2018); 360º multiple viewpoints (Jurgens and Fernandes 2018); and Virtual Reality: embodied interactions and interactive storytelling (Jürgens, Fernandes, and Kuffner 2020).
We would like to discuss with the audience the future usability of those idiosyncratic multimodal annotations in new AI and machine-learning approaches (as in semi-automatic recognition of human movements, for instance) to documenting, archiving and sharing intangible cultural heritage video content. Since more and more AI models are relying on multimodal real-world data, companies will also require multimodal annotations to curate a larger variety of datasets. This will eventually lead museums, art galleries, researchers and artists to innovate in this aspect as well.
Follow us
