Artist Statement
《Deep.PhotoPlay》 is an interactive VR artwork that uses deep learning-based cooperative AI and immersive VR as its main strategies. It reenacts a silent-film, a Korean cultural heritage, disseminates artistic values, and ultimately realizes remediation.
The material for this artwork is the silent-film <There is No Poverty at the End of Labor> (12 minutes, estimated to have been produced in the 1920s) discovered by the Korean Film Archive (KOFA) at the Gosfilmofound of Russia in 2019. As the title suggests, it depicts the life and history of people escaping poverty through hard work. In particular, it is an important piece of cultural heritage that fills gaps in knowledge of early Korean film history. It is an action-film in which performances of piano and string instruments and narration are harmonized. Through cooperation with the KOFA, we discovered that the audience's immersion comes primarily from sounds in the film, which convey the scenes' atmosphere and expand the actors' emotions.
The artificial intelligence we designed functions as an actor that creates sounds. It creates Foley, Ambience, and FX sounds by being aware of the narrative through the film's short length, subtitle text, and character movement. We also performed noise and degradation enhancement on the original film to ensure accuracy. The generated sound is selected through the complementary intervention of human composers and film critic experts and combined as background sound. Collaboration between AI and humans adds depth to creation.
《Deep.PhotoPlay》 builds on the work above to provide audio-visual virtual immersion through a real-time engine. It creates a unique experience through interaction with the film's era, narrative, and sounds. The VR this artwork uses creates a personal yet shared world that blends the silent film era (the 1920s) with modern times. Participants enjoy moving and repeatedly interacting as avatars. This approach is valuable not only for creating a new awareness of cultural heritage but also for improving public access. Participants can also tour the restored Dansungsa Theater, a historic building in Korean film history. While a silent movie is shown in the theater, musicians and narrators replaced by virtual humans present an immersive performance based on the above creative sounds. This presentation has a high degree of realism because the artists' live performances were acquired and produced with motion capture technology. Silent films are still screened at film festivals as they used to be. However, this artwork is the first work of silent film creation using artificial intelligence and virtual technology.