Algorithms conquer the stage

He is bringing artificial intelligence to the theater stage: Ilja Mirsky. And, in the opinion of this young digital artist who reveres analog theater, it is okay if the technology fails during a live performance.

Portrait of Ilja Mirsky

A 21st-century drama: Gunter feels attracted to the artificial superwoman Jann. The robot has feminine allure, is witty, and understands him better than he understands himself. The problem: Gunter is married, and his wife is fiercely jealous. “I am a man after all!”, the conflicted protagonist laments during an online session with his therapist, Dr. Eliza. The latter nonchalantly replies: “Do you believe it’s normal to be a man?” The audience laughs, a crack in the otherwise stereotypical story. The source of the comedic retort: a chatbot.

“There are many stage productions about artificial intelligence,” Ilja Mirsky says. “But in order to thematize the human/machine relationship, performances with AI are really much more impactful.” That is why the immersive media artist and dramaturge is bringing AI-based technologies to the stage. In the improvised science fiction play “Almost Human”, the 27-year-old directs two chatbots – one in the role of a psychiatrist, the other as a robotic woman who leads the protagonist astray.

Ilja Mirsky dialog with chatbot

Stage and research

“The dialogs with the chatbots are often intriguing and thrive on unintentional situational comedy,” Ilja Mirsky explains. In order for them to be able to communicate and for their dialog to appear natural, a great deal of technical control is required behind the scenes. On the one hand, Ilja Mirsky uses speech-to-text software to feed the actors’ responses into the chatbots. On the other, the south German dramaturge relays the bots’ responses to the performers’ wireless earbuds. “All of this has to be done very quickly so that there is no noticeable time lag.”

Another quirk of performing with chatbots: “Frequently, the dialogs are so perfect that they come across as being scripted. This can cause the audience to doubt whether they are actually generated in real time.” Consequently, Ilja Mirsky is conducting intensive research into whether and how theater with AI can work at all. He is doing this on stage, but also within the framework of his doctoral thesis. Ilja Mirsky is pursuing a doctorate in digital dramaturgy at the University of Tübingen and the Zurich University of the Arts.

Unfortunately, digital productions often neglect what this millennia-old medium is so good at: telling stories.

However, the young German is not interested in technology for its own sake. “I am an analog theater enthusiast,” he emphasizes. “Unfortunately, digital productions often neglect what this millennia-old medium is so good at: telling stories.” This is why Ilja Mirsky struggles with many digital projects: “There is often a lack of dramaturgical effort, and the result is experiments with no sense of direction.”

Digitalization in the theater

Digital theater is still in its infancy. However, an ever-growing number of theater companies are experimenting with digital technologies. This involves much more than simply streaming performances on the Internet. “The technologies open up additional means of creating new worlds and narratives,” the immersive media artist and dramaturge Ilya Mirsky says. They also give rise to completely new theater formats. “Virtual reality productions, for example, in which the audience is immersed in a computer-controlled 3D reality using VR glasses,” Ilja Mirsky says. He himself is more involved in projects that combine analog theater with digital technologies, for example via a video screen in the stage backdrop.

The dark side of AI

With his work, Ilja Mirsky seeks to combine the best of the analog and digital worlds. This is an obvious choice – the Regensburg native is at home in both environments: After completing his bachelor’s degree in cognitive science in Tübingen, where he also learned computer coding, he enrolled in performance studies in Hamburg. However, Ilja Mirsky gained much more from his cognitive science studies than just technical know-how: “Questions such as:

What makes us human?
How do we perceive?
What is a body?

Ilja Mirsky also wants to address the dark side of AI. For example, in his “Myths of Purposefulness” play at the Klabauter Theater in Hamburg. The members of the theater company are people with impairments. “Here, I employed AI that uses body tracking to identify people as people. But not all of them – the AI failed to recognize some performers.” In this way, the dramaturge highlights the discriminatory potential of algorithms in a straightforward way. “AI systems are usually trained with data that conforms to the norm, and they therefore make unfair decisions.”

Screenshot Augmented Reality

Keyword: augmented reality

First and foremost, however, Ilja Mirsky uses AI to expand reality. Or in other words: He creates augmented reality. In the Klabauter Theater, for example, he used an algorithm that amplified the emotions on the face of a severely impaired performer in a wheelchair. “This allowed us to portray the actress in a completely new way.” Or he generated 3D scans of the performers and projected them on a video screen next to the stage. “Nowadays, such 3D scans are incredibly easy to implement using any device that features Lidar laser sensor technology for environment detection, such as Apple’s iPad Pro.”

"Body Tracking" by Ilja Mirsky

However, it is not actually necessary for digital technologies to perform live on stage at all. Frequently, Ilja Mirsky uses them merely as tools to assist in the production process. “Figuring out how a blue backdrop would look instead of a green one takes just a few clicks in a virtual environment – in real life, it would take several hours of work,” Ilja Mirsky explains. When deciding whether to put technology on stage in a production or keep it behind the scenes, Ilja Mirsky has one main recommendation: “Everyone involved should be aware of the reasons why they are in favor of or against live technology.” In his opinion, fear of technology is a poor counterargument. “It is okay for technology to fail on stage – such limitations even represent an interesting aspect. You simply have to be open about it to the audience.”

Stupid and naive

In the chatbot play “Almost Human”, the live use of technology plays the key role. “We want to illustrate: Machines are interaction partners, not just tools,” Ilja Mirsky explains. For the play, the team decided to use the Eliza chatbot for one of the roles. Eliza was programmed in 1966 and is the world’s oldest chatbot. “Accordingly, he is rather stupid and naive,” Ilja Mirsky says. “In exchange, he answers in a very rule-based manner and is thus the ideal cast for the role of the therapist.”

We trained this chatbot using a special dataset to teach it to act emotionally and passionately

The other part is played by Jann, which is based on the Spotify algorithm and uses a neural network. “We trained this chatbot using a special dataset to teach it to act emotionally and passionately.” Perfect for the role of the robot seductress. In fact, Jann is what is known as a next neighbor algorithm. This means: It seeks its responses in datasets that are closest to what its counterpart said.

A place of understanding

Next neighbor – that is quite apt. “Theater is a place of understanding,” Ilja Mirsky says. One of the places he experienced just how unifying theater can be was in Tel Aviv, where he sat in on the first Hebrew performance of Borchert’s “The Man Outside”. “The actors and actresses performed a text dealing with a German World War II soldier. That really moved me – after all, it is quite likely that some of their grandparents were killed by German soldiers.”

Be it purely analog or with digital elements: For Ilja Mirsky, theater is a place where people with diverse perspectives and biographies come together. Ilja Mirsky’s own background is colorful: A part of his family comes from Israel, his parents are from Moscow. Talking about the latter fact gives the young immersive media artist cause to pause for a moment. “As we speak, my Ukrainian friends are weathering a hail of bombs,” he notes dryly. And he concludes: “Unifying elements such as theater are more important than ever right now."  

Almost Human by the Karlsruhe theater group Stupid Lovers

Written by:

Sarah Hadorn

I am intrigued by how new technologies can improve the world and our society. Quite seriously. I want to tell stories about this and meet people with new ideas and concepts. My main interests: sustainable development and anything associated with sociology – even though I’m no sociologist. Apart from this, I love French cheese, thick American novels, and cats.

Photos: Samuel Schalch

Read more