Delightful Facial Tracker Turns Smiles into Music

Artist and musician Ben Vida experiments with sonification in his new exhibition '[SMILE ON.] ... [PAUSE] ... [SMILE OFF.]'

Before technology like the talk box and vocoder, most musical instruments required either the movement of either extremities or the lips and breath, usually not both. Software, with its keystrokes and touch-based gestures, changed the nature of sound generation. More recently, facial and motion tracking have been changing the sound game, as we recently saw with Eye Conductor, an eye-tracking music app. Artist Ben Vida, of the band Town & Country, is exploring similar sonification territory in his upcoming exhibition, [SMILE ON.] ... [PAUSE] ... [SMILE OFF.].

The exhibition features two new videos and text-based wall works that dig into the dichotomy of internal narratives versus external communications. Soft Systems Music (Video #1) will be projected in the main space of Lisa Cooley gallery, and Speech Act Video will be on a monitor in the back. The text pieces, which are on paper and panel, accompany Speech Act Video. Vida tells The Creators Project that these works were inspired by concrete poetry—that is, poetry or words as visuals or patterns.

Ben Vida, Soft Systems Music (Video #1) [still], 2016. Video with sound, 34 min 29 sec. Courtesy of the artist and Lisa Cooley, New York

“It’s pleasurable to receive language stripped of its basic function—to reduce it back to sound,” he says. “This opens up new avenues for the voice in performance and for the text on the page.”

In the video Soft Systems Music (Video #1), Vida sonifies a smile. To do this, he asked nonprofessional actors to repeatedly smile and frown repeatedly for a set amount of time. Vida then transformed their facial gestures and “microexpressions” into what he calls "soft system”—a method of composing by introducing a human into a series of decisions that constitute the composition, which then creates a “disturbance in the algorithm.” So any facial moment—a blink of an eye or a faint grin—can alter the musical composition.

Ben Vida, Speech Act 1, 2016. 29” x 26”. Ink of rag paper. Courtesy of the artist and Lisa Cooley, New York

“It is a simple process of watching and decoding the micro-expressions of the performers,” Vida tells The Creators Project. “As I watch video playback of their performances, I tap MIDI messages into a sound program by hand.”

“Once the rhythms of the expressions are captured as digital data, I can feed that data as control commands into a synthesizer which produces the audio content,” he adds. “Some of the most basic compositional decisions fall into the hands of the actors and that disrupts a lot of my own subjective formal choices—which I like.”

Ben Vida, Speech Act 8, 2016. 29” x 26”. Ink of rag paper. Courtesy of the artist and Lisa Cooley, New York

The other video, Speech Act Video, is based off of the Speech Act texts, another set of works in Vida’s exhibition. These works are call-and-response composition that mix sounds with few words, and they appear to be a person struggling to communicate. Each wall piece is a separate work, with each being used during a performance at the gallery, to be turned into yet another video.

“The texts act as the loose script that Mary Manning and I perform in the video,” he explains. “Since the script is more impressionistic than concrete it often slip between textual and textural and makes for a video piece that hints at narrative but often dodges meaning.”

In past projects, like Slipping Control, Vida explored language as gesture and language as tempo and rhythm. There was no linguistic content. Initially, Vida thought that the works in exhibition would take that form, but they slipped back into having linguistic content, and lost some of their abstraction as they began to resemble scripts and scenes and conversations.

Ben Vida. Speech Act 16, 2016. 29” x 26”. Ink of rag paper. Courtesy of the artist and Lisa Cooley, New York

“Aurally it's kind of a short distance from ‘eh, itz tah’ to ‘ah, um oh’ to ‘well, okay so,’ but within this distance the function of the language changes and so the compositional logic begins to change as well,” Vida says. “It becomes a completely different engagement in terms of what is being communicated. And for me, to have a text piece play with this and to oscillate between sense and nonsense creates a wide open space to compose within.”

Ben Vida’s [SMILE ON.] ... [PAUSE] ... [SMILE OFF.] runs at Lisa Cooley from April 3 to May 15, 2016.

Click here to see more of Ben Vida’s work.


Your Eyes Make Music with This Face-Tracking Production App

Getting Up: The TEMPT ONE Story Is A New Doc About The EyeWriter

Marshmallows, Vodka Syringes, and Lasers, Oh My