Experiencing in pieces kit index
Experiencing in Pieces: A Physical Computing Approach to Sonic Composition
Steven Smith
North Carolina State University
Jeremiah Roberts
North Carolina State University
Audio Interview
Transcript
Steven: Hello and thank you for joining us. My name is Steven Smith, and I am a second-year student in the Communication, Rhetoric, and Digital Media department at North Carolina State University. The excerpt of music that you just heard was part of an Immersive Installation piece created by Jeremiah Roberts for his master’s thesis for the College of Design at North Carolina State University. Titled Experiencing in Pieces: An Audible and Visual Study in Reactive Spaces, this project’s goal was to bring closer the engagement between a performer and audience. Using TouchDesigner, a node-based visual programming software for multimedia content development, Jeremiah created a hybrid reality environment that acts as an interactive installation by allowing the user, or users, to manipulate the audio and visual content to be able to experience the music in discrete pieces; this is achieved via the TouchDesigner software and the Microsoft Kinect, a motion-sensor input device. Jeremiah will be joining me shortly for a brief interview, but first a bit of background on the development of his project.
Steven: When composing this project, Jeremiah kept a wide target audience in mind, ranging from adults to children, and chose classical music and contemporary film techniques to reach said audience. These choices allowed him to create an installation that emitted a surreal feeling while under the direction of a well-planned empathetic design, or a user centered design approach that pays attention to the users’ feeling towards a final production or product. In the abstract for his project, Jeremiah wrote that he, quote, “believed that the design was what appealed to the viewer and what they could connect with,” endquote. The project was designed so that users would connect with the entire experience, including the audio, which he composed with his brother, the visuals of a dancing woman, which he filmed using the Microsoft Kinect, and the physical layout of the installation.
Steven: In theory, Jeremiah’s project works rather simply. The TouchDesigner software is started and two Microsoft Kinects are placed in a rather large room in order to track the audience’s movement in relation to eight projectors that emit the video of the dancer. For each projector’s panel, a specific instrument is portioned off in the TouchDesigner software; for example, if you are in say, Quadrant “A,” and the flute composition of the musical piece is set up to be in that same quadrant, the Microsoft Kinect tracks your bodily position and, through TouchDesigner, plays that instrument over a computer’s speaker system.
Steven: For example, picture yourself in a large room with 8 projectors and 8 screens. You begin on the left side of the room where there is a video of a sage-green figure dancing. As you stand there, enticed by her movements, this instrument plays: flute.mp3. You might take a few moments to revel in the flute instrumentation before deciding to move to the next panel, where a now lime-green figure dances a different composition. As you make the switch, you notice the flute fades out of audibility, but then begin to hear this: cellobass.mp3. You repeat this process six more times, that is, visiting different panels with various dance recitals and instruments playing. This includes clarinets (clarinet.mp3), a contrabassoon (contrabassoon.mp3), horns (horns.mp3), percussion (percussion.mp3), a piano (piano.mp3), violas (violas.mp3), and finally, two violins (violins.mp3). This composition also has a soundscape which continuously plays during the exhibition. If you are by yourself, you would only hear one instrument playing at a time. If, however, you are with another friend, you would hear two instruments playing together. Three people could hear three instruments of their choosing, and so on and so forth. Ultimately, though, you would need eight people to hear the entire musical composition.
Steven: Jeremiah has agreed to allow me to interview him for this audio component. Jeremiah, why don’t you tell some of the listeners a little bit about who you are as a designer and who you are as a musician.
Jeremiah: Hey, Steven. Thanks for having me. Yeah, I’ve been a musician now for a little over two decades, I guess, playing in different projects, mainly sticking with string instruments. I’ve always been kind of a self-taught musician. I’ve had a lot of influence from my brother Michael Roberts who’s a classically trained musician, so we’ve been fortunate enough to work together a lot over the years, and I’ve… as a designer, with my educational background I’ve always tried to link the two together because it’s always been a love of mine to be a musician and, the fact that I get to be a designer at the same time, has a lot of benefits. And I feel like I try to relate all of my designs basically to things musically. I’ve always tried to write in a simplistic way and then just keep adding elements, and then just keep adding elements, so I do the same thing when it comes to designing projects. I usually start simple and try to add elements and then start refining over time.
Steven: So this was a great project that really blended musicality and design together, so why don’t you tell the listeners a little bit of how you came up with this project idea.
Jeremiah: Well actually I decided to try and combine my skillsets as a musician with my new-found love for media creation. Trying to combine these two interests was always a goal, and creating a hybrid reality of sorts, which was an idea I had in which I could mix the two, basically. It could appeal to both people who loved music and performance and people who had an interest in technology. You know, my greatest love was/is performing and being on stage, so I saw this as a platform in which I could bring someone who had never had that experience and hopefully engage them in the musical creation process in performance.
Steven: Would you happen to remember any of the audience’s experiences as they engaged with your projects?
Jeremiah: Yeah I do actually, it was great. Since I used motion tracking for a lot of the engaging performance parts, I saw people come in and one of the first things they did was start to reach out and cover more space in the area I’d laid out, which means they wanted to take the project further than I actually expected them to, which gave it more of an engaging and rewarding experience. And just inspiring people to come in and listen to the music, people who didn’t want to get out front and just wanted to come in and listen to the music and see what other people had created was a very rewarding thing—and I think for them too, so.
Steven: Yeah, it’s definitely a really cool project and I can’t wait to utilize myself for a project I have in mind, using the Microsoft Kinect and the body and classical delivery, with, you know, motion tracking and gestures and things like that. Can you tell us a little bit about where you would like to see this project go in the future?
Jeremiah: Actually I’d like to see this project go as far as it can. I think it only has the capability of getting bigger, utilizing bigger rooms and bigger auditoriums. I’d like for it to engage as many people who have no real idea what it’s like to be in the middle of different styles of music being made. You know people can find other relations between the audio and video and themselves and maybe more narrative storytelling structure. I really see it as a way of exploring music that’s already there and then it can be broken apart and into different pieces and then pieced back together, and then people can create their own new music… finding, basically making the audience the composer themselves.
Steven: Jeremiah it’s a brilliant project and I really think that listeners of this journal are going to enjoy providing it to their students for future projects. I really appreciate you taking the time to be with me here today and helping me with all the components of this document.
Jeremiah: Thanks a lot Steven for having me. Yeah, I hope it can inspire, and help in this issue…the way that music and technology are now and the way that people can embrace this relationship between the two and help and help take us places we’ve never been before. Thanks a lot.
Steven: Yep, thank you, Jeremiah. That was Jeremiah Roberts providing a bit of context for his installation called “Experiencing in Pieces.” The instructions for composing a miniature version of his project can be found in the praxis portion of this document.