Bodysoundscape : the cognition of a movement - A conversation with Robert Wechsler | Palindrome

0. A movement perception

I would like to start this conversation with a definition about the movement: in the long experience of Palindrome choreographic language, what is composition and what does this concept involve in your imagination of the body?

The « imagination of the body »: I like how you frame the question. 
You are right, that these are two distinct things: the body, and the imagination of the body. We perceive human movement in a very subjective way, and this has always interested me as an artist. We do not really see what is there, but instead we see something that is effectively and transformed by the viewer. This is not because of optical illusions of course, but relates to deeper psychological aspects of perception. Dancers (and I hope maybe sometimes others as well) will twitch in their seats when they watch a dance they like, when the dance moves them: you see?  It literally moves them!  So watching dance has to do with dancing itself. That is, even though we watch dance with our eyes, we can experience it with other parts of our perceptual apparatus; for example, the parts of the brain that move the muscles. Our ears also experience dance. What we see is effected by what we hear, by what we think we hear, and by what we have heard in the past. The word synaesthia refers to a confusion of the senses and thus dance, to me, is synaesthetic. For me, to talk about dance, is also to talk about music.

But to answer your question, « what is composition [choreography]? » 
I must say that it is still a dream to me. I watched the Cunningham Company perform from 1979 until his death and the dissolution of the company, and through all those years I had a vision that the world was going to wake up and say, « Yes, movement can be art! ». 
But this has not happened.  
I’m sorry to be cynical, but I’ve maybe seen an hour of choreography in Europe that was about movement. Instead, there are usually things everywhere getting in the way; bath tubs, chairs, stuffed animals, but also concepts and emotions are there to gum things up. Meanwhile, the music can also be distracting because it is usually dead, not « live », but played from records. So, that finally one does not perceive movement, but someone’s idea about movement. And these are two very different things. So, for me, composition is the design of human movement. It is simple, but in practice it turns out to be a very hard thing to do. It is trench work. 

Can you talk about your approach to movement – using the form of a glossary – in tree key operational notions?

I want to be honest: I do not choreograph much anymore. I still call myself a choreographer, because I can still do it. And I do it sometimes, but when you ask me for my operational notions, it feels a bit like memories. Anyway, here goes:

1) Systems

I like pieces that follow systems, very concrete things from geometry or math, like riddles or science itself. An example of such a system is the linking of movement to sound:  So, for example, when I touch my left foot, in front and to the right, well, that is a bit of a “special” thing to do, or gesture, and so one might link this gesture to a certain sound. When you hear the sound, you see the gesture; when you see the gesture, you hear the sound. Together they build a Gestalt, a unique form in the mind of the perceiver. 
I have made many pieces like this and I am proud of them. I did a series of pieces called “touching” based on the touch between dancers. Sometimes a certain combination of body parts — say, hand to head, would be linked to a sound in scored music. So the touches, in a sense, « played » the music. Later, when I started using electronics, we did the same thing, but the touch literally caused the sound to be triggered. The two dancers were wired so that when they touched it completed an electrical circuit, and this made the sounds.

2) Associations

I did a bunch of things over the years that worked like models of phenomena, like the life cycle of a virus or DNA. Science has always inspired me. Its like art because if you can understand it, even a little bit, and then step back from it, it can seem so wonderful that it can make you cry. 
Science, real science is just like art. It is the most wonderful and most useless thing! And yet, someone once said, exactly these things, the things most useless, are the things that make us most human.

3) Challenges

I like extremes. I’ve made a couple pieces in which there is no movement. No movement is, of course, a kind of movement because it is such a special thing that we rarely do. I am planning a piece called Morning Birds in which 7 or 9 people, an old man, a young girl, etc. The most ordinary people you can imagine, stand on the stage and look outward. When they blink, or make any tiny movement, it triggers the sound of a morning bird. This is possible with the technology I have been using in recent years. So slowly the day begins, there are more and more sounds of birds. You see it will be a very quiet piece, but it will require a lot of rehearsal!  I like this.

In your work perception plays a very important role: can you talk about changes of perception in relation to the audio feedbacks in Flower. Wine. Moon. Me (2010) or in Tiresias(2013), in the work with Andreas Bergsland in Dresden?

Flower… and Tiresias both use motion tracking technology. Both use small movements as well as large ones, to shape and trigger sounds. The sounds give feedback to the dancer; that is, since the movement makes the music, and the dancer dances to the music, there is a kind of feedback loop in the performance. It is, in a sense, the way that the dancer navigates the space. 
Tiresias, like the Greek with this name, is a blind man. Our actor is indeed blind and also uses a wheelchair. His name is Thomas Mader and we met him at one of our workshops. It was a joy to work with him and the dancer, Annika Dörr. Thomas plays music with his arms, reaching out around him, finding the particles of sounds in Andreas Bergsland’s motion tracking controlled composition. Annika then interprets these sounds and represents in this way his dreams. But it is more than that. It is the « reaching out » from an internal to an external world, from a person with disabilities, to those who are not. Thomas made experiences through Annika, as Annika did through Thomas. Dance works like this, we sense or see movement in someone else, seeing it, or in our case also hearing it! In this way dance brings people together, and it can do so with people with and without disabilities. 

Do variations, in all these cases, involve a new composition of movement, a new body’s configuration in the space stage?

Yes and no. I am often asked how technology has affected the kinds of movement I use, and I usually say, “not at all”. Why should it? But this is not completely true. Because I work so much with camera-based interaction, and because our cameras are so sensitive, I have learned to hold very still. Stiller than most dancers!  
Each piece I’ve made uses different tracking parameters. For example, in one piece, the dancer’s position right-to-left on the stage might be very important, whereas in another, it is the transition from stillness into movement that is tracked. 
Thus each method of tracking puts its own constraints on the dancer’s discipline, that is, it is not so much limiting choreographically, as it is crucial to the dancer’s attention to detail in their performance, and this effects the way they that we rehearse together.

I. Logic of technologies

I think a very important aspect in your artistic approach is the logic of technology. In this framework, the technological device isn’t a simple application but a creative process: technology is a mean of expression. In Interactive Studies II (1995-2000) and even in other works this process is evident. Could you talk, in your long experience, about this crucial aesthetic aspect?

I call those (many) pieces “studies” because they were designed to pose questions:  how can a computer relate to our body and our movements? I’m not sure the world has answered this question even today. Most pieces with technology (including most of ours) fail. The failure is to integrate the technology. Far more often it is separate. It is merely “shown-off”, like another stage set.

But, OK, lets look at this: “technology as a means of expression”. It sounds like Marshall Mcluhan’s media being message. Humans build technology. We hold it with our meaty hands, shake it and sniff it, like the monkeys in 2001: a Space Odyssey. Technology can amplify our small gestures and make them powerful. With our motion tracking system, I can lift my finger and literally shake a large theater with the sound for example.
As dancers who work with technology, I believe that we can play a role in initiating a kind of “second digital age”, one which might actually meet the original dream of true empowerment. My own work is most intensively concerned with the question of mapping. Mapping means: “what causes what” in an interactive system. Mapping human expressive movement in an intuitive way, I believe holds the key to this future. I believe there is a “right” and “wrong” way to do mapping. The wrong way, is the arbitrary way, implied by the specific technology being used, and guided by engineers. 
In the case of motion tracking, this has lead, for example, to viewing space in terms of a Cartesian system of coordinates, x, yand z, in 3d space. The theory is that if we can locate certain “key points” on the body, such as its joints or the tips of its extremities, and then give these and x, y and z value, then we have obtained relevant data. But this is not exactly true. This is not how we humans perceive our body or how we sense movement! We do not learn to walk by bending left knee joint 10°, then bending right hip joint 20°.

Another common “wrong” way to do mapping is an approach generally known as “skeleton extraction”. This is something that “InfoMus”, “Microsoft” and others have championed, in which a simple stick-skeleton is generated out of a person’s movements. This is attractive visually, or at least catchy, and some visual artists like this, for example, professional animators and CGI (computer generated imagery) people for the film industry now routinely use this trick since the stick figures actually look and move like a person, especially if you put a « skin » onto them in the final animation. 

But I believe these skeletons, like the Cartesian coordinates they are derived from, are actually quite useless in terms of giving a person the essentially synaesthetic sensation of hearing their movements. 

Finally, there is the concept of gesture recognition. A lot has been made of this in recent years. Think of the smart phone “pinch”. When we speak, we do so with vocal gestures, such as words. Composers speak of musical gestures, and of course there are the physical gestures that, for example, mimes might use. Dancers famously hate mime though. It is in a sense the opposite of dance. Instead of being about movement ambiguity it is about its fixity. Gestures can embody concrete concepts, this is undeniable, but they strike me as not very important if your goal is a synaesthetic and intuitive experience. 
On the whole, seeing gestures as building-blocks of expressive movement or dance seems miss guided. It implies cognitive processes and not the kinds of intuitive ones that can lead most directly to palpable and engaging interaction…

…so, if not Cartesian Coordinates, skeleton extraction or gesture recognition, what then?

This is not an easy question to answer quickly, indeed, it occupies a large part of our research. I usually think of it in terms of compositional devices (“hooks” as they are sometimes referred to in the music business). They do not need to be obvious, they often are not, but they do need to be powerful. They arise through particular combinations of musical sound (including of course its transformation), movement and the basic human psychology that brings them together. There about a dozen of these that we use frequently, and another dozen or so we have found that work in very specific situations. Although working out such “strong mapping” sometimes involves complex planning, multi-modality and so on, when it is done well, i.e. when we are successful, the result for the user is a simple and pure sensation, “oh! that’s me!”.
The simplest example perhaps is the melt down. Lower pitch, lower body position. It seems to be almost universal, most languages of the world say “lower” and “higher” for the different frequencies of sound (as if the music was truly somehow “lower” or “higher”). 
Another basic mapping is larger dynamic of movement, to louder sound. This may sound completely obvious or simple, but it is not. For example, a movement of the entire body – what would involve many pixels changing value – can be very gentle and soft. And on the other side, a small body part moving very fast, a high-energy gesture, will appear to the motion tracking system as a low level of dynamic, since very few pixels are involved. Thus, quite complex algorithms are needed to design systems that « feel right » in practice. What is achieved is synaesthesia, or the mixing and overlapping of our senses. It is what gives us a feeling of having the music “in us” or “coming from us”, and it is always fun when it happens. 

What are, from a technical point of view, the principal aspects of EyeCon system that you use?

It based on frame subtraction that is comparing similar video frames, or still images, and looking for differences. If there are differences, well, that pretty much means that either the lights have changed, or someone has moved. This basic trick can be played out in one of two ways:
a) you compare the current video frame, to one stored in memory;
b) you compare the current video frame, to the previous frame;
The former gives you position and shape, while the latter gives you activity and direction of motion 
It was developed by Frieder Weiss and one can download it and experiment with it at www.frieder-weiss.de.
I should say though, these days I more often use “EyesWeb”, which is being developed by Antonio Camurri and his team “InfoMus” in Genoa, Italy.
 

In this direction, your approach to technology is also didactic. In this framework a project like Motion Composer is very important or, in the same context, the motion tracking used with persons with disabilities. Could you speak about this and the aims of the project in somatic field?

The MotionComposer project is to build a device (the motioncomposer) that turns movement into music for persons with disabilities. The major requirements are these:

1. Intuitive Systems (that need no explanation)
2. Keep people from looking at the screen, by keeping the interface simple and uninteresting
3. Easy to operate, most therapists want to concentrate on their patients, not on a cool device
4. Music that is
   1. Beautiful;
   2. Clearly under the control of, or influence by the mover;
   3. Varied;

II. Bodysoundscape

The body is a primary source for the sound in your work? 

Well, yes and no. I mean, “yes” in the metaphorical sense. In the sense that this is my dream, yes. But obviously I work with composers, and thus, in a sense “music is the primary source for sound in my work”.
I do like “found” music: birds, wind, wind chimes. Childrens’ voices. I have in this sense sometimes been the composer and choreographer, but really the best work grows out of collaboration. Composers today, the ones I work with, bring an amazing wealth of knowledge to their work which I cannot match. You know the field of Psychoacoustics?  The sounds we hear are not the same thing as the sound that is really there. 

This question concerns an implied sonorous dimension of the body. I’m asking this question because I feel that awareness of such an aspect can open up a process in which listening itself can give the interpreters a strategy to reorient the perception, and create something that is out of the repetition of the same motor patterns in the composition of gesture…

That sounds exciting, but lets go one step at a time. First we need to find a language, a grammar, for this movement-to-sound mapping. Some sounds feel “right” for certain body parts, or certain gestures. This is point. Certain sound-gesture combinations work intuitively. They do not need to be learned. If you look at some of the children with disabilities in the videos at motioncomposer, you can see how direct, and automatic this effect can be. That is, I know it is possible, we have done it! This is probably partly cultural and partly biological, but this doesn’t concern me. I just want to get it to work!  It is of the utmost importance for the goal of our project. 
I have my collection of little tricks, but there does not seem to be anything resembling a “reader”, or a lexicon. Remarkably, there is almost no body of knowledge in this area, no research taking place.

III. The stage is a perceptive device

Your approach to the composition concerns multimodality devices: performance and installations. You designed different formats on the basis of themes or suggestions. I find this way of working on the scenic device very interesting. Can you talk about the common elements and the difference between devices through examples such as, in certain sense, Jenseits der Schattene (2006) and e-touching (2004-2006)?

Jenseits was full of projections: projections on the floor, the wall, on the performer and on the sculpture which was the stage set. This was the beginning of the end for me, with projections. E-touching was different short dances all dealing with this very concrete idea of touch. It used a physical controller, a bio-sensor, that measured when two dancers were touching each other. So it is not concerned with illusions, but rather if the dancers are actually touching:  yes or no. I am not a big fan of illusions, and thus I am not a big fan of theater in general. I like to give people a real experience. This was how Cunningham worked. Its why he did not have narrative in his work. It is why I love his work, and it is something that has informed my own. 

Your work, in technological terms, has an aspect in particular that for me is very interesting: a particular dimension of presence, understood as a sort of double. Can you talk about the way in which you operatively can reach the depiction of re-materialization of the body?

This would seem to contradict what I was just saying above….
That is, I say that physical reality is more important to me than illusion, and yet I have used illusory “doubles” of the dancers. But such representations of the performer are not based on fictional characters, but rather on the dancer herself or himself doing what they are doing in that moment, and it is because the audience knows that these mediated representations are live, and thus are perceived as a part of the performer. 
But this aspect is present in my experience, in the case of some of the early work that Frieder Weiß did with Palindrome, this “doubled” presence, was itself doubled, as we would project a mirrored image of the performer, in Shadows (2003) for example. The performer’s image, and the mirror of that image were both projected. This doubling, or re-doubling catches the eye. I and I again. Two me and two eyes. We thus see ourselves, again, as in our shadow image. Jenseits der Schatten (2006) was another piece dealing with this idea.

IV. Coda: around the future

The workshop is a central moment in your production. This is – for my point of view – an important aspect for different reasons: education on creativity and on performing arts; sensibility to use technologies in a compositional field; education of a perception body and exploration of his potentials. For me these aspects are the elements of a new “ethic of creativity”. Could you talk about the elements that move your workshops and its motivations?

Yes, I consider workshops important. There are almost as many misconceptions about interactive media as there are conceptions. Many confuse automation with interaction, for example, but the larger challenge, as you suggest, goes beyond misconceptions to something more fundamental, yes, it concerns a new “ethic of creativity”. 
Humans use tools in work, in play and in art making and these tools are going through a revolution. Mechanical tools extend our limbs, for example, and make our arm movements more powerful. But the digital age does not offer a new degree of power, but rather a new kind of power, a new meaning of what power is.

Dancers traditionally (and still) go on stage where they use special lighting equipment and other effects to make them seem important, exaggerated, somehow more than human. A tweet can reach a thousand people in a split second, but my point is that digital tools can go beyond a mere amplification of gesture. Interactive systems play with our perception and proprioception (our sense of our body in space), and this “play” has serious implications. Sounds can be confused with movements, movements can be confused with lights. This is what I meant earlier when I spoke of synaesthesia. These kinds of confusions are not interesting because they trick us, but because they reflect and touch upon deeper realities in the way we experience music, dance, art and society. 

So it is a small wonder that it is taking artists and engineers a little time to figure it out. Longer than I might have thought, I must admit. I am still seeing the same failings in interactive art that I saw ten years ago. To me this means that knowledge of the body’s role, and the role of human movement in human expression, is still under-rated, and under-used. 
So I would like to join with you in welcoming a new age, and a new ethic of creativity. One in which body is brought back in the picture, back into the fray, one in which movement is regarded and respected again as art.