Sometimes you have to go back to the sand pit and have some fun. We also play with sand although it is slightly modified. We play with silicon. We use it to create things beyond engineers’ imagination. Monika Redzisz conducted an interview with Wiesław Bartkowski, director of Creative Coding studies at University of Social Sciences and Humanities

The Creative Coding laboratory at the Warsaw Campus of University of Social Sciences and Humanities feels pleasantly chaotic. It looks like a mix of multimedia studio, carpenter’s shop and lecture hall. Unidentified objects the purpose of which I am not even trying to guess are leaned against the walls, scattered on the tables and hanging from the ceiling. My host in the lab is Wiesław Bartkowski*, director of post-graduate Creative Coding studies, interaction designer, and complex systems researcher.

What is your job actually? Is it coding?

Not in the traditional sense of that word…

Art, perhaps?

No, although sometimes it is almost art.

Design?

Kind of, but… Well, it is hard to explain… It is something at the cross-roads of several areas but also something that does not belong to any of them. Let me put it this way: we do unnecessary things here.

I am interested in physicality of programming or, if you wish, code matter. I code, but I try to explore such forms of interfaces that are more similar to interactions with the physical world than interactions with the digital one. I create a physical manifestation of the effect of a code. In other words, I try to connect bits and atoms, the physical world and the digital world. So as to something that is physical could also have an effect on something that is digital and vice versa.

I do not understand.

Think about it. How do we usually come into contact with the digital world? By touching the screen, right? It is hard for us to imagine that there may exist other manifestations of digitization apart from those with a graphic form. I like comparing interaction with the digital world via a graphical interface with kissing someone through glass. We have bodies, we are tangible beings adapted to physically interact with the world. This is our world.

We have known for a long time that René Descartes was wrong about his idea of separating body from mind. By the way, I highly recommend the book called “Descartes’ Error” by Antonio Damasio. Body and mind are one. They form one cognitive system. If we had a different body, the way we think would be different. Meanwhile, more often than not, we interact with the digital world immaterially. Before our eyes there exists only a projected image of the world consisting of pixels. That is not our world! That world is non-human.

It does not sound nice…

No, it does not. Neurobiologists compare our attention to a muscle – you can train it but eventually it gets tired. We do not have unlimited resources of attention. One of the characteristics of screens is that they are highly compelling. They require a lot of our energy, we must control them all the time. because that is not our natural world, and we do not feel it. It is different from the physical world in which we do many things automatically, do not have to stay concentrated, and so we do not tire that much.

The code is just another type of material, a matter; it is just like wood for a sculptor. This is how we see it and this how we inspire students with that approach. We use the code for sculpting, our hands get dirty with the cod

People working in the Creative Coding laboratory are most interested in exploration of physicality. We study alternative, tangible forms of interaction off the screen.

That sounds very much related to modern conceptual art. Your projects can be even admired at exhibitions.

Art is not the path we want to take. We do not need any concept, we do not even care about aesthetics. We just do experiments. Because we are simply curious. Because it is our passion. But we have no goal.

How is it possible not to have a goal? Having no goal seems outrageous, anarchistic, and even immoral. This is how kids or madmen act.

But very often also artists or scientists. You are right. If you follow these studies, you sometimes want to act like a child. Sometimes you need to go back to a sandpit and play. We also play with sand, although it has been slightly modified. We play with silicon. We use it to create things beyond engineers’ imagination as engineers are focused too much on optimization, efficiency and reliability. It makes me happy that when they sometimes visit us, they cannot hide their admiration for our work and want to play the way we do. We also greatly admire their work. It is worth meeting halfway.

To do something really ground-breaking, you must be brave enough to do something useless. Random wandering may lead you to discovering something completely new and not only to an improved version of something that has already existed. A lot of our objects are the result of an error. Students sometimes discover with surprise that something works differently and is better than what they expected. To err is important, but to stay open to an error is even more valuable.

Any example?

Klementyna Jankiewicz did for example a project entitled “Urban Echo”. She was curious how, with the use of such electronic beings, or, to put it differently, electronic graffiti, you can interfere with the public space. Ultimately, after many experiments and errors, the graffiti transformed to a swarm of electronic objects – transparent spheres with exposed electronic elements that deconstructed human voice and then reconstructed it in the space. By talking to the swarm, we affect the way it behaves, we “play” it, and the swarm responds to us with sound and light. You could “chat” with Echo at Maker Faire in New York and at the Tomorrownauts finals in Warsaw. Echo also performed in a play about artificial intelligence in Powszechny Theater directed by Monika Popiel, boldly experimenting with theatrical arts.

That is a bit odd. Where does it lead to?

Interactions. People are curious about experiencing interactions with such objects. It becomes even more apparent in the project of Gosia Nierodzińska, who is creating a unique collection: the collection of hugs! She has created an outfit that detects the way we hug somebody and makes a poster, a sort of an image of the hug. Gosia wants to draw our attention to the unique nature of the moment of a hug. Each of us is different. We hug in a different manner so we generate different images. People become interested and they start hugging. Gosia has hacked the technology. The effect of her project consisted in “bringing together” rather than “setting apart”, which is associated with technology. She is still hacking by bringing people together in her project “Meeting, warming”.

When the object is material, it appeals to our senses. We are close to emotions. In a way, we create a social actor (a notion from the theory of embodied cognition) which moderates people’s behavior. This can clearly be seen in “Magnetio” by Magda Kurowska and Kamila Koźmiańska. A tube with balls at the top. Balls are nice, you want to touch them. Once you have touched them, you can feel that they react to what you have done; they get warmer and they start sticking to one another. It is not only relaxing. It also creates a bond between people who start building something together. You deal here with several psychological effects. For instance, there is a chameleon effect which, in essence, consists in the fact that when people synchronize their behaviors, it brings the feeling of closeness between them and leads to more trust. After interactions with “Magnetio” I often hear people say: “I am shy but thanks to “Magnetio” I got to know more people during a conference break than I usually do”.

Who decides to choose Creative Coding studies?

Very different people. Some have a degree in arts, some are designers, architects, creative directors, copywriters, but in fact education is irrelevant. Whether someone is admitted or not depends on an interview with us and on that person’s portfolio. We care about what your passion is and what you have done and we do not care what your formal education has been. The same goes for lecturers. Due to the fact that we have been following different roads, the teaching style is completely different from the one that is commonly accepted.

I joke that the only group we do not admit is professional programmers. I have been teaching programming for over 27 years, most willingly those who say that they would not fit in that area. I think I have found a way to get to that group. I describe the code as if it was a type of material that you can form by experimenting. My methods of teaching are certainly different from those used at IT studies. I do not think that it is the right approach to understand exactly how it works. It may be useful at a later stage, but at the beginning the most important thing is to create something.

Generally speaking, I am an antidisciplinary man. I believe that you should pursue your dreams and, on the way, discover different disciplines that are necessary to achieve your goal. If we do that, we learn more efficiently, because we use things we care about. We learn best if we act and if we get experience. The most important thing is to stay open to experimenting and to be willing to learn new things.

Youtube movie URL: https://youtu.be/8vunNWQorFQ

But how are such projects handled by persons who cannot code? And why would they need the code anyway?

As I said earlier, the code is only another type of material, like wood for a sculptor. This is how we see it and this how we inspire students with that approach. We use the code for sculpting, our hands get dirty with the code. And there is a bit of rebellion in that. I like referring to the words of Douglas Rushkoff: “Program or Be Programmed”, because it is a chance to get access to the control panel of the civilization and, in consequence, to have a say in the discussion about our future.

Technologies are provided by engineers, but somebody has to decide which technologies will become a part of our life. This is a job for a designer who should understand both the technology and the world of humanistic values. This is why I want the graduates of the Academy of Fine Arts, industrial design, architecture, and humanistic studies to open up to the world of technology and to present their perspective. They will then be able to help the engineers to create the world in which the technology will not scream at us and degrade us but will help to improve our skills. And, just as in Gosia’s projects, to bring people together and not to set them apart.

You are looking for visionaries…

Yes! We like for example Hiroshi Ishii and his Vision-Driven Design Research. Hiroshi is the creator of the idea of radical atoms – atoms connected with bits. Meaning that if a man marks its presence in the digital space, it will change the condition of the physical space. And if we do anything in the physical world, that will be reflected in the digital world. His doctoral students are trying to create prototypes that analyze the vision of radical atoms. One of the most successful ones is called Transform. We can say that it is a table, but a table which is physical and digital at the same time.

That is very impressive. Does it use artificial intelligence?

Not in this prototype. Ready-to-use scenarios have been implemented. But this is an interesting idea for interacting with AI off the screen. Machine learning will allow to interpret movements and gestures around the table. As a result, the table starts moving. The goal of creating that prototype was different. It is hard to imagine what connection of bits and atoms means until you see or touch it. It is even more difficult to imagine what purpose it may serve. That kind of prototype makes us wonder how such a technology could be used. How do people react to that? Does it even make sense to follow that path?

Do you use AI in your projects?

I would say we explore it at the point of contact. In recent years, a lot of ready-to-use AI tools have been developed, including the ones that have been created for artists or designers, that allow us to experiment with artificial intelligence similarly to how we experiment with programming, i.e. without immediately getting too much into technical and theoretical details concerning construction of artificial neural networks and methods of teaching them. Instead, we try to hack what already exists and change its purpose. We do not practice science, we are inspired by science. Interestingly enough, we often encounter feedback. Such artistic or design experiments with technology give rise to an idea that is further explored by scientists. Art & science is a very interesting combination.

As for the experiments with artificial intelligence, I particularly like the one to which Kaiji Moriyama, a well-known Japanese performance dancer, was invited.

Youtube movie URL: https://youtu.be/tLFe2AzCodk

This is an example of how new forms of expressions or even new forms of art can be created with the use of AI. An artificial neural network learnt to convert impulses from Moriyama’s muscles to sounds emitted by a Yamaha pianola. A dancer has become a pianist. After the concert he said that when he was a little boy he dreamt about playing the piano and that now he could finally feel what it was like to become a pianist.

I saw that. Beautiful. How was that done?

First, there was the training process. He had electrodes attached to his body which for an extended period of time gathered impulses from different groups of his muscles. While he was dancing, the network was trying to translate his moves into sounds. However, he could hear what sounds were coming from particular moves, so he was able to control them. That was a very interesting type of machine learning: supported by a human. In situations like that, the network often learns a lot and finds more “human” solutions than it would if it learned from mathematically formulated reward function.

There is this cool example of research on machine learning conducted by OpenAI.com: a leg, simulated by a computer, is learning how to backflip. The computer is showing two attempts of a backflip: one on the screen on the left and one on the screen on the right. With information received from a human evaluator about which attempt was better, the computer learned a backflip after only about 900 bits of feedback. When the computer had to learn alone, not only did the learning process take more time, but also the final backflip looked unnatural and distorted.

What does exploring at the point of contact mean?

It means that we operate between disciplines, or, to put it differently, we operate in an anti-disciplinary manner as Joi Ito defines it. Now we also use methods of and are inspired by artificial intelligence.

It is an interesting domain. Here, the complexity is even bigger and the effects are harder to predict than in the case of generative algorithms. But it does not mean that we operate in the area of AI, because what gives scientists sleepless nights, for us is just an interesting error. We do not need big data sets. On the contrary we like the small ones although we know that those lead to unexpected effects in machine learning. But we generate errors on purpose to confront an effect which is useless for scientists and interesting for us.

We stop understanding why we chose a particular movie and why we follow a given road and not another one. We stare at our mobile phone to get weather information instead of looking out of the window. What is next? Are we going to attach sensors to our bodies and ask a cloud system how we are doing just to discover one day that we have completely lost contact with our corporeality?

One of the graduates of Creative Coding experiments with GANs, i.e. generative adversarial networks – networks that have been invented to generate realistic images, for example images of a face. Theoretically, you would need there at least 10 thousand images but she persisted in her decision of trying to teach the network the images she would draw herself just to see what would happen. So far she has managed to draw one thousand images…

Why do we even say “artificial intelligence”? This term is carries negative energy because it is substantially antagonistic: artificial means non-human and non-human means alien. Us and them. And certainly “them” does not mean anything good; what is even worse, “they” may be dangerous. Instead of saying “artificial”, I prefer using the term “enhanced”. If we learn how to cooperate with a machine, we can do more with it than without it. I would opt for development of artificial intelligence so as it could support ours. Let the non-human intelligence support the human one in all aspects people are weak at. So the enhanced intelligence, but not the one that replaces us, can solve a problem for us without telling us how it is done. In that situation we will soon stop understanding how this world functions!

But it has already happened. We turn on GPS and it chooses the best route for us; we turn on Netflix and it chooses the best film for us.

Exactly. But that is only the beginning. We can still make wise choices to better shape the future but everyone should get involved in the discussion and, maybe, just to start the things off, learn how it works? Meanwhile, we stop understanding why we choose that film and we stop understanding why we follow that route and why we do not choose another one. If that continues, we will lose our sense of orientation in space and we will not be unable to cover the distance of two streets without having our GPS switched on (it is already a problem for some people). We stare at our mobile phone to get weather information instead of looking out of the window. Nobody looks up to the clouds anymore because why would they? What is next? Are we going to attach sensors to our bodies and ask a cloud system how we are doing just to discover one day that we have completely lost contact with our corporeality?

Technologies are provided by engineers, but somebody has to decide which technologies will become a part of our life

It is the matter of our comfort and our laziness, which is probably impossible to eradicate.

But maybe it is possible to create smarter designs? In my opinion designers and researchers should be responsible for convincing people to and helping them with creation of a system that would develop humans and not degrade them. You can for example design a tool that would try to replace a doctor, a tool that would provide a doctor with a ready diagnosis. In that world the doctor would soon lose his skills and intuition because of no training and because of relying on a device. He would transfer his skills to the system. But you can also design tools that would work like glasses or a microscope and that would make it possible for the doctor to see something that would be unlikely to see otherwise. Although that system would help better understand relationships and make a better decision, it would not offer ready solutions. It would develop the doctor. The same analogy might be applied to any learning system, even to GPS we have just discussed.

It has always been like that. In the past doctors were able to read more from the eyes and skin of a patient; today they cannot do that because, in fact, why would they? Instead they refer patients to undergo thorough medical tests. But they are highly specialized, each in their field.

I think that nowadays professionals are overspecialized and that, because of that, we can overlook something important. Specialized surgery? I am all for it. But sometimes all you need is simple physiotherapy. There are not enough people that would combine different specialties into a whole. Each person should be highly specialized in a certain domain but should also have a general idea about other areas. And they should not say: “I’m a programmer and I create superalgorithms. And you are too stupid to understand that so I won’t even bother to explain anything to you”. At the IT faculty where I studied I am trying to prove that this way of thinking leads to nothing good. In other places I am trying to fight the clichéd division between humanist and scientific mind. Those who are interested in how I do that might want to see the “Code Matter” exhibition.

---

*Wiesław Bartkowski – director of post-graduate Creative Coding studies offered by School of Form in Warsaw. He is a staff member of Shool of Form and School of Ideas at University of Social Sciences and Humanities. Creator of Polish pioneering education programs combining information technology with humanities and creative use of technology. He teaches artists and designers how to use the material consisting of the combination of the code with electronics and digital fabrication.

---

Przeczytaj polską wersję tego tekstu TUTAJ

• art
• education
• explainable AI
• Joi Ito
• OpenAI
• SWPS
• Wiesław Bartkowski