A Little Story About Ourselves
(Automata and Artificial Intelligence in the Swiss art and technology scene)
The little boy looks focused on his task, holds a stylograph in his hand and slowly writes the next letter in an elegant calligraphy, attentively following the movements with his eyes. The result looks exactly as it looked three centuries ago when the boy was created. Automata are designed to repeat their actions over time.
The miraculous mechanisms of The Writer –a boy automata crafted by the Jaquet-Droz family in 18th-century Switzerland — are not an exception. However, the specificity of this piece of clockwork is that it can be programmed to write any forty-letter sentences. The machine is on exhibition at the Neuchâtel Museum of Art and History, alongside two other masterpieces from the Jaquet-Droz collection: The Musician and the Draughtsman.
Despite the marvel of their mechanisms, there’s always an unsettling quality about automata. In an article for Aeon Magazine, Ed Simon reflects on the history of a 16th-century machine, now in exhibition at the Smithsonian Museum. The machine is a wooden monk that ambles in a circle, lifting a cross and moving its mouth in prayer, occasionally kissing the crucifix and striking its chest. Commissioned by Phillips II to Juanelo Turriano, it was intended to fulfill the duties of a monk and offer prayers for the life of a dying child. While the monk’s mechanical motions may be repetitive, Simon delves into its disquieting nature:
“One reason for his unsettling quality is that the monk’s purpose isn’t to provide simulacra of prayer, but to actually pray. Turriano’s device doesn’t serve to imitate supplication, he is supplicating; the mechanism isn’t depicting penitence, the machine performs it.”
Simon’s words made me realize that our fascination for automata isn’t solely about their complex mechanisms, but also about the enduring intentions of their creators transcending time through the repetitive actions of the machine. Automata, like any other human endeavor, are imbued with intention. Turriano’s monk continues to pray for the life of a child who passed away over four hundred years ago.
The simulacrum of the eye movement in Jaquet-Droz’s Writer has also an unsettling quality. It performs the action of looking at the writing while raising the arm to charge the stylus with ink, before adding another letter to the sentence. Maybe those movements convey the patience and dedication the creators put into the fabrication of the machine. Nevertheless, the boy is not seeing its handwriting. Is he?
When I arrived in Zurich on June 10, 2023, I went straight to visit the exhibition Data Alchemy. Observing patterns from Galileo to AI, which was part of the Zürich Art Weekend. With works and installations occupying the whole building of the Collegium Helveticum, home to the Semper Observatory, and curated by Liat Grayver, Junior Fellow at the Collegium Helveticum, and Adrian Notz, Curator at the ETH AI Center, the exhibition delved into the creation of meaning through pattern recognition –something we humans are wired to do.
The works on the exhibition spanned from Augmented Reality experiences –Oliver Sahli & Chris Salter’s Possible Worlds… was installed at the observatory’s dome, allowing the visitors to blend the physical world surrounding them with an imaginary vision of the cosmos inspired by Giordano Bruno’s proposition of an infinite Universe- to works on paper and mixed media –like Blue Transmutations, a mixed media installation by Liat Grayver Robert Nißler & Marcus Nebe that uses the results of Grayver working with Egyptian Blue pigment on Japanese paper, coexisting with the processing of video footage of those paintings captured by a special camera used by the Nanoparticle Systems Engineering Lab at ETH Zurich.
The first piece we ran into upon arriving at the exhibition immediately grabbed my attention. It wasn’t until a couple of months later, when I exchanged emails with Liat Grayver, that I confirmed the connections I was making at that moment weren’t far from what the artist and her research group had explored during their process.
While I watched Grayver working in a room full of huge rolls of painted paper, alongside a robotic arm equipped with a brush at its end –almost as if she were its assistant- I couldn’t help but think of Jaquet-Droz little automaton, writing with the stylo in its hand. A narrative arc in the history of Swiss art and technique was unfolding right before my eyes. But something else was going on in that room.
The piece, by Liat Grayver, is called Synaesthetic Strokes and explores the development of a generative system that uses autonomous active perception to learn from its actions and improve its skills at brush painting, through a visual feedback system.[i] In its updated version, the E-David robot can even learn how much pressure to apply to a sheet of paper to achieve different widths of brushstrokes and, by doing so, create different pressure profiles that the system categorizes and stores to use when needed.
Mechanical automata –such as the Jaquet-Droz’s Writer- have a physical record of their motor memories, stored in little plates that, read by a mechanical hand, translate the shape of the plates into the movement of their parts –for the Writer, three different sets of plates are responsible for the movements of the arm, one for each spatial axis.
It’s fascinating to consider that the metallic plates utilized in the construction of those automata to store and reproduce their movements repeatedly –essentially their physical memory- are in a way a precursor to vinyl record technology, as also observed in the previously cited article. Less evident, the grooves in a vinyl record are the physical shapes that guide the needle’s movements. As the needle glides along, its rapid and precise perturbations are transduced into sound, unveiling what is engraved in the physical memory of the record.
I pondered the infinite possible architectures of human memory, capable not only of storing instructions for executing actions but also responsible for the subtle variations in every repetition of those actions. This memory system is a co-evolved, organic, and fallible mechanism rooted in the biology of complex, living organisms like ourselves.
Over a century ago, somewhere between Jaquet-Droz automata and vinyl records, German zoologist Richard Semon introduced the concept of the “engram,” referring to the tangible mark a memory must imprint within the brain, likened to leaving behind a subtle trace, much like a footprint, or a groove. A recent study conducted in mice reveals that the DNA of brain cells undergoes specific folding patterns to facilitate memory retrieval. Different as we are from automata, there seems to be a physical record of our memories too. This suggests that the spatial encoding of memories, as proposed by Semon, is something that holds significance.
In the case of E-David, the system utilized by Grayver at the exhibition, executing actions is something entirely different from reading physical memories. This process relies on a deep-learning algorithm initially trained with images and then with the analysis of its actions through its sensory system. Finally, this information is translated into the movements of the robotic arm. It could be said that the entirety of this complex system constitutes the robot’s motor memory.
In the article on the development of E-David, it is mentioned that Grayver collaborated with the team to “investigate methods to redefine one of the primitive forms of art — painting — in our current technology-based era”[ii]. In fabricating these machines –being the Jaquet-Droz automata, Turriano’s wooden monk or the E-David system- we’re always seeking to understand ourselves, building simplified mechanisms that can replicate our actions and imbuing them with our intentions. Pierre Jaquet-Droz wanted his automaton to write, not to simulate the actions of writing. Turriano needed his monk to pray for a boy, not to simulate the praying. The team behind the development of E-David is trying not only to make the robot learn how to paint but also to gain insight into the implications of robot learning.
Machines serve as mirrors that help us understand human actions, offering metaphors to deepen our understanding of our place in the world and our interactions within it. By constructing cellular automata and neural networks, we try to unravel the workings of our brains and explore the profound questions raised by the phenomenon of consciousness.
Artificial Intelligence is a name we give to a broad group of tools and systems that perform different tasks, yet, fascinating as they are, we continue to impose anthropocentric concepts –such as intelligence, thought, or consciousness- to describe the way those machines operate. Conversely, we also draw on machine-derived concepts to understand our biological processes, such as referring to our brains as computing information.
The exhibition at Collegium Helveticum was an opportunity to reflect on the way we perceive the world, which is in fact, a way of reflecting on what kind of creatures we are. Witnessing Grayver work with the robotic arm made me think of the machines themselves. Machines that pray, machines that write, machines that paint and learn. What kind of creatures are they?
Gottfried Wilhelm Leibniz proposed a thought experiment known as the mill argument. He envisioned a machine capable of thinking, expanded to the scale of a walk-in structure akin to a mill. According to his argument, while we could watch the machine’s parts interacting, we could never access its thoughts or perceptions. This concept resonates with the modern idea of emergence: a single neuron is not aware that it is part of a person. It is only through the collective firing and interaction of all neurons that the entity capable of recognizing itself as a person emerges into existence.
A question that arises now is: if a machine were to develop a new form of consciousness: would we be able to recognize it by only looking at its parts working? To what extent that new consciousness would be willing to let us know about its presence?
There’s a poem by Wisława Szymborska called Astonishment, composed of sixteen lines of questions on the nature of self and the enigma of recognizing oneself as a sentient being, existing in the world. The opening lines of the poem read:
Why after all this one and not the rest?
Why this specific self, not in a nest,
but a house? Sewn up not in scales, but skin?
Not topped off by a leaf, but by a face?
As if life and consciousness could arise anywhere, Szymborska questions facts that we take for granted. We have a face and a skin, we are who we are, and we are alive and conscious. But those concepts –life, consciousness, intelligence- are anthropocentric and defined in our terms. Poets –and artists- role in the world, as Szymborska herself stated in her Nobel acceptance speech, is to ask questions, and to raise them. They should embrace a perpetual state of uncertainty, living in a space of not knowing. This constant exploration of the unknown is what drives art to expand the boundaries of what can be explored, defined, and contemplated.
As I watched the robot learning to paint, in a room filled with countless rolls of paper displaying an array of brushstroke styles it had created, I felt a sense of awe, not at the robot but at ourselves. As if something had shaken the certainties we hold about our existence, I thought about the amount of work and time put into the fabrication of the Jaquet-Droz automata and the amount of work put into the development of the E-David system. All of it serves as a testimony to our relentless yearning to understand ourselves.
[i] Gülzow, J.M.; Grayver, L.; Deussen, O. Self-Improving Robotic Brushstroke Replication. Arts 2018, 7, 84. https://doi.org/10.3390/arts7040084
[ii] Ibid.
