What Do Artists Who Paint With AI Want — and Are These Paintings Art at All?

There may be no right answer, but the question is worth asking.

The American artist Pindar Van Arman won a prize in a painting competition. A friend told him he wasn't sure whether to be moved or disgusted.

The argument is about his painting tool: artificial intelligence. At this year's International Robotic Art Competition, two abstract paintings and one piece in the style of Cézanne won Van Arman first place and a $40,000 prize.

The Canadian artist Joanne Hastie came fifth. She met the same reaction Van Arman did: “This isn't art!”

She thinks that's unfair, but she also grants that this is an unfamiliar category, and that people haven't had time to think it through.

Before her 10 years as an artist, Hastie was an engineer who designed medical devices. Last year she began to fuse the two halves of her life, looking for a way to stand out among the several hundred good artists working in Vancouver.

Dip the brush in paint, work it over the canvas for a while, rinse the brush, wait for the canvas to dry, then get ready for a second and a third layer of color. That is how Hastie's robotic arm paints. Its movements are set by code, so she first has to make a mock-up in painting software and split it into foreground, middle ground and background — the same layered approach she uses by hand.

A painting like this usually takes 15 hours, and that is before you count the long break-in period. Teaching the arm to draw a curve — a natural movement any person makes without thinking — cost Hastie a great deal of time writing code and adjusting coordinates and brush angles. Before that it could only draw purely mechanical straight lines, laying the paint down thin and even, looking “very much like a robot.” The earliest arm was off by ±5mm when the brush touched the paper; it could not even manage a straight line, and it snapped a lot of markers along the way.

Under a tradition centuries old, art has always come from human hands, and so Hastie believes a work has to imitate a human style before it feels familiar and approachable — and before it can be said to have any artistic character.

Sometimes she asks herself what all this trouble is worth. Why spend 10 times as long on a painting that isn't as good as one she could make herself?

In fact, leaving aside interest and artistic ambition, awkward as the method is, it makes Hastie think harder about every detail. She started digging into technique: how to tell when a painting is finished, how to mix colors.

“Painting is usually intuitive. Painting with a machine has made me understand and define that process more explicitly,” she told QDaily.

The changes went further. She used to paint only realist and impressionist work, feeling her way forward, adjusting as she went until the picture under her brush took shape. But the arm has no “eyes.” It cannot see how it is doing, so it paints “blind,” following orders set in advance.

That limit, together with brushwork and color that are unconventional to the point of recklessness, ended up giving the artist an idea. She decided to make the most of the mistake, dropped realism altogether, and used the arm to paint abstracts instead.

She says it is the most artistic project she has done.

Van Arman is now in his 15th year of painting with artificial intelligence, and this month he is opening a show near his home. His machine partner has been through several generations, and a single painting still takes a week.

It is fine, slow work.

At the start the robot has a target image, which a person can feed it or which it can generate itself. In the second case it keeps adjusting the target as it paints. Either way, once a new image forms, the brush does everything it can to imitate it. The machine also carries a camera, which gives it visual feedback. A live heat map conducts the work, showing in real time the gap between what is on the canvas and the target, and issuing instructions: where the color is too dark, add a corrective color; where it is too light, thin it out. Now and then it behaves just like a person — steps back to look at the effect, then steps forward to make a change.

When the gap finally stops shrinking no matter what it does, it puts down the brush, knowing it has done all it can.

Van Arman is broadly happy with his partner's work, but every time it finishes a piece he cannot resist stepping in to add a few strokes of his own. He thinks of the robot as a high-school art student: good at copying, not yet able to form a style of its own — and with no idea what looks good.

These flashes of inspiration go into a database, so the robot can train itself to imitate them through a deep neural network. Getting it to inherit his eye and his taste this way, a drop at a time, may take forever. But in the end what Van Arman wants is a machine that paints more like Van Arman, not more like itself.

For what it's worth, by the time Van Arman adds his few dozen strokes, the robot may already have laid down more than ten thousand. That does not stop him from treating his own contribution as the more important one. He once wrote: “My decisions are what finally make the artwork whole. What's interesting is that I know this is the only decision in the creative process that really matters.”

CMIT Robotics, a robotics lab in the electrical engineering department at Kasetsart University in Thailand, took third place. Its approach could hardly be more direct: make the robot copy a human.

A person paints the picture first, and while they do, the brush records the position and pressure of every movement. Once the machine has that data it doesn't even need extra code. It follows the record stroke for stroke, repeating almost every technique and reproducing the whole painting.

Using the same principle, the team is now working with dancers who perform khon, traditional Thai masked dance, hoping to teach a robot to dance.

The engineer Chowarit Mitsantisuk told us the copy already matches the original by more than 90%, and that it could one day reach 100%. Repeatability plus fidelity means their robots could stand in museums and “replay” an artist's process for the public, and it also makes it possible to analyze technique and style from a technical angle. The team has put the data files online, where anyone can download and use them, or alter the movements themselves.

Mitsantisuk said more than once that the team wants to give artists better tools and to get children interested in machines and programming.

Now, back to where we started: is this art?

Hastie finds the reflex dismissal unfair. To the people making it, digital art is art, a photograph taken on a DSLR is art, and so is a sculpture Rodin had assistants help build. So why doesn't a work she finished with a robot count?

The same rejection, she says, showed up in the early 20th century. Impressionism was new then, and not much liked. In the same years the invention and spread of the camera gave artists a different kind of anxiety: if a camera does it better than I do, what am I for?

That panic and confusion have long since passed. Photography ended up pushing every school of art to innovate. “Representation” was replaced by “expression” as the main aim of modern painting, and Impressionism itself flourished. Artists have always experimented with new technology, Hastie says, and they always will — the more tools there are, the more chances to make something.

In the end she thinks the definition of art has nothing to do with how a work is made and everything to do with what it does: “What matters is that it makes the viewer think... If someone reacts to my work, then whether it was made by hand or by robot, it's art.”

Still, she directs the whole process, and she finds it ironic that an arm with no creativity of its own has taught her something. (She has given it a name, by the way: Mozzie.)

Earlier this year the New York art critic Jerry Saltz said that Van Arman's painting was the first computer-made work that struck him as “not looking like it came from a computer.” Then he changed tack: “That's no good at all.”

To Van Arman, even that is very satisfying. A few years ago nobody in the art world considered these things art; now at least some people call them “bad art.” That, he says, counts as progress for artificial intelligence.

He defines the works himself as something between a painting and a print: each one takes time, each one is particular, and yet copying it isn't hard. That doesn't make them factory goods either. Yes, they can be mass-produced. But can't something mass-produced be art? And even “printing” the same picture twice gives a slightly different result each time.

Some of the negative reaction is fear: who is finishing the work of art? If it's the artificial intelligence, have humans been demoted?

Mitsantisuk, who comes out of a lab, says that as long as robots show no creativity they are only carrying out instructions, and “instructions are a tool for the artist” — nothing more. So has creativity appeared?

The competition's website sorts the robots' methods into three kinds. The first imitates a human through remote control, eye trackers and the like. The second has a robotic arm paint an existing picture, either straightforwardly, as in paint-by-numbers (same number for every area of the same color, then fill it in), or more finely, using a camera or sensor to form an error-correcting feedback loop. In the third, the AI generates the image itself through deep learning.

Take CAN, Creative Adversarial Networks, a deep learning model introduced in June last year. The robot studies a large number of paintings across schools, then, through an adversarial mechanism, produces work that departs from existing styles without being too strange — the result has to be neither boring nor unacceptable. Some of the pictures it made not only passed for human work; viewers rated them higher.

That pushes the question of what art is further still.

The subtle part is that Van Arman's robot can already use deep learning to decide on its own what to paint and how. In the course of one painting, the shifts of the target image mentioned above happen thousands of times, and all of those decisions come from the robot. Van Arman thinks it already has creativity; the only question is how much.

Ken Goldberg, a professor of industrial engineering and operations research at UC Berkeley, disagreed: “They'll always do unexpected things, but that's just the nature of most machines... as long as there's randomness in the program, you'll get unexpected behavior.”

Van Arman also thinks AI is not yet qualified to be an artist, though not for reasons of creativity. The largest gap between AI and people, he says, is whether it can tell clearly “when a painting is finished.” That, plainly, is why he insists on adding his own strokes.

His second reason: all art, whether music, painting or writing, is one person communicating with another. Without a person and without that exchange there is no artistic creation. However complicated it gets, a robot will never become an artist.

“Plenty of things stir human emotion without being art. Being startled makes you afraid, and that isn't art,” Van Arman went on. “What really matters is that art tries to stir emotion, tries to make its own work into art. Because the robot has no such intention, it isn't an artist. I, on the other hand, am trying to use the robot to stir emotion, to make the viewer think.”

“So yes, my paintings are art. It's just that I'm the artist and the AI is my tool,” he says.

Even so, Van Arman would probably grant that the similarities can't be ignored either. His essay “Creativity Is Probably Just a Complex Mix of Generative Art Algorithms” turns that over, from the title down. The way his partner “examines” a work, the way its different decision-making “lines of thought” complement one another — both look faintly human.

Andrew Conru, who runs the competition, teaches control systems and robotics at San Jose State University and Santa Clara University. “Most people probably wouldn't believe how similar human artists and AI are,” he says. “Their fixed skills work like algorithms, and their techniques are repetitive too. Just as every human artist searches for a distinctive style, every AI will end up with a style of its own, because its algorithm is different.”

His own view is that anything essentially beautiful, or carrying an idea, is art. But art is subjective, so the final standard should differ from person to person. Conru gives an example. Say four works hang on a gallery wall: the original Mona Lisa, an artist's copy of it, a photograph of it, and a different portrait. Is only the original a work of art? He leaves that call to the viewer, which is why he built public voting into the competition's judging.

From that he is confident this unsettled subjectivity will quietly let artificial intelligence into the art world, and let it contribute. He goes further: in time, AIs will be able to explain what their own paintings mean — and if they can't, gallery agents can do it for them.

For now, though, most artistic decisions are still set in advance by humans working the controls from a distance, so Conru sees AI's exploration as necessary but very early. Most of the work is also technically immature: the brushstrokes don't blend, and it lacks the power to move anyone. Once it becomes easier to use, artificial intelligence may finally be able to handle the whole process of making a work.

By then, he thinks, AI will very likely compete inside the art world, and that will be normal. His reasoning is that the money in this business is relatively fixed, and besides, “‘starving artists' has been a set phrase for centuries.” But he believes AI together with humans will have an explosive effect on art, and may even create business opportunities to match, because after all, “there are still plenty of dreamers in the world.”

Unless otherwise noted, images in this article are courtesy of ROBOTART, Pindar Van Arman, Joanne Hastie and CMIT Robotics

Cover image courtesy of Pindar Van Arman