Research

The robot painter

The owner asked: "Use an image to trace, and run a set of tests until you can get a master painter outcome." A robot painter now paints a reference photo with the app's own engine, using the same tools, loads and strokes a finger drives. Each run is scored against the reference, the result is inspected, and whatever falls short is fixed, whether in the painter's method or in the engine itself. There were sixteen runs.

The first sixteen runs painted a rendered still life (a bottle, a jug, pears and a cloth), as no photo site could be reached. Its images are no longer shown here; the photographs from run 20 on took its place.

The method: how a trained painter works

1. Trace. The photo goes under Trace, as tracing paper. The main contours are drawn with a thin round in a thin, warm mid-tone, like a raw-umber lay-in.
2. A colour chart. 150 mixes are spread across everything the six pigments make, painted as swatches on this ground under this light, and measured, the way painters paint charts (Schmid). Every mix's predicted colour is then corrected by its nearest measured neighbours.
3. Block-in to finish: large flats, then filberts, then rounds, and a last small round at the edges. Each pass places strokes where the painting still differs most from the reference at that brush's scale (Hertzmann's layered painterly method). Strokes curve along the form, bend but never coil, and get a fresh load each time. Lights go on thicker than darks: fat over lean. Fine brushes only work where there's form or an edge, so flat passages stay quiet and broad.
4. Accents. The brightest lights go on last as single thick strokes along the highlight's own shape; the deepest darks go on lean.
5. Edges. Strokes carry paint right to the canvas edge.

Run it with NODE_PATH=$(npm root -g) node lab/robot-run.cjs <reference.jpg> [out]. It writes the painting, a reference | painting | error sheet, and a log with the score.

Sixteen runs

The score is the mean colour difference (ΔE, lower is better) between the painting and the reference, both slightly blurred so it measures the picture, not the canvas texture.

Run 20: real photographs, and painting like a painter (2026-09-28)

The owner: "I want to see far better art than what I see in the robot painter." Two things held the robot back: it had only ever painted one rendered still life, and it painted every pixel with equal care. Run 20 changes both.

Real references. Three photographs from scikit-image's sample data, all free to use: a cup of espresso (Rachel Michetti, CC0), a cat's face (Stéfan van der Walt, CC0) and an astronaut's portrait (NASA, public domain).

What changed in how it paints:

1. All eight pigments. It had still been mixing from the first six, so a brown cup came out cadmium red. With burnt sienna and burnt umber the browns and deep warm darks are right, and the lay-in is done in the earths, as a painter's is.
2. Masses, not pixels. Each pass paints from the reference simplified at its own brush's scale by a Kuwahara filter, which keeps the edges of shapes and flattens their insides: the painter's squint. Big brushes see big shapes, small brushes smaller ones.
3. Strokes along the form. Directions come from the structure tensor of the simplified image, smoothed over the brush's size: a calm field along edges and across forms, with a confidence measure. Where the form has a clear direction, strokes run longer.
4. Intent: where the detail goes. A focus map (local contrast, weighted toward the middle) decides where the fine brushes and the final accents work. Elsewhere the broad strokes are left alone, and only a glaring miss is fixed. Accents went from about 400 scattered flecks to about 90, all in the subject.
5. Fat over lean. The block-in goes on thin and oily, so it levels and its stroke edges don't stand up; the paint thickens pass by pass and only the late lights go on thick.

Mean colour difference from the photo (ΔE, lower is closer). Each image: the photo, before, after.

The paintings at full size:

The astronaut's paintings are no longer shown: the photograph is public domain, but it shows a real, identifiable person, and a likeness can't be used to market the app. The espresso led the app's sample paintings and the App Store screenshots instead (2026-10-01), until the four below replaced it and the cat.

The app's sample paintings (2026-10-01)

I chose four free photographs from Pixabay for the paintings that come with the app, the App Store screenshots and the website: a latte, an alley in Bologna, a farmhouse in the Alps and a turquoise scooter (credits below the pictures). Each was cropped to the phone's shape and painted by the method above.

Each image: the photo, and the painting.

Photo credits (all on Pixabay, under the Pixabay Content License): the latte by Engin_Akyurt, the alley in Bologna by Marion_22, the farmhouse in the Alps by MARTINOPHUC, and the scooter by Coba1406. Each photo is cropped; the scooter's small maker's badge was taken out before painting.

The scooter shows the palette's limit: none of the eight pigments mixes a turquoise, so ultramarine and white give a cooler, greyer blue. Before painting, the small maker's badge on its front was painted out of the crop, so no brand shows. A fifth, gondolas in fog, was painted and left out.

Tried and left out:

• Drying the block-in, and the finished painting, to take away wet paint's gloss. The dried block-in's relief then showed through everything painted over it, and the painting went dull and lumpy (cat 5.8 → 7.6).
• A last pass of the finest round (r 2.5) on the features. No visible gain for 11,700 more strokes.
• Less oil in the lean passes. More glints, not fewer.

What still stands between this and a master:

1. Glints on every ridge. Fine white streaks run along the brushwork everywhere, strongest over the darks, and they soften faces. They follow the stroke direction, so they are the engine's bristle-groove (anisotropic) highlights, not the robot: dense brushwork has hundreds of them. Whether they are too strong is a question for the app as a whole, looked at on the phone.
2. The darkest, most saturated reds. A cup wall at L 4 and a deep red saucer are beyond these eight pigments (the darkest mix is about L 20). A painter would reach for alizarin crimson; that needs a ninth pigment, and a third paint texture.
3. Faces want finer drawing. The features are soft; the grid's resolution and the glints both limit how crisp an eye can be.

What the tests fixed in the engine (for everyone, not just the robot)

1. Overpainting is bounded. A stroke over wet paint now counts all the wet paint beneath it, including what earlier strokes pushed under their colour (DRYBASE), and the reservoir stops adding where wet paint is already about 2 mm thick (WET_MAX).
2. A brush holds paint in proportion to its size. At size 1 nothing changes; small brushes carry less, big ones more.
3. Lifting a loaded brush leaves a soft lip, not a flat-topped disc with a cliff.
4. A loaded brush fills hollows; only a nearly spent one dry-brushes over them.
5. Ridges are rounded (a Sobel slope in the bake), so stroke edges don't cast razor-thin shadow lines; bristle grooves are softer.
6. A NaN guard where every cell under a lift is already full.

The verdict, honestly

The final painting is a convincing, well-lit oil study. It has Chardin-like chiaroscuro, a modelled white jug, a green bottle with its highlight, calm broad passages and real impasto. It is not yet a master's painting. Here's what stands between them, in order:

1. The palette's gamut. (Earths added since, and used from run 20.) The six pigments can't reach deep warm darks. Ivory black renders about L 18, and a deep olive or brown shadow is out of reach (the colour test missed a pear shadow by ΔE 12.5 and the table front by 13.6). Masters rely on earth pigments (burnt umber, burnt sienna, raw umber) exactly here. Adding them is the single biggest step, for the robot and for people.
2. Small round forms. The pears still read as yellow rosettes. Stroke placement follows the isophotes around a highlight, and a master models small forms with fewer, deliberate strokes across the form.
3. Intent. A master chooses what to lose and what to find: soft edges into shadow, one sharp accent, a simplified passage. The robot is error-driven, so it chases everything evenly.
4. Reference quality. The reference is a rendered still life, because the network blocks photo sites. A real photograph, or the owner's own, is the next test.

Next

• Add burnt umber and burnt sienna to the palette. This needs a third paint texture (the two-texture layout holds six pigments).
• Give the robot form-aware strokes for small objects (across the form, fewer and longer), and a lost-and-found edge pass (soften edges in shadow, sharpen one accent).
• Run it on a real photograph the owner supplies.