back

Stylus

Stylus looks at a sketch and draws it again the way a person at a whiteboard would: one stroke after another, in a human order and at a human pace, with the labels written in by hand. Everything below was drawn by the model, live in your browser from its own pen strokes, at the speed the model predicted. The test-set drawings are the first ones of the evaluation, not picked. The two text requests are the first seed. The bananas are the best of six seeds, and they still fail.

A formulatest set · 5.8 s (human 8.6)
Stylus redrawing the handwritten formula X = |X| e^(i arg x) stroke by stroke
A diagramtest set · 25 s (human 22)
Stylus drawing a small diagram: a node, a box labelled Model and two arrows with labels
An ML pipeline, from a sentence13.8 s
Stylus drawing three boxes labelled Data, Model and Evaluation, connected by arrows
A pipelinetest set · 32 s (human 36)
Stylus drawing Worker 1, an L7 Proxy and a Cluster, top to bottom with arrows
Another formulatest set · 3.6 s (human 7.3)
Stylus redrawing a handwritten fraction with an exponent
A web app, from a sentence55 s
Stylus drawing four boxes for a frontend, an API, a cache and a database
A doodletest set · 11 s (human 6.8)
Stylus redrawing a Quick, Draw! doodle
Where it breaks: bananasfrom text · best of 6
Stylus drawing the client and load balancer boxes, then collapsing into scribbles at the banana icons

How it learns

There is no dataset of pictures paired with how someone drew them. But there are recordings of the drawing itself: Quick, Draw! doodles and MathWriting formulas, every pen point with a timestamp. Stylus is trained backwards on them. A recorded drawing is rendered into an image, the image is made worse on purpose, and the model has to recover the original pen movements. I call that render-and-invert.

Training pairs: a recorded drawing coloured by time, next to the degraded image the model is given
From the paper. Left of each pair: the recorded drawing, dark strokes first, yellow last. Right: the worsened image the model sees. Its job is to get back to the left.

What it is

parameters   934 million
eyes         SigLIP vision encoder, reads the image
hand         Qwen2.5-0.5B decoder, one position per pen eventevent        move · cubic Bézier in n ms · write label 2 · lift
diagram      a few hundred eventspointer      which of 27 × 27 image patches the stroke ends in

The pointer was the change that mattered. Before it picks a coordinate, the model points at the image patch where the stroke ends. Without it, Stylus drew nice strokes that had nothing to do with the picture.

From a sentence to a drawing

For a text request, a language model plans the drawing and its labels, a diffusion model I fine-tuned draws a whiteboard sketch with grey bars where the text goes, and Stylus redraws it with a pen and writes the labels in. That works for boxes and arrows. It doesn't work for objects yet: the banana load balancer above gets its boxes and labels, then the model loops on the banana icons and scribbles. That happened on all six seeds I tried, so it stays on the page.

How good it is

On held-out drawings, Stylus orders its strokes much more like a person than a program that traces the image: a rank correlation with the human order of 0.90 against 0.69 on formulas, and 0.75 against 0.26 on diagrams. Its motion is closer to human on formulas and doodles, and about the same on diagrams. What it can't do is trace precisely. The tracing program copies the image almost exactly, while Stylus is off by about 8 thousandths of the canvas width on diagrams and formulas, and much more on free doodles. Its letters are often hard to read, and it splits lines into too many short pieces.

The full story, including the hour where it learned the wrong thing, is in the blog post and in the paper.