back

5 min read

How I turned myself into pixels

1triangle

Hover over him, or hold a finger on him, to see the texture.

On the home page there is a small version of me standing next to the text, made of grey squares. He turns his head, looks around, and stands with his hands on his hips. None of it is a video. It is a real 3D model, rendered live in your browser and turned into pixels every frame. This is how I made him.

A photo, standing for a scan

It started with two photos: one of the outfit (the black track jacket, light jeans, white sneakers) and a selfie. The model that makes the 3D shape needs one clean picture of a person standing straight, arms away from the body.

My first try used a normal photo with the arms hanging down. The arms came back melted into the jacket, and when I rigged it they tore the body apart. So I gave both photos to GPT Image 2 and asked for a studio photo of the same person in the same clothes, standing like for a 3D scan, arms out at about 35 degrees.

A generated full-body photo of me in an A-pose

From a picture to a body

Microsoft's TRELLIS.2 turns one image into a 3D model: a mesh of triangles and a texture to paint it. I ran it on fal. What came back was 58,888 triangles and one 2048-pixel texture, the whole surface cut into patches and packed into one square.

The model in plain clay, no textureThe whole model as a wireframeThe texture atlas: jacket, jeans and skin cut into patches

A skeleton

The skeleton inside the model

A mesh can't move on its own. Meshy's rigging, also on fal, put a skeleton inside it: 24 bones, from the hips up the spine to the head and out to the hands and feet. Every triangle knows which bones pull on it and how much.

The pose is set in code. The head points straight up, the upper arms go out and a little back, and the forearms point in to the waist, so the hands rest on the hips. The model has no finger bones, so open fingers stay open, which is one reason the hands are out of sight.

My face

TRELLIS.2 is good at bodies and clothes, but it made up its own face: squinting, with pressed lips. That wasn't me.

So the face is its own layer. I found the eyes, brows, nose, mouth and jaw in a front render of the head and in my selfie with Apple's Vision framework, which gives a mapping from one picture to the other. Then I copied the 1,781 triangles on the front of the head into a second mesh and painted it with the selfie through that mapping. The edges fade into the hair and neck, and the colours match the skin around them.

The face layer's triangles in red on a clay head

Pixels

On the page, three.js renders the model into a small hidden image, two samples per cell in each direction. The pixels come back from the graphics card through two buffers that take turns, so the page never waits for the GPU. Each cell gets one of ten grey levels, spread over the range the frame actually uses, and the cells are drawn as squares with small gaps, black on white or white on black.

The textured modelThe render as ten grey levelsThe grey levels drawn as cells with gaps

Alive

Every frame he breathes a little, sways, and slowly turns between left and right, with the head leading. You can drag him round with the mouse. On phones the home page leaves him out, so they don't download the 3 MB model.

And there is a key that makes him dance. I'm not telling which one.