Sending a sketch to a model as words, not just pixels

SketchMason turns rough boxes on a canvas into a finished interface. You press generate and a design streams in beside your sketch. This is the path a sketch takes from your frame to the screen, and the few decisions that shaped it.

The obvious move is to send the model a picture of the frame. On its own, that loses a lot. Two boxes at 40% and 55% of the width are an exact split in the shape data, but in a picture of flat blocks they're just two columns. A box inside a box can turn into one solid slab.

So before anything is sent, the browser writes a manifest. It opens with the frame's size and orientation, then gives one line per shape in whole-number percentages of the frame. It's deterministic and costs nothing, because no model writes it. It's capped at 120 elements and 40 arrows, and the server cuts it at 12,000 characters whatever the client says.

The image still goes along. Filled boxes get a faint white edge so nested boxes stay apart, and the longest side is capped at 1568 pixels, since Anthropic downsamples anything bigger and a flat 2x export was three times the bytes.

The generate route runs in a fixed order. Rate limit first, then the balance. A request that brings its own Anthropic key isn't charged. The instruction is cut to 600 characters. Only then is one credit taken, before the first token arrives.

The design streams through the Vercel AI SDK, and if you close the tab the upstream request is cancelled with it. Things go wrong mid-stream, so the refund rules follow what you actually got.

If the stream fails on the model's side, the credit goes back. If enough markup already arrived to be usable, it's kept with a marker saying it was cut off, so you can continue instead of starting over. If you leave, you're only refunded when nothing usable came back. An empty result is marked and refunded.

Model output is never put on the page as it is. It's parsed and walked against an allow-list of tags and attributes, and its styles are scoped to the design. A single body rule from a model could otherwise blank the whole editor.

None of these pieces is clever on its own. Together they mean the model gets exact geometry, you pay for what you receive and a bad response can't break the app.