How virtual try-on actually works — and where it still lies to you

Image-based virtual try-on has a simple brief: take a photograph of a person and a photograph of a garment, and produce a third photograph in which the person is wearing the garment. I have spent a good part of the last few years building systems that do this, and the thing I most want people to understand is how much the method changed around 2023 — and how little the hardest problem changed with it.

For most of the field’s history the approach was literal. First, understand the body: estimate its pose as a skeleton of keypoints, segment it into regions — arms, torso, hair, the existing clothes — and often fit a parametric body model to it. Second, take the flat product photo of the garment and warp it onto that geometry, usually with a thin-plate spline or a learned flow field, so the sleeves go where the arms are. Third, hand the warped garment and the person to a second network that blends them, repaints the seams and fills in what the warp could not reach. The families of models named after this pipeline — VITON, CP-VTON, VITON-HD and their descendants — got steadily better at each stage.

The warp was always the weak link. When the garment and body did not align — a loose shirt on a raised arm, a wrap dress with no clean silhouette to map to — the print stretched, the collar tore, and a viewer spotted it in a heartbeat. Human beings are extraordinarily sensitive to the geometry of clothes on bodies, because we have looked at little else all our lives.

Diffusion models changed the architecture rather than tuning it. A latent diffusion model such as Stable Diffusion already knows, from training on enormous numbers of photographs, what fabric does on a body. StableVITON showed that you could condition such a model on the garment image through cross-attention and let it learn the correspondence between garment and body itself, with no separate warping stage at all. IDM-VTON, presented at ECCV 2024 and built on SDXL, went further: it feeds the garment in twice — once as a high-level semantic embedding through an adapter, once as low-level pixel features through a parallel network — and adds a detailed text caption of the garment alongside. The caption turns out to matter; it measurably improves how faithfully logos, prints and textures survive the trip from product photo to person. The results look like photographs because, in a sense, they are painted as photographs by a model that has seen millions.

That is also the problem, and it is the part the demos do not dwell on. A diffusion model is rewarded for plausibility, not truth. It produces the most likely image consistent with its inputs, and “most likely” is not the same as “what would actually happen”. The published limitations of the current generation are candid: fine details of the person inside the region being repainted — tattoos, jewellery, the exact fall of hair against a collar — are imperfectly preserved; full-body outfits and consistency across several views remain open problems; and you cannot yet reliably ask the model to change a pattern or a colour with text without it inventing something else. Newer work on disentangled warping and measurement-guided generation is attacking exactly these gaps, which tells you they are real.

The deepest limitation is not a rendering artefact. It is that fit is a measurement, and an image is a picture. A convincing photograph of you in a jacket is not evidence that the jacket fits you; the model will happily paint a size 8 onto a size 14 body and make it look tailored, because tailored is what jackets look like in its training data. No amount of image quality fixes this, because the information was never in the inputs.

The datasets tell you a good deal about the limits before you run a single model. The public benchmarks that the field trains and reports on are overwhelmingly frontal, well-lit, studio photographs of slim models in upper-body garments against plain backgrounds — because that is what e-commerce catalogues contain, and catalogues are where the paired images come from. A system that scores beautifully on those benchmarks has never seen a phone photograph taken in a bedroom, a body outside the sample-size range, a floor-length dress, or a shalwar kameez. Building a try-on for real wardrobes means building the dataset first, and that is most of the work.

That is why ZIMRICS treats the two questions separately and refuses to let one answer the other. The try-on room draws on the wearer’s own photographs and the garment’s own photographs to show how a piece reads on you — the colour against your colouring, the proportion against your frame, the length against your height. Fit comes from the other side of the app: measured garments, a measured body, and a comparison in centimetres that no generated pixel is permitted to overrule. When the two disagree, the centimetres win, and the app says so.

A note from the security side of my work, because it belongs in any honest piece about this technology. A try-on system needs photographs of you, and photographs of a person’s body are among the most sensitive data anyone hands to software. They should be stored with the care you would give a password, deleted when you delete the result they produced, and never become training data without a switch the person set themselves. Whether a product does that is not a detail of its privacy policy. It is a design decision, and you can see it: look for whether the Settings screen has the switch, and whether deleting a look deletes the photograph that made it.

What to expect next is fairly clear from the research direction. Models that keep the person intact and change only the garment; multi-view consistency so a look holds from the side and the back; garment models that carry real measurements so the render is constrained by fit rather than by plausibility; and — the one I care about most — try-on that is honest about uncertainty, rendering what it knows and declining what it does not. Until then, treat a try-on image the way you treat a shop mirror with flattering lights and a helpful assistant. Useful. Not sworn testimony.