A virtual try-on mobile app on free GPUs: React Native, FastAPI and diffusion models with graceful fallbacks

What it does

• 2D try-on. A photo of a person plus a garment image (top, bottom or dress) produces a photoreal image of that person wearing it.
• A capture coach. Try-on quality depends almost entirely on the input photo. A guided full-body capture is scored out of 100 by a pose checker built on MediaPipe, with concrete tips — feet visible, facing the camera, phone at chest height. A photo passes at 70; the user can still force "use anyway".
• Size recommendation from manual measurements, a height/weight estimate or a measured photo, with a slim / regular / relaxed preference applied as a ±3.5 cm bias against four size charts.
• 360° view. A try-on result is turned into a rotatable 3D model (GLB) shown in a self-contained viewer page inside a WebView.
• Looks, wardrobe, family. History capped at 30 looks, 40 saved garments, several body profiles per account, server-side "compare two looks" images for sharing, and a kiosk slideshow mode.
• Guest mode with results that expire after 24 hours, plus account export and deletion that really removes the stored photos.

Architecture

No model runs on my server. The backend calls public Hugging Face Spaces, which means free GPU time — and a shared quota that can run out at any moment. The whole design follows from that.

2D: a synchronous call with a silent fallback

/tryon is a plain synchronous request: an asyncio.Semaphore(4) caps concurrency and the blocking inference runs in a threadpool. The default model is Leffa (VITON-HD weights, 40 steps, fixed seed so a retry gives the same image); IDM-VTON is the fallback for upper-body garments.

When the GPU quota is exhausted the Space returns an error string. The backend parses it into a typed QuotaExhausted exception, silently moves to the next verified free model that supports the garment category, and reports both model (the one actually used) and requested_model in the response. If nothing is left, the user is told the exact time the quota resets — parsed out of the upstream message — instead of "something went wrong".

3D: a job, a ladder and a cache

A 360° build takes one to three minutes, so it cannot be a request. POST /spin starts a background job (at most two at once), and the app polls every 3 seconds with a 6-minute deadline. Engines are tried in order of quality, and TRELLIS.2 has its own ladder: 1536 px first, then 1024 px if the quota rejects it. Finished HD models are cached per source image with a quality version, and a stale HD model is served when quota blocks a rebuild — an older model beats a spinner.

Before 3D, the image is cleaned on the CPU: non-local-means denoise, FSRCNN ×2 super-resolution, then an unsharp mask (fail-open — if any step breaks, the original goes through). The resulting GLB is compressed with Draco and WebP textures; on my test model that took it from 18.3 MB to 3.9 MB, which is the difference between usable and not on mobile data.

Bugs that taught me something

One user's body photo shown to the next account. The photo URL was identical for every account and the auth token travelled only in headers, so the image library's disk cache happily served account A's photo to account B on the same phone. Fix: scope the URI with the user id, and add a session-generation counter, bumped on every sign-in and sign-out, that makes any in-flight async handler from the previous session bail out.

Rate limiting behind a tunnel. Every request arrives from the tunnel's IP, so per-IP limits throttle everyone at once, and X-Forwarded-For can be forged. Fix: a per-email throttle (8 attempts a minute) on login, register and change-password, and X-Forwarded-For ignored unless the deployment explicitly says it sits behind a trusted proxy.

A disk leak with a three-minute fuse. A 360° build could finish after its history row had been evicted by the 30-look cap, leaving an orphan 3D file forever. The attach step now checks the affected row count, and a sweeper removes leftovers after a 15-minute grace period.

Sideways phone photos. Phones store orientation in EXIF instead of rotating pixels, and the models ignore EXIF. Baking the orientation into the pixels on upload was the single biggest quality fix in the project.

A file called .jpg that wasn't. MediaPipe dispatches on the file extension; PNG bytes behind a .jpg name crashed it. Every body photo is now re-encoded to a true baseline JPEG, which also strips the EXIF metadata (location included).

Third-party API drift. One Space describes booleans as strings in its own API description, another rejects keyword arguments, and gradio_client renamed its token parameter between major versions (detected at runtime with inspect). When you build on someone else's demo, adapters and contract tests are not optional.

Security and privacy

bcrypt passwords, HS256 JWTs with a machine-generated secret in a 0600 file, streamed uploads capped at 12 MB and verified as decodable images, path-traversal guards on every file route, and private data (body photos, wardrobe, history) served only with a bearer token. Swagger is off by default. Account deletion removes the database rows and the photos and any shared composites. The Hugging Face token lives only in the process environment.

Running it

Three processes under pm2 — the backend, a tunnel and the Expo dev server — plus three cron jobs: a tunnel watchdog that restarts the tunnel only when the local backend is healthy but the public URL is not (and rewrites the app's config with the new URL), a health monitor that sends a Telegram alert on state changes only, and a nightly backup that uses SQLite's backup API and keeps 14 archives.

What I would do next

The ceiling is the shared free GPU quota, which 2D and 3D compete for. The next step is a paid GPU endpoint behind the same adapter interface — the Replicate adapters are already written and inactive — and an EAS build for TestFlight and the Play Store internal track.