envisage / independent medical ai research / 2d + 3d
rhinoplasty goal visualization,
and the right ruler to measure it
jan 2026 - present / independent researcher / arXiv:2606.28628
Envisage turns one frontal photo into a photorealistic, surgeon-grounded preview of a rhinoplasty goal, a pre-consult conversation starter, not an outcome prediction. Its real contribution is methodological: full-face ArcFace is structurally confounded for a localized edit, so every method scores a negative identity gap against the unedited input. That is a finding about the metric, not a failure. We build SurgicalScore, a mask-decomposed 0–1 protocol that measures the edit region instead of the copied pixels around it, and Envisage leads on it. A parallel 3D track turns patient photographs into rotatable meshes for region-aware, interactive rhinoplasty modeling.
the thesis
Localized generative edits need localized evaluation. The headline result is not a leaderboard win, it is a diagnosis: the standard ruler for facial identity is the wrong tool for an edit that touches roughly 5% of the pixels. Once you measure the right thing, Envisage is the strongest system on it.
the structural confound
When a localized edit is hard-composited, the nose regenerated, every other pixel copied verbatim from the input, full-face ArcFace is dominated by the copied non-surgical pixels. Outside-mask SSIM exceeds 0.999, so the score barely reflects the edit at all. Measured against the unedited input as a proxy, every method posts a negative paired ArcFace gain. That is a property of the metric, not a failure of the models.
the prior work that exposed it
The author's earlier system, LandmarkDiff, regenerated the entire face and composited the surgical region back. Decomposition revealed that over 95% of its identity score came from those composited pixels: 0.509 with compositing, 0.023 without. The number measured the copy, not the model. That confound is exactly what SurgicalScore was built to remove.
surgicalscore, the fix
A mask-decomposed 0–1 protocol that scores only what the surgery changes: edit direction, edit magnitude, masked perceptual fidelity, realism, and outside-mask preservation. A perfect-predictor control (ground-truth paste) scores 0.919, anchoring the ceiling. On this ruler Envisage leads at 0.599, well clear of the strongest baseline at 0.502.
honest scope
Envisage is a pre-consult goal-visualization tool, not a patient-specific outcome predictor. It shows a believable, affordable, surgical-scope-faithful preview from a single photo, no stereophotogrammetry rig, no CT, no per-clinic license. The open problem we name plainly: turning the candidate-space headroom into a deployable, non-oracle ranker.
the pipeline
One frontal photo in, a photorealistic goal preview out. Three stages, no task-specific training for the base pipeline, ~20s per image on a single NVIDIA L40S (48GB, BF16). Identity preservation is architectural, outside the mask, pixels are copied, not generated.
1 / tps pre-warp
A procedure-specific thin-plate-spline warp displaces nasal landmarks 2–4px to seed the lateral geometry that depth alone cannot express, bridge thinning, tip refinement.
- scipy RBF interpolation, thin-plate-spline kernel
- MediaPipe 478 landmarks index the displacement field
- Boundary anchors confine the warp to the surgical region
2 / modified depth
Depth Anything V2 estimates a monocular depth map, then landmark-indexed Gaussian kernels edit it to encode the intended tissue displacement, e.g. flattening a dorsal hump.
- Kernel size and intensity scale with measured anatomy
- Encodes the surgical change as a physically meaningful signal
- Resolution-independent, no fixed pixel offsets
3 / flux.1-fill + depth controlnet
FLUX.1-Fill-dev (12B rectified-flow fill) with a pretrained depth ControlNet regenerates only the masked region, conditioned on the modified depth and a preset-specific prompt.
- Pretrained weights, no fine-tuning of the base pipeline
- 24 anatomically-grounded clinical presets condition the prompt
- ~20s per image on one NVIDIA L40S, BF16
4 / hard-mask composite + gating
A hard-mask composite copies every non-surgical pixel verbatim from the input, outside-mask SSIM > 0.999, so identity is preserved by construction. A 5-seed sweep and a 7 hard-gate scorer select the output.
- Identity preservation is architectural, not a backbone property
- 5-seed candidate sweep × 7 hard gates pick the result
- Operational PASS on 208/211 cases (98.6%)
Released as a reference framework: 24 presets across three procedures, 8 rhinoplasty (Daniel's taxonomy, the headline), 8 blepharoplasty (Tessier), 8 rhytidectomy (SMAS). Rhinoplasty is the evaluated headline procedure.
surgicalscore, the right ruler
The paper's primary contribution is the evaluation protocol, not the pipeline. SurgicalScore is a mask-decomposed 0–1 score that grades the edit region against paired post-operative ground truth instead of letting copied background pixels inflate a full-face number.
five components, one score
Directional alignment A (0.40), edit-magnitude fit B (0.30), masked LPIPS C (0.15), realism D (0.10; SER-FIQA + CR-FIQA), and outside-mask preservation E (0.05), with a passthrough floor of 0.30. The weights put 70% of the score on getting the surgical change right, not just on looking real.
anchored to a ceiling
A perfect-predictor control, pasting the real post-op nose with no model in the loop, scores SS_raw = 0.919. That fixes the top of the scale, so a method's score is read against an achievable ceiling rather than a hypothetical 1.0 that the protocol itself never reaches.
why full-face arcface fails here
ArcFace was trained to verify identity across changes orders of magnitude larger than a rhinoplasty. At a 5%-of-pixels edit, paired against the input, no method posts a positive gain, Envisage's gap is the smallest at −0.048 (95% CI −0.055, −0.042), vs ICEdit −0.139, FLUX.1-Kontext −0.242, InstructPix2Pix −0.294, all p<1e-4. The metric, not the model, is the bottleneck.
the in-mask shift, isolated
Because copied outside-mask pixels are identical between input and output, the paired full-face difference is dominated by the in-mask edit, a correct edit that shifts the nose toward the post-op target reads as lower identity. SurgicalScore isolates that shift and rewards it, which is precisely the signal full-face ArcFace destroys.
results
Rhinoplasty headline cohort: N=211 (ASPS public gallery 202 + private clinical archive 9). Four-way SurgicalScore comparison on the N=205 intersection; all gaps paired and significant at p<1e-4.
| method | surgicalscore | arcface gap | beats input |
|---|---|---|---|
| Envisage (ours) | 0.599 | −0.048 | 16.1% |
| ICEdit | 0.502 | −0.139 | 4.3% |
| InstructPix2Pix | 0.337 | −0.294 | 0.0% |
| FLUX.1-Kontext | 0.229 | −0.242 | 1.0% |
SurgicalScore (mask-decomposed, higher is better) and full-face ArcFace gain over the unedited input (every method is negative, a finding about the metric). Rhinoplasty cohort; four-way SurgicalScore on the N=205 intersection, all paired gaps p<1e-4. A perfect-predictor GT-paste control anchors SS_raw at 0.919; a 5-seed GT-oracle upper bound reaches 0.743.
external validation & surgeon review
A single benchmark can hide a confound. The negative gap and the operational gate were re-checked on an external corpus and read by the surgeon co-author.
457-pair external corpus
On an out-of-cohort ASPS/PCA corpus of 457 pairs, the structural negative gap replicates (pooled GT-to-pooled 0.597 vs baseline 0.664), the confound is not an artifact of the headline cohort. 95.8% (438/457) of cases pass the identity gate.
surgeon-in-the-loop
A board-certified facial plastic surgeon, the co-author, Dr. Amit D. Bhrany, reviewed six representative outputs as a feasibility check on surgical plausibility. A blinded multi-rater GAIS/ROE study is named explicitly as future work, not claimed here.
Baselines: InstructPix2Pix, ICEdit (FLUX.1-Fill-dev + MoE-LoRA diptych), FLUX.1-Kontext-dev; internal diagnostics Direct Copy, TPS Warp, and FLUX inpainting without ControlNet. Data: HDA Plastic Surgery Database + external ASPS Photo Gallery + a private clinical archive (PCA).
my role
engineering
I built the full pipeline: the procedure-specific TPS pre-warp, landmark-indexed depth modification, FLUX.1-Fill + depth-ControlNet inpainting, the hard-mask composite, the 5-seed sweep and 7-gate scorer, the 24-preset library, and the Gradio + FastAPI live demo on Hugging Face Spaces.
research
I designed and validated SurgicalScore, ran the four-way benchmark and component ablations, established the full-face ArcFace confound and the LandmarkDiff decomposition that motivated it, ran the 457-pair external validation, and wrote the paper. First author; Dr. Bhrany contributed the surgical review.
envisage 3d / interactive facial modeling
The 3D track extends a static goal image into a spatial interface: patient photographs become a rotatable facial mesh, rhinoplasty regions remain anatomically addressable, and the model can be inspected and edited in the browser.
photograph-to-mesh path
The prototype converts patient photographs into a rotatable 3D facial surface so the nose can be understood as geometry rather than only as a 2D mask.
anatomical region mapping
MediaPipe landmarks anchor rhinoplasty-relevant regions to the reconstructed face, keeping interactive controls tied to recognizable anatomy.
geometry service
trimesh handles mesh operations behind a FastAPI service, separating geometry processing from the interface and making each transformation inspectable.
browser interaction
Three.js renders the model as an interactive scene for rotation, inspection, and region-aware manipulation without requiring a desktop modeling package.
Envisage 3D is an active research prototype and a first-author manuscript in preparation. It is an interactive scanning and modeling feature, not a validated patient-specific outcome predictor.
papers, demo + code
Envisage: Diffusion-Based Rhinoplasty Goal Visualization with Mask-Decomposed Evaluation
First author with Amit D. Bhrany, MD. arXiv:2606.28628.
Envisage 3D: Interactive Facial Scanning and Rhinoplasty Modeling
First-author manuscript.
paper
Envisage: Diffusion-Based Rhinoplasty Goal Visualization with Mask-Decomposed Evaluation. Mudit Agarwal (University of Washington) and Amit D. Bhrany, MD (UW School of Medicine, Otolaryngology–Head & Neck Surgery). arXiv:2606.28628, submitted June 26, 2026.
live demo
Run the pipeline on your own frontal photo across the 24 clinical presets, rhinoplasty, blepharoplasty, or rhytidectomy. Hosted on Hugging Face Spaces.
source code
Released as infrastructure: the Envisage pipeline, the SurgicalScore implementation, the preset definitions, and matched split manifests. Every reported number is reproducible from the repository.
Envisage is the successor to LandmarkDiff, the author's earlier landmark-conditioned diffusion system. Decomposing LandmarkDiff's identity score, 0.509 with compositing, 0.023 without, revealed the structural confound that motivated SurgicalScore and the inpainting-first redesign.