Auto-recomp on actor plate

📖 User Guide

Overview

This tool replaces an actor in an original plate video with a CG/replacement character in a scene where the actors do not interact. The pipeline handles alignment, segmentation, inpainting, and compositing automatically. It outputs the original plate + the inpainted version with the masks where inpainting happened, and the replaced character + their masks.

Quick Start

Upload — Drop the original plate (4K or 1080p) and the replacement video.
Settings — Defaults should work well. Use 1080p for better masks, 720p for speed.
Run — Click "Run Pipeline". Progress shows each phase in real-time.
Review — Check the final composite result.
Tweak — Adjust feather/expansion in the Tweak panel, preview on 4 frames, then apply.
Download — Grab the ZIP with all outputs.

Pipeline Phases

Phase 1	LoFTR Alignment — Warps replacement to match original camera.
Phase 2	Nanobanana2 Segmentation — Detects character masks (4 images, best MAE pick).
Phase 3	SAM2 Tracking — Propagates masks bidirectionally across all frames.
Phase 3.5	Mask Refinement — SAMRefiner + depth-aware expansion + feather/ViTMatte.
Phase 4	Inpainting — Bbox magenta + Kling/HappyHorse fill (best-of-N with quality check).
Phase 5	Color Harmonization — Percentile color match of replacement to plate.
Phase 6	Final Composite — BG harmonize + character via foreground estimation.
Phase 7	Package — ZIP archive with all outputs.

Settings Reference

Processing Height: 720p = fast iteration, 1080p = better mask quality. Output is always at original resolution.
Nanobanana2 Reference Frame: Which frame Nanobanana2 uses for initial mask detection. "Auto" picks the best of 5 candidates.
Depth Expand Iters: How many 1px dilation steps the mask grows outward (constrained by depth). Higher = captures thin limbs better but may bleed into BG.
Edge Feather (px): Gaussian blur on the final alpha mask. 0 = hard edge (aliased), 2-4 = smooth, 6+ = very soft/glowy.
ViTMatte: Neural alpha matting for hair/fur. Much slower but produces true semi-transparency at edges.

Upload Videos

🎞️

Original Plate Video (4K/1080p)

🎭

Replacement Video

Settings

Processing Height

Nanobanana2 Reference Frame

Color Correction Method

🎭 Mask Refinement Settings

SAMRefiner cleanup Depth-aware expansion ViTMatte alpha (slow)

Depth Expand Iters ?

5 (tight) 20 60 (wide)

Edge Feather (px)

0 (sharp) 2 8 (soft)

How mask refinement works

SAM2 Tracked Mask

Binary mask from bidirectional tracking. May have jagged pixel-staircase edges and miss thin parts (hair, fingers).

binary

0 or 255, no anti-aliasing

SAMRefiner Cleanup

Neural network smooths the contour, fills small holes, removes stray pixels. Still binary.

raw

→

clean

Depth-Aware Expansion

Dilates the mask outward 1px at a time. At each step, only accepts pixels whose depth matches the character. Rejects pixels at background depth. Controlled by Expand Iters.

clean

→

grown

stops at depth edge

Edge Feather

Gaussian blur converts the hard binary edge into a soft alpha gradient. Controlled by Feather px. Result: float alpha matte [0.0 – 1.0].

binary

→

soft α

Output: Per-frame soft alpha matte. Interior = 1.0 (opaque), exterior = 0.0 (transparent), edge = smooth gradient. Used to composite the replacement character onto the inpainted plate.

Run Pipeline

Results

Final Composite

🔧 Tweak Composite

Adjust mask expansion and feather, then preview on 4 sample frames. Click to zoom. When happy, apply to full video.