Development snapshot · 20 October 2026
Bind to the person.
Not their place in the frame.
Face detection, identity matching and face replacement solve different problems. Separating them is essential when several people share the same scene.
Detection finds a face. Recognition decides whose face it is.
A detector returns a face region and landmarks. It does not, by itself, say that the person is role A or role B. The prototype extracts an identity embedding from the original video face and compares it with each role’s original-face references.
The replacement photograph is used to generate the new appearance. It is not the reference used to identify the original person. Mixing these two inputs would make a role’s identity depend on the face you want to insert.
One role can hold several original-face samples.
A front-facing reference may not match a strong head rotation reliably. Additional original-face samples let a role represent several observed appearances. The prototype supports up to 12 references per role, with thumbnail inspection and per-sample deletion. Samples are added by an explicit user choice rather than automatic learning from uncertain playback.
For each candidate role, the highest similarity across that role’s reference samples is considered. The winning role still needs to satisfy its threshold, separation from competing roles, and temporal confirmation. A cosine similarity is a matching score, not a probability of being correct.
Different people keep different replacements.
Role A can use replacement photo 01 while role B uses photo 02. Changing detector output order or crossing positions should not exchange those assignments. Unbound people remain unchanged; disabled roles continue to help prevent their identities from being claimed by another active role.
The same person can appear more than once.
A live person holding a photograph may create two detected faces with the same identity. Each visible instance has its own track and confirmation state. Both can then be assigned to the same role and use the same replacement. A confirmation belonging to one face does not automatically authorize another newly appearing face.
Thresholds trade recall against mistaken matches.
The detection threshold controls which faces enter the pipeline. A separate recognition threshold controls whether a detected face matches a particular role. Lowering either can reduce missed faces or identities, while increasing the chance of false detections or incorrect matches. The current controls cover 0.10–0.95.
Diagnostics display the actual face-to-role similarity matrix, the winning reference sample, competing-role separation, and acceptance or rejection reasons. This helps distinguish a missing detection from a detected face that was not recognized.
Why a replacement may briefly stop.
Motion blur, occlusion, extreme rotation, scene cuts and an inadequate reference set can interrupt identity confirmation. The pipeline uses a bounded continuation window for selected weak observations, while identity conflicts or large position changes stop continuation. It does not keep applying an old face patch indefinitely.
The prototype supports simultaneous role instances, but robust recognition across arbitrary movies and difficult poses remains an evaluation task. A successful still-frame test is not evidence that every moving scene will work.