Recognising familiar characters is within reach
A single reference image per character already matches closed-set retrieval.
A Framework For Open-Set Comic Character Re-Identification
Aaditya Baranwal, Madhav Kataria*, Yogesh S. Rawat, Shruti Vyas
Institute of Artificial Intelligence, University of Central Florida
*Work done as an intern at the University of Central Florida.
01 · Abstract
Comics and manga tell stories through characters whose appearances change from panel to panel, yet readers recognise them hundreds of pages later without ever consulting a cast list. Character re-identification models rarely face the same challenge: existing evaluations give them a fixed set of identities before the story begins. Re:Cognize instead asks models to recognise characters as the story unfolds, inferring the cast incrementally.
Across three manga corpora, we evaluate five Re-ID backbones and MeCha, a memory-augmented baseline we introduce, and find that the central difficulty lies somewhere unexpected. Recognising familiar characters is within reach. Knowing which matches to trust is not. When models build a gallery and grow it from their own predictions, their performance deteriorates, even though correct identity labels would improve it by over twenty points. This raises a simple question: when should a model trust its own match?
We derive a single comparison that determines when gallery updates improve recognition and use it to design Vouch, which updates its memory only when the page provides supporting evidence. Without fitting additional parameters, Vouch recovers between one-third and two-thirds of the improvement available from perfect labels. Re:Cognize measures whether a model can recognise characters as it reads; Vouch shows how to maintain their identities reliably.
02 · Key findings
Recognising familiar characters is within reach. Knowing which matches to trust is not.
A single reference image per character already matches closed-set retrieval.
Growing a gallery from the model's own matches degrades it on every backbone, whereas correct labels would add over twenty points.
An addition improves recognition exactly when it is correct more often than the gallery already was on the queries it captures. The comparison has no fitted parameters, and measured on half of a new corpus it makes the correct decision for the other half wherever the change matters.
One running average per character, grown only where the page itself vouches for a crop, recovers a third to two thirds of what perfect labels would.
“Re:Cognize measures the gap. Vouch shows how to narrow it.”
03 · The framework
Every protocol answers the same stream of query crops in reading order; they differ only in the gallery and whether it may change.
A fifth of each character's crops form the gallery and the rest are queries, ranked by cosine similarity and scored by mAP and Rank-k. The closed-set ceiling.
k labelled crops per character, drawn at random (Seq-R) or as the character's first k appearances (Seq-T); every other crop is a query.
Crops arrive unlabelled into an empty gallery and join the nearest cluster above a novelty threshold or initialise a new one. A diagnostic of emergence: one fixed threshold compares every representation.
P2's gallery, but in reading order each query is added with its top-1 label. Static, predicted and oracle policies isolate the contribution of growth, scored by identity Rank-1.
04 · The commit condition
The limiting factor is what the gallery admits, not the encoder.
A change to the gallery can alter the answer only for a query whose nearest entry it supplied: the queries it captures. Over the stream, the change in accuracy is exactly
Δ = c (peff − a+)
c
Capture rate
The share of queries whose nearest entry the change supplied.
peff
Precision where it acts
The share of captured queries whose capturing entry is labelled with their own identity.
a+
The gallery it replaces
The accuracy the unchanged gallery would have had on those same queries.
So a change improves accuracy exactly when peff > a+: when its additions are correct more often than the gallery already was on the queries they capture, not on average. A crop assigned to the correct character can still capture other characters’ queries, which is why neither term is the intuitive one.
05 · Vouch
A cast sheet of one running average per character, grown only where the page itself vouches for a crop, with no parameters fitted to data.
Each character is one ℓ2-normalised average of the crops assigned to it, so a new crop refines its character's entry instead of becoming a rival entry that captures other characters' queries.
A crop joins a character only when another crop in its page group is already committed to that character. The rule consults only crops already read, and abstains wherever the page offers no evidence.
Before the stream starts, the crops on a seed's page that share its character join it: 1.43 per seed on average, 90.6 % of them correct.
Chronological seeding
When the seeds are each character’s first appearances, binding supplies the reference: page groups merged in reading order and named by their first seed, committed only where the commit condition predicts a gain.
06 · Results
Five backbones, two corpora: one seed recovers the closed-set mAP, correct growth would add over twenty points, and the commit condition predicts which changes improve recognition.
aWhat one seed recovers, by metric
bWhat correct growth would add, k = 1
cEncoder-side adaptation
dEvery gallery operation at k = 5
07 · Code & data
The harness, the per-tuple results and the three corpora. With the results archive, every table and figure of the paper regenerates without a GPU.
The evaluation harness with all four protocols, Vouch, the MeCha baseline, and the analyses behind every number in the paper.
github.com/Per-tuple results behind every table and figure (23 MB). scripts/fetch_results.py downloads and verifies them; the tables then regenerate without a GPU.
recognize-results.tar.xzThe public PopCharacters subset of PopManga: 23 series, with 4,058 test crops of 70 characters over 8 held-out series.
huggingface.co/27 held-out volumes (784 characters, 29,315 crops), evaluated zero-shot.
www.manga109.orgThe Re:Zero annotations behind the Vouch and binding figures.
re-verse.vercel.app08 · Citation
@inproceedings{baranwal2026recognize,
title = {Re:Cognize - A Framework For Open-Set Comic Character Re-Identification},
author = {Baranwal, Aaditya and Kataria, Madhav and Rawat, Yogesh S. and Vyas, Shruti},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
year = {2026},
eprint = {2609.34032},
archivePrefix = {arXiv}
}