NeurIPS 2026 · Evaluations & Datasets Track

Re:Cognize

A Framework For Open-Set Comic Character Re-Identification

Aaditya Baranwal, Madhav Kataria*, Yogesh S. Rawat, Shruti Vyas

Institute of Artificial Intelligence, University of Central Florida

*Work done as an intern at the University of Central Florida.

Recognising characters while the story is read. On the left, a reader's question such as where Takagi asks Mashiro depends on who appears on which page. In the centre, standard Re-ID matches each query against a gallery built before reading starts, so a new character finds no match. On the right, Re:Cognize streams crops in reading order against four galleries that hold every character (P1), a few labelled examples of each (P2), none (P3) or a few that grow as it reads (P4), and Vouch grows the gallery only where the page vouches for a crop.
protocols on one query stream
4
protocols on one query stream
Re-ID backbones evaluated
5
Re-ID backbones evaluated
corpora: POPCharacters, Manga109, Re:Verse
3
corpora: POPCharacters, Manga109, Re:Verse
points of top-1 that correct labels would add
20+
points of top-1 that correct labels would add

01 · Abstract

Recognition as the story is read

Comics and manga tell stories through characters whose appearances change from panel to panel, yet readers recognise them hundreds of pages later without ever consulting a cast list. Character re-identification models rarely face the same challenge: existing evaluations give them a fixed set of identities before the story begins. Re:Cognize instead asks models to recognise characters as the story unfolds, inferring the cast incrementally.

Across three manga corpora, we evaluate five Re-ID backbones and MeCha, a memory-augmented baseline we introduce, and find that the central difficulty lies somewhere unexpected. Recognising familiar characters is within reach. Knowing which matches to trust is not. When models build a gallery and grow it from their own predictions, their performance deteriorates, even though correct identity labels would improve it by over twenty points. This raises a simple question: when should a model trust its own match?

We derive a single comparison that determines when gallery updates improve recognition and use it to design Vouch, which updates its memory only when the page provides supporting evidence. Without fitting additional parameters, Vouch recovers between one-third and two-thirds of the improvement available from perfect labels. Re:Cognize measures whether a model can recognise characters as it reads; Vouch shows how to maintain their identities reliably.

02 · Key findings

Where models fail

Recognising familiar characters is within reach. Knowing which matches to trust is not.

01

Recognising familiar characters is within reach

A single reference image per character already matches closed-set retrieval.

02

Knowing which matches to trust is not

Growing a gallery from the model's own matches degrades it on every backbone, whereas correct labels would add over twenty points.

03

One comparison decides it

An addition improves recognition exactly when it is correct more often than the gallery already was on the queries it captures. The comparison has no fitted parameters, and measured on half of a new corpus it makes the correct decision for the other half wherever the change matters.

04

Vouch narrows the gap

One running average per character, grown only where the page itself vouches for a crop, recovers a third to two thirds of what perfect labels would.

“Re:Cognize measures the gap. Vouch shows how to narrow it.”

03 · The framework

Four protocols, one stream

Every protocol answers the same stream of query crops in reading order; they differ only in the gallery and whether it may change.

ONE QUERY STREAM · BAKUMAN CH. 1 · READING ORDER →p. 8p. 17p. 25p. 29p. 43p. 44p. 47p. 50P1full gallery–P2one seed each–P3starts emptyjoins at ≥ 0.55cluster 1cluster 2cluster 3–P4grows by top-1–
readyTakagiMashiroAzuki
One stream, four galleries. MagiV2’s own decisions (its released encoder) on eight crops of Bakuman chapter 1, against the galleries of the paper’s protocols figure; the protocols run on whole chapters. The page-29 Mashiro crop lies closer to Azuki’s seed: P1 still identifies it correctly, P2 does not, and P4 assigns it to Azuki, where two later Mashiro crops match it.
P1

Closed-set retrieval

A fifth of each character's crops form the gallery and the rest are queries, ranked by cosine similarity and scored by mAP and Rank-k. The closed-set ceiling.

P2

Seeded static gallery

k labelled crops per character, drawn at random (Seq-R) or as the character's first k appearances (Seq-T); every other crop is a query.

P3

Online clustering

Crops arrive unlabelled into an empty gallery and join the nearest cluster above a novelty threshold or initialise a new one. A diagnostic of emergence: one fixed threshold compares every representation.

P4

Seeded gallery that grows

P2's gallery, but in reading order each query is added with its top-1 label. Static, predicted and oracle policies isolate the contribution of growth, scored by identity Rank-1.

04 · The commit condition

When a change to the gallery improves recognition

The limiting factor is what the gallery admits, not the encoder.

A change to the gallery can alter the answer only for a query whose nearest entry it supplied: the queries it captures. Over the stream, the change in accuracy is exactly

Δ = c (peff − a+)

c

Capture rate

The share of queries whose nearest entry the change supplied.

peff

Precision where it acts

The share of captured queries whose capturing entry is labelled with their own identity.

a+

The gallery it replaces

The accuracy the unchanged gallery would have had on those same queries.

So a change improves accuracy exactly when peff > a+: when its additions are correct more often than the gallery already was on the queries they capture, not on average. A crop assigned to the correct character can still capture other characters’ queries, which is why neither term is the intuitive one.

05 · Vouch

Grow only where the page vouches

A cast sheet of one running average per character, grown only where the page itself vouches for a crop, with no parameters fitted to data.

ONE QUERY STREAM · BAKUMAN CH. 1 · READING ORDER →p. 8p. 17p. 25p. 29p. 43p. 44p. 47p. 50P4grows by its own top-1–Vouchone entry each, page-checkedthe seed alonethe seed alonethe seed alone–
readyTakagiMashiroAzuki
Growth by top-1 against Vouch, on the same stream. Both start from one seed per character. P4 assigns every query to its top-1 match, so the page-29 error remains in the gallery and is matched again. Vouch maintains one averaged entry per character and adds a crop only when MagiV2’s page grouping links it to a crop already committed, here the seeds on pages 17 and 50; everywhere else it abstains.
Vouch on Re:Zero crops from Re:Verse, with Rom as identity A. The top row shows the three changes. A page vouches for a character when one of its crops is already committed to it, and commitment then adds that crop's page-group sibling. In the bottom row, over six pages the cast sheet is updated only on the four that vouch for Rom, and elsewhere the rule abstains.
a

A cast sheet, not a bag of crops

Each character is one ℓ2-normalised average of the crops assigned to it, so a new crop refines its character's entry instead of becoming a rival entry that captures other characters' queries.

b

Commitment under the page constraint

A crop joins a character only when another crop in its page group is already committed to that character. The rule consults only crops already read, and abstains wherever the page offers no evidence.

c

Seed expansion

Before the stream starts, the crops on a seed's page that share its character join it: 1.43 per seed on average, 90.6 % of them correct.

Chronological seeding

Binding, and pricing the binder

When the seeds are each character’s first appearances, binding supplies the reference: page groups merged in reading order and named by their first seed, committed only where the commit condition predicts a gain.

Binding, and pricing the binder. The top row uses Re:Verse’s Re:Zero annotations. (a) A page’s crops are grouped above one similarity threshold, (b) page groups merge in reading order into the best earlier match above a second or start new ones, and (c) a group’s first seed labels it. In the bottom row any frozen encoder binds with threshold τ, and its crops are committed when the commit condition predicts a positive Δ, with peff the additions’ precision on the queries they capture and a+ the static gallery’s accuracy there. That prediction, never the measured gain, chooses τ.

06 · Results

What the protocols measure, and where the headroom is

Five backbones, two corpora: one seed recovers the closed-set mAP, correct growth would add over twenty points, and the commit condition predicts which changes improve recognition.

aWhat one seed recovers, by metric

bWhat correct growth would add, k = 1

cEncoder-side adaptation

dEvery gallery operation at k = 5

What the protocols measure, and where the headroom is. (a) One seed per character recovers the closed-set mAP but only part of its Rank-1. (b) Growth by the model's own top-1 matches remains below a static gallery, whereas correct labels would raise it far above (labels give oracle minus static). Panels (a) and (b) use MeCha with one random seed per character over three training runs. (c) Encoder-side adaptation benefits most the backbones weakest in this domain. (d) Each point is one change to the gallery on one backbone and corpus, whether adding under the page constraint, adding by top-1 match or restricting the candidate characters. The stronger the gallery already is, the less any change adds, and the commit condition predicts which points fall below zero.

08 · Citation

Cite Re:Cognize

BibTeX
@inproceedings{baranwal2026recognize,
  title     = {Re:Cognize - A Framework For Open-Set Comic Character Re-Identification},
  author    = {Baranwal, Aaditya and Kataria, Madhav and Rawat, Yogesh S. and Vyas, Shruti},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year      = {2026},
  eprint    = {2609.34032},
  archivePrefix = {arXiv}
}