Person Matching & Deduplication on Import

Related reading

1. Why this exists

EventParticipant.person is @ManyToOne(optional = false), the column is NOT NULL, and there is a unique constraint on (event_id, person_id). The Person is the spine: race numbers, results, leaderboard standings, membership and security filtering all hang off it. Every imported row must therefore end up attached to exactly one Person, and the importer has to decide — with no operator present, at several hundred rows a minute — whether the human in front of it is one we already hold.

That decision has two failure modes, and they are not symmetric.

Failure What it looks like Cost

Under-merge (duplicate)

The same human is created twice. Two Persons, two histories.

Visible and recoverable. Both records exist, the operator can see them, and merge tooling can collapse them.

Over-merge (false link)

Two different humans are collapsed onto one Person.

Often invisible, and expensive to undo. The second entrant inherits the first’s identity number, date of birth, results history and any future external-reference write-back. Nothing in the data says a mistake was made.

The design follows from that asymmetry: where the evidence is weak, a duplicate is the cheaper error.

2. Scope: three matching paths, one vocabulary

Three code paths resolve "who is this person?", and they are tuned for different threats. Confusing them is the most common source of wrong reasoning about this area.

Path Entry point Caller & context Optimised against

Import ladder

PersonService.matchPersonForImport

Bulk EP import from an operator-supplied file, with a RegistrationSystem context. Headless — no human per row.

Wrong linkage. The operator is trusted; the data is not. Throughput matters, so the decision must be deterministic and unattended.

Legacy single-add

PersonService.matchOrAdd

Admin-UI direct add, and any import caller with no sourceSystemId. Returns a list; the caller takes the first.

Nothing in particular — retained verbatim for back-compat. Matches on identity number and DOB+gender only. Callers migrate to the ladder as they are touched.

Progressive matching

ProgressiveMatchService

Registration Portal self-service: a logged-in principal linking a family member they claim to know.

Enumeration by an adversary. The person typing is not trusted at all. Hence weighted scoring, a uniqueness requirement, masked reveals and opaque tokens — see Progressive Person Matching.

The two matching designs look similar and are not. Progressive matching asks "has this user proved enough knowledge that we may reveal a person to them?" and answers conservatively by refusing to answer at all (AMBIGUOUS) when several candidates fit. The import ladder asks "is this row the same human as one we already hold?" and must return a Person either way, because the EP cannot exist without one. A tier that is acceptable here would be a data leak in the portal; a threshold that is right there would reject most of a legitimate entry list here.

What they do share is the underlying evidence model — the same fields, ranked by the same notion of uniqueness — and the same sensitivity to pre-existing duplicates in the database, which degrade both.

3. The matching ladder

For each row the importer walks the tiers in order and stops at the first that resolves. The tier that fired is carried back on ImportPersonMatchResult, so the outcome records how the link was made and not merely that it was.

Only Tiers 1 and 2 carry the authority to link an existing Person. Below the identity number the evidence establishes resemblance, not identity, so ADR-0012 reserves that decision for a human: an interactive flow asks the operator, and a headless flow creates a new Person and files a merge candidate.

The ladder below is the implementation as it stands, and it does not yet honour that rule — Tiers 3 and 4 still link. See Limitations of the current design.

 
row
 │
 ├─► Tier 1  External UID      ──hit──► Person ──► returns (mapping already known; last_seen_at refreshed)
 │
 ├─► Tier 2  Identity number   ──hit──► Person ──┐        >1 hit → DUPLICATE_SAID, row fails
 │                                               │
 ├─► Tier 3  DOB + gender      ──hit──► Person ──┤        weak — skipped when an identity number was supplied
 │                                               │
 ├─► Tier 4  First + last name ──hit──► Person ──┤        weakest — skipped when a validating ID matched nobody;
 │                                               │        a candidate whose stored DOB contradicts the row's is refused
 └─► Tier 5  Create new Person ───────► Person ──┘
                                                 │
                                                 ▼
                                 Person-XID write-back (tiers 2 and 5 only —
                                 a weak resolution mints no mapping, ADR-0012 rule 6)
Tier MatchTier Condition and lookup Confidence

1

EXTERNAL_UID

Runs only when a RegistrationSystem is in context, is_self is not true, and the row carries a non-blank sourceSystemPersonId (trimmed). Looks up PersonExternalReference by the (registration_system_id, external_uid) pair; on a hit refreshes last_seen_at and returns.

Highest. A deliberate, previously-confirmed system-of-record claim — not an inference.

2

IDENTITY_NUMBER

Runs when identityNumber is non-blank. Delegates to the shared matchPerson pipeline, which trims the value once ahead of all three identity types — a blank or whitespace-only number matches nothing at all rather than colliding with every other record that has no number. NATIONAL matches on (id_number, id_country) with the country forced to the tenant’s; PASSPORT matches the same pair then falls back to the number alone; OTHER searches id_number only — the legacy other_number fallback for WordPress-migrated records was retired 2026-09-02 after its 12-month transitional window; its data is stale. More than one hit throws DuplicateSaidException.

Note the trim is leading and trailing only. Internal spaces are not removed, so a number written 850101 5009 087 will not match a stored 8501015009087.

High. A national identity number is close to a natural key for the population we serve.

3

DOB_GENDER

Runs only when no identity number was supplied, and both dateOfBirth and gender are present. Matches on the (date_of_birth, gender) meta pair.

Weak. A shared birthday is a coincidence, not an identity.

4

NAME_LOW_CONFIDENCE

Runs when firstName and lastName are both non-blank and the row did not supply an identity number that validates as a national ID in the form Tier 2 searched with. Matches on the (first_name, last_name) meta pair, then refuses any candidate whose stored date of birth is present and differs from the row’s — a namesake with a contradicting birth date is a different human, not a fuzzy match (Bug #1129).

Weakest. Named as low-confidence in the result and surfaced as a row warning.

5

CREATED

Nothing matched. A new Person is persisted from the row’s own data, scoped to the import’s Organisation.

n/a — asserts nothing about an existing record.

3.1. Collision between the two strong tiers

Tiers 1 and 2 are both evaluated before either is allowed to win, so that their disagreement is detectable. When the external UID resolves to Person A and the identity number resolves to a different Person B, the result is a collision: the import proceeds against A — the stronger, deliberately-stored claim — and B is carried back on getConflictingMatch(). The row is warned as EXTERNAL_UID_COLLISION and a MergeCandidateRef(A, B, EXTERNAL_UID_COLLISION) is attached to the response.

This is the pattern the design aims at throughout: a disagreement is recorded and routed, never silently resolved.

3.2. Duplicate identity numbers stop the row

If the identity number matches more than one Person, the importer does not choose. It throws DuplicateSaidException, the row fails with DUPLICATE_SAID, and the pre-existing duplicate must be resolved before the file is re-run. Picking one of two Persons who claim the same identity number would be a guess with no evidence behind it, and the failure is cheap: the row can be re-imported unchanged once the duplicate is merged.

3.3. The pool is global, by design

Person matching deliberately spans organisations. The system holds one pool of mostly-unique Person records, and a person entering through any organisation reuses an existing record rather than minting a parallel one — which is what makes an athlete’s identity, history and results follow them across the federations and events they enter.

The queries reflect that: they join wp_users to wp_usermeta on the meta key/value pair alone, with no organisation predicate. (Tier 2’s NATIONAL branch narrows by the tenant’s country code, which is about interpreting the number, not about scoping the pool.)

The organisation boundary governs disclosure of personal detail, not lookup. A match may resolve to a Person first created under another organisation; what that organisation’s operators may then see of that Person is a separate, access-controlled question. This is intended behaviour and not a scoping gap — narrowing these queries to the importing organisation would fragment the pool and defeat the deduplication the ladder exists to perform.

3.4. Weak-tier matches collapse to an arbitrary row

Tier 3 reads a page of up to 50 candidates and Tier 4 a page of up to 5, and each takes the first element. Neither query carries an ORDER BY, so "first" is whatever the database returns. Two runs of the same file can in principle resolve to different Persons, and the other candidates are never shown.

4. The supplied-but-unmatched identity rule

The governing rule of the ladder:

An identity number that validates and matches nobody is positive evidence that this is a new person.

Roughly 98% of the South African population we serve holds a national identity number. If a row carries one that passes the checksum and it matches no Person we hold, the overwhelmingly likely explanation is that we do not hold this human — not that we hold them under a number we never recorded. Falling through to a weaker tier at that point does not add information; it discards a strong negative signal in favour of a weak positive one.

Two qualifications carry most of the design.

Validity, not presence. A number that fails the checksum is a typo, not an identity claim. It says nothing about whether we already hold the person, so it must not suppress the weaker tiers — the name match is exactly what correctly re-attaches a returning entrant whose identity number was mistyped on a later entry. Tier 4 is therefore keyed on PersonIdentityUpdatePolicy.isValidSaId, not on the field being populated.

The number must be in the form Tier 2 searched with. isValidSaId trims and strips all spaces before validating; Tier 2 trims only the ends. A number carrying internal spaces — as spreadsheet exports often do — would therefore validate here while never having had a chance to match at Tier 2, and suppressing the name match on that basis would create a second Person for someone already on file. The guard requires value.equals(value.trim().replace(" ", "")) as well as validity.

The practical consequence for anyone preparing an import: strip all whitespace from identity numbers before submitting. A number with internal spaces misses at Tier 2 and fails this guard, so the row falls through to the name match — the one path where namesakes can be collapsed. Normalising the Tier 2 lookup instead is not a safe substitute: stored identity numbers are themselves inconsistently formatted, so making the two agree is a data migration rather than a change to this method.

4.1. Why the two guards differ

Tiers 3 and 4 are guarded on different conditions, and the difference is deliberate.

Tier Guarded on Reasoning

3 — DOB + gender

Identity number supplied

A row that carries any identity number is asserting an identity. Matching it to whoever shares a birthday is an anti-match whether or not the number is well-formed, and there is no case in which a birthday collision is the right answer for such a row.

4 — Name

Identity number validates, in the form Tier 2 searched with — plus a per-candidate date-of-birth contradiction check (date of birth only; see the note below on gender)

A validating number that matched nobody says "new person". A malformed one says nothing: it is absence of evidence, not evidence of a returning entrant. The rows behind it are two populations the number cannot separate — a returning entrant who mistyped (for whom the name match is right) and a genuinely new person whose number is unusable (for whom it silently binds them to a namesake). The date of birth is the discriminator: a candidate whose stored DOB contradicts the row’s is refused, and only an agreeing (or underdetermined) candidate may still take the low-confidence link.

A mistyped identity number therefore skips Tier 3 and reaches Tier 4, where the name match may still resolve it onto the existing Person — but only past the DOB check. Tier 3’s condition is deliberately left wider than Tier 4’s rather than narrowed to match: there is no case in which a birthday coincidence is the right answer for a row asserting any identity number, while the name tier retains a genuine (DOB-vetted) re-attachment case.

Gender is deliberately not a contradiction signal. It is a single low-cardinality field: two namesakes agree on it half the time by chance, so it refuses far fewer true mismatches than a birth date while manufacturing duplicates from ordinary data noise — a mis-keyed or absent gender on an otherwise correct row would push a returning entrant to a new Person. Date of birth separated every observed false match from every observed benign one, and it does so without that cost. Revisit only with evidence of a mismatch class that gender would catch and DOB would not.

The residue is the row that carries a malformed number and no contradicting date of birth — there the two populations remain inseparable and the low-confidence link is still taken, wrongly for the new-person case. This is accepted interim behaviour, eliminated only by ADR-0012 rules 2–4 (candidates, not links). Until then a clean import summary is not evidence of a clean import: read the merge candidates row by row.

4.2. Illustration

Three entrants sharing a first and last name — two of them distinct humans, the third a returning entrant.

Row Row data Ladder outcome

A

Validating identity number, matching nobody

Tier 3 skipped (number supplied). Tier 4 skipped (number validates). Tier 5 creates Person P.

B

A different validating identity number, same name as A

Identical path. Tier 5 creates a second, distinct Person. The namesakes stay separate.

C

Same name as A, identity number mistyped so the checksum fails, date of birth agrees with P’s

Tier 3 skipped (number supplied). Tier 4 runs — a typo is not an identity claim — P’s stored DOB agrees, and the name match re-attaches the returning entrant.

D

Same name as A, malformed identity number, date of birth contradicts P’s

Tier 3 skipped. Tier 4 runs, finds P, and refuses the candidate on the DOB contradiction — Tier 5 creates a distinct Person. Before Bug #1129 this row was silently bound to P, inheriting P’s identity number, birth date and history; observed in production-shape data 29 years apart.

E

Same name as A, malformed identity number, no date of birth on the row

Tier 3 skipped. Tier 4 runs and links P low-confidence — rows C and D are indistinguishable here, and the link is the accepted interim residue (see the warning above). The row warning is the only trace.

Row B is the case the Tier 4 validity guard exists for; row D is the case the DOB contradiction check exists for. Without it, B would have been attached to P by name. Within a single event the (event_id, person_id) unique constraint rejects the second EP insert and the row fails visibly; across two events there is no constraint, no error, and the second entrant’s entry is silently attached to the first’s Person, carrying across the identity number, date of birth, results history and any future external-reference write-back. That silent variant is the failure the guard prevents.

5. What the importer reports

Every non-obvious outcome is surfaced structurally, not only as prose, so the import summary can count and link them rather than scrape warning text.

Signal Carried on Meaning

MatchTier

ImportPersonMatchResult.getTier()

Which tier fired. Feeds the row response DTO.

LOW_CONFIDENCE_MATCH

Row warning + MergeCandidateRef(matchedId, null, …)

A weak tier fired — the link rests on a name alone (Tier 4) or a birthday+gender coincidence alone (Tier 3; warned since Bug #1129 — previously a Tier 3 link was completely silent). personIdB is null because only one Person is in play; the candidate means "verify this assignment", not "merge these two".

EXTERNAL_UID_COLLISION

Row warning + MergeCandidateRef(A, B, …)

Tiers 1 and 2 disagreed. A was used; B needs review.

SAID_COLLISION

MergeCandidateRef

The row’s identity number resolved to several Persons — each pair surfaces so they can be reviewed independently.

DUPLICATE_SAID

Row error

As above, but as the row’s terminal outcome: nothing was committed.

MergeCandidateRef is a transport type on the import response, not a persisted record. candidateId is not populated: the pair is reported, and nothing is written to a review queue.

6. The ladder improves itself: Person-XID write-back

Tiers 2 through 5 funnel into a single tail; only Tier 2 and Tier 5 resolutions reach the write, which records the (RegistrationSystem, external_uid) → Person mapping as a PersonExternalReference with source = IMPORT. The next import from that same source system resolves the person at Tier 1 in one hop, without re-deriving identity from the row.

A weak-tier resolution is excluded by ADR-0012 rule 6: the write-back is the mechanism that mints the very facts rule 1 treats as system-of-record, and combined with the never-repoint rule below it would promote a DOB or name guess to the strongest tier, silently and permanently — with the low-confidence warning never appearing again. That promotion was observed live (Bug #1129) before the suppression was added: a name-tier misattribution wrote the victim’s external uid against the wrong Person, and every subsequent import would have resolved it at Tier 1 with nothing left to review. A repeat entrant resolved by a weak tier therefore re-runs the ladder on each import instead of gaining a Tier-1 shortcut — the intended cost.

This is what makes cross-event deduplication work: without it, the only durable link between two imports of the same human is the identity number, and rows without one fall back to the weak tiers every time.

The write-back never repoints an existing mapping at a different Person — an existing row only has its last_seen_at refreshed. Repointing would be exactly the over-merge the collision path exists to prevent. Tier 1 returns before the tail because its mapping already exists and was refreshed on the way in. The never-repoint rule cuts both ways: it protects a good mapping from corruption and preserves a bad one against correction, which is why what may write one is now gated at the tier and why a person merge migrates person_external_reference rows to the surviving Person (they carry the losing record’s import identity) and restamps their source to MERGE, so the audit trail does not read a merge-driven transfer as an import that never mapped that Person.

The upsert is a best-effort read-then-insert rather than a constraint-safe one. Rows within a job are processed sequentially so an intra-job race cannot occur; two concurrent jobs carrying the same external UID could collide on the unique constraint and would surface as an ordinary failed row, not as corrupted data.

7. Design rationale — alternatives weighed

Rejected alternative Why this design instead

Match on the strongest available signal, always

Treats "no identity number matched" as absence of evidence when it is evidence of absence. The supplied-but-unmatched rule exists because a strong negative outranks a weak positive.

Pick one when the identity number matches several Persons

A guess with nothing behind it, applied to the tier we trust most. Failing the row is cheap and re-runnable; a wrong merge at Tier 2 is neither.

Let the identity number win over the external UID on collision

The external UID is a claim a system of record deliberately stored and we previously confirmed; the identity number is a value retyped into a spreadsheet. The stronger claim wins and the weaker is preserved for review — neither is discarded.

Score-and-threshold the whole ladder, as the portal does

The portal can refuse to answer; the importer cannot — EP.person is NOT NULL. A threshold that rejects ambiguity would reject a large fraction of a legitimate entry list. Ordered tiers with an explicit create-new floor give a deterministic answer for every row.

Repoint an existing Person-XID mapping when the row disagrees

That is over-merge with extra steps. Existing mappings are refreshed, never redirected; disagreement becomes a merge-candidate ref.

Nullable EventParticipant.person for unresolved rows

An honest model — "identity not yet known" — but person_id is NOT NULL, is half of the (event_id, person_id) unique constraint, and is the spine for number assignment, results, leaderboards and security filtering. Making it nullable means null-handling in every consumer and a reworked uniqueness model. The create-new floor reaches a usable answer without the schema change.

8. Limitations of the current design

Properties of the ladder as built, stated so that callers and operators can reason about what it does and does not guarantee:

  • Weak tiers link an existing Person, which they must not. ADR-0012 rules that only a human may resolve a weak match: Tier 3 and Tier 4 may raise candidates but never set the participant’s Person. The implementation predates that rule and still links — Tier 3 for rows carrying no identity number, Tier 4 for rows with no validating identity number in searched form and no contradicting date of birth. This is the largest gap between the design and the code, and everything below follows from it. The Bug #1129 interim guards narrow the blast radius (a DOB contradiction refuses the candidate; a weak link mints no Person-XID mapping and now warns on both tiers) but do not close the gap: closing it is US #773 and US #1118.

  • There is no adjudication step, and the interactive path is deferred. The import mapping flow pauses for unresolved foreign-key values but not for an ambiguous person, so an operator who is present is never asked. ADR-0012 rule 3 classifies that flow as interactive and requires the operator resolve the row there. Until US #773 lands, no flow takes the rule-4 headless behaviour either — a weak match still links directly (see the first bullet), and the operator reviews the low-confidence warning after the fact rather than resolving the ambiguity beforehand. When both land, the ADR’s core rule will hold in every flow: no flow links an existing Person on weak evidence.

    The structural reason it is not a small change: person matching runs inside row processing, while the mapping flow’s pauses all happen before processing begins. The candidates therefore do not exist at the point a dialog would show them, so the work needs either a matching pre-pass in its own stage or a processing stage that can halt across an operator round-trip.

  • Weak-tier results are non-deterministic. First-of-page with no ORDER BY, and the other candidates are discarded rather than offered.

  • Merge candidates are reported, not persisted. They exist for the duration of the import response; there is no review queue behind them and no deep-link from the summary.

  • Pre-existing duplicates fail rows outright. A duplicated identity number in the database stops every row carrying it, and can only be cleared by merging the duplicate.

9. Related documentation

10. Source code

File Role

admin-service/…/service/PersonService.java

matchPersonForImport — the ladder. recordPersonExternalReference — the Person-XID write-back. matchPerson — the shared identity/DOB lookup pipeline. matchOrAdd — the legacy single-add path.

admin-service/…/service/ImportPersonMatchResult.java

Result carrier: chosen Person, MatchTier, conflicting match.

admin-service/…/service/DuplicateSaidException.java

Raised when the identity number resolves to several Persons.

admin-service/…/service/EventParticipantServiceEx.java

The caller. Converts tier + collision into row warnings and MergeCandidateRef entries.

admin-service/…/service/dto/MergeCandidateRef.java

Structured merge-candidate ref surfaced on the import response.

admin-service/…/service/ProgressiveMatchService.java

The unrelated interactive path — listed here only to keep the two apart.

wordpress-database/…/wordpress/UserBaseRepository.java

findByMeta / findByMetaAndMeta — the meta-pair queries every tier below Tier 1 is built on.

11. Change history

Date Change

2026-09-07

Recorded that gender is deliberately excluded from the Tier-4 contradiction check, and why — the reasoning previously lived only in a code comment. Noted that a merge restamps a repointed external reference’s provenance to MERGE.

2026-09-02

Bug #1129 corrections. Rewrote "Why the two guards differ" — a malformed identity number is absence of evidence, not evidence of a returning entrant; added the two-population account, the date-of-birth contradiction check on Tier 4 candidates, and illustration rows D and E. Constrained the Person-XID write-back to Tiers 2 and 5 (ADR-0012 rule 6): a weak resolution mints no mapping. Tier 3 links now warn like Tier 4. Retired the OTHER other_number fallback. Noted that person merge migrates person_external_reference to the surviving Person.

2026-09-01

Corrected Tier 2 against the current implementation: the shared pipeline trims the identity number ahead of all three identity types (blank now matches nothing rather than colliding with every record lacking a number), and OTHER searches id_number first with other_number as a legacy fallback. Added the whitespace guidance for import preparation.

2026-09-01

Recorded the authority rule from ADR-0012 — only Tiers 1 and 2 may link an existing Person — and restated the weak tiers' current linking behaviour as the gap against it rather than as normal operation. Corrected the global person pool from a stated limitation to the design property it is.

2026-08-31

Created. States the matching ladder as implemented in PersonService.matchPersonForImport: the five tiers and their conditions, the collision and duplicate-identity paths, the Person-XID write-back, the supplied-but-unmatched identity rule and the differing guards it places on tiers 3 and 4, and the boundary against progressive matching.