nameparser 2.0 implementation#288
Draft
derek73 wants to merge 176 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping typing_extensions, CI matrix) remains tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds two composition tests over the real RU and TR_AZ packs (the synthetic composition tests in test_parser.py stay as-is): the union of both packs' patronymic rules parses one name from each pack's case segment correctly, and a custom-lexicon base survives the parser_for fold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spec §5.2: each pack (and both together) must parse the full default corpus -- the shared case table's untargeted rows plus the 486-name differential corpus at tools/differential/corpus.jsonl -- identically to the default parser, or the deviation must be declared via the pack's DEVIATES predicate. Ran clean: no undeclared deviations across RU (1 declared), TR_AZ (1 declared), or the combined pack (2 declared) -- the be3f25e predicate hardening already covers this corpus. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prefix reasoning per new NON_FIRST_NAME_PREFIXES/PREFIXES entry
(nameparser/config/prefixes.py):
- بن ("bin"/"ibn", son of): added to NON_FIRST_NAME_PREFIXES. Latin
'bin' stays ambiguous only to protect the unrelated Chinese given
name "Bin"; that collision risk doesn't exist in Arabic script, and
بن is never a bare given name on its own.
- بنت ("bint", daughter of): NON_FIRST_NAME_PREFIXES, mirroring the
existing Latin 'bint' entry.
- ابن ("ibn", son of, alternate spelling): NON_FIRST_NAME_PREFIXES,
mirroring the existing Latin 'ibn' entry.
- أبو / ابو ("abu", father of): left in plain PREFIXES (ambiguous),
mirroring the existing Latin 'abu' entry -- "Abu Bakr" reads "Abu"
as a given name.
- آل ("aal", family/clan of, e.g. "Al Saud"): NON_FIRST_NAME_PREFIXES.
Distinct from the excluded definite article "ال" (#269 explicitly
excludes standalone "ال"); a clan prefix, never a bare given name.
- בן ("ben", son of) / בת ("bat", daughter of): NON_FIRST_NAME_PREFIXES,
same reasoning as بن -- no Latin collision risk, never a bare given
name in Hebrew usage.
Verification notes:
- Every parametrized test row was verified live against a
runtime-augmented Lexicon.default() before the data landed. Two
rows came out differently than the task's first guess: a
single-alphabetic-character conjunction (Cyrillic "и"/"і") hits v1's
issue #11 initial carve-out and does not join a short 3-piece title
chain, so the "и" conjunction row uses a longer, 5-piece chain
instead; "الشيخ" was added to FIRST_NAME_TITLES (like its
transliterated cousin "sheikh") so a lone following name reads as
given, not family.
- Geresh/gershayim gate (#269 step 2): probed 'ד"ר'/"גב'" (ASCII
quotes) against extract_delimited live. Both survive untouched --
the quote sits mid-word and never satisfies _open_ok/_close_ok's
boundary test -- so both the ASCII-quote and the typographic
Unicode spellings ('ד״ר' U+05F4, 'גב׳' U+05F3) ship.
- Cyrillic мл/ст (junior/senior) deliberately NOT shipped, per the
issue's own collision caveat; deferral comment left in
nameparser/config/suffixes.py.
- Differential harness: adding native-script بن as a prefix changes
one existing non-Latin corpus row ('محمد بن سلمان', RTL-marked) from
v1's plain-middle-name reading to 2.0's prefix-chained family --
exactly the new-recognition #269 exists to add, not a Latin-corpus
regression. Classified in tools/differential/expected_changes.toml
rather than reverting the entry; harness exits 0 with 2 classified
diffs (the pre-existing comma-family diff plus this one), 0
unexplained.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up to 6028ab4: - Bare 'κ' removed from TITLES: _normalize strips the edge period, so the abbreviated initial 'Κ.' matched the entry and 'Κ. Παπαδόπουλος' parsed as title='Κ.' with an EMPTY given -- degrading the very common initial+surname shape. The single-letter #11 carve-out guards conjunctions only, and Latin TITLES deliberately has no bare single-letter entries for the same reason. Deferral note left next to the Greek entry group; regression test pins the correct initial+surname reading; the bare-κ test row is replaced with κος/κα rows (verified live before pinning). - expected_changes.toml: the #269 rule's name_regex tightened from "بن" to "\bبن\b" -- the bare form matches بن as a substring anywhere (including inside لبنان, 'Lebanon') and would silently absorb unrelated middle/last diffs. Verified live: re.search matches 'محمد بن سلمان' and NOT 'لبنان'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four customize.rst doctests pinned the titles set's LAST element
('zoologist') through the ELLIPSIS form -- #269's non-Latin entries
sort after it and broke all four. The ellipsis now swallows to the
closing brace, so vocabulary additions cannot break the docs again
(the constant-content rule, applied to doctests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
casefold()'s caseless-matching folds mutated stored vocabulary: the
new Greek title 'κος' arrived in the lexicon as the misspelling 'κοσ'
(final sigma flattened) and v1's 'großfürst' as 'grossfürst' -- while
the v1 CONSTANTS path (lc() = lower()) kept both as authored, so the
two surfaces disagreed about the vocabulary's spelling. lower() follows
Unicode SpecialCasing contextually ('ΚΟΣ' -> 'κος'), matches v1's lc()
exactly, and _normalize is the single fold for storage and match-time
lookups, so matching stays symmetric. Pin the preserved spellings and
the default-lexicon fidelity of both entries. (Review finding,
2026-07-17.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared corpus exercised TR_AZ with a single name, leaving that
gate nearly vacuous. Add synthetic rotator names -- one per alternation
branch of each pack's marker regexes, both scripts -- prove each
deviates AND declares individually, and floor every gate's declared
count at its rotator count. Also: TR_AZ per-token DEVIATES regression
(mirror of the RU one), whole-token negative ('Ogluev'), rows for the
untested #269 entries (أبو/ابو kunya ambiguity split, آل leading and
chained, ابن, Ukrainian і), pack-policy-survives-custom-base and
parser_for-chaining composition tests, a facade-path #269 smoke test,
and monkeypatch-based module-cache handling in the laziness test.
(Review findings, 2026-07-17.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Truthiness swallowed an empty code (empty shell variable, typo) into the default-parser branch; 'is not None' routes it to locales.get(), which exits 2 listing the available codes like any unknown code. (Review finding, 2026-07-17.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the invented 'protect the Chinese given name Bin' rationale from prefixes.py -- no source in the repo history attests it; state only the observable fact that Latin 'bin' stays outside NON_FIRST_NAME_PREFIXES. Soften the мл/ст deferral comment to what #269 actually says (both need within-script collision vetting; the 'ст' specificity was ours, not the issue's). Cite issue #185 and its test bank in tr_az.py for parity with ru.py's data-sources citation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop ~13 function-scope re-imports that restated module-scope names (hoisting Policy to the top); share one default-parser baseline (functools.cache) and module-level packed parsers across the gate and rotator tests, so the suite scales corpus + packs instead of corpus x packs; factor the per-pack layering ALLOWED tuples into a shared _LOCALE_PACK_ALLOWED constant mirroring _PIPELINE_STAGE_ALLOWED; and replace the rotator lists' per-row branch comments as the only coverage guarantee with a mechanical test that every alternation branch of every marker regex is exercised by some rotator token. Document why the corpus loader stays a local copy of compare.py's convention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The non-interference gate, rotator lists, and branch-coverage test now iterate the registered packs (_REGISTRY -> _PACKS/_PACKED/_ROTATORS dicts) instead of hand-wiring each pack by name: the per-pack gate tests collapse into one parametrized test, the combined test derives its predicate by OR-ing every registered DEVIATES, and marker regexes are discovered as the pack module's re.Pattern attributes (anchoring read from the pattern's own shape). A contract meta-test requires every registered pack to declare DEVIATES and ship a rotator list, so pack #3 fails structurally until it does -- accepted option C of the 2026-07-18 DEVIATES design note, keeping Locale pure data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1's patronymic_name_order was a bool flag, so True is the likeliest wrong value a migrator passes to the 2.0 field; it failed loudly but with the raw "'bool' object is not iterable". Probe with iter() -- NOT a try/except around tuple()/frozenset(), which would also rewrite exceptions raised inside a caller's generator (pinned property) -- and raise a message naming the v1 flag and both working replacements. Same guard on PolicyPatch's union fields, with the v1 hint only where it applies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1.4's comparison_key()/matches() lowercased; 2.0 casefolds (both the new API and the facade). The deviation is strictly more permissive -- ß/SS and final-sigma case pairs now compare equal -- and comparison is the one surface where casefold's aggressive folds are wanted: the key is opaque, so the vocabulary spelling-mutation casefold caused in storage (fixed to lower() in 3e6e08d) cannot happen here. Release-log behavior-changes entry + a pin test on both API paths, so the next casefold audit finds the intent recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The facade (and v1.4 before it) already indent the multi-line repr body with 4 spaces; ParsedName was the lone tab hold-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shim translates patronymic_name_order=True to {EAST_SLAVIC, TURKIC}
(v1's single bool governed both detectors); the True-guard's hint said
EAST_SLAVIC only, which would migrate users to narrower behavior than
the compat layer gives them. Caught by the migrate-page table audit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
patronymic_name_order=True maps to BOTH patronymic rules (what the shim actually does); the maiden field landed in 1.3, not 1.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness) — full v1 suite (1,240+ tests) reconciled and passing against the facade; differential harness (tools/differential/) verifies 1.4-on-PyPI vs the facade over 486 corpus names with one classified diffru,tr_az) — shipped with the non-interference gate, Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269 default vocabulary, and CLI--localeNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code