Skip to content

Ko TN: Add serial grammar and review fixes - #457

Merged
tbartley94 merged 16 commits into
NVIDIA:staging/ko_tn_v2from
bbae0312:ko-tn-fix
Jul 27, 2026
Merged

Ko TN: Add serial grammar and review fixes#457
tbartley94 merged 16 commits into
NVIDIA:staging/ko_tn_v2from
bbae0312:ko-tn-fix

Conversation

@bbae0312

@bbae0312 bbae0312 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds Korean TN serial grammar support for number-like strings with clear context signals such as 번호 and 연락처.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

bbae0312 and others added 9 commits April 7, 2026 14:08
@bbae0312
Add Korean TN post-processing rules for particle agreement and month …
98d4da2 
…handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
19619a9 
for more information, see https://pre-commit.ci
@bbae0312
Add Korean TN fraction test cases for particle agreement
420fb96 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@bbae0312
Fix Korean fraction verbalization with particle-aware handling and re…
ecf34cc 
…move post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
7acdb88 
for more information, see https://pre-commit.ci
@bbae0312
Fix date and fraction normalization issues based on review feedback
f47b370 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
be9a45c 
for more information, see https://pre-commit.ci
@bbae0312
Add Korean TN serial and review fixes
cbf4bd4 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c747daf 
for more information, see https://pre-commit.ci
@tbartley94
tbartley94 self-requested a review July 10, 2026 17:48
tbartley94
tbartley94 requested changes Jul 14, 2026
View reviewed changes

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serial appears more appropriate as part of the cardinal class no? What's the rationale for a standalone class?

Comment thread tests/nemo_text_processing/ko/test_serial.py Outdated
@@ -1,4 +1,4 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

@tbartley94 tbartley94 Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change copyright

@bbae0312 bbae0312 Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread nemo_text_processing/text_normalization/ko/taggers/serial.py Outdated
"""

def __init__(self, deterministic: bool = True):
super().__init__(name="serial", kind="classify", deterministic=deterministic)

@tbartley94 tbartley94 Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't believe serial is sparrowhawk compliant no?

@bbae0312 bbae0312 Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed serial.py

Comment thread nemo_text_processing/text_normalization/ko/taggers/telephone.py
Comment thread nemo_text_processing/text_normalization/ko/verbalizers/fraction.py Outdated

@tbartley94 tbartley94 Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from what i'm seeing is that this is just replacing a single korean character. couldn't you compress the three together for simplicity?

@bbae0312 bbae0312 Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I combined those three rules into a single one.

@bbae0312

bbae0312 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Serial appears more appropriate as part of the cardinal class no? What's the rationale for a standalone class?

I made it standalone because the digit-by-digit reading is triggered by context such as 번호 or 연락처, while regular numbers should remain standard cardinal readings. I agree the numeric logic is closer to cardinal, though. Would you prefer that I move it into CardinalFst and remove the standalone class?

@tbartley94

tbartley94 commented Jul 20, 2026

Copy link
Copy Markdown
Member

Serial appears more appropriate as part of the cardinal class no? What's the rationale for a standalone class?

I made it standalone because the digit-by-digit reading is triggered by context such as 번호 or 연락처, while regular numbers should remain standard cardinal readings. I agree the numeric logic is closer to cardinal, though. Would you prefer that I move it into CardinalFst and remove the standalone class?

Take a gander if it's feasible to just make it in Cardinals and work from there. If it's too messy then we can keep it. I'm just trying to limit semiotic spread

@bbae0312

bbae0312 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Serial appears more appropriate as part of the cardinal class no? What's the rationale for a standalone class?

I made it standalone because the digit-by-digit reading is triggered by context such as 번호 or 연락처, while regular numbers should remain standard cardinal readings. I agree the numeric logic is closer to cardinal, though. Would you prefer that I move it into CardinalFst and remove the standalone class?

Take a gander if it's feasible to just make it in Cardinals and work from there. If it's too messy then we can keep it. I'm just trying to limit semiotic spread

Sounds good. I’ll see if I can integrate it into the cardinal class cleanly and push the update. If it gets too messy, I’ll let you know.

bbae0312 and others added 5 commits July 23, 2026 15:13
@bbae0312
Address Korean TN review feedback
34b8698 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@bbae0312
Merge branch 'staging/ko_tn_v2' into ko-tn-fix
95f08fc 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
33b6c0a 
for more information, see https://pre-commit.ci
@bbae0312
Resolve Korean telephone conflict and remove serial references
5586dcd 
Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
7b5ae47 
for more information, see https://pre-commit.ci
tbartley94
tbartley94 requested changes Jul 27, 2026
View reviewed changes

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor formatting changes. GTG but please see question on serial weights

Comment thread nemo_text_processing/text_normalization/ko/taggers/cardinal.py Outdated
# Require at least three digits.
serial_body = (
serial_digit
+ pynini.closure(serial_separator, 0, 1)

@tbartley94 tbartley94 Jul 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(serial_digit + pynini.closure(serial_separator, 0, 1) ** 2 . Cleaner. Don't need to add more lines to cardinal class.

@bbae0312 bbae0312 Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread nemo_text_processing/text_normalization/ko/taggers/cardinal.py
Comment thread nemo_text_processing/text_normalization/ko/taggers/telephone.py Outdated
Comment thread nemo_text_processing/text_normalization/ko/taggers/tokenize_and_classify.py Outdated
Comment thread nemo_text_processing/text_normalization/ko/verbalizers/fraction.py
bbae0312 and others added 2 commits July 27, 2026 10:54
tbartley94
tbartley94 approved these changes Jul 27, 2026
View reviewed changes

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tbartley94
tbartley94 merged commit 541c668 into NVIDIA:staging/ko_tn_v2 Jul 27, 2026
2 checks passed
folivoramanh pushed a commit that referenced this pull request Jul 30, 2026
@tbartley94 @bbae0312
@pre-commit-ci @mgrafu
Ko TN: Add serial grammar and review fixes (#457) (#465)
acacc21 
* Ko TN: Add serial grammar and review fixes (#457)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* KO TN: Update Jenkins cache (#467)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Korean TN Jenkins cache

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tbartley94 tbartley94 tbartley94 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bbae0312 @tbartley94