Skip to content

fix: convert nano timestamps to nanoseconds in partition representation#3695

Open
mittalpk wants to merge 1 commit into
apache:mainfrom
mittalpk:fix/partition-nano-timestamp-representation
Open

fix: convert nano timestamps to nanoseconds in partition representation#3695
mittalpk wants to merge 1 commit into
apache:mainfrom
mittalpk:fix/partition-nano-timestamp-representation

Conversation

@mittalpk

Copy link
Copy Markdown

Summary

_to_partition_representation() in pyiceberg/partitioning.py has no registered handler for TimestampNanoType/TimestamptzNanoType. A datetime partition value for a nano-precision timestamp column passes through unconverted instead of being converted to nanoseconds since epoch, the same way TimestampType/TimestamptzType already convert to microseconds.

That unconverted datetime object would reach DataFile.partition, where TimestampNanoWriter calls write_int — an int is expected, not a datetime.

Fixes #3652

Changes

  • Register a handler for TimestampNanoType/TimestamptzNanoType in _to_partition_representation, using the existing datetime_to_nanos() from pyiceberg/utils/datetime.py (mirrors the existing micros handler exactly).
  • Added test_to_partition_representation_timestamps (8 parametrized cases covering None, int passthrough, and datetime conversion for both the existing micros types and the new nano types) and test_to_partition_representation_unrecognized_type_raises (confirms unrecognized input types raise ValueError for both micros and nanos types — previously nano types silently fell through to the generic PrimitiveType handler, which returns the value unchanged rather than raising).

Test plan

  • New tests verified to fail against pre-fix code (git stash the fix in partitioning.py, rerun — 3 of 9 new tests fail as expected) and pass post-fix.
  • uv run pytest tests/test_transforms.py: 274 passed (up from 265 pre-fix, the +9 are the new tests); the 17 pre-existing failures (missing pyiceberg-core extension in this environment) are unchanged before/after — confirmed identical failure set on unmodified main.
  • uv run pytest tests/table/test_partitioning.py: 19 passed / 11 pre-existing failures (same extension-availability cause), unchanged before/after.
  • uv run ruff check / ruff format --check: clean on both changed files.
  • uv run mypy pyiceberg/partitioning.py: no errors attributable to this file (pre-existing unrelated errors in other files are due to optional dependency stubs not installed in this environment).

_to_partition_representation() had no handler for TimestampNanoType/
TimestamptzNanoType, so a datetime partition value passed through
unconverted instead of being converted to nanoseconds since epoch (the
same way TimestampType/TimestamptzType already convert to micros). The
resulting datetime object would reach DataFile.partition where an int
is expected, since TimestampNanoWriter calls write_int.

Fixes apache#3652
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partition values for timestamp_ns columns are not converted to nanoseconds

1 participant