Skip to content

test: cover ArrayExists three-valued logic config#5000

Open
manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/audit-array-exists-three-valued-logic
Open

test: cover ArrayExists three-valued logic config#5000
manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/audit-array-exists-three-valued-logic

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

Relates to apache/datafusion-comet#4180.

Rationale for this change

spark.sql.legacy.followThreeValuedLogicInArrayExists changes the result of exists when no predicate result is true and at least one result is null. The existing SQL test did not exercise that divergent behavior under both configuration values, and result comparison alone did not directly assert that the JVM codegen dispatcher ran.

What changes are included in this PR?

  • Run expressions/array/exists.sql with both values of spark.sql.legacy.followThreeValuedLogicInArrayExists.
  • Add column and literal no-match-plus-null cases, plus an always-null predicate case.
  • Add a focused Scala test that runs ArrayExists under both config values inside assertCodegenRan and checkSparkAnswerAndOperator, so silent fallback to Spark fails the test.
  • Document the config's semantics, JVM codegen-dispatch path, disabled-dispatcher fallback behavior, and test coverage in the supported Spark configurations contributor guide.

How are these changes tested?

  • make core
  • Focused CometSqlFileTestSuite exists matrix on Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2: both configuration cases passed on every profile.
  • CometCodegenHOFSuite on Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2: 5 tests passed on every profile.
  • Documentation generation completed for all four Spark profiles under JDK 17; the final HTML render was not available because sphinx-build is not installed locally.
  • Spotless and git diff --check
  • The pre-commit Rust formatting and Clippy checks passed. The hook's Scala/scalafix step could not resolve the unavailable org.scalameta:semanticdb-scalac_2.13.17:4.13.6 artifact; the commit was amended with --no-verify after the focused cross-version tests above passed.

Co-authored-by: Codex <codex@openai.com>
@comphead

Copy link
Copy Markdown
Contributor

might be related to #4799

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants