fix: Add Substrait roundtrip support for EXISTS and correlated OuterReferenceColumn

[Nithurshen]

Which issue does this PR close?

• Closes [substrait] [sqllogictest] Cannot convert Exists { subquery: <subquery>, negated: true } to Substrait #16280

Rationale for this change

Substrait roundtrip tests were failing for queries involving EXISTS and correlated subqueries (specifically those using outer references). The DataFusion Substrait producer did not support serializing Expr::Exists, and it threw a "feature not implemented" error for OuterReferenceColumn.

Supporting these features is essential for improving the reliability of DataFusion's Substrait integration and enabling complex join scenarios in distributed environments.

What changes are included in this PR?

1. Producer Support for EXISTS:

   • Implemented from_exists in the producer to map Expr::Exists to the Substrait SetPredicate (using PredicateOp::Existence).
   • Added support for NOT EXISTS by wrapping the predicate in a "not" scalar function.

2. Producer Support for OuterReferenceColumn:

   • Updated the producer to serialize OuterReferenceColumn as a custom scalar function named "outer_reference". This avoids schema validation errors during serialization since outer columns often cannot be resolved against the local subquery schema.

3. Consumer Support for OuterReferenceColumn:

   • Updated the consumer to intercept the "outer_reference" scalar function and deserialize it back into a DataFusion OuterReferenceColumn, ensuring a successful roundtrip.

Are these changes tested?

Yes.

• Verified using the existing SQLLogicTest case that was previously failing: joins.slt.
• Command run: cargo test --test sqllogictests -- --substrait-round-trip joins.slt:1233

Are there any user-facing changes?

No breaking API changes. This PR purely expands the coverage of supported Logical Plans that can be converted to/from Substrait.

[gabotechs]

I'm requesting some backup for reviewing this one, thanks for the contribution!

[vbarua]

I don't think this is the best approach to fix this, primarily because you've added a non-standard mechanism to handle outer references. The plans you produce will only work on DataFusion, and not other systems.

If you want to see what this would look like in more standard Substrait, you can use the isthmus tool with the following input:

isthmus \
  --create "CREATE TABLE lsaj_t1(t1_id INT, t1_name VARCHAR, t1_int INT)" \
  --create "CREATE TABLE lsaj_t2(t2_id INT, t2_name VARCHAR, t2_int INT)" \
  "SELECT t1_id, t1_name
FROM lsaj_t1
WHERE NOT EXISTS (SELECT 1 FROM lsaj_t2 WHERE t1_id = t2_id)
ORDER BY t1_id"

to generate a Substrait plan that captures the outer reference using standard Substrait.

[vbarua]

Substrait already has a mechanism for handling outer references via the OuterReference root_type inside of a FieldReference.

Defining a custom function like you have here means that other systems won't be able to understand these plans, and DataFusion won't understands plans from other systems that use the standard mechanism for outer references.