refactor(workflow-compiler): unify duplicate compilers into common/workflow-compiler

[Yicong-Huang]

What changes were proposed in this PR?

This is a source-level de-duplication only — the runtime behavior of two compilations stays. A workflow is still compiled twice: once at editing time (through workflow-compiling-service) and once at execution time (through amber). This PR only makes both paths share one compiler implementation instead of two hand-synced copies.

amber and workflow-compiling-service each maintained a separate copy of the workflow compiler and its models (WorkflowCompiler, LogicalPlan, LogicalLink, LogicalPlanPojo), which had to be kept in sync by hand (e.g. #6288 had to patch both copies). This PR consolidates them into a single shared common/workflow-compiler module that both services depend on; the old per-service copies are removed.

The unified WorkflowCompiler.compile returns a WorkflowCompilationResult (logical plan, optional physical plan, per-port output schemas, per-operator errors, and the output ports needing storage) and takes a CompilationErrorHandling mode: Lenient for the editing-time path (accumulate per-operator errors so the UI can render them) and Strict for the execution path (fail-fast before a run). amber's engine Workflow becomes a thin wrapper over the compilation result, built via Workflow.fromCompilationResult.

Two known minor behavior changes (both flagged in review, both intentional):

• Plan expansion now uses scala.util.Try (non-fatal only) where the old compiling-service caught Throwable, so a fatal JVM error during expansion now fails the /compile request instead of being recorded per-operator.
• Strict now also fails fast on schema-propagation errors (e.g. a Projection on a missing column). The legacy execution path silently launched such plans and they only failed at runtime; per review discussion the fail-fast contract now covers this case too, pinned by a test.

Any related issues, documentation, discussions?

Closes #6142. Design rationale lives in the scaladoc of WorkflowCompiler.

How was this PR tested?

• Migrated the compiler and model unit specs into the new module and added strict-mode coverage (success path and the schema-propagation fail-fast path).
• amber and workflow-compiling-service compile and test against the shared library.
• Rebased over fix(frontend): improve static error msgs to be user-friendly for unco… #6288 and verified its error-message changes carry over into the shared module (they now exist in one place instead of two).

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8 (Claude Code); review follow-ups by Claude Fable 5 (Claude Code)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @aglinxinyuan, @Ma77Ball, @roshiiiz
  You can notify them by mentioning @aglinxinyuan, @Ma77Ball, @roshiiiz in a comment.

[aglinxinyuan]

Let's get input from @bobbai00 and @Xiao-zhen-Liu as well.

[Copilot]

Pull request overview

This PR consolidates the previously duplicated workflow compiler implementations into a single shared common/workflow-compiler module, repointing amber and workflow-compiling-service to depend on and use the unified API with explicit Lenient (editing-time) vs Strict (execution) error-handling semantics.

Changes:

• Introduced a shared common/workflow-compiler module exposing WorkflowCompiler.compile(..., CompilationErrorHandling) returning WorkflowCompilationResult (logical plan, optional physical plan, schemas, per-op errors, storage ports).
• Updated amber execution/adapters and workflow-compiling-service to import and use the shared compiler/models; removed amber’s legacy compiler/model copies.
• Migrated/expanded compiler unit coverage into the shared module and updated affected amber tests/imports accordingly.

Reviewed changes

Copilot reviewed 35 out of 36 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
workflow-compiling-service/src/test/scala/org/apache/texera/service/resource/WorkflowCompilationResourceSpec.scala	Updates model/spec references to the shared compiler package.
workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowCompilationResource.scala	Switches resource to use shared WorkflowCompiler + shared LogicalPlanPojo.
common/workflow-compiler/src/test/scala/org/apache/texera/compiler/WorkflowCompilerSpec.scala	Migrates and expands compiler behavior tests (physical plan shape, storage ports, strict mode).
common/workflow-compiler/src/test/scala/org/apache/texera/compiler/model/LogicalLinkSpec.scala	Moves test into the unified org.apache.texera.compiler.model package.
common/workflow-compiler/src/test/scala/org/apache/texera/compiler/LogicalPlanSpec.scala	Repoints plan tests to unified LogicalPlan/Pojo/Link model types.
common/workflow-compiler/src/main/scala/org/apache/texera/compiler/WorkflowCompiler.scala	Implements unified compiler API/result, adds storage-port computation and strict/lenient handling.
common/workflow-compiler/src/main/scala/org/apache/texera/compiler/model/LogicalPlanPojo.scala	Moves LogicalPlanPojo into shared compiler model package.
common/workflow-compiler/src/main/scala/org/apache/texera/compiler/model/LogicalPlan.scala	Moves LogicalPlan and adds getTerminalOperatorIds used for storage-port selection.
common/workflow-compiler/src/main/scala/org/apache/texera/compiler/model/LogicalLink.scala	Moves LogicalLink into shared compiler model package.
common/workflow-compiler/src/main/scala/org/apache/texera/compiler/CompilationErrorHandling.scala	Introduces explicit strict vs lenient compilation error-handling mode.
common/workflow-compiler/DESIGN.md	Documents rationale, API contract, and module architecture for the unification.
common/workflow-compiler/build.sbt	Adds module-local sbt settings/deps for the new shared compiler module.
build.sbt	Registers WorkflowCompiler project and rewires amber/workflow-compiling-service dependencies.
amber/src/test/scala/org/apache/texera/workflow/WorkflowCompilerSpec.scala	Removes legacy amber compiler spec now covered in the shared module.
amber/src/test/scala/org/apache/texera/web/service/WorkflowExecutionServiceSpec.scala	Updates imports to shared LogicalPlanPojo.
amber/src/test/scala/org/apache/texera/amber/engine/faulttolerance/CheckpointSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/e2e/TestUtils.scala	Executes compilation in strict mode and adapts to WorkflowCompilationResult for workflow construction.
amber/src/test/scala/org/apache/texera/amber/engine/e2e/ReconfigurationSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/e2e/PauseSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/e2e/DataProcessingSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/e2e/BatchSizePropagationSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/resourcePolicies/ResourcePoliciesSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/ExpansionGreedyScheduleGeneratorSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/DefaultCostEstimatorSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/CostBasedScheduleGeneratorSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/controller/WorkflowSchedulerSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/test/integration/org/apache/texera/amber/engine/e2e/ReconfigurationIntegrationSpec.scala	Repoints LogicalLink import to shared compiler model.
amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala	Removes legacy amber compiler implementation (replaced by shared compiler + thin adapter).
amber/src/main/scala/org/apache/texera/workflow/LogicalPlan.scala	Removes legacy amber logical-plan model (replaced by shared compiler model).
amber/src/main/scala/org/apache/texera/web/service/WorkflowService.scala	Repoints LogicalPlan import to shared compiler model.
amber/src/main/scala/org/apache/texera/web/service/WorkflowExecutionService.scala	Adapts execution path to strict compilation + uses shared compilation result and storage ports.
amber/src/main/scala/org/apache/texera/web/resource/SyncExecutionResource.scala	Uses strict compilation for validation paths and repoints model/compiler imports.
amber/src/main/scala/org/apache/texera/web/model/websocket/request/WorkflowExecuteRequest.scala	Switches request to shared LogicalPlanPojo type (removes inline duplicate model).
amber/src/main/scala/org/apache/texera/web/model/websocket/request/EditingTimeCompilationRequest.scala	Repoints LogicalLink/LogicalPlanPojo imports to shared compiler model.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/controller/Workflow.scala	Repoints LogicalPlan to shared compiler model in engine workflow wrapper.
.github/workflows/build.yml	Adds jacoco aggregation for the new WorkflowCompiler module.

Comments suppressed due to low confidence (1)

common/workflow-compiler/src/main/scala/org/apache/texera/compiler/WorkflowCompiler.scala:262

• In Strict mode, schema-propagation failures from collectOutputSchemaFromPhysicalPlan are currently written into a throwaway buffer and then ignored (no throw, and they don’t land in operatorIdToError). This violates the Strict "fail-fast" contract and can let execution proceed with schema errors that would only surface later.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 72.22222% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.69%. Comparing base (f02dd2f) to head (929e78e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...org/apache/texera/amber/util/StackTraceUtils.scala	0.00%	9 Missing ⚠️
...ache/texera/common/compiler/WorkflowCompiler.scala	89.13%	1 Missing and 4 partials ⚠️
.../texera/web/service/WorkflowExecutionService.scala	0.00%	4 Missing ⚠️
...he/texera/web/resource/SyncExecutionResource.scala	0.00%	1 Missing ⚠️
...era/common/compiler/CompilationErrorHandling.scala	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6143      +/-   ##
============================================
+ Coverage     77.30%   77.69%   +0.39%     
- Complexity     3524     3595      +71     
============================================
  Files          1161     1160       -1     
  Lines         45922    45999      +77     
  Branches       5101     5099       -2     
============================================
+ Hits          35501    35741     +240     
+ Misses         8840     8659     -181     
- Partials       1581     1599      +18     

Flag	Coverage Δ
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)
amber	70.04% <71.83%> (+0.94%)	⬆️
computing-unit-managing-service	20.49% <ø> (ø)
config-service	66.66% <ø> (ø)
file-service	67.21% <ø> (ø)
frontend	82.58% <ø> (ø)
notebook-migration-service	78.94% <ø> (ø)
pyamber	92.28% <ø> (+0.13%)	⬆️
workflow-compiling-service	26.31% <100.00%> (-28.84%)	⬇️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main f02dd2f benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	393	0.24	25,510/30,665/30,665 us	🔴 +16.1% / 🔴 +99.0%
🔴	bs=100 sw=10 sl=64	788	0.481	125,586/151,931/151,931 us	🔴 -5.1% / 🔴 +39.3%
⚪	bs=1000 sw=10 sl=64	907	0.553	1,097,818/1,166,293/1,166,293 us	⚪ within ±5% / 🔴 +11.5%

Baseline details

Latest main f02dd2f from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	393 tuples/sec	430 tuples/sec	754.55 tuples/sec	-8.6%	-47.9%
bs=10 sw=10 sl=64	MB/s	0.24 MB/s	0.263 MB/s	0.461 MB/s	-8.7%	-47.9%
bs=10 sw=10 sl=64	p50	25,510 us	21,968 us	12,816 us	+16.1%	+99.0%
bs=10 sw=10 sl=64	p95	30,665 us	32,947 us	16,594 us	-6.9%	+84.8%
bs=10 sw=10 sl=64	p99	30,665 us	32,947 us	19,806 us	-6.9%	+54.8%
bs=100 sw=10 sl=64	throughput	788 tuples/sec	830 tuples/sec	969.38 tuples/sec	-5.1%	-18.7%
bs=100 sw=10 sl=64	MB/s	0.481 MB/s	0.507 MB/s	0.592 MB/s	-5.1%	-18.7%
bs=100 sw=10 sl=64	p50	125,586 us	119,930 us	103,584 us	+4.7%	+21.2%
bs=100 sw=10 sl=64	p95	151,931 us	152,060 us	109,097 us	-0.1%	+39.3%
bs=100 sw=10 sl=64	p99	151,931 us	152,060 us	117,304 us	-0.1%	+29.5%
bs=1000 sw=10 sl=64	throughput	907 tuples/sec	913 tuples/sec	1,004 tuples/sec	-0.7%	-9.6%
bs=1000 sw=10 sl=64	MB/s	0.553 MB/s	0.557 MB/s	0.613 MB/s	-0.7%	-9.7%
bs=1000 sw=10 sl=64	p50	1,097,818 us	1,089,093 us	1,002,357 us	+0.8%	+9.5%
bs=1000 sw=10 sl=64	p95	1,166,293 us	1,177,515 us	1,046,463 us	-1.0%	+11.5%
bs=1000 sw=10 sl=64	p99	1,166,293 us	1,177,515 us	1,073,661 us	-1.0%	+8.6%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,508.80,200,128000,393,0.240,25509.57,30665.31,30665.31
1,100,10,64,20,2539.43,2000,1280000,788,0.481,125585.80,151931.12,151931.12
2,1000,10,64,20,22054.99,20000,12800000,907,0.553,1097817.56,1166293.11,1166293.11

[bobbai00]

LGTM! Left few minor comments

[chenlica]

@Xiao-zhen-Liu Please chime in.

@Yicong-Huang A general question: should we create a discussion first? Or in what cases should we create a discussion before raising a PR?

[Yicong-Huang]

@Xiao-zhen-Liu Please chime in.

@Yicong-Huang A general question: should we create a discussion first? Or in what cases should we create a discussion before raising a PR?

We create discussions when there could have different opinions, for instance when we are unsure if support a feature or not, when we are not sure whether support something or not, or whether we want to remove/disable something. The discussion usually is on the direction.

For this particular case, I feel having two copies of the compiler source code is problematic and adds maintenance cost. Thus, I did not ask for input to unifying them by creating a shared lib.

There is no change on the design, or experience: we still do two compilations, one at editing time, through compilation service, one at execution time, through execution service. It's only a code level refactoring.

[Xiao-zhen-Liu]

Nice refactor — merging the two hand-synced compiler copies into one module is a clear win, and the boundary is clean (common/workflow-compiler depends only on WorkflowOperator; the WorkflowCompilationResult / CompilationErrorHandling split reads well).

One ask: could you update the PR description to make clear this is source-level de-duplication, not a change to how many times a workflow compiles? We still compile twice at runtime (editing + execution). As written it reads like it might be collapsing them, which threw me off — a line like "two compilations stays; this only shares the code" would clear it up.

I checked both callers against their old versions — faithful almost everywhere. A few non-blocking things inline: Strict doesn't actually fail fast on schema errors (drops them) though the docs say otherwise; one real behavior change on the editing path (Try vs the old catch Throwable); the failed-compile path falls through to a null workflow (bobbai00 spotted it); and Strict is thinly tested. One more small one: the compiling-service caller relies on the default Lenient while amber passes Strict explicitly — passing it explicitly there too would read symmetrically.

Also +1 to bobbai00 on hoisting getUpstreamLinks and the DESIGN.md question (no other module ships one). Good to merge once the Strict/schema wording is sorted or a follow-up is filed.

[chenlica]

Thanks @Xiao-zhen-Liu.

@Yicong-Huang : how easy is it to further improve the system by compiling a workflow only once?

[Xiao-zhen-Liu]

Thanks for the updates, these all look good to me. Making Strict actually fail fast on schema errors instead of deferring it was the right call, and the two new strict tests pin the behavior well. The rewritten description also makes the "still two compilations" point clear up front, which was my main confusion.

The red CI is not from your changes. The branch is now 165 commits behind main, and main has since picked up LoopIntegrationSpec.scala (from #5700), which still imports the org.apache.texera.workflow.LogicalLink that this PR removes. That single import is the only thing breaking the merged build:

amber/src/test/integration/org/apache/texera/amber/engine/e2e/LoopIntegrationSpec.scala:49
- import org.apache.texera.workflow.LogicalLink
+ import org.apache.texera.common.compiler.model.LogicalLink

A rebase plus that line should turn amber green. Worth doing soon: that file is actively changing on main (#6661 touched it again), and since this PR deletes a widely used symbol, the longer it sits the more likely something else picks up the old import.

One thing to watch after the rebase: the amber tests never actually ran, they failed at compile, so the Strict fail-fast change hasn't been exercised against the existing e2e suite yet.

[Yicong-Huang]

@Yicong-Huang : how easy is it to further improve the system by compiling a workflow only once?

@chenlica At this stage, I still think we will need two compilations. one for editing time one for execution time. Whether we can safely push the compiled artifact (e.g., Physical Plan) to frontend is a separate discussion.

[Yicong-Huang]

Thanks @Xiao-zhen-Liu for review. If no more comments do you mind stamp on the PR?

[chenlica]

@Yicong-Huang : how easy is it to further improve the system by compiling a workflow only once?

@chenlica At this stage, I still think we will need two compilations. one for editing time one for execution time. Whether we can safely push the compiled artifact (e.g., Physical Plan) to frontend is a separate discussion.

OK. We will have a separate discussion about this topic.

[Xiao-zhen-Liu]

LGTM!