refactor(pyamber): replace PyFilesystem2 with tempfile and unpin setuptools

[aicam]

What changes were proposed in this PR?

fs (PyFilesystem2) is unmaintained — 2.4.16 (May 2022) is its final release, and it calls __import__("pkg_resources").declare_namespace(__name__) at import time. setuptools 82.0.0 removed pkg_resources, so import fs hard-crashes there, which is the only reason amber/requirements.txt has carried a setuptools pin since #4199. This PR removes the dependency and the pin.

Before:  ExecutorManager -> fs.open_fs("temp://") -> pkg_resources -> setuptools held < 82
After:   ExecutorManager -> tempfile.mkdtemp + shutil.rmtree -> fs gone -> setuptools unpinned

ExecutorManager was the only consumer, and fs's TempFS is internally tempfile.mkdtemp + shutil.rmtree, so the swap is behavior-preserving.

File	Change
amber/src/main/python/core/architecture/managers/executor_manager.py	cached property fs → tmp_dir (a Path from tempfile.mkdtemp); self.fs.open(...) → builtin open; self.fs.close() → shutil.rmtree(..., ignore_errors=True). Lazy creation, sys.path append/remove, the never-materialized early-return guard, and the tolerated leak-on-force-kill semantics are all preserved.
amber/src/test/python/core/architecture/managers/test_executor_manager.py	the two manager.fs.getsyspath("/") reads become manager.tmp_dir; both close tests now also assert the directory is gone from disk, and a third test covers the early return when no tmp directory was ever created.
amber/requirements.txt	drop fs==2.4.16 and the setuptools==80.10.2 pin plus its comment.
amber/LICENSE-binary-python	drop fs==2.4.16 and appdirs==1.4.4 (an fs-only transitive). six stays — python-dateutil still needs it.

Implementation notes:

• tempfile.mkdtemp is deliberate, not tempfile.TemporaryDirectory: the latter attaches a finalizer that would silently remove the directory on GC or interpreter exit, changing the leak-on-force-kill behavior the existing TODO documents.
• shutil.rmtree(..., ignore_errors=True) matches TempFS's default ignore_clean_errors=True.
• Directories are created with a texera-udf- prefix so a leaked one is identifiable in /var/tmp.
• setuptools is not restored to requirements.txt: nothing in the closure needs it at runtime any more. It still arrives transitively at 81.0.0 through torch's setuptools<82 in operator-requirements.txt, which is harmless and invisible to the license check (pip-licenses excludes system packages).

One extra fix, in the same code path: load_executor_definition now calls importlib.invalidate_caches() before importing the file it just wrote. The import system caches a directory listing per sys.path entry and only refreshes it when the directory mtime changes, so a second UDF written into the same tmp directory within one mtime tick can be invisible to the finder and raise ModuleNotFoundError: No module named 'udf-vN'. This is not introduced here — checking out main's executor_manager.py and test_executor_manager.py with fs installed reproduces it locally (4 TestUpdateExecutor failures in the same environment). It is fixed in this PR because the write-then-import path is exactly what is being rewritten, and a reviewer running the suite locally would otherwise attribute the flake to this change.

Any related issues, documentation, discussions?

Closes #6917. Supersedes #6911 (the setuptools v83 bump, closed — it could not merge while fs was present) and #6412 (setuptools<82, the interim bound). Also makes #6880 unnecessary: with fs gone there is no pkg_resources deprecation warning left to silence. Original pin: #4199, re-affirmed by the #6110 pin audit.

How was this PR tested?

Existing test cases plus the assertions added above, run locally on Python 3.12 in a clean venv built from amber/requirements.txt + amber/dev-requirements.txt with proto bindings generated via bin/python-proto-gen.sh (import fs in that venv raises ModuleNotFoundError, confirming nothing reaches it):

• pytest -m "not integration" → 862 passed, 1 deselected, 1 xfailed. This exercises ExecutorManager directly (test_executor_manager.py) and end to end through UDF loading (pytexera/udf/, handlers/control/test_initialize_executor_handler.py, test_update_executor_handler.py, runnables/test_main_loop.py).
• test_executor_manager.py run 5× consecutively → 23 passed each time, confirming the invalidate_caches fix is stable (the same file fails 3–4 cases per run without it, on main as well).
• ruff check and ruff format --check over src/main/python src/test/python → clean.
• License drift, the same check CI runs: pip-licenses over the runtime closure + ./bin/licensing/check_binary_deps.py --ignore-transitive-version python → OK: 109 Python packages match LICENSE-binary. Running that same package list against main's LICENSE-binary-python reports exactly STALE: appdirs==1.4.4, fs==2.4.16 — i.e. the two removals here are precisely what is required, and nothing else.
• Dependency-closure diff (pip resolution, py3.12, requirements.txt + operator-requirements.txt) between main and this branch: exactly fs==2.4.16 and appdirs==1.4.4 removed, nothing added, six retained, setuptools 80.10.2 → 81.0.0 via torch.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[aglinxinyuan]

Can you check if it's same as #6928?

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Ma77Ball
  You can notify them by mentioning @Ma77Ball in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.96%. Comparing base (49f9e2c) to head (309a28b).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #6946   +/-   ##
=========================================
  Coverage     78.96%   78.96%           
  Complexity     3798     3798           
=========================================
  Files          1161     1161           
  Lines         46084    46085    +1     
  Branches       5110     5110           
=========================================
+ Hits          36388    36389    +1     
  Misses         8076     8076           
  Partials       1620     1620           

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)		Carriedforward from 49f9e2c
agent-service	76.76% <ø> (ø)		Carriedforward from 49f9e2c
amber	72.23% <ø> (ø)		Carriedforward from 49f9e2c
computing-unit-managing-service	20.49% <ø> (ø)		Carriedforward from 49f9e2c
config-service	66.66% <ø> (ø)		Carriedforward from 49f9e2c
file-service	67.21% <ø> (ø)		Carriedforward from 49f9e2c
frontend	82.91% <ø> (ø)		Carriedforward from 49f9e2c
notebook-migration-service	78.94% <ø> (ø)		Carriedforward from 49f9e2c
pyamber	95.38% <100.00%> (+<0.01%)	⬆️
workflow-compiling-service	55.14% <ø> (ø)		Carriedforward from 49f9e2c

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

✅ No material benchmark regressions detected

🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline

Compared against main 49f9e2c benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
⚪	bs=10 sw=10 sl=64	543	0.331	17,682/25,760/25,760 us	⚪ within ±5% / 🔴 +58.7%
🟢	bs=100 sw=10 sl=64	1,197	0.731	84,144/94,151/94,151 us	🟢 -8.5% / 🟢 +22.9%
⚪	bs=1000 sw=10 sl=64	1,419	0.866	708,614/754,598/754,598 us	⚪ within ±5% / 🟢 +41.3%

Baseline details

Latest main 49f9e2c from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	543 tuples/sec	552 tuples/sec	767.9 tuples/sec	-1.6%	-29.3%
bs=10 sw=10 sl=64	MB/s	0.331 MB/s	0.337 MB/s	0.469 MB/s	-1.8%	-29.4%
bs=10 sw=10 sl=64	p50	17,682 us	18,348 us	12,502 us	-3.6%	+41.4%
bs=10 sw=10 sl=64	p95	25,760 us	24,724 us	16,234 us	+4.2%	+58.7%
bs=10 sw=10 sl=64	p99	25,760 us	24,724 us	18,919 us	+4.2%	+36.2%
bs=100 sw=10 sl=64	throughput	1,197 tuples/sec	1,217 tuples/sec	974.8 tuples/sec	-1.6%	+22.8%
bs=100 sw=10 sl=64	MB/s	0.731 MB/s	0.743 MB/s	0.595 MB/s	-1.6%	+22.9%
bs=100 sw=10 sl=64	p50	84,144 us	82,744 us	102,449 us	+1.7%	-17.9%
bs=100 sw=10 sl=64	p95	94,151 us	102,894 us	108,652 us	-8.5%	-13.3%
bs=100 sw=10 sl=64	p99	94,151 us	102,894 us	116,310 us	-8.5%	-19.1%
bs=1000 sw=10 sl=64	throughput	1,419 tuples/sec	1,410 tuples/sec	1,004 tuples/sec	+0.6%	+41.3%
bs=1000 sw=10 sl=64	MB/s	0.866 MB/s	0.86 MB/s	0.613 MB/s	+0.7%	+41.3%
bs=1000 sw=10 sl=64	p50	708,614 us	708,011 us	999,606 us	+0.1%	-29.1%
bs=1000 sw=10 sl=64	p95	754,598 us	752,207 us	1,046,770 us	+0.3%	-27.9%
bs=1000 sw=10 sl=64	p99	754,598 us	752,207 us	1,076,937 us	+0.3%	-29.9%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,368.59,200,128000,543,0.331,17682.42,25759.69,25759.69
1,100,10,64,20,1670.49,2000,1280000,1197,0.731,84144.16,94150.73,94150.73
2,1000,10,64,20,14090.30,20000,12800000,1419,0.866,708614.04,754597.96,754597.96

[Yicong-Huang]

@aicam please mark this as a draft as we are discussing on the solution. Thanks