Skip to content

fix(bin): replace lsof with /dev/tcp in server check_port; remove dea… - #3105

Open
bitflicker64 wants to merge 6 commits into
apache:masterfrom
bitflicker64:fix/lsof-port-check
Open

fix(bin): replace lsof with /dev/tcp in server check_port; remove dea…#3105
bitflicker64 wants to merge 6 commits into
apache:masterfrom
bitflicker64:fix/lsof-port-check

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Purpose of the PR

Fix a startup hang in kind/Kubernetes environments — the case that surfaced while bringing up the Helm chart. In a pod where ulimit -n is very large, lsof -i :PORT walks an enormous file descriptor table and start-hugegraph.sh stalls before the server ever binds.

The fix replaces that lsof call in the server's check_port, and removes the dead check_port copies from PD/Store.

Design

Following the minimal design agreed in review, the preflight answers exactly one question: is anything already listening on this port? It is best effort — the server's own bind remains authoritative.

  1. Parse only enough of the configured URL to obtain a validated port: case-insensitive scheme, explicit or default port, scheme-less value, bracketed IPv6. Ambiguous unbracketed IPv6 is rejected with a warning rather than guessed.

  2. Detect per OS, inspecting only LISTEN rows and only the local-address column:

    • Linux: ss -H -ltn, falling back to netstat -ltn
    • macOS/BSD: netstat -an -p tcp

    Splitting the address on its last separator keeps IPv6 hextets ([2001:db8:8080::1]:9090) from being read as the port.

  3. Report busy, free or unknown. A probe that is missing, fails, or yields no recognisable listener row is unknown: warn and let the bind decide. Only positive evidence blocks startup.

Port-level detection covers the dual-stack and wildcard cases conservatively — matching the original lsof -i :PORT semantics — without reproducing kernel socket behaviour in shell.

This also strengthens the original fix. DNS resolution, address canonicalization, cross-family endpoint matching, /dev/tcp and its watchdog are all gone, so there is no longer any unbounded call left in the startup path — not just no lsof.

Main Changes

hugegraph-server/.../bin/util.shcheck_port rewritten as above (parse_port_from_url + port_listen_state); normalize_addr() and run_with_deadline() deleted; wait_for_startup() bounds each probe with --connect-timeout / --max-time by the time left in the deadline.

hugegraph-pd/.../util.sh, hugegraph-store/.../util.sh — dead check_port removed; command_available inversion fixed; atomic download(); a failed rename no longer reports success, so LD_PRELOAD is never exported for a library that did not install; mktemp instead of a PID-derived temp name.

.github/workflows/pd-store-ci.yml — preflight checked lsof while cleanup used fuser; aligned.

Review items

All items from the last review round are addressed:

# Item Resolution
1 macOS false occupied Fixed — LISTEN rows and local-address column only
2 Linux dual-stack false free Removed — port-level detection is conservative by construction
3 Wildcard false free Removed with the /dev/tcp fallback
4 Unknown treated as free Fixed — unknown is a distinct state that never blocks
5 URL normalization partial Fixed — case-insensitive scheme, scheme-less, ambiguous IPv6 rejected
6 Resolution not bounded Removed — no DNS resolution in this path
7 CI prerequisites disagree Fixed in pd-store-ci.yml
8 No end-to-end no-timeout coverage Removed — no watchdog left to cover
9 Store mv failure swallowed Fixed
10 Startup timeout not hard Fixed
11 PID-based temp names Fixed — mktemp
12 Scope and hygiene process_num / process_id / free_memory / get_ip reverted to master
13 Oversized port wraps into range Fixed — digits bounded before arithmetic conversion

One deviation: netstat is used as the Linux fallback rather than fuser. fuser cannot distinguish "not found" from "insufficient privileges", which would collapse free and unknown into the same result and break the contract.

Size

Converging on the port-only design shrank the change substantially:

Before Now
PR diff +1731 / −189 +870 / −188 vs. master
server util.sh 849 lines 622 lines
test-check-port.sh 984 lines 320 lines

Tests

test-check-port.sh replaced with a compact contract suite: URL-to-port cases, Linux ss busy/free/failure, BSD LISTEN vs ESTABLISHED vs TIME_WAIT, the unknown fallback, one real ephemeral listener (binds port 0 and asserts the child actually bound), rename failure, and a bounded startup probe.

48 passed, 0 failed.

Verified non-vacuous by mutation — each of these makes the suite fail:

  • dropping the LISTEN row filter
  • converting the port with $((10#$port)) before bounding its digit count
  • downgrading unknown to free
  • reading the foreign-address column instead of the local one

The macOS false positive was reproduced before the change and re-checked after: with 37 ESTABLISHED connections to :443 and no listener on 443, check_port "http://0.0.0.0:443" previously exited 1 and now correctly reports the port free.

Known limitations

Converging on the port-only design traded away capability on purpose. Each of these is marked in the code with a greppable TODO(check_port) so it can be picked up later:

Limitation Effect Where
Port-only matching ignores the listener's address A listener on 127.0.0.1:8080 reports the port busy even when the server would bind 192.168.1.5:8080 — refuses a bind that would have succeeded. Fails safe, and matches what lsof -i :PORT did. port_listen_state
Zero LISTEN sockets is indistinguishable from a restricted table Both report unknown, so a genuinely idle host warns on every start port_listen_state
Neither ss nor netstat present No probe remains, so the preflight is permanently unknown and never detects a busy port. A dependency-free fallback needs a bounded connect — which is exactly what was removed here, so adding one back means re-solving the hang this PR fixes. port_listen_state
Unbracketed IPv6 (::1:8080) Ambiguous, so no preflight runs at all for that form parse_port_from_url
Scheme-less value with no port (plain 127.0.0.1) No derivable port, so it is skipped rather than guessed parse_port_from_url
Startup deadline overshoot is bounded, not zero The retry loop still sleeps 2s after a probe before re-reading the clock, so the total can exceed timeout_s by about one interval. Previously it could overrun indefinitely. wait_for_startup
Detection branches are mock-driven On any one runner only that host's branch runs against a real kernel; the real-listener case covers whichever OS the job is on test-check-port.sh

None of these can block a legitimate startup except the first, and that one is the pre-existing lsof behaviour rather than a new regression.

grep -rn "TODO(check_port)\|TODO(wait_for_startup)\|TODO(test-check-port)" \
  hugegraph-server/hugegraph-dist/src/assembly/

Out of scope

Two findings raised on this PR are not addressed here, deliberately:

  • libjemalloc_aarch64.so MD5 mismatch in start-hugegraph-store.sh — the expected_md5 value is pre-existing master content; this PR only added quoting around the download_and_verify arguments. Changing a published checksum belongs in its own change.
  • cluster-test / MultiClusterDeployTest.testServerNodesDeployment (org mirror only) — fails with an identical signature on branches that do not touch this code, and passes on master, so it is not a regression from this PR.

Documentation Status

  • Doc - No Need

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Jul 23, 2026
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 35.95%. Comparing base (c10779d) to head (ea81ad7).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3105      +/-   ##
============================================
- Coverage     37.48%   35.95%   -1.53%     
+ Complexity      520      424      -96     
============================================
  Files           808      770      -38     
  Lines         69594    65772    -3822     
  Branches       9169     8725     -444     
============================================
- Hits          26089    23650    -2439     
+ Misses        40720    39474    -1246     
+ Partials       2785     2648     -137     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bitflicker64

Copy link
Copy Markdown
Contributor Author

@copilot review

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The new probe can miss or stall on real port conflicts, and the corrected command detection exposes a broken curl-only path. Evidence: static review of 6395c0c; bash -n passed; codecov/project is failing on this head.

Comment thread hugegraph-pd/hg-pd-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The fallback probe can execute configuration text as shell code and misparses a supported IPv6 URL form; the no-lsof branch matrix also remains untested. Evidence: exact-head static review, bash -n on all three changed scripts, and controlled reproductions for command execution and [::1]:8080 parsing.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Jul 23, 2026

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The listener-table probes can reject valid binds or silently skip collision detection, and the PD startup test still does not enforce its new lsof-free cleanup path. Evidence: exact-head static inspection, targeted shell reproductions, the 19-test check_port suite, and successful visible checks.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh Outdated
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 24, 2026

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes — 6.0/10 on current head d315ae8. The main direction and CI are good, but the following issues still need to be addressed before merge.

New current-head findings:

Existing comments that remain applicable on d315ae8 (not reposted; +1 used for deduplication):

Please do not treat these linked threads as fully resolved merely because some are outdated or marked resolved against older revisions; the specific residual behaviors above were revalidated on the current head.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/travis/test-check-port.sh Outdated
@bitflicker64

Copy link
Copy Markdown
Contributor Author

fixing remaining issues

@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch 2 times, most recently from 9d75341 to 745a8bb Compare July 24, 2026 20:38
bitflicker64 added a commit to bitflicker64/hugegraph that referenced this pull request Jul 24, 2026
…-test suite

Replace lsof with layered probe (ss → netstat → /dev/tcp) in server
check_port to eliminate startup hang in Kubernetes pods where ulimit -n
can be ~1 billion causing lsof to scan the entire FD table.

Core changes (server util.sh):
- ss -ltn: kernel socket table query, no FD scan
- netstat -ltn / netstat -an: fallback for systems without ss
- bash /dev/tcp with timeout 1: last resort, null-command probe
- Hostname resolution (getent/dscacheutil) for localhost → numeric IP
- Address-family-aware wildcard matching (no cross-family false conflicts)
- ss non-zero exit fallthrough to netstat
- No-timeout watchdog with 2s hard deadline
- URL whitespace stripping (handles ServerOptions edge case)
- IPv6 bracket notation parsing
- Octal-safe port validation via $((10#port))
- curl -fL for HTTP error detection (was curl -L)

PD and Store util.sh:
- Remove dead check_port (never called by PD/Store startup scripts)
- Fix command_available return-value inversion bug
- Fix curl download path from broken concatenation to basename output
- curl -fL for HTTP error detection

CI and cleanup:
- fuser preflight check in server-ci.yml (Linux port cleanup)
- OS-aware cleanup in test-start-hugegraph*.sh (lsof on macOS, fuser on Linux)
- Warning when neither tool available

Tests (test-check-port.sh, 34 tests, 0 failures):
- ss branch: IPv4/IPv6 occupied/free, wildcard, dot-escaping guard
- netstat branch: Linux/BSD formats, IP octet false-positive guard
- /dev/tcp: timeout mock, real ephemeral-port listener with liveness check
- Host/port conflict: cross-family wildcard, specific IP vs loopback
- Tool failure fallthrough: ss→netstat, ss+netstat→/dev/tcp
- Hostname collision: localhost resolution, unresolved fallthrough
- URL normalization + IPv6 hextet guard
- Edge cases: empty URL, port >65535, port 0, non-numeric
- download() curl fallback: PD, Store, failure propagation
- SKIP exit-code handling for setup failures

Closes apache#3105
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch 2 times, most recently from 937624c to 6f87b8f Compare July 24, 2026 22:28

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: This head reverts a merged query-correctness fix and also leaves new URL parsing, resolver timeout, and test-cleanup gaps. Evidence: exact-head static review by 6 independent lanes; bash -n passed; focused shell suite reported 33 pass, 0 fail, 1 environment-limited skip; visible exact-head checks are green.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/travis/test-check-port.sh Outdated
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch 3 times, most recently from 4cf5886 to 8307426 Compare July 25, 2026 06:09
@bitflicker64
bitflicker64 requested a review from imbajin July 25, 2026 06:11
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch from 8307426 to 9fdc941 Compare July 25, 2026 06:28
@bitflicker64

Copy link
Copy Markdown
Contributor Author

Pushed a follow-up addressing the outstanding threads on the current head. Status per open item:

Fixed:

  • Endpoint/dual-stack conflict semantics — kept as-is intentionally. Treating IPv4/IPv6 wildcard binds as a conflict is conservative (fails safe: blocks startup on a legitimate coexisting bind rather than missing a real conflict). Left the existing code comment explaining the tradeoff; happy to make this dual-stack-aware in a follow-up if you would rather not carry the false-positive risk.
  • fuser cleanup remains unenforced — this was already addressed in 8307426: server-ci.yml now gates on fuser for non-Darwin runners in the preflight step, and test-start-hugegraph*.sh warns explicitly when neither fuser nor lsof is available instead of silently no-opping.
  • download() missing directory creation on the wget path — the earlier fix only covered curl (added mkdir -p + basename). Applied the same guard to the wget branch in all three util.sh files (server, PD, store) for consistency.

Documented, not changed:

  • Unbracketed IPv6 (::1:8080 without brackets) would still misparse in check_port authority extraction. Confirmed ServerOptions/HugeConfig only trims whitespace and prefixes a default scheme — it does not enforce bracket notation, so this is not provably unreachable. Added a comment at the parse site documenting the assumption and the failure mode, so it does not need rediscovering. Low blast radius since it only affects the pre-flight check, not the actual bind.

Re-requesting review on the latest commit — let me know if any of the above needs more than a comment.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes — 5.5/10 on current head 9fdc941d.

Why 5.5/10

Only two new inline comments were added because the other blockers already have review threads; the score reflects the whole current implementation, not the number of new comments.

actual port conflict / stalled resolver
                  |
                  v
probe output is missing, ambiguous, or not bind-equivalent
                  |
                  v
check_port returns "free" — or never returns
                  |
                  v
Java bind fails with EADDRINUSE / startup hangs

Expected design

Make every probe return occupied, free, or unknown:

  1. Normalize and validate the configured endpoint once; explicitly reject unsupported ambiguous forms such as unbracketed IPv6.
  2. Parse listener records into address-family/address/port tuples and apply wildcard plus v6only bind semantics.
  3. Treat execution failure and empty/unparseable output as unknown, then continue to a bounded fallback.
  4. Put both resolution and connection attempts under a TERM-to-KILL deadline.
  5. Test the actual production branches and align CI prerequisites with the cleanup tool being used.

Only validated, bind-equivalent evidence should produce free.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/travis/test-check-port.sh Outdated
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch from 9fdc941 to 1f3f01e Compare July 25, 2026 10:58
@bitflicker64

Copy link
Copy Markdown
Contributor Author

All three 5.5/10 items addressed in 1f3f01e:

  1. netstat -an empty output[[ -n "$out" ]] guard added. Empty/diagnostic-only output now falls through to /dev/tcp instead of reporting port_checked=1.

  2. No watchdog test — extracted inline watchdog into run_with_deadline() helper (positional params, no injection). Added deterministic test: run_with_deadline "sleep 5" 2 asserts completion in ≤3s. Proves the watchdog actually kills a stuck child.

  3. EXIT trap cleanup — already addressed in the current head. The trap at line 340 includes kill -9 $PY_PID; wait $PY_PID so the Python listener is reaped even when check_port calls exit 1.

Also fixed a regression from authority parsing: invalid ports like http://127.0.0.1:abc were silently falling through to default port 80 instead of being rejected. Added a guard that rejects authority with exactly one colon and no valid numeric port.

35/35 tests pass. Re-requesting review.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The no-timeout port probe passes its arguments in the wrong order, and its new regression test is invalid; the latest-head server CI is failing. Evidence: exact-head static inspection; a controlled helper invocation printed 8080|2; HugeGraph-Server CI reported 34 passed and 1 failed.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Comment thread hugegraph-server/hugegraph-dist/src/assembly/travis/test-check-port.sh Outdated
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch 5 times, most recently from 5237843 to 0e60673 Compare July 26, 2026 09:50
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jul 26, 2026
…lbacks and harden startup scripts

Replace lsof with layered probe (ss, netstat, /dev/tcp) in server
check_port to eliminate startup hang in Kubernetes pods where ulimit -n
can be ~1 billion causing lsof to scan the entire FD table.

Server util.sh:
- Token-based listener matching with normalize_addr() for IPv6 canonicalization
- IPv6 addresses canonicalized via getent ahosts (Linux) or python3 (macOS/BSD)
- Authority extraction strips ? and # in addition to / and path
- Same-family wildcard matching (IPv4/IPv6 isolated)
- run_with_deadline() helper: deadline-bounded /dev/tcp without timeout
- Atomic download(): temp file + mv on success, rm on failure
- curl -fL with correct -o before -- argument ordering
- wget progress flags in indexed array for set -u / Bash 3.2 safety
- Helper functions scoped with _check_port_ prefix and unset -f cleanup
- netstat -an fallback uses Bash built-in case-insensitive match
- normalize_addr() wraps getent in timeout 2, tr fallback for case
- process_num() returns boolean 0/1, process_id() echoes result
- command_available() uses command -v instead of [[ -x ]]
- get_ip() falls back to loopback on empty parse
- free_memory() Darwin branch guards empty values
- wait_for_startup() separates local/assignment, quotes http_code
- read_property() localizes file_name/property_name
- port_checked=1 set when ss/netstat succeeds and comparison is reliable

PD and Store util.sh:
- Remove dead check_port, fix command_available
- Atomic download() matching server pattern
- wget progress flags in indexed array
- read_property() localizes file_name/property_name

CI and test scripts:
- test-check-port.sh runs as fast gate after Compile
- fuser preflight on Linux, lsof removed
- OS-aware port cleanup (fuser on Linux, safe lsof loop on macOS)
- PD/Store/Server tests include fuser in non-Darwin prereqs
- PD/Store tests call STOP_SCRIPT before manual cleanup
- test-start-hugegraph.sh no longer masks java_home exit status

Tests: test-check-port.sh - 40 mock-driven tests, 0 failures
- timeout mock makes ss/netstat free cases deterministic
- IPv6 canonicalization (compressed vs expanded)
- download tests verify temp file + atomic rename + cleanup
- wget success-path tests for PD and Store
- Bash 3.2-safe array expansion throughout
@bitflicker64
bitflicker64 force-pushed the fix/lsof-port-check branch from 0e60673 to e960cc5 Compare July 26, 2026 09:51
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Jul 26, 2026
@bitflicker64 bitflicker64 reopened this Jul 26, 2026
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Jul 26, 2026
- Localize read_property variables (match PD/Store)
- Use if ! download / if ! tar in ensure_package_exist so atomic
  download failures are detected without a fragile $? check

Also merges upstream master so the branch is no longer diverged
(RISC-V libatomic + CI from apache#3102 kept intact with check_port).
@bitflicker64

Copy link
Copy Markdown
Contributor Author

Status since imbajin’s last review

Addressing the blocking items from the later review rounds (including heads 9fdc941d / 1f3f01e5 / 9b60fbb4). Current tip: 986c95f1.

Port check correctness

  • Fixed: listener matching is endpoint-based (not bare port-only); hostname → numeric resolution before ss/netstat compare
  • Fixed: IPv6 canonicalization via normalize_addr() (compressed vs expanded, brackets, zone scope, IPv4-mapped)
  • Fixed: same-family wildcard conflicts only (IPv4 0.0.0.0 / IPv6 :: isolated)
  • Fixed: tool failure fallthrough — nonzero ss/netstat does not count as “port free”; falls through to next probe
  • Fixed: /dev/tcp bounded with timeout or run_with_deadline() (no open-ended hang)
  • Fixed: run_with_deadline arg order (cmd, deadline, args…) — wrong order that printed 8080|2 is fixed
  • Fixed: /dev/tcp host/port passed as "$1"/"$2" (no shell reparse of config text)

URL / authority parsing

  • Fixed: authority cut at first /, ?, or # (query/fragment no longer poisons port)
  • Fixed: bracket IPv6 forms ([::1]:8080 and scheme-less variants)
  • Fixed: whitespace stripping on URL/host; octal-safe port via $((10#port))

download() / PD–Store

  • Fixed: atomic download (temp + mv on success, rm on failure) — no partial-file poison
  • Fixed: curl -fL, correct -o before --; wget progress as Bash 3.2-safe array
  • Fixed: dead check_port removed from PD/Store; command_available inversion fixed

CI / tests / cleanup

Also

  • process_num / process_id no longer use truncated exit codes for PIDs/counts
  • Darwin free_memory empty-value guards
  • Server read_property / ensure_package_exist polish aligned with PD/Store

Local: bash -n clean; test-check-port.sh40 passed, 0 failed.

Happy to take another look when you have time — thanks for the thorough reviews.

@bitflicker64
bitflicker64 requested a review from imbajin July 26, 2026 10:20

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes — 5.5/10 on current head 986c95f1.

The implementation has improved, but the preflight still has verified false-positive and false-negative behavior. Below is the complete current-head status so resolved threads do not hide residual cases.

Fixed on this head

  • Shell configuration values are passed as positional parameters instead of being reparsed as code.
  • Authority parsing now stops at /, ?, or #; surrounding whitespace and bracketed IPv6 are handled.
  • Hostnames and equivalent IPv6 spellings are normalized before listener comparison.
  • Nonzero ss / netstat execution falls through instead of being treated as “free”.
  • The no-timeout argument-order regression, invalid local declarations, and listener cleanup were fixed.
  • Normal downloads use a temporary file and rename only after successful transfer.
  • The accidental rollback of the merged count-optimization fix was removed.

Still required

  1. ‼️ macOS false occupied: netstat -ltn includes non-listening states, while the parser scans every token in every row. An unrelated ESTABLISHED peer port can block startup. BSD wildcard rows also carry their family in tcp4 / tcp6; discarding that field creates cross-family false positives.
  2. ‼️ Linux dual-stack false free: the dual-stack thread remains applicable. [::]:PORT can also occupy IPv4 when IPV6_V6ONLY=0, but the implementation and test currently assume the families are independent.
  3. ⚠️ Wildcard false free: the /dev/tcp fallback checks loopback only, so a listener bound only to another local interface is missed before a wildcard bind.
  4. ⚠️ Unknown is treated as free: a successful but empty or unparseable listener table can still set port_checked=1; this is reproducible for the ss / Linux-netstat paths.
  5. ⚠️ URL normalization is still partial: the default-port thread still applies to uppercase schemes and scheme-less values accepted by ServerOptions. Ambiguous unbracketed IPv6 should be explicitly rejected rather than guessed.
  6. ⚠️ Resolution is not always bounded: the resolver deadline thread still has a no-timeout path that can run getent synchronously.
  7. ⚠️ CI prerequisites disagree with the implementation: the PD/Store workflow still checks lsof, while Linux cleanup uses fuser.
  8. ⚠️ The production no-timeout path lacks end-to-end coverage: the existing watchdog test tests the helper, not check_port → run_with_deadline with a real occupied/free endpoint.
  9. ⚠️ Verified Store download can report success after installation failure: mv failure is overwritten by return 0.
  10. ⚠️ The configured startup timeout is not a hard deadline: wait_for_startup() calls synchronous curl without --connect-timeout / --max-time, so one blackholed request can exceed the outer timeout indefinitely.
  11. ⚠️ PID-based temporary names are not exclusive: ...tmp.$$ is shared by concurrent background subshells. Current callers are sequential, so either use mktemp as a small fix or explicitly keep concurrency unsupported; no locking framework is needed.
  12. 🧹 Scope and hygiene: unrelated process_num / process_id / memory and shared-helper contract changes have no focused compatibility tests, and test-check-port.sh:319 contains trailing whitespace.

Minimal design for this PR

Keep this as a best-effort startup preflight; the real server bind remains authoritative.

  1. Normalize only enough to obtain a validated port: case-insensitive http / https, explicit/default port, scheme-less value, and bracketed IPv6. Reject ambiguous input with a clear warning.
  2. Detect any LISTEN socket on that port, matching the original conservative behavior:
    • Linux: filtered ss -H -ltn (or fuser fallback).
    • macOS/BSD: netstat -an -p tcp, inspecting only LISTEN rows and the local-address column.
  3. Return busy, free, or unknown. Tool failure and unparseable output are unknown; warn and let the server perform the authoritative bind.
  4. Remove DNS resolution, address canonicalization, cross-family endpoint matching, /dev/tcp, and its watchdog from this path. Port-level detection makes the dual-stack and wildcard cases conservative without reproducing kernel socket semantics in Bash.
  5. In the same PR, check mv, use mktemp, bound the startup curl by the remaining deadline, align CI prerequisites with the selected OS tool, and revert or split unrelated helper refactors.
  6. Replace the broad mock matrix with a compact contract suite: URL-to-port cases; Linux ss busy/free/failure; BSD LISTEN versus ESTABLISHED; unknown fallback; one real ephemeral listener; rename failure; and a blocking startup request.

This handles the current blockers in one PR without adding a new abstraction or expanding support for rare socket configurations.


if [[ $in_use -eq 0 && $port_checked -eq 0 ]] && command_available "netstat"; then
matched_token=0
if out=$(netstat -ltn 2>/dev/null) && echo "$out" | grep -qi "listen"; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

‼️ On macOS, netstat -ltn succeeds but includes non-listening TCP states. This branch only checks whether the complete output contains any LISTEN, then scans every token in every row, so an unrelated ESTABLISHED peer port can be mistaken for a local listener. I reproduced this on macOS: no listener existed on 443 (lsof -iTCP:443 -sTCP:LISTEN was empty), but an outbound connection to :443 made check_port "http://0.0.0.0:443" exit 1. Please select the parser by OS and inspect only the local-address field of LISTEN rows; a conservative port-only comparison is sufficient here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5f4a70d1.

The parser no longer scans every token in every row. Detection is now OS-selected and inspects only LISTEN rows and only the local-address column:

  • macOS/BSD: netstat -an -p tcp, rows filtered on $NF == "LISTEN"
  • Linux: ss -H -ltn (already listen-only), falling back to netstat -ltn

The address is split on its last separator, so an IPv6 hextet such as [2001:db8:8080::1]:9090 can no longer be read as port 8080.

Reproduced your case on macOS before and after. With 37 ESTABLISHED connections to :443 and netstat -an -p tcp | awk '$NF=="LISTEN" && $4 ~ /\.443$/' empty:

before: check_port "http://0.0.0.0:443"  ->  "The port 443 has already been used", exit 1
after : check_port "http://0.0.0.0:443"  ->  exit 0 (free)

Per your note that a conservative port-only comparison is sufficient, I also took the rest of the minimal design: DNS resolution, address canonicalization, cross-family matching, /dev/tcp and its watchdog are removed, and the result is now busy / free / unknown.

Two regressions in the suite cover this specifically: a TIME_WAIT row whose local port is the target must read free, and a row where only the foreign address carries the port must read free. Both fail if the LISTEN filter or the column restriction is removed — I verified that by mutation rather than assuming.

rm -f -- "$tmp"
return 1
fi
mv -f -- "$tmp" "$filepath"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ A failed atomic rename is overwritten by the unconditional return 0 below, so callers can export LD_PRELOAD as if the verified library were installed even when mv failed. Keep this simple: mv -f -- "$tmp" "$filepath" || { rm -f -- "$tmp"; return 1; }, and add one rename-failure test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5f4a70d1, using your suggestion as written:

mv -f -- "$tmp" "$filepath" || { rm -f -- "$tmp"; return 1; }

The unconditional return 0 is gone, so a failed rename can no longer make callers export LD_PRELOAD for a library that never installed.

The same bug was present in download() in all three copies (server, PD, store) — the rename there was also unchecked — so it is fixed in each.

Covered by a rename-failure test that stubs mv to fail and asserts three things: download returns non-zero, no destination file is left behind, and the temp file is cleaned up.

Rework the startup port check to the minimal design agreed in review. The
preflight is best effort - the server's own bind stays authoritative - so it
now answers one question only: is anything already listening on this port?

check_port:
- Parse just enough of the configured URL to obtain a validated port:
  case-insensitive scheme, explicit or default port, scheme-less value,
  bracketed IPv6. Ambiguous unbracketed IPv6 is rejected with a warning
  instead of guessed.
- Detect listeners per OS, inspecting only LISTEN rows and only the
  local-address column: ss -H -ltn on Linux, netstat -an -p tcp on BSD.
  Splitting on the last separator keeps IPv6 hextets from being read as
  the port.
- Return busy, free or unknown. A probe that is missing, fails, or yields
  no recognisable listener row is unknown: warn and let the bind decide.
- Drop DNS resolution, address canonicalisation, cross-family endpoint
  matching, /dev/tcp and its watchdog. Port-level detection covers the
  dual-stack and wildcard cases conservatively without reproducing kernel
  socket semantics in shell, and removes the last unbounded call from the
  startup path.

This fixes the macOS false positive where netstat -ltn returned
non-listening rows and an unrelated ESTABLISHED peer port aborted startup.

Also in this pass:
- store: a failed atomic rename no longer reports success, so LD_PRELOAD is
  never exported for a library that did not install. Same fix applied to
  download() in all three copies.
- download(): mktemp instead of a PID-derived name, which is shared by
  concurrent background subshells.
- wait_for_startup(): bound each probe with --connect-timeout / --max-time
  by the time left in the deadline, so one blackholed request cannot
  outlive the configured timeout.
- pd-store-ci: preflight checked lsof while cleanup used fuser; align it.
- Revert unrelated process_num / process_id / free_memory / get_ip changes.

netstat is used as the Linux fallback rather than fuser: fuser cannot
distinguish "not found" from "insufficient privileges", which would break
the busy/free/unknown contract.

Tests: test-check-port.sh replaced with a compact contract suite - URL to
port, ss busy/free/failure, BSD LISTEN vs ESTABLISHED vs TIME_WAIT, unknown
fallback, a real ephemeral listener, rename failure, and a bounded startup
probe. 41 passed, 0 failed. Verified non-vacuous by mutation: dropping the
LISTEN filter, downgrading unknown to free, and reading the foreign-address
column each fail the suite.
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Jul 27, 2026
Converging on port-only detection removed capability on purpose. Mark each
gap in place with a namespaced TODO so it can be found and picked up later
rather than rediscovered:

- port-only matching ignores the listener's address, so a listener on one
  local address reports the port busy even when the server would bind a
  different one. This is the old `lsof -i :PORT` behaviour and fails safe,
  but it can refuse a bind that would have succeeded.
- a host with genuinely zero LISTEN sockets is indistinguishable from a
  restricted or unparseable table; both report unknown and warn.
- with neither ss nor netstat present there is no probe left, so the check
  is permanently unknown. A dependency-free fallback needs a bounded
  connect, which is exactly what was removed here.
- unbracketed IPv6 and a scheme-less value with no port are skipped rather
  than guessed.
- wait_for_startup overshoot is now bounded but not zero: the loop still
  sleeps between probes before re-reading the clock.
- the Linux and BSD detection branches are mock-driven, so only the host's
  own branch runs against a real kernel on any one runner.

Also fix a real parsing defect found while auditing those gaps: a URL
carrying userinfo (http://user:pass@host:port) was matched by the
unbracketed-IPv6 check, so it skipped the preflight and printed a
misleading warning about bracket notation. Strip userinfo from the
authority before that test, and cover the userinfo forms in the suite.

Suite: 44 passed, 0 failed.
@bitflicker64

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed design in the last review — I took it in full rather than patching around the edges. Current head: ea81ad76.

What changed

check_port is rewritten to the minimal design. It now answers one question — is anything already listening on this port? — and reports busy, free or unknown, with the bind left authoritative.

  • Parse: case-insensitive scheme, explicit/default port, scheme-less value, bracketed IPv6. Ambiguous unbracketed IPv6 is rejected with a warning instead of guessed.
  • Detect: ss -H -ltn on Linux (falling back to netstat -ltn), netstat -an -p tcp on BSD — only LISTEN rows, only the local-address column, address split on its last separator.
  • Removed: DNS resolution, address canonicalization, cross-family matching, /dev/tcp and its watchdog. normalize_addr() and run_with_deadline() are gone entirely.

Worth noting this strengthens the original goal rather than only preserving it: there is no longer any unbounded call left in the startup path, not merely no lsof.

All 12 items from your list are addressed — the resolution table is in the PR description. Items 2, 3, 6 and 8 are resolved by removal rather than by more shell.

One deviation

You suggested fuser as the Linux fallback; I used netstat instead. fuser cannot distinguish "not found" from "insufficient privileges", so it would collapse free and unknown into one result and break the tri-state contract. Happy to switch if you would rather have fuser there.

Size

Before Now
PR vs master +1731 / −189 +859 / −188
server util.sh 849 617
test-check-port.sh 984 320

The convergence commit itself is +426 / −1297.

What this gives up

Port-only detection trades away real capability, so rather than leave that implicit I marked each gap in place with a namespaced TODO:

  • Port-only matching ignores the listener's address, so a listener on 127.0.0.1:8080 reports the port busy even when the server would bind 192.168.1.5:8080. That is the old lsof -i :PORT behaviour and fails safe, but it can refuse a bind that would have succeeded — it is the one limitation here that can block a legitimate start.
  • Zero LISTEN sockets is indistinguishable from a restricted table; both report unknown and warn.
  • With neither ss nor netstat present there is no probe left. A dependency-free fallback would need a bounded connect, which is exactly what was removed — so that one is not a simple follow-up.

grep -rn "TODO(check_port)" finds them; the full table is in the description.

Tests

test-check-port.sh is now a compact contract suite (984 → 320 lines): URL-to-port, ss busy/free/failure, BSD LISTEN vs ESTABLISHED vs TIME_WAIT, the unknown fallback, one real ephemeral listener that asserts the child actually bound, rename failure, and a bounded startup probe. 44 passed, 0 failed.

I checked it is not vacuous by mutating the implementation — dropping the LISTEN filter, downgrading unknown to free, and reading the foreign-address column each fail the suite. The first mutation initially passed, which is how the TIME_WAIT case got added.

Your macOS repro, before and after on this head:

37 ESTABLISHED to :443, no listener on 443
before: check_port "http://0.0.0.0:443"  ->  "The port 443 has already been used", exit 1
after :                                  ->  exit 0 (free)

Auditing those gaps also turned up a defect worth mentioning: a URL with userinfo (http://user:pass@host:port) was caught by the unbracketed-IPv6 check, so it skipped the preflight and printed a misleading warning about bracket notation. Fixed and covered.

Out of scope

Two findings are deliberately not addressed, with reasons on their threads: the libjemalloc_aarch64.so MD5 (pre-existing master value; this PR only added quoting) and the mirror-only cluster-test failure (identical signature on branches that do not touch this code).

@bitflicker64
bitflicker64 requested a review from imbajin July 27, 2026 15:01

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Oversized numeric ports can wrap into valid values before range validation. Evidence: exact-head static review and a controlled reproduction where 18446744073709551617 is accepted as port 1; the 44-case shell suite otherwise passes.


[[ "$port" =~ ^[0-9]+$ ]] || return 1
# Normalise leading-zero forms; Java reads 08080 as decimal 8080.
port=$((10#$port))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ parse_port_from_url() converts the digit string before bounding its size, so Bash integer overflow can turn an invalid configured port into a valid one. On this exact head, parse_port_from_url "http://127.0.0.1:18446744073709551617" prints 1 and returns 0 because $((10#$port)) wraps before the 1..65535 check; the 44-case suite only covers 70000. Please strip leading zeroes and reject values longer or lexically larger than 65535 before arithmetic conversion, then add an oversized-integer regression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in a413325b. Your repro is correct, and the case is a bit wider than the value you found.

18446744073709551617 wraps to 1, but the worse shape is a value that wraps into a plausible port:

before: parse_port_from_url "http://127.0.0.1:18446744073709559616"  ->  8000
after : rc=1 (skipped)

That one mattered more than the wrap to 1: the preflight would have aborted startup with "The port 8000 has already been used", naming a port the operator never configured, for what is really a malformed value.

The fix normalises the leading-zero form textually and bounds the digit count before any arithmetic:

port="${port#"${port%%[!0]*}"}"
[[ -z "$port" ]] && return 1
(( ${#port} <= 5 )) || return 1
(( port >= 1 && port <= 65535 )) || return 1

Leading zeroes are still accepted, since new URI("http://127.0.0.1:0000000000000008080").getPort() returns 8080 — I checked rather than assumed. 00000 returns 0 there and is skipped.

Four cases added to the URL-parsing table: both wrapping values, 00000, and the long leading-zero form. The suite is 48 passed, 0 failed, and restoring port=$((10#$port)) fails exactly the two overflow cases, so they are not vacuous.

parse_port_from_url() converted the digit string with $((10#$port))
before bounding it. Bash evaluates in 64 bits and wraps silently, so an
out-of-range value could re-enter 1..65535 and be accepted as a real
port: 18446744073709551617 parsed as 1, and 18446744073709559616 as
8000. The preflight would then abort startup naming a port that was
never configured.

Normalise the leading-zero form textually instead, reject anything
longer than five digits, and only then convert and range-check. Java
reads 08080 as 8080, so that form is still accepted.

Covers both wrapping values, an all-zero port, and a long leading-zero
form in the URL-parsing table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants