Skip to content

fix(bigtable): fix session creation leaks - #13887

Merged
mutianf merged 1 commit into
googleapis:mainfrom
mutianf:fix/session-budget-goaway-leak
Jul 27, 2026
Merged

fix(bigtable): fix session creation leaks#13887
mutianf merged 1 commit into
googleapis:mainfrom
mutianf:fix/session-budget-goaway-leak

Conversation

@mutianf

@mutianf mutianf commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Fix budget leaks:

  1. GO_AWAY before open handshake — session goes STARTING → WAIT_SERVER_CLOSE and closes with prevState == WAIT_SERVER_CLOSE, so the release never fired.
  2. Synchronous failure in createSession — factory.createNew()/constructor/metadata-merge throws before any listener is wired, so no terminal callback ever runs to release the slot.
  3. Session wedged in STARTING forever — a stream that connects but never sends OpenSession or a GO_AWAY leaves the session stuck with no terminal callback.

@mutianf
mutianf requested review from a team as code owners July 24, 2026 16:30

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes a budget slot leak in SessionPoolImpl when a session receives a GO_AWAY before completing the open handshake. It introduces a set to track session handles holding a budget reservation, ensuring exactly-once release on success or failure. A test case and fake server support are added to verify this scenario. The feedback suggests using System.nanoTime() instead of System.currentTimeMillis() in the test's polling loop to avoid potential flakiness from system clock adjustments.

Comment on lines +694 to +695
long deadlineMs = System.currentTimeMillis() + 5_000;
while (System.currentTimeMillis() < deadlineMs) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using System.currentTimeMillis() to measure elapsed time or timeouts can make tests flaky if the system clock is adjusted (e.g., via NTP synchronization) during execution. It is safer to use System.nanoTime() for monotonic time measurements.

Suggested change
long deadlineMs = System.currentTimeMillis() + 5_000;
while (System.currentTimeMillis() < deadlineMs) {
long deadlineNs = System.nanoTime() + 5_000_000_000L;
while (System.nanoTime() < deadlineNs) {

@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from 48a2217 to 2e2cd0b Compare July 24, 2026 16:40
@mutianf
mutianf requested review from a team as code owners July 24, 2026 16:40
@snippet-bot

snippet-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from 2e2cd0b to 3a1200c Compare July 24, 2026 21:01
@mutianf mutianf changed the title fix(bigtable): release session-creation budget on GO_AWAY during STAR… fix(bigtable): fix session creation leaks Jul 24, 2026
@mutianf

mutianf commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an open-handshake timeout to prevent sessions from remaining in the STARTING state indefinitely, which would otherwise leak session-creation-budget slots. It also improves budget tracking by associating reservations directly with session handles in a new sessionsHoldingBudget set, ensuring that budget slots are released exactly once during both normal and abnormal session lifecycles (such as receiving a GO_AWAY before the handshake completes or encountering synchronous failures during creation). The review feedback identifies two critical concurrency issues in SessionPoolImpl.java where non-thread-safe collections and budget operations are accessed or modified without acquiring the poolLock, which could lead to data races.

Comment on lines +484 to +487
handle = sessions.newHandle(session);
// Bind the budget reservation made by tryReserveSession() above to this handle so it is
// released exactly once when the session becomes READY or terminates.
sessionsHoldingBudget.add(handle);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The sessions list and sessionsHoldingBudget set are non-thread-safe collections guarded by poolLock. Since createSession runs asynchronously and does not hold poolLock during stream creation, calling sessions.newHandle(session) and sessionsHoldingBudget.add(handle) directly here introduces a data race with other threads that access these collections under poolLock (e.g., in onSessionReady or onSessionClose).

Please wrap these registration steps in a poolLock critical section.

        poolLock.lock();
        try {
          handle = sessions.newHandle(session);
          // Bind the budget reservation made by tryReserveSession() above to this handle so it is
          // released exactly once when the session becomes READY or terminates.
          sessionsHoldingBudget.add(handle);
        } finally {
          poolLock.unlock();
        }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong. createSession is guarded by pool lock.

Comment on lines +521 to +526
if (handle == null || sessionsHoldingBudget.remove(handle)) {
budget.onSessionCreationFailure();
}
if (handle != null) {
sessions.removeUnstartedHandle(handle);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The operations in this catch block modify sessionsHoldingBudget, budget, and sessions (via removeUnstartedHandle), all of which are non-thread-safe resources guarded by poolLock. Since createSession runs asynchronously without holding poolLock, executing these modifications without synchronization introduces a critical data race with other threads (e.g., those executing onSessionClose or onSessionReady).

Please wrap these state updates in a poolLock critical section.

      poolLock.lock();
      try {
        if (handle == null || sessionsHoldingBudget.remove(handle)) {
          budget.onSessionCreationFailure();
        }
        if (handle != null) {
          sessions.removeUnstartedHandle(handle);
        }
      } finally {
        poolLock.unlock();
      }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above. CreateSession is already guarded by poolLock.

@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from 36289ac to af5e173 Compare July 26, 2026 23:58
@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from af5e173 to 80ee204 Compare July 26, 2026 23:59
@mutianf
mutianf enabled auto-merge (squash) July 26, 2026 23:59
@mutianf
mutianf merged commit d586d07 into googleapis:main Jul 27, 2026
212 of 214 checks passed
@mutianf
mutianf deleted the fix/session-budget-goaway-leak branch July 27, 2026 13:40
whowes pushed a commit that referenced this pull request Jul 30, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>1.89.0</summary>

##
[1.89.0](v1.88.0...v1.89.0)
(2026-07-29)


### Features

* **agentidentity:** onboard v1 and v1beta API versions
([#13796](#13796))
([1b1300d](1b1300d))
* **auth:** add JSpecify Null annotations to Auth
([#13842](#13842))
([f6f24d4](f6f24d4))
* **bigquery-jdbc:** add `SSLTrustStoreType` and `SSLTrustStoreProvider`
connection properties
([#13858](#13858))
([9449be1](9449be1))
* **bigquery-jdbc:** add otel trace and span IDs to local logs
([#13935](#13935))
([2801fbd](2801fbd))
* **bigquery-jdbc:** implement BigQueryParameterMetaData and dynamic
type mappings
([#13812](#13812))
([3d67dba](3d67dba))
* **bigquery-jdbc:** implement parameter setters in PreparedStatement
([#13792](#13792))
([94f7404](94f7404))
* **bigquery-jdbc:** Migrate `getImportedKeys` and `getCrossReference`
to BQ API
([#13692](#13692))
([082b046](082b046))
* **bigquery-jdbc:** migrate `getPrimaryKeys` to use BQ API
([#13691](#13691))
([1951f49](1951f49))
* **bigquery-jdbc:** OpenTelemetry integration in BQ JDBC
([#12902](#12902))
([af18f65](af18f65))
* **bigquery-jdbc:** optimize memory footprint for JSON result set
streaming
([#13660](#13660))
([11f26d3](11f26d3))
* **bigquery-jdbc:** standardize parameter handling and calendar
defensive copying across statement interfaces
([#13805](#13805))
([ccd13eb](ccd13eb))
* **bigtable:** add view_parameters support to BoundStatement
([#13673](#13673))
([d5cc437](d5cc437))
* **bigtable:** BigtableDataClientFactory session support
([#13829](#13829))
([284ce13](284ce13))
* **commerceproducer:** onboard v1beta API
([#13814](#13814))
([61b89b3](61b89b3))
* Default to least-in-flight balancing for Bigtable unary clients
([#13802](#13802))
([ac9ccd1](ac9ccd1))
* **firestore:** Add support for 16MB documents
([#13478](#13478))
([1b7c2e0](1b7c2e0))
* **gapic-generator:** add JSpecify Null annotations to the generator
classes
([#13769](#13769))
([843bd7c](843bd7c))
* **gapic-generator:** Add Nullable annotation to generated classes
([#13558](#13558))
([e3e9d0b](e3e9d0b))
* **gapic-generator:** Add NullMarked annotation to generated classes
([#13584](#13584))
([b7a8504](b7a8504))
* **gax-httpjson:** Add Post Quantum Cryptography (PQC) Support by
default via Conscrypt
([#13853](#13853))
([550df81](550df81))
* **gax-java:** add JSpecify Null annotations to gax
([#13799](#13799))
([65aee08](65aee08))
* **google/cloud/sql:** onboard a new library
([#13864](#13864))
([38e272e](38e272e))
* **google/maps/navconnect/v1:** onboard a new library
([#13927](#13927))
([2256394](2256394))
* **maps-isochrones:** onboard v1 API
([#13817](#13817))
([3037ab3](3037ab3))
* port secure_context testing support to executor proxy
([#13522](#13522))
([0f81bf0](0f81bf0))
* **productregistry:** onboard v1 API
([#13816](#13816))
([9517313](9517313))
* **storage:** allow checksum on appendable upload finalization
([#13833](#13833))
([ddf9add](ddf9add))
* **storage:** enable App-Centric Observability (ACO) support in Otel
([#13248](#13248))
([4329896](4329896))


### Bug Fixes

* **bigquery-jdbc:** Add PerConnectionHandler to list of excempted
logging classes
([#13888](#13888))
([50b3c24](50b3c24))
* **bigquery-jdbc:** add preferIPv4Stack to argLine for Kokoro
reliability
([#13923](#13923))
([d5e33d6](d5e33d6))
* **bigquery-jdbc:** add service resource transformer for standalone IT
([#13893](#13893))
([dc80fe8](dc80fe8))
* **bigquery-jdbc:** align metadata methods error handling with spec
([#13793](#13793))
([d85fb10](d85fb10))
* **bigquery-jdbc:** fix WriteAPI when running in restricted environment
([#13856](#13856))
([66ba925](66ba925))
* **bigquery-jdbc:** refine temporal timezone coercion and
PreparedStatement parameter setters
([#13813](#13813))
([6f68c4d](6f68c4d))
* **bigquery-jdbc:** resolve `ITOpenTelemetryTest` pipeline and trace
validation failures
([#13898](#13898))
([c18141d](c18141d))
* **bigquery-jdbc:** resolve failing otel IT in nightly
([#13915](#13915))
([ac79713](ac79713))
* **bigquery:** resultSet.getLong() does not truncate for large int64
values
([#13718](#13718))
([bc19822](bc19822))
* **bigquery:** support optional fields in BigLakeConfiguration to
prevent NPE on Iceberg/Lakehouse tables
([#13733](#13733))
([e2cca4d](e2cca4d))
* **bigtable:** add materialized view routing param to ReadRows and Sa…
([#13918](#13918))
([4ddf250](4ddf250))
* **bigtable:** bound SessionPoolImpl lock to prevent pod-wide wedge
([#13890](#13890))
([ed87a68](ed87a68))
* **bigtable:** fix session creation leaks
([#13887](#13887))
([d586d07](d586d07))
* **bigtable:** prevent ClientConfigurationManagerTest from wedging on…
([#13907](#13907))
([725086d](725086d))
* **bigtable:** stop installing DirectpathEnforcer on the directpath
pool
([#13880](#13880))
([5f2e056](5f2e056))
* **bom:** make release-note-generation Java 8 compatible
([#13837](#13837))
([bc18390](bc18390))
* **ci:** fix java-cloud-bom release-notes workflow errors
([#13682](#13682))
([b679835](b679835))
* deprecate resource detector
([#13844](#13844))
([aba4f01](aba4f01))
* **deps:** align logback versions and add java8 profile in storage
([#13678](#13678))
([7e57092](7e57092))
* do not start stream with direct executor
([#13945](#13945))
([630e790](630e790))
* fix java-cloud-bom README update workflow after monorepo migration
([#13892](#13892))
([5b8e295](5b8e295))
* **oauth2_http:** Avoid retrying on 4xx errors during GCE metadata ping
([#13715](#13715))
([537c16c](537c16c))
* regenerate
([#13714](#13714))
([8a72860](8a72860))
* regenerate libraries
([#13703](#13703))
([a29ea79](a29ea79)),
refs
[#13690](#13690)
* **release:** handle missing release tags gracefully in
release-note-generation
([#13795](#13795))
([43005fb](43005fb))
* **release:** resolve first-party-dependencies SNAPSHOT in
libraries-bom
([#13790](#13790))
([d5bfe21](d5bfe21))
* **spanner:** avoid data race on DIRECTPATH_CHANNEL_CREATED by using
volatile
([#13727](#13727))
([1e05ea5](1e05ea5))
* **spanner:** prevent fastpath tablet routing flaps
([#13803](#13803))
([4dabdac](4dabdac))
* **storage:** BidiAppendableUpload Takeover operation fixes
([#13776](#13776))
([f2d0474](f2d0474))
* **storage:** correctly insert explicit nulls for json patch updates
([#13716](#13716))
([4fb3f4b](4fb3f4b))
* update group id mapping
([#13698](#13698))
([50a72e1](50a72e1))
* use a new managed channel builder when creating channels
([#13684](#13684))
([a049999](a049999))


### Performance Improvements

* **bigquery-jdbc:** optimize getExportedKeys performance using hybrid
metadata lookup
([#13734](#13734))
([c9738ea](c9738ea))


### Dependencies

* Add Conscrypt to shared-deps
([#13838](#13838))
([05ce6ce](05ce6ce))
* move conscrypt from third-party-dependencies POM to gax-java POM
([#13948](#13948))
([1e634ec](1e634ec))
* **shared-deps:** migrate awaitility to shared-dependencies
([#13671](#13671))
([abb91ef](abb91ef))
* **shared-deps:** switch conscrypt shared dependency to
conscrypt-openjdk-uber
([#13845](#13845))
([2e3f208](2e3f208))
* Update gRPC-Java to v1.82.2
([#13877](#13877))
([da228d8](da228d8))
* Update http-client to v2.2.0
([#13854](#13854))
([334a2c5](334a2c5))
* Update Protobuf-Java to v4.33.6
([#13876](#13876))
([5c6478c](5c6478c))
* Upgrade Guava to v33.6.0-jre
([#13875](#13875))
([41a7a52](41a7a52))


### Documentation

* add ErrorProne and NullAway integration guide and JSpecify migration
playbook
([#13882](#13882))
([d5ea739](d5ea739))
* add JSpecify nullness guidelines to AGENTS.md
([#13881](#13881))
([54b846b](54b846b))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants