Change Python's InternalQueue to use LinkedBlockingMultiQueue - #1725
Merged
Conversation
Yicong-Huang
changed the title
zuozhiw
approved these changes
Nov 29, 2022
zuozhiw
left a comment
zuozhiw
left a comment
Contributor
There was a problem hiding this comment.
did a side-by-side comparison with the original java version and this implementation looks very good, left some minor changes,
yangzhang75
pushed a commit
to yangzhang75/texera
that referenced
this pull request
Jun 22, 2026
…#1725) This PR updates the Python engine's `InternalQueue` implementation from `DoubleBlockingQueue` to `LinkedBlockingMultiQueue`. ## Motivation A major reason for the change is that we need to satisfy a few more requirements: 1. multiple consumers (including those accessing size of the queue), for CP+DP separation and flow control. 2. multiple sub-queues, for chaining operator's in-operator tuples. ## Implementation The `LinkedBlockingMultiQueue` is mimicking from Java's version [marianobarrios/linked-blocking-multi-queue](https://github.com/marianobarrios/linked-blocking-multi-queue/blob/master/src/main/java/lbmq/LinkedBlockingMultiQueue.java) (at commit [9209d8](marianobarrios/linked-blocking-multi-queue@9209d83)), with some modifications: 1. For locks, since Python does not have an equivalent `lockInterruptibly()` method, we are using `Lock.acquire()`. 2. Since we do not need capacities on queues, the capacity-related concept is intentionally not implemented.
This was referenced Jul 29, 2026
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Jul 30, 2026
…che#7034) ### What changes were proposed in this PR? Deletes `amber/src/main/python/core/util/customized_queue/double_blocking_queue.py`, which has contained no code since 2022. All 16 remaining lines are the ASF license header. | When | What | | --- | --- | | 2021-07-23 | `DoubleBlockingQueue` added by apache#1262 | | 2022-11-29 | apache#1725 switched `InternalQueue` to `LinkedBlockingMultiQueue` and emptied the class — but left the file behind | | 2025-05-03 | apache#3415 stamped an ASF license header onto the now-empty file | Nothing imports it. The package barrel `customized_queue/__init__.py` exports only `LinkedBlockingMultiQueue` and `IQueue`, so no `__init__` change is needed. −16 lines, no behaviour change. ### Any related issues, documentation, discussions? Closes apache#7033 ### How was this PR tested? Existing tests only — there is nothing to add or remove, since the file has no code and no spec. From `amber/`: - `python -m pytest src/test/python/core/util/customized_queue/` — 54 passed. - `python -m ruff check src/main/python/core/util/customized_queue/` — all checks passed. - `python -m ruff format --check src/main/python/core/util/customized_queue/` — 4 files already formatted. Verification that the file is empty and unreferenced, re-runnable by a reviewer: ``` grep -vc '^\s*#\|^\s*$' amber/src/main/python/core/util/customized_queue/double_blocking_queue.py grep -rn "double_blocking_queue\|DoubleBlockingQueue" . --include=*.py --include=*.scala --include=*.txt --include=*.toml ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates the Python engine's
InternalQueueimplementation fromDoubleBlockingQueuetoLinkedBlockingMultiQueue.Motivation
A major reason for the change is that we need to satisfy a few more requirements:
Implementation
The
LinkedBlockingMultiQueueis mimicking from Java's version marianobarrios/linked-blocking-multi-queue (at commit 9209d8), with some modifications:lockInterruptibly()method, we are usingLock.acquire().