Skip to content

Complete rewrite of database layer - #2

Open
TheTechRobo wants to merge 8 commits into
masterfrom
new-queue
Open

Complete rewrite of database layer#2
TheTechRobo wants to merge 8 commits into
masterfrom
new-queue

Conversation

@TheTechRobo

@TheTechRobo TheTechRobo commented Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

First commit was from February, geez...

things still to do (off the top of my head):

  • Show depth on page endpoint
  • Add !ig/!unig aliases for reject + skip rules and their corresponding removal
  • Display a relation graph on the dashboard (this one can maybe wait)
  • Go through the code and try to clean up some of the ugly parts (especially the dashboard)
  • Make !df report whether or not all pipelines have at least 3GiB of space
  • Find all the overengineered stuff :-)
    • Probably remove claim locks. UNTIL_FINISHED is covered by job tags (just make a new one that only has the pipeline(s) you want) and I can't really see a use case for INDEFINITELY. The only thing that tags don't cover for UNTIL_FINISHED is being able to fine-tune exactly how many of each pipeline there will be for a job, if concurrency > 2. Given that it's not much code, I'm open to keeping it in there for awhile in case it is considered useful.

@TheTechRobo
Remove JobType enum and column
452a3e8 
Systems other than brozzler will probably be implemented with a
configuration option, not this enum.

Actionless jobs (i.e. JobType.NONE) are a hack. If they are desired,
they can be emulated with a tag that owned by no pipelines, which
avoids tainting the entire system's design with them. (This could
be useful for, e.g., a simple way of storing results from custom JS
over multiple pages, where otherwise you would need to merge the
CUSTOM_JS result for each page.)
@TheTechRobo
Separate job rules into one array for each setting
33be6bc 
This makes the system (at least IMO) much easier to understand.
Having one set of rules that could change four different settings
took a lot more thinking than needed.
@TheTechRobo
Worker version 20260717.01. Fix some bugs and reimplement some dashbo…
42544c0 
…ard helpers.
@TheTechRobo
Remove some old files
623b831
@TheTechRobo
Add support for recursion
ba8f933
@TheTechRobo
Worker version 20260718.01. Report disk space to server. Remove ancho…
b501e99 
…rs from URLs before queuing.
@TheTechRobo
Worker version 20260718.02. Various fixes.
2b12ad4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@TheTechRobo