Exploring Solutions to Tackle Low-Quality Contributions on GitHub

[moraesc]

Hey everyone,

I wanted to provide an update on a critical issue affecting the open source community: the increasing volume of low-quality contributions that is creating significant operational challenges for maintainers.

We’ve been hearing from you that you’re dedicating substantial time to reviewing contributions that do not meet project quality standards for a number of reasons - they fail to follow project guidelines, are frequently abandoned shortly after submission, and are often AI-generated. As AI continues to reshape software development workflows and the nature of open source collaboration, I want you to know that we are actively investigating this problem and developing both immediate and longer-term strategic solutions.

What we're exploring

We’ve spent time reviewing feedback from community members, working directly with maintainers to explore various solutions, and looking through open source repositories to understand the nature of these contributions. Below is an overview of the solutions we’re currently evaluating.

Short-term solutions:

• Configurable pull request permissions - This has been a long-standing request (since 2016) and is a first step in giving maintainers more tools to customize their PR experience. Repository owners will be able to control pull request access at a more granular level by restricting contributions to collaborators only or disabling PRs for specific use cases like mirror repositories. This also eliminates the need for custom automations that various open source projects are building to manage contributions.
• The ability to delete a PR from the UI - This provides maintainers with the ability to remove spam or low-quality PRs directly from the interface to improve repository organization.

Long-term direction:

As AI adoption accelerates, we recognize the need to proactively address how it can potentially transform both contributor and maintainer workflows. We are exploring:

• Enhanced permission models - Provide maintainers with more tools so they can customize their PR experience:
  • More granular controls for determining who can create and reviewing PRs beyond blocking all users or restricting to collaborators only.
  • Ways for maintainers to define criteria that PRs must meet before they can be opened.
• Improved triage tools - Potentially leveraging AI to evaluate contributions against project guidelines and standards to identify which contributions maintainers should focus on reviewing.
• Transparency in AI-assisted contributions - Improved visibility and attribution when AI tools are used throughout the PR lifecycle.

Next Steps

These are some starting points, and we’re continuing to explore both immediate improvements and long-term solutions. Please share your feedback, questions, or concerns in this thread. Your input is crucial to making sure we’re building the right things and tackling this challenge effectively. As always, thank you for being part of this conversation. Looking forward to hearing your thoughts and working together to address this problem.

[xavidop]

such a great intiative

[patrickhlauke]

explore solutions to tackle this sort of low-effort spammy crap as well, maybe...

[PaulNewton]

@patrickhlauke I'd also like to have a way to quickly block this type nonsense without having to build userscripts.
For real this is like "who's here in 2026" engagement bait and it should just be qualify as spam not upvoted over everything.
But the people upvoting are part of why so much spam exists too and i'd like to be able to block them all.

[xavidop]

I know this is a pretty ambitious idea and not trivial to implement, but it would be really powerful to have an AI-detection mechanism with a configurable threshold at the repository or organization level. That way, teams could decide what percentage of AI-generated code is acceptable in pull requests.

Another possible approach would be to define a set of rules or prompts and evaluate pull requests against them. PRs that don’t meet those rules could be automatically flagged or potentially even closed.

[moraesc]

Another possible approach would be to define a set of rules or prompts and evaluate pull requests against them. PRs that don’t meet those rules could be automatically flagged or potentially even closed.

This is definitely something we’re exploring. One idea is to leverage a repository’s CONTRIBUTING.md file as a source of truth for project guidelines and then validate PRs against any defined rules.

In regards to AI-generated code, have you seen cases where the code is AI-generated but still high-quality and actually solves the problem? Or is it always just something you want to close out immediately? I'm curious if an AI-detection mechanism would rule out PRs where AI is used constructively, but interested in investigating this more and understanding what sensible would thresholds look like.

[PaulNewton]

"AI-detection" is a backwards approach to a human problem filled with false positives.
Any system good enough to actually identify AI is a system that can be used to train undetectable AI and that would encourage MORE spam.
QED

The only actual test that ends up mattering is whether the code works and the contributor is allowed to contribute that code.
And that is the problem, code that doesn't work wasting humans time on slop.
The goal is less AI noise not more insisting on itself at every point in the maintainers life.

[who]

Judge work on its merits and its logic, no matter if the source is human, AI, or a combination thereof.

[xavidop]

As of today, I would say that 1 out of 10 PRs created with AI is legitimate and meets the standards required to open that PR. On 28 Jan 2026, at 18:41, Camilla Moraes ***@***.***> wrote:  Another possible approach would be to define a set of rules or prompts and evaluate pull requests against them. PRs that don’t meet those rules could be automatically flagged or potentially even closed. This is definitely something we’re exploring. One idea is to leverage a repository’s CONTRIBUTING.md file as a source of truth for project guidelines and then validate PRs against any defined rules. In regards to AI-generated code, have you seen cases where the code is AI-generated but still high-quality and genuinely solves the problem? Or is it alwaays just something you want to close out immediately? I'm curious because I'm wondering if an AI-detection mechanism would rule out PRs where AI is used constructively, but that's where we'd want to test this thoroughly and understand what sensible thresholds look like. — Reply to this email directly, view it on GitHub<#185387 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABBWEYEKF6WLNDKE376L3GD4JDYFXAVCNFSM6AAAAACS7B7C7OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNRTGEZTMMI>. You are receiving this because you commented.Message ID: ***@***.***>

[eli-schwartz]

5. Produces a hash-chained evidence bundle - cryptographic proof of what was reviewed, when, and what the models found. Independently verifiable offline.

I think that for extra security you should make sure that this uses the blockchain distributed consensus model. Otherwise how do we know which proofs to trust?

Eager to hear the name of your new cryptocurrency.

[sirosen]

I don't think it's a scam. I think it's a silly.

You are wading into a discussion which is primarily about harm being done by vibe coding.¹ So, uh... pushing a vibe coded project here is super ironic.

Footnotes

1. In the El Reg article about this discussion, a GitHub manager said that this isn't about LLMs, which is so patently dishonest that it's offensive. ↩

[bxb100]

Damn, MF should hire you as their PM.

[tsjensen]

Deleting a PR would be a great feature, I've looked for this since before AI

[rmacklin]

@moraesc As the OP alludes to it's a legitimate flaw that all prior public issues become inaccessible if a public repository decides to disable issues going forward. Are you open to making changes to this? This discussion has been about a potential upcoming feature to disable PRs for a repository, but I want to explicitly ask if GitHub is open to considering improving the behavior for the existing disable-issues feature, too.

[Mossaka]

Hey! I am from Azure Core Upstream and we have a lot of OSS maintainers who mainly maintain repositories on GitHub. We held an internal session to talk about copilot and there is a discussion on the topic where maintainers feel caught between today’s required review rigor (line-by-line understanding for anything shipped) and a future where agentic / AI-generated code makes that model increasingly unsustainable.

below are some key maintainer's pain points:

1. Review trust model is broken: reviewers can no longer assume authors understand or wrote the code they submit.
2. AI-generated PRs can look structurally “fine” but be logically wrong, unsafe, or interact with systems the reviewer doesn’t fully know.
3. Line-by-line review is still mandatory for shipped code, but does not scale with large AI-assisted or agentic PRs.
4. Maintainers are uncomfortable approving PRs they don’t fully understand, yet AI makes it easy to submit large changes without deep understanding.
5. Increased cognitive load: reviewers must now evaluate both the code and whether the author understands it.
6. Review burden is higher than pre-AI, not lower.

[moraesc]

Thanks for sharing @Mossaka ! Great to get more feedback. This looks along the lines of what we've been hearing from maintainers too, so validates what some of the key pain points are. Were there any feature requests mentioned during the session?

[Tibor17]

1. I have a problem with the following statement. It is very leftish. The GitHub has no rights to impose such limitations. You have to configure your repo and TTL of a PR:
   limited timeframe to delete a PR. Maybe a week in case of low activity

2. Regarding the next statement, you can delete a PR when you close it. How different is close and delete?
   Next question, why you mentioned low-quality PRs? I hope you do not want to employ AI to determine what low quality means! We, maintainers, have to determine it.
   The ability to delete a PR from the UI - This provides maintainers with the ability to remove spam or low-quality PRs directly from the interface to improve repository organization.

3. Who is an external contributor?
   Example, you mean a situation when XLibre developers are removed from Xorg?
   Restrict PRs to collaborators - This provides more granular access control, allowing contributions exclusively from existing collaborators while blocking external contributors.

4. This would segregate the maintainers and we don't want it. We have flags on the PRs to indicate what to do with them.
   If the AI is leveraged to this job, we have segregated the maintainers again in the Open Source. But the rule is that we are all equal, and this might be a problem if one is notified and the second guy is not. You may not totally trust the AI due to finally the result may change without the second guy or the quality would be bad without his opinion since he does not attend the review.
   Improved triage tools - Potentially leveraging AI to evaluate contributions against project guidelines and standards to identify which contributions maintainers should focus on reviewing.

[AlanGreene]

Many of the badges can easily be gamed. I've seen plenty of users who've followed some blog post / YouTube video instructing them on how to gain badges in an automated manner through a provided script and pool of user / bot accounts.

[jamisonbryant]

Wow, that sucks!! I had no idea. I have never really chased the badges, just cracked a half-smile whenever I earn one from going about my day-to-day, but IMO they should be harder to obtain than this, especially with botting!!

[eli-schwartz]

Um...last I checked, labeling, assigning, and closing PRs are "write access" permissions. GitHub repo permissions are already very granular, in the places where such things are implemented. I don't think the proposal to include PRs in this is as big a lift as you're thinking it is.

They certainly are not, and I know this because I'm a member of an org where all org members have triage permissions, and only a dedicated mirroring bot has write permissions to push new commits to the repository. All org members must push to our in-house gitolite repository instead, and the reason for only handing out triage roles on GitHub is to prevent anyone from accidentally pushing to the wrong remote and causing the two to diverge (which would mean that GitHub gets force pushed, which would be disruptive to users).

Issues are disabled because we use bugzilla. PRs are allowed for the sake of users who don't wish to email patches to a mailing list.

I can:

• add various labels to PRs, such as the labels that cause our in-house CI system to bump a PR to the top of the queue.
• add assignees,
• add requests for reviewers,
• manage milestones (even if that particular repo doesn't use milestones),
• close them

I cannot:

• edit the PR title,
• edit or delete comments,
• approve workflows,
• mark my own review comment as needing action after the PR author responded to my review by saying "no" and clicking the "resolve" button (in my not so humble opinion, this is a grave usability flaw in GitHub PRs -- the author of a PR must not be permitted to win in a governance dispute with a project member, nor hide reviews by default in the hope that a different maintainer will not notice that the first maintainer has outstanding concerns)

The permission system in question is a toggle that allows administrators to select whether a user has read-only access, "triage", "write", or admin. I am entirely confused, what you seem to be calling triage that isn't what I just described. It is also funny to hear you describing permissions as ,"very granular" when it is a straight line going up at an angle and each level simply adds additional permissions on top of lower levels -- are you seriously suggesting that there's a GitHub repository role that "granularly" permits users to, for example, merge PRs but not add an assignee or a milestone?

[eli-schwartz]

Wow, that sucks!! I had no idea. I have never really chased the badges, just cracked a half-smile whenever I earn one from going about my day-to-day, but IMO they should be harder to obtain than this, especially with botting!!

Badges are something that GitHub implemented to copy Xbox games, are you really that surprised that they are vulnerable to cheating? :)

The literal definition of those badges is that you have e.g. "got X number of PRs merged", but you can get them by creating your own personal repo and submitting PRs to it, last I checked.

A badge that perhaps means a bit more is the one-off badges that you can get if you have had PRs merged by repositories whose code is in particular NASA missions, as you can't game that without knowing it will exist in advance, and the list of repositories whose code is used in some well known government-funded event is unlikely to include outright script-generated cheats.

[chadlwilson]

I think the message was probably ineloquently phrased and comes from a misunderstanding of the term "write access [to code]", but the intent was probably to say that there's no particular reason that GitHub can't consider someone with a triage role also able to submit PRs even if disallowed from fully external folks?

But the triage role certainly is a mess and unsuitable for actually doing triage right now (not being able to edit titles or descriptions is crazy) and very far from granular for public free orgs (as opposed to enterprise orgs, I believe). That shouldn't have been a heavy lift to resolve either, but it has not been 😅

[sirosen]

An option to limit new contributors to one open PR would be nice.

Just today I had to batch-close several AI generated PRs which were all submitted around the same time.

For this protection, defining "new contributor" is probably not possible to do perfectly. But anyone who has no interactions with a project prior to the last 48 hours seems like a good heuristic. The point is to catch such a user at submission time and limit the amount of maintainer attention they can take up.

For a different type of problem, I'd like to be able to close PRs as "abandoned", similar to the issue close statuses. It's a clear UI signal to the contributor that their work isn't being rejected but I'm not going to finish it for them. Several of the low quality contributions I have handled, dating back to before the Slop Era but getting worse, are simply incomplete and need follow through.

[who]

Giving repo owners the ability to "rate-limit" PRs from contributors is probably a good idea.

You can automate this today using on: pull_request_target GitHub Actions.

[webknjaz]

@sirosen look what I've just found: https://github.com/marketplace/actions/good-egg-trust-scoring-prs. Not sure it'd score me very high (saying hi to corner cases 😃) but I'm seriously considering trying it out. The methodology there reminds me of what my ML friend @LukaIhn was suggesting in our chats strategy-wise.

[sirosen]

A lot of the solutions people have raised here are some form of "reputation system", which I find a bit uncomfortable because I think it will be wrong a lot of the time. And any rep system results in some people karma-farming.

I'm definitely open to the idea that we might incorporate some tools which give us signals about contributors. e.g., A comment bot that simply notes "warning signs" without forcibly closing would let us continue to exercise our judgement.

---

But before we discuss more here...

I have unsubscribed from this discussion and I recommend/urge anyone reading to not bother reading most of it, and potentially to unsubscribe as well. It started out somewhat-promising, but then the trolls came out to play.

The more limited maintainers community or per-project discussions are more usable, as there's less spam/junk.

[timrichardson]

We learn in physics that friction is good. AI has made PRs almost zero friction. Any solution means adding friction back. Thanks for the link to the "good egg" project. Very Bertie Wooster.

[projekta2]

I'm definitely open to the idea that we might incorporate some tools
which give us signals about contributors.

The distinction you're drawing between "signal" and "verdict" is the
right one. A tool that surfaces warning signs without forcing a decision
keeps the maintainer in control — which matters a lot when the signal
is probabilistic, not deterministic.

The contributor-level signals you mentioned are useful, but I've found
PR-level signals more actionable for daily triage. Contributor history
tells you something about intent; PR signals tell you something about
the specific cost of reviewing this PR today.

The two that have surprised me most in practice:

Author engagement velocity: if a contributor opens a PR and doesn't
respond to any automated checks or comments within 48–72 hours, the
review cost goes up significantly — you're now doing the debugging for
them. Surfacing this early means you can reorder your queue before
you've invested time.

Scope/size mismatch: a PR that touches 12 files for a "minor fix"
is a flag regardless of who submitted it. Human or AI, the description
is wrong, and that means either the author doesn't understand the
change or they do and didn't explain it. Either way, it needs more
attention, not less.

Neither of these requires AI to compute. They're just signals that
get lost in a flat list view.

[dgoldman0]

For the long term horizon: Implement a reviewer LLM that first does an initial scoring of the PRs? Critique is far easier than creation of a correct result. That automated pre-moderation should give the edge needed to handle. Depending on whether you just use rich prompting or fine-tuning, you can even start building an "oracle vox" for your project, which acts as a reasonably informed, reasonably on point virtual representative for the project/organization.

[chadlwilson]

Yes, I understand and broadly agree with your sentiment - but without going into a long sidetrack into the ethics (and irony) of coding-LLMs actively undermining the very thing that even made them possible in the first place (open source code and contributors) ...I wanted to add a different voice with perhaps some more nuance,

In the sense that different tools can be applied to different jobs with different trade offs. Don't need a Ferrari to get to the grocery store, and all that - and if the tool can bring time back for actually fixing bugs or improving the product, working with well meaning contributors - I'm not as fussed with whether it incorporates AI or not, just as whether I don't mind much whether a tool is built in Python or C or Ruby if it works well relative to its costs/downsides.

[eli-schwartz]

Essentially heuristics defined in clear text rather than a classic deterministic scoring system. I don't think that would be that bad (and perhaps such things already exist) - and neither more or less impersonal as the bots in some OSS projects that nag you to sign CLAs; sign off commits; tell you when a merge conflict has appeared, close stale issues, respond and label based on PR templates /areas of code touched etc.

I will say, that I'm extremely opposed to bots in OSS projects that ''close stale issues" and I've never once seen it work out to the mutual satisfaction of both the maintainers and the community. Sometimes it doesn't even work out to the satisfaction of a single side -- e.g. a single developer forces through the bot and angers other developers.

CLA bots and signoff bots aren't heuristics based, any more than something like os.path.exists() uses heuristics to determine whether a file exists. People are attuned to treating such bots as expressions of inarguable fact.

I haven't actually seen bots that tell you when a merge conflict has occurred, although it's much the same (and effectively just serves as a notification pipeline for when the forge internal flag for displaying a merge conflict in the Web UI, changes status). It sounds incredibly painfully noisy and I would object to such a bot in projects I contribute to, but not because I think it's "impersonal", I just think it's far too prone to generating hundreds of new comments that anyone seeking to read the PR discussion will have to scroll through. Generating a new comment for a frequently changing status flag is not an effective communication channel; better would be a forge native feature to email the author when a merge conflict occurs, and then people can also see the summary at the bottom...

respond and label based on PR templates /areas of code touched

Labeling based on PR templates doesn't require a bot, last I checked. GitHub natively lets you associate templates with labels? Labeling based on paths modified in the diff is not really something I'd call a "heuristic" given it is a simple key/value mapping from paths to labels, manually defined in some script that is then reproducible and idempotent.

So none of this is really anything that I feel would map well to give a hint about how users might respond to a heuristics based LLM review tool.

Especially keep in mind that people who don't like LLMs (such as me) will absolutely flip out any time such a tool erroneously gives them a failing grade and "penalizes" them. And a review tool that doesn't do anything to hide PRs with a failing grade from developer attention, does not accomplish anything to solve the problem of developers being forced to expend their attention on what they consider spam.

and if all comments added have an appropriate disclaimer/disclosure and ability for a contributor to "get help" with a bit of friction if it goes awry - that could perhaps be a useful piece of the toolkit?

A disclaimer/disclosure will do nothing because the victim knows that the project chose to impersonally:

• act without knowledge,
• use the tool that they said is bad, to leap to the incorrect conclusion that the victim is using the exact same tool and penalize them in some manner for using it

An appeal button will not help because people have a natural aversion to challenging authority and this goes quadruple for cases where a first time contributor feels slighted and "on the defensive" from the initial interaction and doesn't have some motivation to get the change in "at all costs and no matter what I have to swallow to do it" (such as a critical work-blocking bug that cannot be worked around in a dependency that cannot be replaced or reimplemented).

[marcindulak]

I will say, that I'm extremely opposed to bots in OSS projects that ''close stale issues" and I've never once seen it work out to the mutual satisfaction of both the maintainers and the community.

There is currently at least one running example of this problem anthropics/claude-code#16497

[chadlwilson]

Yes, I am mainly referring to heuristics where a bot (AI or otherwise) tries to use various aspects of a contribution and the contributor's history to assess signal vs noise for a contribution - not existing deterministic bots doing simple automation, or "rules" for contributions.

I'm aware templates can be used for labels, but some projects use heuristic based bots for this (e.g grpc, many others) depending on how they use labels.

I wasn't making any particular judgment on the merit of all these various bots, nor claiming that an AI agent should be the only tool available - just highlighting that there are many things you might want to do in terms of automation, but they all require setup and configuration, and in some cases exposing your project to supply chain risk for that bot due to the privileges they run with.

If people want to block all AI contributions they should be able to do so, sure, but existing contribution prioritisation is opaque for the vast majority of open source projects anyway, so a signal visible only to maintainers is no worse than the current situation of opacity.

I'm in no way claiming that this could be suitable for all projects - no tool (AI or otherwise) would be.

[PaulNewton]

That assumes you have the resources to evaluate every contribution to determine if it is something you don't need/want.

AHAHAHAAHAHAAH haha ahuehue heu omg I can't.
That's literally the point of this entire discussion, NOT having the resources.

Humans don't have infinite time, and LLM's do NOT have infinite energy resources.
Even if you are the type to let AI go at full rip you will burn time somewhere in some sort of review, backfilling, or defending against security breaches etc etc etc ad nauseum; all the while turing your time into a free training vector for other businesses software.

Productivity gains cannot be infinite in a mortal world using probability distributions.
We need deterministic tools for human to human contribution.
Yeeeesh.

[PraiseTechzw]

This is a very real problem, and I appreciate that it’s being treated as systemic rather than blaming maintainers or contributors individually.

One concern I have with repo-level PR restrictions is that they may disproportionately impact first-time contributors who do want to engage meaningfully but don’t yet have collaborator status.

Personally, I think the most promising direction here is criteria-based PR gating rather than blanket restrictions things like required checklist completion, passing CI, linked issues, or acknowledgement of contribution guidelines before a PR can be opened.

On AI usage specifically, transparency feels more scalable than prohibition. Clear disclosure combined with automated guideline checks could help maintainers focus on high-intent contributions without discouraging responsible AI-assisted workflows.

Looking forward to seeing how these ideas evolve especially solutions that preserve openness while respecting maintainer time.

[moraesc]

One concern I have with repo-level PR restrictions is that they may disproportionately impact first-time contributors who do want to engage meaningfully but don’t yet have collaborator status.

Completely agree with this concern and just want to emphasize that this is just a starting point and not the final solution. The plan is to develop more tools and frameworks for both maintainers and contributors to help address the "slop" problem.

Personally, I think the most promising direction here is criteria-based PR gating rather than blanket restrictions things like required checklist completion, passing CI, linked issues, or acknowledgement of contribution guidelines before a PR can be opened.

Some other ideas we're exploring are requiring an issue to be created first before a user can open a PR or leveraging AI to validate a PR against rules defined in a CONTRIBUTING.md file or something similar. Any thoughts on that?

[robmen]

Some other ideas we're exploring are requiring an issue to be created first before a user can open a PR

Requiring that an issue be assigned to the user before they can post a PR would work well with our process. We already request that users who open a PR also open an issue so it can be triaged into the appropriate milestone. It would be great if GitHub had a way to normalize that workflow.

[taranion]

I like the idea of "issue first".
Allow PRs

• either from project members
• or for existing issues that the maintainer manually put in some kind of "expecting PR" state.

Potentially make the order irrelevant. Have PRs from non-members in some kind of "hold" state, until the issue has been opened and accepted. Close PRs that link to rejected issues. Auto-close PRs in "hold" state from non-members that do not link to an issue after a configurable period.

[PraiseTechzw]

One concern I have with repo-level PR restrictions is that they may disproportionately impact first-time contributors who do want to engage meaningfully but don’t yet have collaborator status.

Completely agree with this concern and just want to emphasize that this is just a starting point and not the final solution. The plan is to develop more tools and frameworks for both maintainers and contributors to help address the "slop" problem.

Personally, I think the most promising direction here is criteria-based PR gating rather than blanket restrictions things like required checklist completion, passing CI, linked issues, or acknowledgement of contribution guidelines before a PR can be opened.

Some other ideas we're exploring are requiring an issue to be created first before a user can open a PR or leveraging AI to validate a PR against rules defined in a CONTRIBUTING.md file or something similar. Any thoughts on that?

I think leveraging AI to validate PRs against CONTRIBUTING.md rules is promising, but only if it’s deterministic in enforcement. If the output is advisory rather than blocking, maintainers still carry the review burden.

[PraiseTechzw]

One concern I have with repo-level PR restrictions is that they may disproportionately impact first-time contributors who do want to engage meaningfully but don’t yet have collaborator status.

Completely agree with this concern and just want to emphasize that this is just a starting point and not the final solution. The plan is to develop more tools and frameworks for both maintainers and contributors to help address the "slop" problem.

Personally, I think the most promising direction here is criteria-based PR gating rather than blanket restrictions things like required checklist completion, passing CI, linked issues, or acknowledgement of contribution guidelines before a PR can be opened.

Some other ideas we're exploring are requiring an issue to be created first before a user can open a PR or leveraging AI to validate a PR against rules defined in a CONTRIBUTING.md file or something similar. Any thoughts on that?

I think leveraging AI to validate PRs against CONTRIBUTING.md rules is promising, but only if it’s deterministic in enforcement. If the output is advisory rather than blocking, maintainers still carry the review burden.

[funnelfiasco]

Thinking along the lines of the discussion first approach that Ghostty uses, I think one way to create just enough friction would be to have an opt-in where a PR has to be linked to an open issue or discussion topic. So when an unprivileged (i.e. does not have elevated privileges on the repo) user tries to create a PR, there's a required field that takes an issue/discussion number. If that's not provided (or the corresponding issue/discussion is closed), then the PR can't be created.

This could be trivially worked around by throwing in any old issue/discussion (or by creating one), but it may cause just enough friction to help. To guard against this, perhaps maintainers could set a "minimum age" for the issue/discussion (e.g. 12 hours) to prevent creating fake issues to support a spammy PR.

[moraesc]

@jamietanna does the discussion-first approach that you follow require all users to create a discussion first, or only "non-collaborators" (users without write access) ?

[jamietanna]

@moraesc generally non-collaborators. The maintainers and collaborators are able to create an Issue and remove the needs-discussion label that auto-closed an Issue raised by a non-collaborator

That being said, non-collaborators will often go through the Discussion process, as it's pretty good at distilling what we want, and a place to work through ideas before they're in an Issue

[marcindulak]

The problem with github discussions is that they are isolated - they don't automatically create links https://github.com/orgs/community/discussions/2971 like issues do, meaning information is scattered in two places and difficult to find.

[weisdd]

The issue with this approach is that pretty much any popular project would already have enough of pre-existing issues/discussions to choose from.

In our case, we've noticed that a good chunk of AI slop was targeting old issues with the "good-first-issue" label. - They're attractive in a sense that, by definition, it's not something complex to do, so you don't have to spend much time prompting AI and, thus, can generate dozens or even hundreds of PRs a month (we've seen such accounts) across a random list of popular repositories (automated tooling makes it easy to find a target). We ended up removing the label from all issues.

[PaulNewton]

Agree but the permissions names might need a different messaging:

So when an unprivileged

Try this optics nit on for size:
"The unprivileged cannot code on github. ; unless their AI is good enough to pass."
Only being partially sarcastic in order highlight while less privilege from a security standpoint is 100% valid there are some very real problems in how we phrase permissions with AI in the mix reallllly muddying things more and more while also wanting/needing new people to join in.

There is non-zero chance for this slop problem to grow into an economic/cultural divide in haves good AI's vs have no-AI's.
Making under-privelaged excuse for some odd twisted ladder pulling on a "social coding" platform; that could weirdly encourage more AI use by any possible contributor just so they can escalate privilege's making the actual learning seem more and more less useful.
Instead of security/spam prevention tools for maintainers to vet new contributors who do want to learn but can no longer afford the entry fee.
ugh 🤮 .

[upgradeQ]

Implementing a secondary, muted counter for bot-authored PRs would help maintainers distinguish real community activity from automated noise at a single glance.

[wqqkakuiii]

I totally agree that low-quality and AI-generated PRs have become a huge burden for open source maintainers. It wastes lots of time on reviewing meaningless, non-compliant and abandoned contributions.
It’s necessary for GitHub to roll out both short-term and long-term solutions: granular PR permission control, spam PR deletion ability, and clear rules to validate PRs against project contribution guidelines.
Also, instead of simply detecting AI-generated code, we should judge contributions by actual code quality, logic and whether it solves real problems. Adding proper entry friction for new contributors would also effectively reduce mass low-effort PR submissions.

[hopeseekr]

I desperately need a Pull Request review panel for all of my orgs and pull requests.

For instance:

• PHPExpertsInc - 75 open sourced repos
• BetterRimworlds - 26 open sourced repos
• Autonomo AI - 21 open sourced repos

122 repos in just 3 of my orgs. I have about 175+ open sourced github projects in total. In order to find the Pull Requests right now, I basically have to manually cycle through the projects, and sometimes that can take years to find (like this one or this 7 year 5 day-old PR only discovered because the author Assigned it to me (which no one ever does)).

What I need is either (or both) an organization-level Pull Requests page, or a universal-level : Pull Requests page for every project I have merge rights on.

I also want the ability to FULLY DELETE accidental or slop PR requests.... instead of closig them and then littering the Closed section... or at least a way to Archive them (remove from active list and search).

[hopeseekr]

The problem for me personally is way worse than I assumed. I knew it was bad, but not the 50+ PRs spanning 15 years and 20+ repos :O

[yurishkuro]

I built a helper recently that lets me prioritize PRs for review: https://github.com/jaegertracing/maintainer-tools. The list is first sorted by repo because I only look at 3 of them, but it's easy to change to render differently.

npm run triage
# → writes ./triage.html
open triage.html

[hopeseekr]

@yurishkuro I am so installing this first thing tomorrow morning (Colombia time)!

[projekta2]

The cross-repo triage problem is brutal at that scale. Your script output
actually illustrates the core issue well — mixed repos, mixed staleness,
PRs from 2023 sitting next to ones from last week, no way to know at a
glance which ones need attention today.

The thing I've found missing from most list-based approaches (including
ones I've tried) is context about why a PR should be prioritized, not
just when it was opened. A 2023 PR with a bot dependency bump is a
different problem than a 2023 PR from a contributor who's still actively
responding.

A few signals that have changed how I work through queues like this:

• Whether the author has commented in the last 7 days (live vs. dead PR)
• Whether CI was passing at the time the PR was opened, not just now
• Whether the files touched overlap with anything else currently open

The last one is the most underrated — two PRs touching the same files
are not independent review decisions, even if they look unrelated.

@yurishkuro's maintainer-tools approach of rendering to HTML is smart
for the overview layer. The piece I'd add is a priority signal per PR
so you're not just seeing the list but knowing where to start.

[yurishkuro]

The priority is indirectly built into how this report organizes the output, but priority is not a single dimension, so the report is focused on classification rather than a straight ranking.

[CJCrafter]

The automated spam is getting more and more frequent, and less usable. We really need a solution that is better than "just turn off PRs"

Here is an example project I found that has been spamming me. These systems are so naive and dumb that a simple honeypot solution would almost certainly catch 90% of them... you just have to mention "payment" and they leap on it. If GitHub can autogenerate honeypots for legitimate user accounts, it might become cost prohibitive to run these bots.
https://github.com/asaadnashed/bounty-autopilot

[CJCrafter]

another potential solution is to let authors disable API write access to their repositories, and add captchas to their PRs for first-time contributors. This might just add enough friction to stop bots

[Davincc77]

One missing layer in this discussion is portable governance context for agents.

A repo can have CONTRIBUTING.md, CI, branch protection and review rules, but an agent still needs a runtime-readable boundary that travels with it:

• tests it must not delete
• CI checks it must not weaken
• files it cannot modify without human approval
• release gates
• security constraints
• allowed tools
• architectural decisions from previous work

The important part is that the agent should not hold the keys.

In the x.klickd model, the passphrase belongs to the human operator. A trusted local runtime decrypts the file, applies policy/redaction/human-veto rules, and only injects the safe subset of context into the model.

Memory belongs to the user.
Authority stays governed.
The agent gets enough context to work, but not enough authority to rewrite the rules of the system.

[Faheemff]

Thanks for addressing this. The low-quality PR problem is real maintainers are spending more time rejecting spam than reviewing genuine contributions.
The most useful short-term fix would be configurable PR permissions letting maintainers restrict who can open PRs is long overdue since 2016.
For long-term, AI triage tools sound promising, but transparency in AI-assisted contributions is equally important. Knowing whether a PR was AI-generated upfront would save a lot of review time.
Looking forward to seeing these changes ship.

[RyogiMutsuki]

https://github.com/PCL-Community/PCL-CE/issues/new?template=xxx.yml
https://github.com/Meloong-Git/PCL/issues/new?template=xxx.yml

I personally think you should fix some bugs first, such as being able to create a blank issue that only maintainers have permission to create by using a non-existent template. (Blank issues have been disabled in the configuration, but the permission checks for some GitHub features are almost the same as if they weren't.)

[projekta2]

The CONTRIBUTING.md-as-source-of-truth idea is interesting. The challenge is
that low-quality contributions often technically follow the guidelines — they
just don't solve a real problem, or they solve it in a way that creates
maintenance debt.

Something that's helped me more than detecting "is this AI-generated" is
asking a slightly different question: "how much reviewer attention does this
PR actually require?" A small, well-scoped fix from a first-time contributor
might need more explanation but less technical scrutiny than a 600-line
refactor from a known contributor. Those are different types of review load.

The signals I've found most useful for estimating review load before opening
a PR:

• Size vs. scope claim (does the diff match what the description says it does?)
• CI status at the time of triage (not just at merge)
• File overlap with recent changes (touching the same files as an open PR
  is an automatic complexity multiplier)
• Author engagement after submission (PRs where the author stops responding
  within 48h have a very different trajectory than ones where they stay active)

I've been tracking these manually for a while and eventually automated it for
my own workflow. Happy to share specifics if it's useful context for what
you're evaluating here — the tricky part was making the scoring consistent
enough to actually rely on daily, without it feeling like a black box.

[henry-collins101]

This is a real pain point for maintainers, so it’s good to see it being addressed. Overall, the balance between keeping repos healthy and keeping them open is the hard part, but these ideas are a solid starting point.

[freakynit]

One approach I have been experimenting with is a lightweight triage layer rather than an "AI detector".

I built PRAS Bot for this: https://github.com/marketplace/actions/pr-anti-spam-bot

Source code: https://github.com/freakynit/pras-bot

The reason I avoided pure AI-detection is that it feels like the wrong target. AI-assisted code can be good, human-written code can be bad, and any "was this generated?" detector will be noisy. What maintainers usually need is more practical:

Does this PR look suspicious, low-effort, or expensive to review before I spend real maintainer time on it?

PRAS Bot runs as a GitHub Action on incoming PRs, scores each one from 0-100, and applies one label:

• likely-spam
• needs-review
• looks-good

It can also post a scorecard comment explaining why the PR was flagged. It does not close PRs, reject contributors, or make final moderation decisions. The goal is first-pass triage, not automated judgment.

The current approach is signal-based. It is not perfect, but I think this is the most useful shape of solution right now: combine many weak-but-useful signals instead of pretending there is one magic spam detector. Maintainers already do this mentally; the bot just makes it consistent and visible.

Category	Signals
PR shape	Lines changed, files changed, account age, recent cross-repo PR volume
Contributor trust	GitHub author association, prior merged PRs in this repo, closed-unmerged PR ratio, issue participation, review engagement, duplicate/repeated PR titles, bio-positioning patterns, bursty activity across repos
Repo fit / review burden	Tests included, change scope, risky paths touched, generated/deprecated file maintenance, linked issue/reference, duplicate work, DCO sign-off when required
Optional repo-context / LLM checks	Related prior work, contribution-rule adherence, diff credibility, PR template completion, scope/roadmap/architecture alignment, PR body quality

The optional LLM checks are off by default. Repos can keep the bot fully heuristic, or enable LLM-backed checks only where they are comfortable with the cost and trust tradeoff.

Basic usage is:

- uses: freakynit/pras-bot@v1

Repo-level config lives in .github/pras-bot.yml, so maintainers can tune weights, thresholds, labels, and which signals matter for their project. Almost everything is tuneable.

Here is a sample run with the bot scorecard:
freakynit/whissle-ai-nodejs-sdk#2

I would be interested in feedback from maintainers here, especially on which signals are actually useful in daily triage, which ones are too noisy, and what kind of scorecard output would make this more actionable.

[BenjamenMeyer]

Great to see this being addressed; though this is not simply due to AI - though that probably makes the problem worse.

In the ISO C++ Core Guidelines there was a lengthy discussion around this before AI was even a thing (see isocpp/CppCoreGuidelines#2258) because the project gets PRs and Issues opened by what seems like students at universities in China and India doing a homework assignment to create a GitHub account and open a PR or Issue; but as discussed in that issue (at least at that time) there was not real good way of identifying those and auto-closing them at least from our perspective as maintainers/contributors than someone just manually doing it.

I don't know if there's been specific discussion of it, but I've also seen that on the Flutter repositories.

This is perhaps a bigger issue for high profile projects which most are not - I've part of a number of FLOSS projects and communities and most don't see much of this. However, AI is certainly contributing to issues as pointed out. I'd just like to request that these other pre-existing issues also be considered in this.

Also, perhaps an auto-close instead of delete is more appropriate so that people don't feel like their contribution is completely ignored - rather a closed as spam by automation kind of comment is more appropriate; they can always are argue whether its spam, but it would at least ease the burden.

[pombredanne]

+1 for that. In fact a good bunch of the Ai-generated PRs we receive are from students-aspiring-contributors that do not mean bad by inflicting AI-slop on us, they just do not know that what they are doing is not helping. To that extent, we (my small FOSS org) are now partnering with Codeday that sends us qualified human college students that genuinely want to learn and are instructed not to use LLMs.

[GuillaumeDua]

+1.
As the author of isocpp/CppCoreGuidelines#2258, I'd add that my investigations tends to point towards at least 3 schools which might encourage students to open shallow/empty issues/PRs on this repo.

For now, it's quite obvious to detect, and Github tools works fine to avoid that (repo configuration).
But when such contributions (issues/comments/PRs) are performed by AI bots crawling random repos, it's sometimes more difficult to detect.
Just had the case today on a personal repo (I'm the only developer on that one) with a AI-generated comment on a random issue produced by that account -> reported it immediately.

[rossbuckley1990-hash]

Maintainer-adjacent builder here (I ship dev tooling, repo-proofer, BootProof). Most proposals in this thread try to detect slop after submission. I keep wondering about the inverse: let legitimate contributors prove work at submission time. Concretely: a CLI/Action run alongside an agent session that attaches a signed receipt to the PR, tests actually executed with output hashes, files read before editing, whether existing tests were modified, declared AI-assistance level per project policy. No receipt on an AI-assisted PR becomes a triage signal, like a missing DCO sign-off. Question for maintainers: if receipts existed, would “receipts required for AI-assisted PRs” be a policy you’d adopt? Which single piece of evidence would save you the most review time? Validating before building.

[eli-schwartz]

This would save me zero review time whatsoever. My policy is that if you used an LLM for anything, including anything at any stage of development, then simply opening the PR constitutes a Code of Conduct violation and the only real question is whether it was a knowing violation that also deserves disciplinary measures (such as a ban from all project spaces).

No, "I didn't even know I was using an LLM" isn't a valid justification either. If your text editor nefariously upgraded behind your back and switched from using clangd or python-jedi to "ask an LLM to analyze the current file and list what function names exist in the current scope" in order to power tab autocompletion, then YOU HAVE VIOLATED THE BAN and you are ethically required to self-report once you discover this, close your PR as "anandoned', and optionally opening a new PR that was done properly. You may not do the ethical thing, but that doesn't mean the Code of Conduct doesn't expect it of you.

I suppose that users offering a cryptographically signed proof of violating the code of conduct isn't detrimental to the project, I just don't see why any user would ever bother to offer one if they could simply refrain from violating the code of conduct.

As far as demonstrating actual compliance with the code of conduct goes, it is nonsensical to conceptualize "cryptographic proof that one has NOT done something" unless you're proposing ankle monitors. I don't think the situation has gotten that bad yet. ;) As far as non-cryptographic proof goes, we've had a checkbox in the PR template for years, requiring people to check the box and acknowledge that they understand the LLM ban and agree to abide by it. If you check that box, but have used an LLM at any point to aid your contribution, then you are a dishonest person who lies, and liars have violated the code of conduct and deserve to be banned from all project spaces.

[BenjamenMeyer]

thoughts:

1. AI is kind of here and likely to stay; tooling/policies has to account for it some how. Avoiding it is untenable because its entirely reliant on self-reporting which is easy to avoid. CoC aside, not everyone has the ethics to not lie about stuff like that.
2. (And more pertinent) @rossbuckley1990-hash how would this be done? ... this would require all client-side tooling and you have no control over the client's platform. What tools work for you may not work for me. Are you going to mandate which operating systems developers can use? That's also untenable. So while an interesting idea, I don't think its practical to implement.

[rossbuckley1990-hash]

@BenjamenMeyer your (1) is the actual argument here: a disclosure rule with no evidence layer is self-reporting, and self-reporting is exactly what fails.

On (2): no OS mandate, no client control needed, because it’s voluntary rather than enforced. Same shape as git commit -s: client-side, forgeable in theory, and a universal convention anyway, because maintainers made its absence the triage signal rather than its presence a guarantee. A receipt says “these commands ran against this exact tree and produced these outputs,” signed. It can’t prove comprehension and can’t stop a determined liar with local key access, it raises the cost of lying from zero to non-zero and lets honest contributors be distinguishable. Anything beyond that would be overclaiming.
I’ve got a rough prototype running (Python CLI + Action, no LLM calls, deterministic). Not linking it here because it’s early and I’d rather it be judged when it’s honest than when it’s loud, happy to share with anyone who wants to kick holes in it. And the question still stands: if a PR carried a signed record of the tests that actually ran against that tree, would it change your triage, or is the bottleneck elsewhere?

[awdemos]

The “asymmetry of cost” point is the real insight here. Right now it is almost free for someone to fire an agent at every repo, but it is expensive in maintainer time to clean up the result. That asymmetry exists because most AI coding tools are metered per request, so the cost to the spammer is tiny and the cost to the maintainer is unbounded.

One way to change the shape of that asymmetry is to move from per-request billing to a flat-rate compute model. If the agent is running on a GPU you control and pay a fixed monthly price for, the incentive shifts: the cost is bounded for you, and the spammer still has to operate within a real resource budget. That is the model we built UltraWork around — a flat-rate GPU-backed coding agent at https://vibecodingagency.com/gpu-cloud/ — so teams can set a predictable ceiling on their AI spend instead of watching a meter spin.

Not a magic fix for low-quality PRs, but it does remove the “burn credits on trivial noise” dynamic that makes the asymmetry worse.

[BenjamenMeyer]

@awdemos that assumes that the spammer has not done the same. They could use: local AI deployment, fixed price agents, API billed agents, etc... there's no predicting what they're using.