🤖 Why I Still Trust LLMs (Even When the OSS Crowd Is Closing Their PRs)

The Dissonance Is Real, And It’s Not New

I sat in a Berlin hall, watched Armin Ronacher field a question about PR overload, and heard him say “we auto‑close almost all LLM‑generated submissions.” The audience applauded. A few rows back, engineers were feverishly typing away in Claude Code. The scene felt like a tech‑religious revival: people preaching against the very tool they were clutching.

I felt that familiar tug—the same one I get every time I open a new issue in a repo that’s been flooded with ChatGPT‑styled bug reports. The critique is spot‑on: copyright leakage, carbon footprints, and the danger of a “sloppy‑as‑a‑service” model that churns out code that looks right but breaks silently. Yet, I still run LLMs through my CI pipelines, ask them to generate Cypress tests, and let them draft pull‑request descriptions. Why?

The Good (and the Ugly) of LLM‑Generated PRs

The core gripe is trust. Before LLMs, a decent PR meant a human spent hours—research, debugging, polishing the commit message. That effort acted as a low‑cost filter against trolls. Now a bot can spin up a fresh GitHub account, crank out a PR, and the maintainer is left guessing: did a junior spend a day on this, or did Claude just copy‑paste a snippet from StackOverflow?

Projects like Zig and Gentoo have started rejecting LLM PRs outright. It’s a knee‑jerk reaction that feels safe, but it also throws away a huge pool of potential contributors—especially those in regions with limited access to mentorship.

How I Make LLMs Work for Me (And Not Against Me)

The answer isn’t “stop using them.” It’s designing guardrails that let the LLM do the heavy lifting while humans stay in the loop where it matters.

A Mini‑Pattern I Swear By

1. LLM‑only branch – Generate the code on a dedicated branch named llm‑draft/<feature>.
2. Static‑analysis gate – Run a full suite of linters, type‑checkers, and mutation tests in CI. If the mutation score drops below a threshold, the PR is auto‑rejected.
3. Human‑review badge – Require a reviewer with the +LLM‑approved label to add a short comment confirming they ran the changes through a local IDE and verified behaviour.

This three‑step dance preserves the trust signal (a human‑signed badge) while still harvesting the speed advantage of LLMs.

What the Rest of the News Is Telling Us

The LLM hype train isn’t the only thing chugging along. Two other headlines caught my eye because they echo the same trust‑vs‑automation tension.

• Kimi K3: Open Frontier Intelligence – Kimi positions itself as an “open‑source” alternative to the big‑AI vendors, promising transparency. The open‑source angle is promising, but without the guardrails I mentioned above, even an open model can become a flood of low‑quality contributions. Think of it as a more democratic Claude that still needs a moderator.

• Microsoft Comic Chat Goes Open Source – This project turns chat into illustrated dialogues, a fun experiment that could become a testing tool for UI/UX flows. But the moment the repo is public, you’ll see a wave of PRs that simply replace every speech bubble with a GPT‑generated caption. Again, without a clear contribution policy, the repo risks being buried under novelty.

Both stories reinforce a single truth: open tools invite open chaos unless we embed quality checks into the workflow.

The Automation‑Testing Lens

As an SDET, my day revolves around making sure everything that runs does so reliably. LLMs can write tests, but they can also write flaky ones. Here’s what I do to keep my test suite honest:

• Prompt‑to‑test contracts – I store the exact prompt used to generate a test in a comment block. When the test fails, I can replay the prompt, tweak it, and regenerate the test. This creates a reproducible pipeline that’s auditable.
• Differential testing – Run the LLM‑generated test alongside a hand‑written baseline. If the new test diverges in coverage or execution time, flag it for review.
• Metric‑driven pruning – Use a simple script that measures flakiness (retries > 2) and automatically closes PRs that exceed a flakiness threshold.

These steps turn the LLM from a wild west contributor into a controlled assistant.

Practical Takeaway for Engineers

If you’re going to let LLMs touch your codebase, put a human signature on the final gate.

In practice, that means adding a lightweight checklist to your repo’s CONTRIBUTING.md:

• [ ] Did the PR pass all static‑analysis and mutation tests?
• [ ] Did a reviewer add the +LLM‑approved label with a brief sanity‑check note?
• [ ] Is the generated code covered by at least one non‑LLM test?

Once this checklist is automated (GitHub Actions can enforce it), you’ll see the volume of PRs stay high, but the noise drop dramatically. Maintainers can finally breathe again.

Closing the Loop

The critics are right: LLMs produce slop, they burn energy, and they can swamp open‑source maintainers. But the same tools also give us a productivity multiplier that we can’t afford to ignore—especially when testing frameworks evolve faster than our manual capacity.

The sweet spot is human‑in‑the‑loop automation. Let the model draft, let the CI validate, and let a real engineer give the final nod. That’s how I keep my CI/CD pipeline fast, my test suite trustworthy, and my sanity intact.

So the next time you hear a speaker at a conference proclaiming the death of OSS because of LLMs, smile, raise your hand, and whisper: “I’ve got a PR with a +LLM‑approved badge. Let’s merge it.”