This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound.
Not saying that this way to review is a silver bullet, but at least it removes some weight.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
> Slop is asymmetric. The cost of producing a thousand-line PR or a ten-page doc is approaching zero, while the cost of reviewing it remains constant.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...
Plus, I find reading slop docs takes cognitively much more effort. When I find problems in human docs, it's easier for me to point out the issue, comment etc. But on these AI slop docs, I find it overwhelming to comment on, because the issues are often woven throughout the document in a way that makes it so I can't isolate the comment in a single place. It's hard for me to describe.
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound. Not saying that this way to review is a silver bullet, but at least it removes some weight.
The discipline of splitting them up helps in almost every dimension.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...