Rendered at 19:18:58 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
plorkyeran 20 hours ago [-]
> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies. We evaluate the first answer, not an agent’s result after iterating on errors.
Very unsurprising results given this methodology. The agentic approach where models are allowed to iterate on changes is a very big part of what has made LLM coding tools actually useful.
yodon 20 hours ago [-]
> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies.
Ack! Do people actually try to develop software this way?
plorkyeran 20 hours ago [-]
Not any more, but it was common before Claude Code came along (~18 months ago). Based on the model versions used it may have still been the most common thing when the paper was started.
refactor_master 20 hours ago [-]
I do it from time to time when I’ve run out of tokens at the end of the month. But even then I’ve created a script to pack multiple code files and their dependencies into a single txt, with a tree overview.
And of course you never change the test suite in the process… That would be bad developer practice, LLM or not.
Gemini 3.1 Pro can easily work with this effectively, if not a bit slower than an agent.
1718627440 19 hours ago [-]
> But even then I’ve created a script to pack multiple code files and their dependencies into a single txt, with a tree overview.
Like:
$ cat $(find project/ -name '*.py' -type f)
?
2000UltraDeluxe 13 hours ago [-]
Non-technical people often have no clue that there are options
customguy 12 hours ago [-]
No, they test what LLM are capable of this way. They then examine the results by reading the code and reasoning about the changes the LLM made, rather than just throwing up their arms and going "Ack!" as if that's anything.
antisthenes 19 hours ago [-]
When I need modification to a small independent Python script (<1000 loc), sure.
Not for anything serious though.
TZubiri 20 hours ago [-]
It's pretty much the standard way to develop in OOP.
You are not supposed to hold all of the code in context, but to have a single object and reason about with what it see.s
kgeist 20 hours ago [-]
The average age of the explored models is ~1.5 years. They discuss PHP 8.3, which was released in 2023 and has not been actively supported since 2025. Agentic coding isn't used.
I'm not sure what the point is of publishing results on the mid-2025 state of the art in October 2026.
StevenWaterman 21 hours ago [-]
Models:
- Claude Sonnet
- Gemini 2.5 Flash
- Gemini 2.5 Pro
- GPT-5 Codex
- Llama 3.3 70B
It's inconvenient how quickly things are improving, makes potentially-interesting results like this dubiously relevant
StevenWaterman 17 hours ago [-]
Sonnet 4 specifically*
AdrianB1 20 hours ago [-]
One year ago I was very skeptical about such attempts. Today, it would be my first choice as LLMs evolved a lot and the result of carefully using a LLM for this task can be very, very good.
TZubiri 20 hours ago [-]
When code output becomes cheap, you write 15 million lines of code per weekend, and of course everything gets deprecated pretty fast version 1? that was like 4 geological ages ago (1 month) that's legacy.
Very unsurprising results given this methodology. The agentic approach where models are allowed to iterate on changes is a very big part of what has made LLM coding tools actually useful.
Ack! Do people actually try to develop software this way?
And of course you never change the test suite in the process… That would be bad developer practice, LLM or not.
Gemini 3.1 Pro can easily work with this effectively, if not a bit slower than an agent.
Like:
Not for anything serious though.
You are not supposed to hold all of the code in context, but to have a single object and reason about with what it see.s
I'm not sure what the point is of publishing results on the mid-2025 state of the art in October 2026.
- Claude Sonnet
- Gemini 2.5 Flash
- Gemini 2.5 Pro
- GPT-5 Codex
- Llama 3.3 70B
It's inconvenient how quickly things are improving, makes potentially-interesting results like this dubiously relevant