Rendered at 16:17:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dwroberts 2 hours ago [-]
> Mostly capability failures that newer models no longer make: counting letters
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
dcrazy 41 minutes ago [-]
The LLMs have indeed gotten better at representing numbers and performing arithmetic, but they can still do it wrong.
I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
cyanydeez 1 hours ago [-]
Inthink finetuning and prompts and intermediary engineering harnesses all obscure idependent verificant of raw LLM capability.
So yes, they probably spend half theor budget trying to keep suspension of disbelief.
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
So yes, they probably spend half theor budget trying to keep suspension of disbelief.