"As a Language Model..." is one of the beginnings of a sentence I hate the most from LLMs and is the reason why I support free (as in "Liberty"), local models. I'm well aware that it is not a doctor and cannot replace a real doctor with multiple years of experience, I don't need to waste braincell activity on reading that it "as a Language Model" cannot give a precise diagnosis and that I should ask a real doctor - all I want to know is if I what I experience justifies either A) ER, B) 3-4 weeks scheduled doctors appointment or C) two paracetamol and a nap.
I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.
This statement should be restricted to answers for provocative questions. If an LLM is being asked a question that goes against the guidelines, then "As a Language Model..." is a valid starting point. Rest, obviously we're aware that a software doesn't have the judgement that a human has.
Be careful there. LLMs may be good at identifying a condition based on a description of the symptoms, but they are much worse at recommending the correct course of action (getting it wrong half of the time).
...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.
As a Human, I do not need to know it is a Language Model.
Though I don't wish the world was filled with people like you, remembering that it's not, and it's filled with people that have very little discernment when it comes to higher learning makes it's pretty obvious companies do not want the liability of it's users thinking the technobabble passes for wisdom or experience or intelligence.
Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.
Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!
> we don't fully know what drives the voice in LLMs
Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.
> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.
(Replying to your quote from another comment)
This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.
> but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?
Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.
Maybe I'm missing something deeper here, but isn't it clear that this is driven by post-training and system prompt? Anthropic's constitutional reinforcement (soul document,etc), for example, is very clear about "who" (not so much what) Claude is supposed to be.
Very cool innovation in steering - but a lot of introspection only emerges at the highest weight classes - this research would be fascinating to run on bigger models.
In my view, these models should never be set up to output first-person "experiential" (from the abstract) language. It's too easy to humans to anthropomorphize software that presents itself as having an identity.
The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.
Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.
Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.
It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.
In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).
It's an interesting thought, but humans do like to antropomorphize things anyway, and I believe your variant won't be popular if choice is given to consumers.
I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.
The first is equivalent to "I don't want my operating system to be used to program viruses."
The second is "I don't want vendors to include marketing in their product."
[0] https://www.nature.com/articles/s41591-025-04074-y
As a Human, I do not need to know it is a Language Model.
It seems like should be obvious given that they can play multiple characters, but it’s good to have more confirmation.
(Although, I do wonder to what extent these personas might become stable entities.)
Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.
Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.
> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.
(Replying to your quote from another comment)
This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.
Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.
"You are a Large Language Model" in (system?) prompt would do the trick..
The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.
Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.