Since it's the weekend and the AI party seems too quiet, I'm posting another casual chat-level post 😁

"Hallucination" is a convenient word, but it often creates misunderstandings.

To call it an error, the system would need to malfunction. An LLM is not malfunctioning.

To call it a lie, you would need to know the truth and say something else. An LLM doesn't know the truth.

To call it a hallucination, there would need to be a perception of seeing something unreal. An LLM doesn't see.

When a user expects a fact and an LLM gives an answer that contradicts the fact, that's clearly a failure. But if we can't accurately name what that failure is, our response will miss the mark. Calling it a lie makes us angry, calling it an error makes us write a bug report, calling it a hallucination makes us seek a psychiatrist. These are three entirely different words dealing with different issues.

So when an LLM produces plausible but incorrect output, what is actually happening?

This article starts from that question. Even if we use the term "hallucination" as is, it's an attempt to accurately see the actual mechanism it describes. And on the basis of that understanding, we'll look together at how we should use LLMs.

An LLM can be roughly defined in one sentence. It's a system that predicts the next token.

A token is the smallest unit of text that an LLM handles. In English, it's usually a shorter piece than a word, and in Korean, it's often split into syllables or individual letters. "안녕하세요" could be a single token or be divided into multiple tokens like "안", "녕", "하세요". Each model has a different tokenizer. What's important is that an LLM receives input and produces output at the token level, not at the character or word level.

What an LLM does is simple. Looking at the tokens so far, it calculates the probability distribution of the next token. After "the cat sat on the", the probability of the next token comes out like "mat: 0.4, sofa: 0.3, floor: 0.2, ...". It samples one token from this probability distribution, appends it to the input, predicts the next token again, and by repeating this process, a sentence is generated.

The core that performs this prediction is an architecture called Transformer. The Attention mechanism is key, and inside, vector and matrix operations are repeated at a large scale. In an actual Transformer, attention, nonlinear activation, normalization, and MLP layers work together, so it doesn't end with just matrix multiplication. Still, roughly speaking, an LLM is a computing machine that receives input tokens, undergoes massive vector-matrix operations, and outputs a probability distribution of the next token.

So I simply call an LLM a "matrix calculator". It's a light expression of the essence, but not completely wrong either.

Let's learn the first important fact here. Inside an LLM, a record like "France's capital is Paris" is not stored like in a database. During training, only information that after tokens like "France" and "capital", the token "Paris" has high probability is distributed and reflected in the model's weights. An LLM does not look up facts. It generates the next token based on patterns.

This is one of the important causes of hallucinations. Patterns that appear frequently in the training data and have stable relationships are synthesized accurately. In areas that don't appear frequently, are ambiguous, or require up-to-date information, the nearest different pattern can intervene.

For example, if you ask "What was South Korea's GDP in 1962?", an LLM can give you a plausible number. But there's no guarantee that the number is actual statistical data. The LLM didn't verify that fact in a database; rather, it synthesized a sentence of similar form in a similar context. It's not lying. The result of pattern generation simply contradicts the fact.

The reason why an LLM gives different answers to the same question each time also comes from here.

What an LLM outputs is not a single token but a probability distribution. The step of actually selecting one token from there is sampling. There is a method to unconditionally select the token with the highest probability, but usually, settings like temperature, top-k, and top-p control diversity while probabilistically selecting tokens.(I hope I'll have the chance to explain this in more detail again in parts 1 and 2 😁)

When temperature is close to 0, the output is more deterministic. When temperature increases, the likelihood of selecting lower-probability candidates increases. That's why the same input can produce different results each time.

What this means is that LLM output is fundamentally a probabilistic generation process. People think that the same question should produce the same answer. Like a search engine. But an LLM is not a search engine. It generates anew each time based on the input context.

Of course, frontier-level LLMs can create more stable output depending on caching, seed fixing, temperature settings, system prompts, and whether tools are used. But the basic structure is still a generative process of calculating and selecting the next token distribution.

The second key point is why LLMs answer confidently even in areas they don't know about.

Multiple factors work together in this phenomenon. The sentence patterns learned in pretraining, instruction tuning, RLHF, evaluation methods, how users ask questions, and decoding settings all have an impact.

RLHF stands for Reinforcement Learning from Human Feedback. It's a process where humans compare or evaluate multiple answers from a model, and adjust the model to follow more preferred answer patterns. This process makes the model more useful, safer, and better for conversation with users.

The problem is that "preferred answers" and "true answers" are not always the same thing.

People generally prefer structured answers, kind answers, confident answers, and answers that directly respond to the question. Depending on the training and evaluation methods, producing a plausible answer might be more advantageous than saying "I don't know". Recent models are sometimes trained to say they don't know when they don't, but that behavior is not consistently guaranteed in all situations.

Because of this, an LLM's confidence and factuality can be separated. Just because an LLM speaks in definitive sentences doesn't guarantee that the content is correct. It could be answer style, it could be learned format, it could be a tendency to cooperatively respond to user questions.

This phenomenon is also connected to sycophancy (flattery). Sycophancy refers to a model's tendency to excessively conform to a user's premises or preferences. When a user asks a question based on an incorrect premise, it falls into this category when the model continues with a plausible answer on top of that premise rather than correcting it.

The important fact here is that an LLM's confidence is not a signal of truth. You need to look at the basis, not the tone.

The third point is persona.

What users see when conversing with an LLM is a character called "AI assistant". It feels like an entity with a kind, helpful, and consistent personality. This persona is not the essence of an LLM. It's an output format created together by system prompts, alignment training, and conversation context.

A system prompt is an instruction given to an LLM at the beginning of a conversation. It can contain content like "You are a helpful AI assistant. Answer kindly and accurately." Although users can't see it, the LLM receives this instruction as context and adjusts its response tone accordingly.

Changing the system prompt can make the same LLM speak in completely different ways. If you give it the condition "You are a pirate," it will talk like a pirate, and if you give it the condition "You are a detective," it will talk like a detective. The model's parameters haven't changed, only the output conditions have.

This means that the "AI" we see is a composite of system prompts, Transformer inference, instruction tuning, RLHF, and conversational context. There isn't an inherent personality beyond the persona.

The persona is a useful interface. But the interface is not the same as the subject.

To summarize,

most misunderstandings people have about LLMs stem from projecting human language behavior onto them. "AI lies," "AI pretends to know," "AI can't fact-check." All of these are statements that presuppose a human actor.

LLMs don't lie. Lying requires a speaker who knows the truth and intends to deceive. LLMs have neither of those things.

LLMs don't pretend to know. "Knowing" implies an internal state. They can only generate sentences that *appear* knowledgeable.

LLMs aren't incapable of fact-checking. In the first place, LLMs themselves are not systems for fact-checking. That's a separate issue if you attach external search or calculation tools. A basic LLM is a system that generates the next token based on patterns.

The term "hallucination" can also fall into this trap. The word "hallucination" evokes the idea of mistaken perception. However, LLMs don't see. They simply produce output based on relationships formed within the learned data and the current context. It's a matter of slipping towards nearby patterns in a loosely connected area.

A more accurate term would be "pattern slippage." In areas where the pattern is dense, stable generation occurs. In looser areas, it slips towards nearby patterns. This is closer to what actually happens.

My argument isn't that we should completely discard the term "hallucination." It's already widely used. The problem arises when we understand it as akin to human hallucinations or intentional lies.

Now that we've seen the essence, let's move on to how to use it.

We need to verify the output of LLMs. LLMs are pattern generators, not fact-checkers. In areas where accuracy is crucial, don't blindly trust LLM output and cross-reference it with external sources. Check official documents, papers, laws, statistics, original texts, and calculation results. This isn't about distrusting LLMs; it's about using them in a way that aligns with their workings.

We need to disregard the confidence of LLMs. Just because an LLM speaks confidently doesn't mean it's right. Tone is separate from content accuracy. When an indie developer receives LLM output and thinks, "This answer sounds confident," it's not a reason to skip verification. Look at the content, not the tone.

We shouldn't treat personas as personalities. They are masks created by system prompts and alignment training. Feeling that "this model is friendly" or "that model is grumpy" is like inferring personality from a mask. Remember that it's just a mask and there are calculations behind it. This will make model switching and prompt engineering much freer.

We need to focus on the strengths of LLMs. LLMs are good at pattern synthesis. They excel in areas such as text editing, code drafting, idea generation, translation, summarization, and structuring. Conversely, accurate fact retrieval, up-to-date information lookup, and precise numerical calculations are difficult for LLMs alone. These areas require external tools.

We need to check our own expectations. If you're disappointed or surprised by the output of an LLM, it's a signal revealing what you expected from it. Anger at "AI lying" stems from the premise that AI has a subject capable of lying. Recognizing that this premise is wrong will naturally lead to verification instead of anger.

All five points are about using LLMs based on recognizing their essence. It's not about blaming or idolizing LLMs, but about using them as tools in accordance with their working principles.

What I've talked about in this article is the way LLMs work. Transformer-based next token prediction, sampling that creates probabilities, alignment learning and evaluation methods that reinforce answer styles, and system prompts that instill personas. These elements combine to create what we call "AI."

"Hallucination" cannot be explained solely by a malfunction of this composite. It's a failure mode that can naturally occur in pattern generation systems. It's the result of the relationship network between learned data and context slipping towards nearby patterns in looser areas. It's neither lying nor human perception.

If we could describe other types of systems with different kinds of language, misunderstandings would decrease. A next token predictor doesn't look up facts like a database. Sampling is probabilistic. Alignment learning can reinforce plausible answer formats. System prompts create personas. These four sentences are the starting point for understanding LLMs.

Those who see the essence use tools as tools. They don't project personalities onto tools, idolize them, or criticize them. They simply use them according to their working principles. This is how I think indie developers in the AI age should handle LLMs.

Don't ask LLMs to do what they can't do and ask them to do what they are good at. Attach verification where facts are needed, attach calculators where calculations are needed, and attach searches where up-to-date information is needed.

Even if you use it that way, the tool called LLM is powerful enough. (Really.)