RUBIKSCORPAI fundamentals
How large language models work: tokens, context and reliable answers
A large language model, or LLM, is a model trained on language-related patterns and used to generate or transform text. Understanding tokens, context and the surrounding application helps explain both its usefulness and its failure modes.
Tokens are the pieces a model processes
A token can be a word, part of a word, punctuation or another encoded unit. Tokenization affects how much text fits into a request and how usage is measured. A page count is therefore a poor substitute for checking the actual input size.
Context is the information available for a particular generation: instructions, conversation history and any retrieved material or tool results included by the application. Information that exists somewhere in a company database is not automatically part of that context.
Training and using a model are different operations
Training adjusts model parameters using examples and an optimization process. During inference, the application supplies inputs and receives an output. A conversation does not, by itself, imply that the model’s underlying parameters are being updated.
Many language models use Transformer-based designs. The original Attention Is All You Need paper introduced the Transformer architecture. Modern systems vary, so that paper is useful background rather than a complete specification for every product.
Fine-tuning is a further training step for adapting behavior. Providing examples in a prompt is a different method: the examples are part of the immediate context rather than a parameter update.
What the application adds around the model
The same base model can behave very differently in two applications because these layers differ. Investigate the input and workflow before attributing every failure to the model.
| Layer | Responsibility | Example failure |
|---|---|---|
| Instructions and context | Supply the task and relevant material | An outdated policy is included |
| Retrieval | Find material the user may access | The wrong document revision is selected |
| Tools | Perform approved operations | A timeout is mistaken for a completed action |
| Validation and interface | Check results and communicate limits | An unsupported answer is presented as confirmed |
Worked example: the assistant that invents a warranty
Imagine a customer asking whether a repaired part has a six-month warranty. The correct answer depends on the product, purchase terms and current approved policy. A model may generate a convincing answer without possessing any of those facts.
A better workflow identifies the part and policy revision, retrieves the permitted source and answers only what that source supports. If the policy does not address repaired parts, the assistant should say the information is missing and route the question to an authorized person.
Test the full sequence with an obsolete policy, an ambiguous product name and an unsupported claim in a customer message. The useful result is evidence about source selection and escalation, not a subjective impression that the response sounds professional.
Why fluency and longer context are not enough
Coherent wording does not demonstrate that a statement is true. Relevant evidence can be absent, retrieved incorrectly or used in the wrong way. Supplying more text can also introduce conflicting versions and irrelevant passages.
Treat longer context as capacity, not a guarantee of correct use. Test whether the right source is selected and whether each important claim follows from it. Verify citations by opening the referenced passage, not merely checking that a link appears.
Record the model version, instruction version, source revision and tool outcomes when testing. A reproducible failure is much easier to diagnose than an isolated screenshot of a wrong answer.
Questions to ask before deploying an LLM feature
- Which facts require an approved external source?
- What happens when the source is absent or contradictory?
- Which tools can the application call, with whose permissions?
- How will users distinguish a draft from a completed action?
- Which changes trigger regression tests?
- Who reviews complaints and corrects the underlying problem?
Common questions
Does an LLM automatically know my company documents?
No. The application must supply authorized information through context, retrieval or another explicitly designed mechanism.
Does chatting with a model automatically fine-tune it?
No. A conversation and a training update are different operations. Provider data-handling practices are a separate question.
Does a citation prove that an answer is correct?
No. Check that the cited source is the right version and actually supports the claim.
Sources and notes
- Google Machine Learning Crash Course — Large language models
Foundational material on language modeling.
- Vaswani et al. — Attention Is All You Need
The original Transformer architecture paper; not a claim that all modern models share an identical design.
- Google — Fine-tuning, distillation and prompt engineering
How adaptation methods differ.
- NIST AI 600-1 — Generative AI Profile
Risks and management considerations for generative AI systems.
Read next
Have a specific question?
Tell us the task, the information available and what a useful result would look like.