Last updated on September 20th, 2026 at 01:54 pm
TL;DR
- Pre-training teaches a language model to predict the next token from context. Repeating that objective across enormous datasets changes the model’s parameters and produces much richer capabilities than the objective itself might suggest.
- Tokens are processing units, not “atomic units of meaning” or simply words. Depending on the tokenizer, a token can represent a word, part of a word, punctuation or another encoded unit.
- There is no universal training-dataset recipe for frontier LLMs. Public research documents mixtures of web text, books, code and other sources for specific models, while proprietary providers often disclose only part of their data construction process.
- Loss minimization does not force a model to reproduce the training corpus exactly. Gradient-based optimization adjusts parameters so the model assigns better probabilities across huge numbers of training examples. Memorization can nevertheless occur.
- A deployed model does not update its weights simply because it sees a new website or prompt. But the wider AI product may use retrieval, search, files, memory, APIs and tools to supply current information at runtime.
- For marketers, a source update and a model update are different events. Publishing accurate information can improve what retrieval-enabled systems have available; it does not give a publisher direct control over another provider’s model parameters.
The foundation of a Large Language Model begins with pre-training: the computationally intensive stage in which the model learns statistical structure from a very large collection of examples.
The important word here is learns, because it is easy to give it a human meaning it does not have. During pre-training, the model is not reading a book, deciding which facts are important and filing them into a database. It is repeatedly solving an optimization problem and changing its numerical parameters according to the error it makes.
The result is a model that can represent language, associations and regularities well enough to generate surprisingly capable continuations. That can include factual recall, abstraction and reasoning-like behavior. But the mechanism is still different from a search index, a knowledge base or a person learning from a new experience.
Pre-training changes the model’s parameters.
Inference later uses those parameters together with the context available at that moment. Giving a deployed model new context can change its answer without becoming another pre-training step.
Enjoy Ad Free
Access the full Research Paper. For free.
This article is just an extract from the full 100 pages independent research I’ve written for Fuel LAB® Research over 4 years of analysis, studying LLMs models, and data collection.
What Does LLM Pre-Training Actually Do?
For GPT-style autoregressive language models, the basic training objective is deceptively simple:
Given a sequence of tokens, predict the next token.
This is usually described as autoregressive language modeling or next-token prediction. OpenAI’s GPT-4 Technical Report, for example, describes GPT-4 as a Transformer-based model pre-trained to predict the next token in a document. The earlier GPT-3 paper by Brown and colleagues describes the same broad autoregressive objective.
The model receives token sequences and estimates a probability distribution over possible continuations. During training, the actual next token in the example provides the target against which that prediction can be evaluated.
That sounds much less impressive than what modern models can eventually do. But predicting language across billions or trillions of tokens requires the network to capture a remarkable amount of structure: grammar, syntax, semantic relationships, styles, recurring factual associations, code patterns and many regularities that help predict what follows what.
The training objective itself should therefore not be confused with the full set of capabilities that emerge from optimizing a sufficiently large model over sufficiently diverse data.
But What Is a Token?
Before a language model can process text, the text must be converted into discrete units that the model can represent numerically. Those units are called tokens.
A token is not necessarily a word and not an “atomic unit of meaning”. Depending on the tokenizer, it may correspond to an entire word, part of a word, punctuation, whitespace-associated text or another encoded unit.
Different model families use different tokenization schemes, so the same sentence does not have one universal token split. Subword approaches such as SentencePiece, introduced by Kudo and Richardson, illustrate why modern tokenization can operate below the word level and across different languages.
So the following example is deliberately illustrative rather than a claim about the exact tokenizer used by a particular current model:
Illustrative tokenization only:\n[\"Chat\", \"GPT\", \" is\", \" smart\", \"!\"]Once tokenized, the model does not operate on the strings themselves. Each token is mapped into numerical representations that the network can transform. We will deal with embeddings, positional information and attention separately in the next articles.
For now, the useful rule is simply this: tokens are the units the model processes, not miniature words with fixed meanings attached to them.
Now consider the basic prediction step:
Illustrative example:\nInput: \"The capital of France is\"\nPossible next-token probabilities:\n\"Paris\" = 0.91\n\"London\" = 0.02\n\"Berlin\" = 0.01\n...The values above are invented for teaching purposes, but they illustrate the type of quantity involved: the model assigns probability mass across possible next tokens and a decoding procedure determines which continuation is generated.
This does not mean that the model simply looks up which word most often followed the exact phrase in its training corpus. The probabilities result from a large network computation over the available context, using representations and patterns learned across training.
Nor does the next-token objective settle philosophical questions about whether a model “understands”. For the purposes of this research, the more useful distinction is operational: fluent generation does not by itself establish factual knowledge, source provenance or human-like comprehension.
Later stages of training can also substantially alter behavior. Ouyang and colleagues’ InstructGPT work, for example, shows how supervised demonstrations and human preference feedback can tune a pre-trained model toward more useful instruction-following behavior. That post-training process is distinct from the base pre-training objective discussed here.
What Data Is Used to Pre-Train LLMs?
Pre-training requires very large and diverse datasets, but there is no single dataset shared by all LLMs.
Depending on the model and provider, training mixtures can include public web content, licensed material, books, code, documentation, academic text and synthetic examples. The exact composition, filtering and weighting are model-specific—and for proprietary frontier models, often only partially disclosed.
It is therefore better to separate documented examples from assumptions about every current model.
For GPT-3, the original 2020 paper documents a mixture derived from filtered Common Crawl, WebText-style data, books and Wikipedia. The models in that study were trained for a total of 300 billion tokens, with the largest GPT-3 model containing 175 billion parameters.
Other published model families disclose different pipelines and scales. Meta’s Llama 3 technical paper, for example, provides a separate description of its data and training process. These public descriptions are useful examples, not templates that can automatically be assigned to GPT-4, Claude, Gemini or another proprietary model.
Useful public resources that have appeared in large-scale language-model research or related corpora include:
- Common Crawl — a large public crawl of the web that has been used as source material in multiple research datasets.
- Wikipedia — structured encyclopedic text used in many language-model and retrieval experiments.
- Project Gutenberg and other book corpora — examples of long-form text sources used in research datasets.
- GitHub and other code sources — relevant to models trained for programming as well as natural language.
- ArXiv, PubMed and technical Q&A sources such as Stack Overflow — examples of specialized public material that can appear in research corpora.
- Synthetic data — increasingly relevant in modern training pipelines, though its generation, filtering and role vary substantially by model.
Do not infer a proprietary model’s training corpus from a generic list of popular datasets.
GPT-3 is unusually useful because its paper documents substantial details of the training mixture and scale. By contrast, OpenAI’s GPT-4 Technical Report deliberately withholds many architecture and training details. Different disclosure levels require different levels of certainty.
This also corrects another common misconception: pre-training is not simply “memorizing the internet”. Large language models learn statistical regularities that allow them to generalize beyond individual examples.
But generalization does not mean memorization never occurs. Research such as Carlini and colleagues’ work on extracting training data from language models demonstrated that models can sometimes reproduce memorized training sequences. The right description is therefore more nuanced: models learn distributed statistical representations, while some training examples or sequences can also be memorized.
The practical consequence is important. We usually cannot inspect a proprietary model and determine whether one particular webpage was present in its training data merely because the model can discuss its subject.
How Loss and Gradient Descent Change Model Weights
For the network to improve, it needs a numerical signal describing how well its predictions match the training objective. This is the role of the loss function.
In autoregressive language-model training, the model predicts a distribution over the next token. The observed next token in the training example provides the target, and a loss such as cross-entropy loss measures how much probability the model assigned to that correct target relative to the alternatives.
The important correction is that this process does not force the model to store and later reproduce every training sentence exactly.
The loss is differentiated with respect to the trainable parameters, producing gradients that indicate how those parameters should change to reduce future error. Across enormous numbers of examples and training steps, small updates accumulate into a network that assigns better probabilities to language patterns represented in the data.
Gradient-based optimizers perform those updates. AdamW and LAMB are documented examples of optimization methods used in deep-learning systems, but the exact optimizer and training recipe depend on the model.
The scale can be enormous, but it should not be described with one universal hardware count or price tag. Different models vary dramatically in parameter count, dataset size, number and type of accelerators, training duration, engineering efficiency and total compute.
- Large training runs can use substantial accelerator clusters for extended periods.
- Model scale varies by orders of magnitude. GPT-3 alone was published as a family ranging from 125 million to 175 billion parameters.
- Training cost is model-specific. Public estimates or reported research costs can illustrate scale, but should not be turned into a fixed “LLMs cost X million dollars” rule.
The conceptual loop, however, remains simple:
The model predicts.
The loss measures the error.
Gradients show how the parameters should change.
The optimizer updates them.
Repeat this process across enormous numbers of examples, and the network gradually becomes a much better statistical model of the training distribution.
Pre-training gives us the base model. It does not automatically give us the assistant behavior users expect from ChatGPT-like products. Later stages can include supervised fine-tuning, preference optimization and other alignment techniques, which I cover separately in Inside LLMs: RLHF, RLAIF, DPO and KTO.
Frozen Models vs Modular AI Systems
Once a particular trained checkpoint is deployed for ordinary inference, its parameters do not change simply because you send it a prompt, edit a webpage or show it a new document.
This is what people usually mean when they say the model’s weights are “frozen” during inference. It is a useful distinction—but it should not be stretched into the claim that the model can never be adapted later. An authorized operator can retrain, fine-tune or otherwise adapt a model through a separate training process.
More importantly for marketers, modern AI products can surround that relatively stable model with dynamic systems that supply information at runtime.
- Search and retrieval: relevant passages can be fetched from a search index, vector store, documentation collection or other source. The foundational Retrieval-Augmented Generation paper by Lewis and colleagues formalized one influential architecture that combines parametric and non-parametric information.
- Files and long-context inputs: a user can supply a PDF, document or other material directly to a specific interaction or workspace.
- Tools and APIs: the application can call calculators, databases, code execution, business systems or other external services when permitted.
- External memory and knowledge stores: information can persist outside the model weights and be retrieved later according to the product’s storage and permission rules.
This creates the effect that often confuses users: a system can answer with current information even though the underlying model checkpoint did not learn that information through pre-training.
The new information was supplied around the model at inference time.
For marketers, separate the layers.
✓ You can update a source you control.
✓ A retrieval-enabled system may later use that updated source.
✓ Supplied files, memory or tools can change a particular workflow’s answer.
✖ None of those events proves that the provider’s base-model weights were globally updated.
✖ A crawler visit does not mean your page became permanent model knowledge.
✖ One successful answer does not establish the same behavior for every user or future interaction.
This distinction is the bridge from model mechanics to search marketing. If the question is whether a current webpage can affect a search-grounded response, investigate the retrieval path. If the question is whether a provider has incorporated something into its model parameters, that is a different question requiring different evidence.
For the broader control boundaries, see Ranking in ChatGPT: What Search, Retrieval and Model Updates Actually Change. For the page-level path, see how LLM systems extract and cite web sources.
Enjoy Ad Free
Access the full Research Paper. For free.
This article is just an extract from the full 100 pages independent research I’ve written for Fuel LAB® Research over 4 years of analysis, studying LLMs models, and data collection.
Summary
| Element | Description — legacy research scope |
|---|---|
| Model goal | In the autoregressive language-modeling reference, learn the probability of the next token from preceding context. The training text supplies the target; the objective alone does not establish factual correctness. |
| Method | Tokenization converts text to IDs; learned representations and transformer operations produce continuation scores. Causally masked training can process known positions in parallel, while autoregressive generation uses preceding outputs as context. |
| Training data | Model-specific mixtures can include web text, books, code and scientific material. Collection, licensing, filtering, deduplication and sampling affect what is retained. A published or crawled page is not proof of inclusion in a particular training set. |
| Optimization | A next-token cross-entropy loss supplies a training objective. Backpropagation computes gradients; optimizers such as AdamW update trainable parameters. The optimizer, schedule and stopping criteria belong to the specific training setup. |
| Scale | Parameters, training-token exposures and context capacity measure different things. Compare documented model configurations rather than assuming one universal parameter range or equating a context window with the training dataset. |
| Limitations | Ordinary inference can keep deployed parameters fixed; publishing a page, crawling it or supplying it as context does not by itself update those parameters. A fluent answer does not reveal a reliable source trail. |
| Extensions | Fine-tuning or authorized adapters can change effective model computation. Retrieval, tools and external memory can instead supply changing context or stored records without retraining. These are different mechanisms, not automatic permanent learning. |
| Outcome | A parameterized model with learned representations and behavior. Generalization and memorization can coexist; useful performance, factual reliability and provenance must be evaluated for the task rather than inferred from the training objective. |
FAQs
What is pre-training in large language models?
Pre-training is the large-scale optimization stage in which a language model learns statistical structure from training examples by adjusting its parameters. For autoregressive GPT-style models, the core objective is next-token prediction. The resulting model can generalize far beyond any one training sequence, while some memorization can also occur.
Which datasets are used in LLM pre-training?
There is no universal dataset shared by all LLMs. Published research has documented mixtures containing web text, books, code, Wikipedia and other sources for specific models, while proprietary frontier providers often disclose only part of their dataset construction. GPT-3, for example, documented filtered Common Crawl, WebText-style data, books and Wikipedia.
How do LLMs actually learn during pre-training?
The model predicts a distribution over the next token, a loss function measures error against the observed target, and gradients are used to update the trainable parameters. Repeating this across enormous numbers of examples gradually improves the model’s statistical representation of language and other patterns present in the training data.
What is the difference between pre-training and fine-tuning?
Pre-training creates the broad base model through large-scale learning from general data. Fine-tuning and other post-training methods modify an already pre-trained model for instruction following, preferences, safety, domain adaptation or other objectives. Both can change parameters, but they occur at different stages and use different data and objectives.
Can an LLM update itself after pre-training?
A deployed model does not normally update its parameters simply because it receives a prompt or sees a webpage. An authorized operator can later retrain or adapt it. Separately, the wider AI product can use retrieval, files, memory and tools to supply new information at runtime without changing the underlying model weights.
Why does pre-training matter for SEO and AI visibility?
It explains why editing a webpage is not the same as updating another provider’s model weights. For current information, the operational marketing question is often whether a search or retrieval system can access, select and correctly use the maintained source. Training exposure, retrieval visibility, citation, traffic and conversion should be treated as separate events.
Pietro Mingotti is an Italian neural science researcher, entrepreneur and technical marketing specialist, best known as the founder and owner of Fuel LAB®, a leading digital marketing and technical marketing agency based in Italy, operating worldwide. With a passion for science, creativity, innovation, and technology, Pietro has established himself as a thought leader in the field of technical marketing and data science and has helped numerous companies achieve their goals.