Last updated on September 20th, 2026 at 02:00 pm
TL;DR
- Neural networks transform numerical representations through learned operations. Training changes parameters; ordinary inference uses those parameters to compute new activations.
- Self-attention lets token positions combine information from other permitted positions through learned queries, keys and values. Attention weights are not truth scores, website-quality scores or citation probabilities.
- Transformers make many computations over available sequence positions parallelizable. Autoregressive generation still extends an answer token by token.
- Causal masking prevents a token position from using future positions during that computation. Later generated tokens can condition on everything that is already present in their permitted context.
- Embeddings and positional mechanisms do different jobs. Embeddings provide numerical representations; positional mechanisms represent sequence structure. Retrieval embeddings are also not necessarily the generator’s own token embeddings.
- For marketers, attention is not a ranking formula. Finding a page, extracting a passage, using its information and citing it are different stages that can involve systems outside the model itself.
- Today’s reading: these transformer fundamentals remain useful, but modern models can use later architectural variants such as RoPE, grouped- or multi-query attention and other proprietary changes. Do not assume that one published architecture describes every current product.
What happens between a sentence entering a language model and the model producing an answer?
At the transformer level, three ideas do much of the explanatory work: numerical representations, attention and sequence structure. Together they help explain how information in the available context can interact while a model computes its next output.
For marketers, this is useful for a very specific reason. It prevents us from confusing what the model does with information it already has with how an AI product found that information in the first place.
A page may first need to be crawled, retrieved, parsed or supplied as context. Only then can the transformer computation operate on the representation it receives. And even after that, correct use of the information and correct attribution remain separate questions.
This article adapts chapter IV.B of AI-Optimized Marketing: Understanding and Influencing LLMs, 1.5.0-LEGACY — Frozen Outlook. The research concerns the 2023–2025 transition from search results to generated answers; its evidence review and master revision are dated 17 September 2026. The research series places this chapter within the wider technical explanation.
How to read this in 2026: this article explains durable transformer concepts, not the complete architecture of a current ChatGPT, Claude, Gemini or other frontier product. The 2017 transformer used multi-head attention and sinusoidal positional encoding; later research introduced alternatives such as rotary positional embeddings (RoPE), grouped-query attention and multi-query attention. Providers may also use undisclosed variants. The purpose here is to understand the mechanism well enough to reason correctly about content and retrieval—not to infer proprietary internals from a product name.
Enjoy Ad Free
Access the full Research Paper. For free.
This article is just an extract from the full 100 pages independent research I’ve written for Fuel LAB® Research over 4 years of analysis, studying LLMs models, and data collection.
What Are Neural Networks in Large Language Models?
A neural network is a computation whose behavior depends on learned numerical parameters.
A simple feedforward layer can be written as h = f(Wx + b). Here, x is an input vector, W and b are learned parameters, and f applies a transformation. Deep networks repeat and combine operations like these across many layers.
The resulting activation is a numerical state produced by the computation. Calling these systems “neural” does not mean that an artificial unit is equivalent to a biological neuron, or that one activation corresponds cleanly to one human concept.
Language introduces an additional problem: what a token means in context may depend on information much earlier in the sequence.
Before transformers, recurrent architectures were a major way of handling sequential information. A recurrent neural network updates a state as it processes successive inputs. Long short-term memory networks (LSTMs), introduced by Hochreiter and Schmidhuber in 1997, added gating mechanisms designed to improve the handling of longer dependencies.
- Earlier information must survive successive state updates. That is not the same thing as keeping an intact copy of all earlier text.
- Long dependency paths complicate optimization. Gradient signals can weaken across repeated transformations; LSTM gating helps address this problem without providing unlimited memory.
- The recurrent dependency constrains parallelism. A later state depends on an earlier one, so many sequence positions cannot be processed independently at the same stage.
Attention itself arrived before the transformer. Bahdanau, Cho and Bengio’s neural machine translation work, first submitted in 2014, allowed a decoder to build a weighted combination of source representations instead of relying exclusively on one fixed-length sentence vector.
The major architectural break came in 2017, when Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin introduced the transformer in Attention Is All You Need.
The transformer removed recurrence from the sequence-transformation architecture described in that paper and made attention central to communication between positions. The practical effect was not that every token suddenly “understood” every other token. It was that information could move between permitted positions through a much shorter computational path, while many operations over already available positions became parallelizable.
This needs one important qualification: parallel processing of an available context is not the same thing as generating an entire answer at once. A GPT-style autoregressive model still generates a continuation token by token. Each new token then becomes part of the context available for subsequent predictions.
The transformer was therefore a major change in computational structure, not a shortcut from fluent language to verified knowledge.
Consider this illustrative sentence: “The cat sat on the mat, but it ran away.”
The model can compute relationships among the token representations available to a given position, which may help associate “it” with “the cat” rather than “the mat”. But there is no guarantee that one attention head is permanently assigned to pronouns, nor that the model will resolve every reference correctly.
How Does Self-Attention Work?
Self-attention lets each permitted position form a context-dependent combination of representations from the same sequence.
In the standard scaled dot-product formulation, learned projections of an input matrix X produce queries, keys and values: Q = XW_Q, K = XW_K and V = XW_V.
The terminology is unfortunate for readers arriving from search marketing. These “queries” and “keys” are mathematical objects inside the network. They are not search queries, keyword lists, database keys or commercial values.
The standard operation can be written:
softmax(QKᵀ / √d_k + M)V
where d_k is the key-vector dimension and M represents the relevant mask. The calculation is easier to understand as three stages:
- Compare. A query representation is compared with key representations to produce compatibility scores.
- Constrain and normalize. The mask removes positions that are not allowed to contribute, and softmax converts the permitted scores into normalized weights.
- Combine. Those weights determine how value representations contribute to the output of that attention operation.
What the operation does not produce is equally important. An attention weight is not a confidence that a sentence is true. It is not a website-authority score. It is not a probability that a source will be cited.
A constructed example from the research: scaled scores of (2, 1, 0) become approximately (0.665, 0.245, 0.090) after softmax. With value vectors (1, 0), (0, 1) and (1, 1), the weighted result is approximately (0.755, 0.335). These are invented teaching values, not activations recovered from a production model.
Later architectures modify the standard mechanism for efficiency or longer-context operation. For example, grouped-query attention (GQA) allows groups of query heads to share key/value heads, while multi-query attention shares keys and values even more aggressively during decoding. These are engineering changes to the attention mechanism, not evidence that one architecture is automatically more factual or more useful for a particular website.
Long context is also not persistent memory. A larger context window makes more information available to a particular computation. It does not mean that the model permanently learns the supplied content into its weights or will retain it for unrelated future conversations.
Multi-Head Attention
The original transformer does not run attention only once. It uses multiple attention heads with different learned projections and combines their results.
This gives one layer several representational perspectives on the same available context. Researchers have observed heads and circuits associated with particular grammatical or reference patterns, but it is safer to think of these as learned computational behaviors than as permanently named modules.
- Syntax: some learned patterns can relate grammatical roles and agreement.
- Reference: attention patterns can contribute to relationships such as a pronoun and a possible antecedent.
- Position and context: different heads can capture different relationships among parts of the available sequence.
That does not imply the existence of a clean “world-knowledge head”, “SEO head” or “brand head” that a publisher can deliberately target.
The head outputs are concatenated and passed through an output projection before later transformations continue processing the representation.
A useful analogy is several readers highlighting different relationships, followed by a combination of their notes. But attention heads are learned mathematical operations, not people with intentions or guaranteed specialties.
An attention heatmap should also not automatically be treated as a causal explanation. Jain and Wallace (2019) challenged that interpretation in their experiments, while Wiegreffe and Pinter (2019) argued that the broader question is more nuanced. The useful lesson is methodological: test an explanation rather than assuming an attractive visualization proves it.
Positional Encoding: Giving Order to the Computation
Attention compares representations according to their content, but language also depends on order. Changing the order of the same words can change the proposition entirely.
“Paris is the capital of France” ≠ “France is the capital of Paris”
The original transformer therefore added sinusoidal positional encodings to its token representations. Later architectures introduced different ways of representing sequence position.
A particularly influential example is Rotary Position Embedding (RoPE), which rotates query and key representations in a way that incorporates relative positional information into their interaction.
For the reader, the architectural detail matters less than the conceptual boundary: position signals tell the network something about sequence structure; they do not tell it whether the content is true, fresh or authoritative.
Do not translate “position matters inside a transformer” into “the first paragraph receives a guaranteed AI ranking boost”. A retriever may extract a passage from the middle of a document, a table may be transformed before it reaches the model, and a chunk may begin far from the top of the original page. The relevant question is what representation actually entered the context.
Embedding Layers: From Tokens to Vectors
Before attention can operate, text must be represented numerically.
A tokenizer first maps the input to token IDs. The model then looks up a learned vector for each ID. If the vocabulary contains V token IDs and the representation width is d, the embedding table can be thought of as a V × d learned matrix.
- Tokenization: text becomes a sequence of token IDs. Token boundaries can correspond to whole words, subwords, punctuation or other encoded units.
- Embedding lookup: each token ID is associated with a learned numerical vector that enters the model computation.
The input embedding should not be confused with the complete contextual meaning of the token.
The token “bank”, for example, can begin from the same token embedding in different contexts. As the representation moves through transformer layers, its state can change according to the surrounding context. The distinction between a river bank and a financial bank therefore emerges from contextual computation, not from two dictionary definitions stored inside the initial vector.
Vector representations can nevertheless contain striking relational regularities. The specific source behind the famous example is Mikolov, Yih and Zweig’s 2013 paper Linguistic Regularities in Continuous Space Word Representations, which showed that learned word vectors could capture relation-specific offsets such that king − man + woman produced a vector close to queen.
Illustrative vector analogy: the expression below is mathematical shorthand, not executable JavaScript and not a universal algebra of meaning.
vec("king") - vec("man") + vec("woman") ≈ vec("queen")The approximate sign matters. The relationship depends on the learned representation, dataset and comparison method; the equation is not a universal property of every embedding space.
Similarity must also be distinguished from factual support. “The product supports offline export” and “the product does not support offline export” can occupy closely related semantic territory while asserting opposite facts. Retrieval or representation systems still need to preserve the negation and its context.
Token embeddings and retrieval embeddings are different components. A retriever can represent passages using one model and representation space, then pass the selected text to a separate generator. Lewis and colleagues’ RAG architecture is a classic example of parametric generation combined with external non-parametric retrieval.
Related representation techniques can also connect modalities. CLIP, for example, learns image and text representations that can be compared through its training objective. That does not imply that every multimodal assistant puts text, images, audio and video into one universal vector space. Each system needs to be described and tested on its own terms.
What Attention Does—and Does Not—Explain About Content Visibility
This is where the technical discussion reconnects with marketing.
Attention explains how representations already inside the model’s permitted context can interact. It does not explain by itself why your webpage entered that context.
A source-to-answer workflow can fail at several earlier or later stages:
- The page may not be available to the relevant retrieval system.
- The wrong passage may be retrieved or reranked above the useful one.
- A parser may detach a qualification from the claim it limits.
- The generator may misinterpret information that was correctly supplied.
- The final interface may omit or misassign attribution.
Calling all of these an “attention problem” makes the architecture sound more explanatory than it is.
The separate article on how LLM systems extract and cite sources follows that information path in detail.
1. Check what information reached the system.
A page existing on the web does not prove that the relevant passage entered the model context.
2. Preserve the conditions that make the answer correct.
Keep entity, product version, scope, negation, units and supporting evidence close enough to survive extraction.
3. Measure use and attribution separately.
Check whether the answer preserved the fact and whether the displayed source actually supports the statement. Record the product surface, prompt, date and relevant conditions and repeat the test.
These are practices to evaluate, not ranking factors derived from the attention equation. The content-optimization article develops the editorial workflow without pretending that headings, bold text or repeated brand names provide direct control over model internals.
The practical boundary: clarity makes information easier to inspect, extract and test; it does not give a publisher an attention-setting dial. The full 1.5.0-LEGACY research connects these mechanisms to retrieval, citations, measurement and the wider marketing implementation framework. This article explains one technical layer of that system, not a formula for guaranteed AI visibility.
Enjoy Ad Free
Access the full Research Paper. For free.
This article is just an extract from the full 100 pages independent research I’ve written for Fuel LAB® Research over 4 years of analysis, studying LLMs models, and data collection.
Summary
| Concept | Role in transformer-based LLMs |
|---|---|
| Self-attention | Combines value representations using learned query/key compatibility over permitted positions. Its weights are not truth or citation scores. |
| Multi-head attention | Uses several learned projections and combines their outputs. Heads are not guaranteed human-readable specialists. |
| Positional encoding | Supplies sequence-position information. It is not a freshness signal or a universal rule that the first paragraph receives more weight. |
| Embeddings | Map token IDs to learned numerical vectors; later layers make representations contextual. Retrieval embeddings can belong to a separate encoder. |
| Transformers | Use attention and feedforward transformations instead of the original recurrent sequence path. Parallel processing of known positions differs from token-by-token generation. |
| Causal masking | Prevents an autoregressive position from using future tokens. Later predictions can use the complete preceding context. |
| Context and key/value cache | Provide an input budget and reusable computation state. Neither automatically updates model weights or supplies persistent cross-session memory. |
FAQs
What are neural networks in large language models?
They are computations with learned parameters that transform numerical representations through many layers. Training adjusts those parameters; inference uses them to compute activations for the supplied context. Artificial neurons and activations should not be treated as direct equivalents of biological neurons or complete human concepts.
What is the attention mechanism in LLMs?
Attention forms context-dependent combinations of numerical representations. In scaled dot-product attention, learned queries are compared with keys; scores are scaled, masked and normalized; the resulting weights combine values. Attention weights are not probabilities that a statement is true, a website is authoritative or a source will be cited.
How do transformers use self-attention?
Self-attention allows permitted token positions within a sequence to exchange information through learned projections. Multiple heads provide several representational perspectives. In a causal decoder, a position cannot use future tokens. Parallel computation over available context does not mean that the entire unknown response is generated simultaneously.
How are transformers different from recurrent networks such as RNNs and LSTMs?
Recurrent architectures propagate state sequentially through the input. The transformer architecture introduced in 2017 removed that recurrent path and used attention to exchange information between positions, enabling substantially more parallel computation over available sequences. Autoregressive generation still extends the output token by token.
What should marketers take from attention and embeddings for GEO or AEO?
The main lesson is to separate retrieval, extraction, generation and attribution. The attention equation does not prove that headings, bold text, repetition or any other page feature directly increases citation probability. Clear, scoped information is useful because it can be inspected and tested; actual visibility must be measured in the relevant product and information path.
Is the original 2017 Transformer architecture still the architecture used by modern LLMs?
It remains foundational, but modern model families can use later variants in positional encoding, attention, routing, context handling and other components. Examples include RoPE, multi-query attention and grouped-query attention. Proprietary providers may not disclose their complete architecture, so the 2017 paper should be read as a foundation rather than a blueprint of every current system.
Continue the technical explanation: how attention, feedforward layers, residual connections and normalization fit inside a transformer. The series overview provides the complete reading path.
Pietro Mingotti is an Italian neural science researcher, entrepreneur and technical marketing specialist, best known as the founder and owner of Fuel LAB®, a leading digital marketing and technical marketing agency based in Italy, operating worldwide. With a passion for science, creativity, innovation, and technology, Pietro has established himself as a thought leader in the field of technical marketing and data science and has helped numerous companies achieve their goals.