Inside LLMs: Understanding Transformer Architecture – A Guide for Marketers

Last updated on September 20th, 2026 at 02:39 pm

The previous articles introduced pre-training and embeddings, positional information and attention. Now we can put those pieces together.

What actually happens between token representations entering a language model and the model producing scores for the next token?

The answer matters because transformer architecture attracts unusually convenient marketing myths. Attention becomes “make these words bold”. Learned associations become “repeat the brand until the model remembers it”. Mixture of Experts becomes “write for the marketing expert”.

None of those recommendations follows from the architecture.

Internal mechanisms can help explain what a model is capable of doing with the information available to it. They do not automatically tell us how an external search or retrieval system selected that information, whether the source will be cited, or which publishing intervention would change the outcome.

A transformer performs repeated numerical transformations that can support language generation, factual recall, abstraction and problem-solving. Describing those transformations does not settle philosophical questions about human-like understanding. Conversely, dismissing the result as “only prediction” does not evaluate the demonstrated capability.

Research context: adapted from AI-Optimized Marketing: Understanding and Influencing LLMs, edition 1.5.0-LEGACY — Frozen Outlook. The research concerns the 2023–2025 period; its evidence review and master revision are dated 17 September 2026.

How to read this in 2026: the concepts below remain foundational, but there is no single modern LLM architecture that can simply be called “the transformer”. Published systems use different normalization placements, positional mechanisms, attention variants, dense or sparse feedforward layers, multimodal components and routing strategies. Proprietary systems can withhold important architectural details.

The purpose of this article is therefore to understand durable mechanisms and their boundaries—not to reconstruct a current commercial model from a 2017 diagram.

The Transformer Stack: What Each Block Contributes

The original Transformer paper by Vaswani and colleagues introduced an encoder–decoder architecture based on attention rather than recurrence. Modern generative LLMs are commonly explained through decoder-style causal computation, but even within that family the exact block design varies.

A useful contemporary teaching model is a block containing two major transformations:

1. Self-attention lets a position incorporate information from other permitted token positions.
2. A feedforward network, or MLP, applies a learned nonlinear transformation to the representation at each position.

Residual paths and normalization are then used around these transformations.

One common pre-normalization arrangement can be summarized as u = h + Attention(Norm(h)), followed by h_next = u + FeedForward(Norm(u)). The original 2017 Transformer instead used a post-normalization arrangement. This distinction matters because “the transformer block” is not one immutable equation; normalization placement itself became an architectural design choice.

Xiong and colleagues’ analysis of Pre-LN and Post-LN Transformers is a useful reference for why the placement of normalization affects training dynamics.

Reading the pre-normalized example step by step:

1. Normalize the incoming representation and apply attention. Queries and keys determine compatibility between permitted positions; weighted values contribute new information.
2. Add the attention result back through the residual path. The incoming representation remains available while the layer contributes a learned update.
3. Normalize again and apply the MLP. The MLP transforms each already-contextualized position through learned nonlinear computation.
4. Add that result through the second residual path. The resulting representation moves to the next block, where the process can continue.

Eventually, the final representation is projected into scores over possible output tokens.

That description is deliberately generic. Architectures can change normalization, activation functions, attention mechanisms, positional systems, feedforward designs and routing. A diagram of one block is therefore an explanatory model, not a specification for every commercial assistant.

The same caution applies to depth. The old claim that GPT-4 contained “96+ layers” was not supported by OpenAI’s public technical report and should not be repeated. Layer counts are model-specific facts that require disclosure, not guesses from apparent intelligence.

It is still useful to think of a deep stack as repeated transformation. But do not turn depth into a fictional organizational chart:

• Representations change through depth. Signals relevant to token identity, syntax, position, entities or learned associations can become available in different forms.
• Different components perform different operations. Attention moves and mixes information across permitted positions; feedforward networks transform representations locally at each position.
• A detected pattern is not automatically a causal explanation. Finding information in a layer does not prove that the model used it to produce the behavior being studied.

The network does not contain a neatly separated “grammar department”, “facts department” and “reasoning department”. Behaviors can depend on distributed and interacting computations across many components.

Think of the stack as a sequence of workstations repeatedly transforming a shared representation. The analogy is useful for composition, not for assigning one human skill to each floor—or inferring an undisclosed layer count from the model’s behavior.

1. Residual Connections: Carrying Information Across Depth

A residual connection adds a learned transformation back to its input. In simplified form:

h + F(h)

If h = (1, −1) and an invented transformation produces F(h) = (0.2, 0.3), the residual result becomes (1.2, −0.7).

The arithmetic is trivial. The architectural consequence is important: a sublayer does not have to reconstruct everything that arrived before it. It can contribute a change while an identity path carries the previous representation forward.

The useful intuition is: keep the incoming representation available, then add a learned update.

This is not the same as a memory database, a chat-history store or permanent preservation of every fact.

Residual paths are also important for optimization because information and gradients have an identity route through deep computation. They help make deep networks trainable, but they are one part of a larger optimization problem involving initialization, learning rates, normalization, data and other design choices.

This distinction also prevents another terminology trap. The residual stream is an internal computational representation. Product memory is external state stored by the surrounding application and later supplied to the model. Both can affect an answer; they are not the same mechanism.

2. Normalization: Conditioning Activations, Not Checking Facts

Normalization operates on numerical activations. It does not inspect whether the proposition represented by those activations is true.

Layer Normalization, introduced by Ba, Kiros and Hinton, computes normalization statistics across features of an activation for an individual example, then applies learned gain and bias parameters.

In simplified notation:

(h − mean) / sqrt(variance + epsilon)

For the illustrative vector (2, 4, 6), the mean is 4 and the variance is 8/3. Ignoring the stabilizing epsilon and learned adjustment gives approximately (−1.225, 0, 1.225).

Other architectures use different normalization methods. RMSNorm, for example, omits mean-centering and normalizes using the root mean square.

The important point is not which normalization acronym is fashionable. It is that normalization is an architectural mechanism for managing activations and optimization dynamics.

The sound-mixer analogy is useful within limits. Normalization conditions the numerical signal moving through the network. A perfectly conditioned signal can still contribute to a false answer, just as a perfectly mixed recording can contain a false statement.

Numerical conditioning and factual verification solve different problems.

MLPs: Learned Transformations and Factual Associations

The feedforward network—often called the MLP or FFN—is the other major transformation inside a conventional transformer block.

A common design expands the representation into a wider intermediate space, applies nonlinear operations and then projects it back toward the model width. This happens independently at each token position, but the representation being transformed is already contextual: attention and earlier layers have influenced it.

Research has found particularly interesting relationships between feedforward computation and learned factual associations.

Geva and colleagues analyzed transformer feedforward layers through a key-value-memory interpretation, while Meng and colleagues’ ROME work used causal tracing and targeted weight edits to investigate factual associations in specified GPT models.

These studies support a much more careful conclusion than “MLPs are the model’s factual database”.

• Parameters can participate causally in factual associations. A familiar fact can be produced without a runtime search or database lookup.
• Representation is distributed. A useful association does not imply one isolated neuron containing one human-readable fact.
• Editing demonstrates causal leverage under controlled access. It does not mean the model exposes a public interface through which publishers can rewrite those parameters.
• Successful generation is not provenance. Producing “Paris” in response to a capital-city prompt does not identify which training document contributed to that association.

A neural-network “neuron” is a mathematical component, not a biological cell. An MLP contains many units and parameter matrices, and interpretability techniques may extract features that span combinations of those units.

A more defensible picture is a learned system of interacting representations rather than a shelf of tiny switches, each holding one fact.

Researchers with model access can intervene on parameters and test causal effects. Editing a webpage is a different intervention entirely.

Sparse Computation with Mixture of Experts (MoE)

A dense feedforward layer evaluates the same feedforward network for each token. A Mixture-of-Experts architecture instead provides several expert networks and uses a learned router to activate only a subset for a particular token or representation.

The word expert creates an unfortunate metaphor. An expert is a parameterized computational pathway, not necessarily an independent language model and not a human-style specialist labelled “medicine”, “French”, “SEO” or “marketing”.

Mixtral 8x7B provides a particularly clear documented example. Each sparse feedforward layer contains eight experts, and its router selects two experts for each token before combining their outputs.

That is Mixtral’s architecture, not the definition of MoE.

Later published models use different arrangements. This is precisely why “two experts per token” should never become a generic explanation of all MoE systems.

An illustrative routed result might combine 0.6 × Expert A and 0.4 × Expert C. Those values are invented.

The point is conditional computation: the architecture can expose more total parameter capacity than it actively evaluates for one token.

This changes the relationship between total capacity and active computation, but it does not make additional capacity free.

• Parameters still need to be stored. Sparse activation does not eliminate model memory requirements.
• Routing has its own costs. Communication, expert placement and load balancing matter in real systems.
• Experts can differentiate without receiving human job titles. Any claim about specialization requires empirical analysis of the particular model.

For marketers, this boundary matters more than the implementation detail. Nothing here establishes the existence of a “marketing expert” that a public webpage can deliberately activate through its wording.

Clear names, explicit product conditions and well-structured explanations can still be valuable because they improve the information supplied to a system. Their usefulness should be demonstrated through retrieval and output evaluation—not retroactively justified with an MoE metaphor.

Directionality, Causal Masking and Token Generation

A GPT-style causal decoder generates text autoregressively: each new token is predicted from the sequence that is available before it.

During training, a causal mask prevents a position from looking ahead at later target tokens. This makes it possible to compute many training positions in parallel without leaking the answer from the future.

During ordinary generation, the sequence dependency becomes visible. The model computes scores for the next token, a decoding procedure selects one, and that token becomes part of the context for the next prediction.

These statements are compatible:

• transformers permit substantial parallel computation over an already available sequence;
• autoregressive generation still depends sequentially on previously generated tokens.

After the final transformer computation, an output projection produces logits over the vocabulary. Softmax can convert those scores into a probability distribution, after which the decoding procedure determines the continuation.

The model computes scores and probabilities over possible tokens.

It does not thereby compute a calibrated probability that every proposition expressed by the resulting sentence is true.

This is why lowering sampling randomness should not be described as a factuality mechanism. A highly probable continuation can repeat a misconception, preserve an incorrect premise or confuse two similar products. A low-probability token can belong to the correct answer when the required name is unusual.

Generation, memorization and factual correctness are different properties.

BERT provides a useful historical contrast because its bidirectional representation objective allows positions to use context on both sides. GPT-style causal language models use a different visibility pattern designed for autoregressive continuation. That difference should not be inflated into the claim that bidirectional components can never appear within a larger generative system.

For website publishing, causal generation establishes no universal paragraph length, heading formula, bold-text advantage or instruction to remove nuance. Those claims require evidence from the actual retrieval and answer-generation system.

Interpretability: What We Can—and Cannot—See Inside a Model

Interpretability research asks a harder question than “which neuron lights up?” It tries to understand what internal representations encode and whether particular computations actually contribute to a behavior.

Those are different levels of evidence.

• Probing: can information be recovered from an internal representation? Recoverability does not prove that the model uses that information to produce the behavior.
• Feature decomposition: methods such as sparse autoencoders can identify interpretable directions or features in activation space. Those features are outputs of an analysis method, not necessarily original individual neurons.
• Circuit analysis: researchers can investigate how several features and operations interact along a computational path.
• Causal intervention: changing an activation, component or parameter and observing a corresponding behavioral change provides stronger evidence about causal contribution.
• Parameter editing: changing learned weights and evaluating both the target effect and collateral effects asks yet another question.

Anthropic’s 2024 feature-mapping work on Claude 3 Sonnet provided an important example of extracting interpretable features from a production-scale model. But a feature map is not a complete explanation of the model.

Interpretability research has continued beyond that stage. Work published in 2025 connected features into partial computational circuits and used interventions to test how those circuits relate to behavior. That progress strengthens the case for causal analysis while reinforcing the same limitation: even sophisticated interpretability tools reveal only part of the computation, and the methods themselves can introduce artifacts.

That distinction matters because attractive visualizations encourage overclaiming.

If an analysis identifies a “Golden Gate Bridge feature”, it does not follow that the model contains a database row named Golden Gate Bridge. If changing a feature changes the output, that tells us something stronger about causal influence—but still under the specified model, prompt, intervention and measurement.

Interpretability therefore supports several legitimate research goals:

1. Diagnose a defined behavior. Locate relevant representations or circuits and test whether intervention changes the outcome.
2. Evaluate controlled adaptation. Determine whether a change produces the intended effect without unacceptable side effects.
3. Test mechanistic hypotheses. Move from correlation toward causal evidence instead of turning a heatmap into a story.

What interpretability does not provide is a universal provenance trail from a generated sentence back to every training source that contributed to it.

For a marketing team operating its own model, internal analysis may complement task-level evaluation when the required model access exists. For a publisher dealing with an external answer engine, the observable evidence is usually much further downstream: which source could be retrieved, what representation reached the system, what was generated, what was cited and whether the final claim was correct.

The architectural lesson is not that influence is impossible, nor that interpretability has revealed a secret targeting interface.

Each intervention acts on a different object.

A source edit changes a source. A parameter edit changes model computation. A retrieval rule changes source selection. A citation rule changes application behavior. Their effects must be tested separately.

Summary

Component	Function and boundary
Transformer blocks	Compose attention and feedforward transformations across depth. Their layout varies; a simplified diagram does not disclose a proprietary model’s architecture.
Residual connections	Add a transformation to an incoming representation and support information and gradient paths. They are not cross-conversation memory.
Normalization	Condition activation statistics or scale, depending on the method. LayerNorm and RMSNorm differ; neither checks factual truth.
MLPs and MoE feedforward pathways	Apply learned nonlinear transformations; some architectures route selected expert contributions. Learned associations do not imply a one-fact-one-neuron database or cost-free extra capacity.
Causal masking and output generation	Restrict access to later target tokens in causal modeling; an output head and decoding procedure produce continuations. Token probabilities are not website ranks or calibrated truth scores.
Interpretability methods	Investigate representations and, with interventions, their role in behavior. A feature map is not a complete provenance record or a public brand-placement control.

A transformer repeatedly transforms contextual representations through mechanisms such as attention, feedforward computation, residual paths and normalization. Some architectures add sparse expert routing or other components. Exact implementations vary.

These mechanisms can help explain capabilities and specific failures. They do not independently explain truth, retrieval, citation or commercial visibility.

The practical lesson is to identify the object you can actually observe or change. Architecture describes model computation; search and retrieval determine what information may reach that computation; application logic can shape what the user finally sees.

Continue with RLHF, RLAIF and preference optimization to see how behavior can be shaped after pre-training. For the complete argument, worked examples and source trail, the reference remains the full 1.5.0-LEGACY research edition.

FAQs