with how fast AI models are evolving, I think it is worth stopping for a moment to understand the basics.
today everyone talks about GPT, Claude, DeepSeek, MiniMax, Gemini, Kimi and many other models, but behind a big part of this evolution there is a similar idea: models based on transformers, trained to predict the next token.
when I say GPT here, I am not talking only about ChatGPT.
I am talking about the Generative Pre-trained Transformer architecture, which became one of the most important foundations of modern LLMs.
each lab changes different parts of this system depending on what they want to solve. some focus more on reasoning, others want faster token generation, longer context windows, better tool calling or lower inference cost.
and this changes the final result a lot.
one model can be better for code, another for agents, another for long tasks and another for running at scale with lower cost.
but deep down, the main question is still the same:
how can a model receive text and generate an answer that feels intelligent?
a GPT works in a way that is simple to explain, but very hard to build well.
it receives a sequence of tokens and tries to predict the next token.
then it takes that new token, adds it to the context and predicts the next one.
and it keeps doing this many times until it forms a full answer.
a token can be a word, part of a word, a number, a symbol or even a space.
so when you ask a model a question, it calculates which continuation makes the most sense based on everything that came before.
the impressive part is that this process, when trained at a massive scale, starts generating answers that feel like reasoning.
the first step is turning text into tokens.
the model works with numbers, so each token gets an ID.
but having only an ID is very limited.
if the letter A becomes number 1 and the letter B becomes number 2, that only identifies the token. it does not show context, meaning or relation with other tokens.
that is why token embedding exists.
an embedding is a richer numerical representation of each token.
instead of the token being just one number, it becomes a vector with multiple dimensions.
this vector gives the model space to learn characteristics of that token: where it usually appears, which tokens normally appear near it, how it behaves in different sentences and what kind of pattern it carries.
it is like each token gets more space to carry information.
without this, the model would only have a label.
with embedding, it starts to have a more useful internal representation.
after embeddings, there is still an important point missing: the position of the tokens.
the order completely changes the meaning of a sentence.
"love your job" and "job your love" use almost the same words, but they do not mean the same thing.
that is why the model also needs positional embedding.
it adds information about where each token is inside the sequence.
so the model combines two things: what that token represents and where it appears in the sentence.
this combination helps the model understand the structure of the sequence better.
after the tokens have representation and position, the model needs to understand how they relate to each other.
this is where the attention mechanism comes in.
attention allows each token to look at other tokens in the sequence and discover which ones are more important in that context.
in a long sentence, a word at the end can depend on something that appeared at the beginning.
without a mechanism to capture this relation, the model would lose a lot of information.
attention solves this by creating a mathematical way to measure relevance between tokens.
to calculate attention, the model creates three main representations: Query, Key and Value.
the Query represents what a token is looking for.
the Key represents how other tokens can be found.
the Value carries the information that will be passed forward.
the model compares Query with Key to discover how much one token should pay attention to another. then it uses this weight to combine the Values.
in practice, this creates a kind of relevance map.
if the sequence has 8 tokens, the model can form an 8x8 matrix, where each position shows the strength of the relation between two tokens.
because GPT generates text from left to right, it uses a causal mask.
this means each token can only look at previous tokens.
it cannot look into the future, because during generation the future does not exist yet.
a single attention mechanism already helps, but there is an advantage in splitting this process into multiple heads.
this is called multi-head attention.
instead of having only one way to look at the sequence, the model creates multiple perspectives at the same time.
one head can capture grammar relations, another can look at short-term dependencies, another can capture longer relationships or more abstract patterns.
you cannot say exactly that each head learns one specific function, but the intuition is this: multiple heads allow the model to analyze the same sequence from different angles.
then the results of these heads are combined.
this is what makes the transformer much more powerful than simple attention.
after attention, the feed forward network comes in.
attention allows tokens to exchange information with each other.
the feed forward network processes this information at each position and helps the model transform the representation that was just built.
think about it like this:
attention is the part where tokens communicate.
feed forward is the part where the model works better with the information after that communication.
that is why each transformer block normally has these two parts together.
first attention.
then feed forward.
when many operations are stacked together, the internal values of the model can become unstable.
some numbers grow too much, others become too small, and this hurts training.
layer normalization helps keep these values in a more controlled range.
this makes training more stable and allows many transformer blocks to be stacked without the model breaking so easily.
in large models, this kind of stability is essential.
another important piece is the residual connection.
it creates a path where the original information can skip part of a layer and be added back later.
this helps the model preserve what was already useful and learn modifications on top of the input, instead of transforming everything aggressively at every layer.
the deeper the model gets, the more important this becomes.
without this type of connection, the information could become too distorted as it passes through many layers.
with residual connections, the model can make progressive adjustments in a more stable way.
when we put attention, feed forward, layer normalization and residual connections together, we get a transformer block.
a GPT is basically many of these blocks stacked together.
each block takes the previous representation, processes the relations between tokens better and passes a more refined version to the next block.
the more blocks, the more depth the model has.
and the more depth it has, the more capacity it has to learn complex patterns.
at the end, the model takes the final representation and projects it back into the vocabulary.
from there, it calculates the probability of the next token.
this process repeats many times.
token by token.
until it forms a complete answer.
having the architecture built still does not mean the model knows how to do anything useful.
to actually work, it needs to be trained on a huge amount of data.
during training, the objective is to predict the next token correctly.
the model sees a sequence, tries to guess the next token and compares it with the real token.
when it gets it wrong, the internal weights are adjusted.
this happens billions or trillions of times.
over time, the model learns language patterns, facts, styles, structures, code, basic reasoning and many hidden relationships inside the data.
behind this, there are techniques like gradient descent, optimizers, batching, loss function, learning rate and many other engineering decisions.
this is where each lab really starts to differentiate itself.
having the architecture is not enough.
the data, training, post-training, infrastructure and inference cost change the final behavior of the model a lot.
even when using similar ideas, each lab makes different decisions.
one team may train the model to be better at code.
another may prioritize long context.
another may focus on agents and tool calling.
another may try to make the model cheaper and faster to run at scale.
these choices appear in the final product.
and they also create pressure on the layers underneath.
if the model needs a larger context, it consumes more memory.
if it needs to generate tokens faster, the infrastructure needs to deliver more throughput.
if it needs to run cheaply, the inference system needs to be very efficient.
if it is going to be used by agents, it needs to handle long tasks, tools, history and multiple steps better.
that is why you cannot look at the model in isolation.
the application asks for more capability.
the model tries to deliver.
the infrastructure needs to handle it.
the chips need to keep up.
and the cost needs to make sense.
for me, understanding GPT is important because it removes some of the magic.
from the outside, it looks like the model simply understands everything.
underneath, there is a sequence of well-defined parts: tokens, embeddings, position, attention, QKV, multi-head attention, feed forward, layer norm, residual connections, stacked blocks and training at scale.
all of this exists to solve a task that seems simple:
predict the next token.
the impressive part is that, when this task is done at massive scale, with a lot of data and a lot of engineering, it starts generating behavior that feels like intelligence.
and maybe this is the most important point to understand LLMs.
Antonioni Nascimento
Software, cybersecurity, data & AI. Building advanced systems at KRX. Writing about technology, venture capital, and how software shapes markets.