If you look under the hood of a tool like ChatGPT, you won’t find anything that looks like a traditional brain or a complex reasoning engine. Instead, you will find layer after layer of computational blocks performing basic math.
To understand where AI is heading—specifically the shift toward autonomous, agentic AI—you have to understand how we got here. It turns out that combining simple operations with massive scale creates behavior that looks a lot like intelligence.
The “Dumb” Compute Behind Modern AI
Modern large language models run on a structure called a Transformer. Each Transformer block performs a fixed set of matrix operations on incoming data.
When you ask an AI a question, it breaks your prompt into words and word fragments. It maps each fragment to a vector (a string of numbers), stacks them into a matrix, and passes that matrix through its first Transformer block. The block returns a new matrix of the same size.
This process repeats dozens of times. For example, older models like GPT-3.5 repeat this 96 times, while GPT-4 reportedly repeats it 120 times.
The next word the AI generates is simply the last column of its final output matrix, mapped back into text. It adds that new word to your original prompt and runs the entire process again. One matrix multiplication after another, it slowly morphs your input into a complete response.
So where does the intelligence come from? The answer lies in how these models are trained and how they learn to organize information.
The 2012 Turning Point
To see how these math blocks learn, it helps to look at computer vision. In 2012, a model called AlexNet shocked the computer vision community. Before AlexNet, winning teams used complex, hand-coded algorithms developed by human experts over decades. AlexNet, however, was built on an old idea: the artificial neural network.
Instead of predicting the next word like a text model, AlexNet predicted image labels. You feed it a picture, and it outputs a probability score for 1,000 different categories, from hot dogs to aircraft carriers.
The first five layers of AlexNet are convolutional blocks. You can think of them as a special, earlier version of the Transformer blocks used today. They work by sliding a small grid of learned numbers, called a kernel, across an image. At each stop, the kernel computes a dot product between itself and the image patch.
Here is a simple way to think about it: the dot product acts as a similarity score. If the image patch looks like the kernel, the score is high.
Learning Without Explicit Instructions
The magic happens as data moves deeper into the network.
- Layer 1: The model learns to detect basic edges and rapid changes from light to dark.
- Layer 2: It combines those edges to form corners and basic shapes.
- Layer 5: The model responds strongly to high-level concepts, like faces or specific animals.
No one explicitly programmed AlexNet to know what a face is. The training data didn’t even have a “face” or “person” category. The model learned entirely on its own that faces are important visual features for classifying images, and it learned how to recognize them.
Organizing the World in High Dimensions
By the time an image reaches the end of AlexNet, it is represented as a single vector—a point in a 4,096-dimensional space. Every image fed into the model maps to a specific coordinate in this massive space.
Researchers found that this high-dimensional space (often called an embedding space or latent space) is organized logically.
If you measure the distance between the coordinates of two images, you find that similar concepts physically group together. An image of an elephant sits right next to other images of elephants, even if the actual pixels in the photos look completely different. The model understands the underlying concept.
We can visualize these complex embedding spaces using tools called Activation Atlases. These 2D maps show how the neural network organizes the visual world. As you pan across an Activation Atlas, you can watch the model’s understanding shift smoothly from zebras, to tigers, to leopards, to rabbits.
The same principle applies to modern text models. Words and phrases map to vectors in an embedding space where similar meanings cluster together. Directionality matters, too. Researchers have found that if you identify the specific activations for a concept—like the “Golden Gate Bridge”—and artificially force those values high, the AI will actually change its behavior and begin identifying itself as the bridge.
Scale and the Shift to Agentic AI
The algorithms driving this revolution are not new. The basic dot product operation used in these networks was proposed in the 1940s as an oversimplified model of human neurons. The calculus technique used to train them, backpropagation, was applied to multiple layers in the 1980s.
For decades, the broader computer science community dismissed this approach. The algorithms were simply too computationally expensive for the hardware of the time.
The difference today is the sheer scale of data and compute power.
| Era | Model | Learnable Parameters |
|---|---|---|
| 1990s | LeNet-5 | ~60,000 |
| 2012 | AlexNet | ~60,000,000 |
| Today | Modern LLMs | 1,000,000,000,000+ |
This massive scale is what turns simple pattern-matching into the foundation for agentic AI. We are moving past models that just answer questions or categorize images. Agentic AI refers to systems that can plan, use external tools, and execute multi-step workflows autonomously. They can do this because they have built incredibly rich, high-dimensional representations of how language, code, and logic intersect.
Predicting where AI goes next is notoriously difficult. A generalization of the compute blocks in AlexNet, scaled up by four orders of magnitude, gave us systems capable of writing code and drafting essays. The next leap toward fully autonomous agentic AI might simply require another massive increase in scale, or it might come from an old, forgotten algorithm resurfacing once again.
But at its core, the intelligence you see is the result of simple operations combining in complex ways, learning the structure of our world from the data we feed it.
Sources
- https://www.youtube.com/watch?v=UZDiGooFs54
- https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf