Artificial Intelligence is on everyone’s lips; recently, Hinton and Hopfield were awarded the Nobel Prize in Physics for their contributions to the field. While much is said about its capabilities, few truly understand what lies beneath: what hardware does it require, and what does its future hold? Join us as we explore these and other questions surrounding Artificial Intelligence.
Artificial Intelligence: from the Brain to Silicon
Artificial Intelligence (AI) consistently dominates the headlines in both specialized and mainstream media. Although it has recently caused a major societal disruption, it might just be one of humanity’s most deeply yearned-for inventions. Throughout history, human beings have continuously developed, devising increasingly complex solutions and tools for survival starting from the invention of the earliest implements.
As projects grew in scale, so did the ambition to create more sophisticated things. As early as the 1st century AD, Hero of Alexandria envisioned the creation of intelligent artificial life in his treatise “Automata,” which is widely regarded as the first book on robotics in history. Many writers followed in his footsteps, imagining and fantasizing about the possibility of intelligent beings capable of developing, communicating, and perhaps even alleviating human loneliness.
Artificial Neural Network
The term AI encompasses a vast array of technologies; however, Large Language Models (LLMs) have garnered the most attention.
These models consist of a collection of artificial neurons grouped into layers and interconnected with one another. In the late 1950s, Frank Rosenblatt developed the Perceptron, forming the foundational basis for the artificial neurons that populate much of modern AI.
The Perceptron is a mathematical model consisting of a linear combination of its numerical inputs and assigned weights for each input, to which a non-linear function—called the activation function—is ultimately applied to de-linearize the result.
The combination of these artificial neurons enables the creation of systems capable of inferring results from highly complex functions. The challenge with these systems lies in determining the value of the weights assigned to each input of every neuron. The process of finding these values using a large dataset of known information is called supervised learning, and various strategies have been developed over the years to determine these specific weights. The implementation of mathematical optimization techniques, such as gradient descent and backpropagation, enabled the creation of increasingly larger artificial neural networks as early as the 1980s.
This increasingly sophisticated application has led to advancements in image recognition, the ability to generate novel text and images, and even the capacity to produce illustrations or paintings mimicking the specific style a particular artist would use if they had drawn a user-provided photograph.
One of the major hurdles researchers have faced during these seventy years of work is the immense computational load required to solve these massive mathematical models. A modern LLM can contain billions of weights, meaning that obtaining an inference requires executing an overwhelming number of mathematical operations. Furthermore, training these models, which relies on a colossal amount of data, can take weeks or even months of intensive work on the highest-performance supercomputers. This inevitably leads us to question the very foundation of this engineering marvel: silicon.
Processors and the von Neumann Model
All the operations required by AI models are executed on microprocessors built into microchips. Whether they are general-purpose (CPUs) or more specialized (GPUs, NPUs, etc.), these digital devices are designed and constructed on chips that integrate an ever-increasing number of transistors.
In 1965, Gordon Moore predicted that the number of transistors integrated into a circuit would double every two years; however, maintaining this growth rate is becoming increasingly difficult for the microelectronics and semiconductor industries.
In the 1940s, John von Neumann established the foundational architecture for modern computers. In his work “The Computer and the Brain,” he outlined one of the core principles of current architecture: “only one organ for each basic operation”.
His computing model, based on sequential operations, made it possible to perform a vast number of different calculations using a single arithmetic logic unit (ALU) and a memory. However, AI tasks, which involve a massive number of operations, turn into highly time-consuming processes for these machines.
The fields of multimedia development, computer-generated imagery, and video games have encountered similar bottlenecks in the past. The development of Graphics Processing Units (GPUs) helped offload a significant computational burden from the CPU. To achieve this, these chips integrate multiple ALUs, enabling them to parallelize a massive number of operations.
A modern GPU can contain over four thousand cores, capable of parallelizing an equal number of operations simultaneously. This is why, whenever AI is discussed, it is common to hear mentions of Nvidia and GPUs. In recent years, AI accelerators utilizing programmable logic devices such as FPGAs and CPLDs have also been developed, further aiding in the execution of higher quantities of operations per second.
Hardware for Artificial Intelligence
For many years, various artificial neural network architectures capable of recognizing images or other data types have emerged; however, the continuous evolution of these architectures made it extremely difficult for integrated circuit designers to create specialized AI devices.
The development of an Application-Specific Integrated Circuit (ASIC) presents numerous challenges and demands considerable time from the initial problem statement to the delivery of a marketable product. Currently, AI-specific ASICs are beginning to be developed, such as Neural Processing Units (NPUs) or chips specifically tailored to the architecture of LLMs.
Despite all the solutions that have been developed, the energy consumption required for training and operating AI continues to soar. Many major tech companies are considering purchasing dedicated power plants simply to supply energy to their data centers. However, how much power does our brain consume? Many researchers estimate the power of the human brain to be between 10 and 20 W. A single operation in the human brain consumes roughly a femtojoule, whereas current technological solutions require between 4 and 5 orders of magnitude more energy.
The Neuromorphic World
In 1989, Carver Mead defined the concept of Neuromorphic Computing, which aims to mimic the brain using microelectronics and semiconductor technology. While it is often said that artificial neurons are based on the neurons in our brain, they actually have very little in common. Our biological neurons do not receive numbers as inputs, nor do they operate sequentially.
Among the solutions presented by this field is In-Memory Computing (IMC); this approach aims to break the von Neumann paradigm by using the neural network’s weights as the system’s memory, performing the computations for weight evolution locally within that same memory. To achieve this, memristive devices[1] can be utilized, as they behave like analog memories.
There are digital solutions within the neuromorphic field designed to mimic biological neural networks: TrueNorth (IBM)[2], SpiNNaker (University of Manchester)[3], and Loihi (Intel)[4] are examples of various ASICs containing event-based neural networks (SNNs, or Spiking Neural Networks).
In the analog realm, SNNs are constructed using memristors as neural synapses alongside a neuron model known as Integrate-and-Fire. Unlike digital systems, these do not require high-speed clocks to function, which drastically reduces their power consumption.
Memristors are micrometric-sized devices that can be integrated into a chip, and their resistance can be altered to function as non-volatile memory. However, this technology still faces numerous challenges before it can truly stand out in the AI landscape.
The Road Ahead
Although artificial neural networks do not closely resemble biological models, our understanding of mathematics has led to significant breakthroughs in training these systems; conversely, neuromorphic solutions face a harsher reality where many of the secrets hidden within our brains remain to be understood.
The use of AI is beginning to yield major benefits for our society, but it is a journey that still has a long way to go. While mainstream AI achieves impressive results, we must keep an eye on emerging technologies that could match or surpass current performance with significantly lower energy consumption. Furthermore, we will see if computing ultimately takes the leap from the von Neumann architecture to a novel architecture based entirely on neural computation.
Energy Efficiency Comparison [5]
| CPU | FPGA/GPU | IMC | analog SNN | Human Brain |
|---|---|---|---|---|
| 70pJ/Op | 50pJ/Op | 1pJ/Op | 13fJ/Op | 1fJ/Op |
References
- https://spectrum.ieee.org/memristor
- https://open-neuromorphic.org/neuromorphic-computing/hardware/truenorth-ibm/
- http://apt.cs.manchester.ac.uk/projects/SpiNNaker/SpiNNchip/
- https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html
- Ultra-Low Energy Neuromorphic Circuits and Systems for Large-Scale Distributed AI - Gert Cauwenberghs
Note: This article is a translation of an original article from “bit” magazine, published on December 20, 2024. Link: https://bit.coit.es/inteligencia-artificial-del-cerebro-al-silicio/
