Note: This is a rewrite of my post of May 17. I did this to make it clearer and to focus on use of power by AI data centers, which I consider the biggest problem with AI today.
One of the biggest problems facing the power industry today
is the huge electricity demands of AI data centers. This includes both those in
place today and, more importantly, those being projected by data center
developers, often without having a firm idea of where the necessary power will
come from. Naturally, most of the discussions about this problem have
focused either on how to greatly expand power supply, at least in the longer
term, or on how to make the data centers more efficient, so they can do what
they’re currently doing while using somewhat less power.
However, it seems to me that the real question we should be
asking about AI is why it’s placing such huge demands on the power grid in the
first place. After all, if “boiling the ocean” is an apt description for
anything, it is for language model training, especially when the models are
complex, the problems are complex, or the data aren’t well defined. Does all of
this come with the territory? That is, if your goal is to recreate
intelligence, is it inevitable that sooner or later you’ll have to throw every
kilowatt you can find at the problem?
I’m sure this doesn’t come with the territory. After all, it’s
well known that an AI model can’t even approach the level of intelligence of
the common housecat. Stated another way, if there were a good way to embed a
smart AI in a cat’s body, it’s highly unlikely that the AI would be anywhere
near as good at catching mice as the cat is.
Of course, many people – including many scientists – used to
consider any sign of animal intelligence to be simply pre-existing programming
- i.e., firmware inherited from the animal’s parents. However, it’s impossible
to hold such a position today. After all, what about chimpanzees that use tools?
What about ravens who go back and dig up a bauble they’ve buried if they notice
that a human, or another raven, was watching them when they buried it? What
about Alex the Grey Parrot, whose last words were “You be good. I love you”? We
should all end so fittingly.
For that matter, a living creature doesn’t need a brain to
exhibit intelligence. A slime mold, which consists of a collection of
disconnected cells without any central intelligence, can seek out fragrant
food, even when there’s an uncomfortable barrier in the way. A Venus fly trap
can count
to five. And it isn’t hard to design cellular automata that exhibit
intelligent behavior, even though their electronic “bodies” don’t contain a
single carbon atom.
However, let’s suppose that an AI model could outwit a mouse
half as intelligently as a cat could. Even if that were true, I would want to
know how the likely huge amount of energy required to train that model compares
with the tiny amount of energy needed to train the cat’s brain to outwit mice.
After all, it seems likely that the only training that’s
required for the cat is the on-the-job training it acquires by watching its mother
catch mice – along with presumably some inherited “training” (firmware vs.
software). On the other hand, I would bet that training the model would require
ingesting millions of documents, videos, etc. Yet the model would almost
certainly prove to be nowhere near as successful as the cat for purposes that
matter to cats: catching mice and other small mammals.
In other words, there is almost certainly a better way to
create artificial intelligence than to boil an ocean of documents. It requires
paying attention to what animals do in their day-to-day activities. The
intelligence that animals exhibit in those activities goes far beyond the
wildest predictions of what AI can do. Even better, no animal needs to consume
huge amounts of power to learn how to perform those activities. Maybe, if we
just pay attention to what animals can do and then “reverse engineer” how they
do it, we will be able to develop truly intelligent systems that will require relatively
little power to train or operate.
Here's an example. Let’s say a dog escapes from its home one
day and wanders seemingly aimlessly around the neighborhood. There’s no way the
dog could remember every tree he watered, every driveway he crossed, every rabbit
he chased, etc. Yet somehow, later in the afternoon (perhaps just before
dinnertime), he shows up at his home. What’s most remarkable is his master’s
reaction to his return: Since the dog wanders off regularly and has never once
not found his way home before dinnertime, there is nothing miraculous about his
return this time. The master barely notices that his dog has returned.
Yet, it is quite miraculous. The dog doesn’t have a map or
GPS. There’s no way the dog can use logic, as we can, to find its way back
home. For example, the dog can’t say to itself, “When I was crossing this
street a few hours ago and was almost hit by a car, I had just set out from my
house. At first, I walked along this side of the street, but I didn’t cross it.
Therefore, if I just keep walking along this side of the street for a little
while, I’ll probably come home.”
Is there some way the dog can utilize a chain of reasoning
like this without “consciously” invoking it? Perhaps, but what is the mechanism
behind that result? It certainly can’t be genetic. Even if the dog was born in
the neighborhood and has lived there ever since, there’s no known process by
which his genome could be altered by that experience.
Could it be training? When the dog was a puppy, did its
mother train it to wander around the neighborhood and find its way home? There
are certainly animals, like bees and ants, that can find food outside of their
“home” (e.g., their beehive) and return to instruct their peers to do the same
(the bee sometimes does that by performing a dance that encodes the route it
took to find a particularly abundant source of pollen for the other bees). But dogs
don’t do that, and of course we’re hypothesizing that the dog was wandering
randomly, rather than being in search of food (which he already knows he’ll get
at home when he returns).
Yet, the dog did find its way home, and given that this is a
common occurrence, it’s clear the dog in question did not utilize any
extraordinary power that its fellow dogs (and other types of animals, of
course) do not possess. How did it get there?
Of course, I don’t know the answer to this question.
However, there are two things I know for sure:
1.
Generative AI is 100% based on statistical
relationships between words. The model learns these relationships and uses that
data to create whatever content it’s been asked to create. However, the model
doesn’t “understand” the words it uses.
2.
Dogs don’t understand words, either. Yet the
fact that a dog can demonstrate at least one type of truly intelligent behavior
– finding its way home without seeming to have some fixed pattern to follow - seems
to indicate there might be a different type of AI that doesn’t require the
boiling-the-ocean approach to learning that Gen AI does.
What could explain why the dog can find its way home,
if neither genetics nor training can? Again, I can’t answer that question for
certain. However, I can point out that infants don’t have any command of words,
yet they seem to be able to “reason” based on symmetry. For example, a baby –
even a newborn – can recognize an asymmetrical face and react differently to it
than a symmetrical face.
It seems that human infants can perform what I call “symmetry
operations” without prior experience, and certainly without training. This makes
it likely that other mammals like cats and dogs, and especially primates, can also
perform those operations. Could symmetry operations contribute to at least some
forms of intelligence, including the cat’s ability to outwit mice and the dog’s
ability to find its way home?
The point is that various functions (mathematical and
otherwise) can be embedded in animals and even plants. An animal might use
these functions to solve certain hard problems like, “Given that I’ve been
wandering all over the neighborhood this afternoon, how can I find my way home
before dinnertime?”
To conclude, how do we create a system that might perform as
intelligently as a cat in catching mice, or as a dog in navigation? First, we
figure out what ingrained capabilities (like recognizing symmetrical objects)
enable this intelligent behavior. Second, we figure out how to recreate those
capabilities, either in hardware or software.
In one of my next posts, I hope to examine what are the ingrained
capabilities that allow a dog to find its way home. We may learn that dogs are
masters of symmetry operations.
To produce this blog, I rely on
support from people like you. If you appreciate my posts, please make that
known by donating here. Any amount is welcome. Thanks!
If you would like to comment on
what you have read here, I would love to hear from you. Please email me
at tom@tomalrich.com.