A cluster of off the shelf boxes has about 160 chips with 320 HyperTransport buses. So we’re about a factor of 60 away from having the right bandwidth in one rack.

You’d definitely need a custom solution that is communication bandwidth oriented to get the right bandwidth per chip. My guess that the bandwidth/chip gap will be closed by the time the memory/chip gap is (10-15 years).

I don’t understand your comment about bandwidth dropping at longer distances. First of all, you don’t need that much, due to locality. Second, you can just route through local interconnects by using multiple hops. There’s plenty of time in a “neuronal cycle” (10ms?) to cross even 100 hops. Basically, you don’t need anything more than connections to immediate neighbors.

Agree completely that connection density probably follows a power law. But you’re still wire-limited at the chip, printed circuit board, and bus levels. As the number of wires goes up, your IC yields go down. Same story with your PCBs. And your backplane is just a nightmare.

So you may be able to lop a couple of orders of magnitude off of the problem due to locality, but that still leaves 2-3 orders of magnitude left to go, with a lot of it at the longer connection distances, where bandwidth necessarily drops due to noise, capacitance, etc.

@Michael–

We certainly don’t know enough today to build AIs out of hierarchical neural pattern matchers, but I suspect that that will change rather quickly when we can simulate neural nets that are big enough to do useful work. And that’s sort of a chicken-and-egg problem, isn’t it? No platform to do interesting neural AI on because there’s no software, no software because there’s no platform. Bottom line is that very little is going to happen until we get some kind of a fab breakthrough–which was more-or-less my original point.

I also agree that we’re unlikely to get genuine neural processing at the neuron level–way too many processing units. However, there’s a lot of territory in between one processor per neuron and all neurons simulated on a single processor. We know enough about neural activation functions to write interesting algorithms. Those algorithms can be simulated millions of times faster than the time it takes a real neuron to fire, so millions of neuron simulations per processor are possible. You are also correct that any accurate neural simulation must also simulate the funny chemicals being squirted out globally, but that’s a pretty minor complicating factor to discovering interesting algorithms.

The most interesting problem to be solved is to figure out how pattern-matching hierarchies self-organize to achieve cognition. There’s a genetic component involved here: Lots of white matter projections grow during development, both from the limbic system and brain stem to the cortex, as well as from place to place inside the cortex itself. But once this rough architecture is in place, plasticity has to prune connections and learn appropriate connection weights. Even more important, something’s going on that must automatically allocate the size and functions of the network of pattern recognizers. But that algorithm has to be really, really simple. Its discovery will be a key enabler for using neural nets for more cognitive tasks.

FPGAs, Cell processors and other interspersed memory/computation architectures have high local bandwidth (within the chip). The highest bandwidth would be required only for the average distance of an axon, which covers only 1/10,000 of the brain area. i.e. you only need to reach 1/10,000 of the other chips with the highest bandwidth.

To achieve power law distribution for bandwidth, it would be enough to lay out the chips in a 2-D network, and to have local buses between adjacent chips. In effect, this mimics the connectivity of the brain.

I’ve been doing software and hardware for almost 50 years now, and I know whereof I speak.

When PCs (and similar microprocessor-based equipment) had memories of 8K or less, the operating systems fit neatly into less than 4K. Try finding an operating system that runs today in under 4K.

The computer that I learned on had 1000 words of memory (it was a big UNIVAC I). It didn’t really have an operating system per se — what was available was part of the bootstrap, a standard practice back then. A year later, I worked with a computer (NCR200) that had all of 200 bytes of memory. The first real operating system that I saw (and worked with) was the executive developed for the Honeywell 800/1800, which was a true multiprocessor (in the late 50s/early 60s!). It ran in under 2K words (IIRC, a word on the Honeywell 800 was 48 bits wide). We also didn’t have the luxury of a standardized USB — that meant dealing with paper tape, punched cards, magnetic tape, and numerous other peripherals.

The idea of “fetches” doesn’t seem to shed any light here. The “program” of the brain, if we may call it that, can’t be written out as a single entity. The brain is made up of of a number of nucleii, each of which is a very generalized pattern-matching system that work together to model the outside world- or so it appears. Beyond that, we have very little idea of how this organizes itself into a conscious being, although there are plenty of philosophers like Dennet who will tell you it’s a simple matter

Looking at small scale neural nets in vivo has given us some insights into how to design mechanical insects that can navigate on their own, but we’re still far from a workable model of how the brain organizes memory and knowledge of events and objects, let alone a sense of self.

The problem isn’t the memory itself; it’s the wires. Wires are the bane of the electronics industry. They’re expensive, hard to fabricate, hard to test, and they break easily. You can have all the FPGAs you want, but you can only present vectors to them serially, and even then, the gate array’s logic is bounded by how many simultaneous inputs it has.

My point was merely that the usual way of moving data around in a computer breaks down under the amount of parallel data needed for proper hierarchical pattern recognition. You can’t trade the number of wires you need for properly parallel data processing simply by emulating the parallelism with really fast serialized buses. It just doesn’t scale.

I believe the average axon length is 10mm. That reaches an area that is less than 1e-4 of the total cortex surface area (~1.6 m^2). That’s pretty far from needing all that memory bandwidth globally.

So yes, I do agree that memory bandwidth will be the bottleneck rather than CPU, but I think we can easily build adequate solutions.