Tag Archives: artificial neural networks

Out of the AI Winter and into the Cold

A logarithmic scale doesn't have the appropriate visual impact to convey how extraordinarily cold 20mK is.

Any quantum computer using superconducting Josephson junctions will have to be operated at extremely low temperatures. The D-Wave machine, for instance, runs at about 20 mK, which is much colder than anything in nature (including deep space). A logarithmic scale like the chart to the right, while technically correct, doesn't really do this justice.  This animated one from D-Wave's blog shows this much more drastically when scaled linearly (the first link goes to an SVG file that should play in all modern browsers, but it takes ten seconds to get started).

Given that D-Wave's most prominent use case is the field of machine learning, a casual observer may be misled to think that the term "AI winter" refers to the propensity of artificial neural networks to blossom in this frigid environment. But what the term actually stands for is the brutal hype cycle that ravaged this field of computer science.

One of the original first casualties of the collapse of artificial intelligence research in 1969 was the ancestor of the kind of learning algorithms that are now often implemented on D-Wave's machines. This incident is referred to as the XOR affair, and the story that circulates goes like this:  "Marvin Minsky, being a proponent of structured AI, killed off the connectionism approach when he co-authored the now classic tome, Perceptrons. This was accomplished by mathematically proving that a single layer perceptron is so limited it cannot even be used (or trained for that matter) to emulate an XOR gate. Although this does not hold for multi-layer perceptrons, his word was taken as gospel, and smothered this promising field in its infancy."

Marvin Minsky begs to differ, and argues that he of course knew about the capabilities of artificial neural networks with more than one layer, and that if anything, only the proof that working with local neurons comes at the cost of some universality should have had any bearing.  It seems impossible to untangle the exact dynamics that led to this most unfortunate AI winter, yet in hindsight it seems completely misguided and avoidable, given that a learning algorithm (Backpropagation) that allowed for the efficient training of multi-layer perceptrons had already been published a year prior, but at the time it received very little attention.

The fact is, after Perceptrons was published, symbolic AI flourished and connectionism was almost dead for a decade. Given what the authors wrote in the forward to the revised 1989 edition, there is little doubt how Minsky felt about this:

"Some readers may be shocked to hear it said that little of significance has happened in this field [since the first edition twenty year earlier]. Have not perceptron-like networks under the new name connectionism - become a major subject of discussion at gatherings of psychologists and computer scientists? Has not there been a "connectionist revolution?" Certainly yes, in that there is a great deal of interest and discussion. Possibly yes, in the sense that discoveries have been made that may, in time, turn out to be of fundamental importance. But certainly no, in that there has been little clear-cut change in the conceptual basis of the field. The issues that give rise to excitement today seem much the same as those that were responsible for previous rounds of excitement. The issues that were then obscure remain obscure today because no one yet knows how to tell which of the present discoveries are fundamental and which are superficial. Our position remains what it was when we wrote the book: We believe this realm of work to be immensely important and rich, but we expect its growth to require a degree of critical analysis that its more romantic advocates have always been reluctant to pursue - perhaps because the spirit of connectionism seems itself to go somewhat against the grain of analytic rigor." [Emphasis added by the blog author]

When fast-forwarding to 2013 and the reception that D-Wave receives from some academic quarters, this feels like deja-vu all over again. Geordie Rose, founder and current CTO of D-Wave, unabashedly muses about spiritual machines, although he doesn't strike me as a particularly romantic fellow. But he is very interested in using his amazing hardware to make for better machine learning, very much in "the spirit of connectionism".  He published an excellent mini-series on this at D-Wave's blog (part 1, 2, 3, 4, 5, 6, 7).  It would be interesting to learn if Minsky was to find fault with the analytic rigor on display here (He is now 86 but I hope he is still going as strong as ten years ago when this TED talk was recorded).

So, if we cast Geordie in the role of the 21st century version of Frank Rosenblatt (the inventor of the original perceptron) then we surely must pick Scott Aaronson as the modern day version of Marvin Minsky.  Only that the argument this time is not about AI, but how 'quantum' D-Wave's device truly is.  The argument feels very similar: On one side, the theoretical computer scientist, equipped with boat-loads of mathematical rigor, strongly prefers the gate model of quantum computing. On the other one, the pragmatist, whose focus is to build something usable within the constraints of what chip foundries can produce at this time.

But the ultimate irony, it seems, at least in Scott Aaronson's mind, is that the AI winter is the ultimate parable of warning to make his case (as was pointed out by an anonymous poster to his blog).  I.e. he thinks the D-Wave marketing hype can be equated to the over-promises of AI research in the past. Scott fears that if the company cannot deliver, the babe (I.e. Quantum Computing) will be thrown out with the bathwater, and so he blogged:

“I predict that the very same people now hyping D-Wave will turn around and—without the slightest acknowledgment of error on their part—declare that the entire field of quantum computing has now been unmasked as a mirage, a scam, and a chimera.”

A statement that of course didn't go unchallenged in the comment section (Scott's exemplary in allowing this kind of frankness on his blog).

I don't pretend to have any deeper conclusion to draw from these observations, and will leave it at this sobering thought: While we expect science to be conducted in an eminently rational fashion, history gives ample examples of how the progress of science happens in fits and starts and is anything but rational.