And Why There Still May be Some Free Cheese in my Future.
Occasionally I like to bet. And in Matthias Troyer I found somebody who took me up on it. I wrote about this bet a while ago, but back then I agreed that I wouldn’t identify him as my betting pal, until his paper was published. Now the paper has been out for a while and it is high time to settle the first part of this bet.
The conditions were straightforward, can the D-Wave machine beat a single classical CPU? But of course we specified things a bit more precisely.
The benchmark used is the time to find the ground state with 99% probability, and then not only the median is considered but also the 90%, 95% and 99% quantile. We then agreed on basing the bet on the 90% quantile. I.e. the test needs to run long enough to make sure that for 90% or more of the instances we find a ground state with 99%.
Assuming that Matthias gets to conduct his testing on the current and next chip generation of D-Wave, we agreed to make this a two part bet, i.e. same betting conditions for each.
Unfortunately, I have to concede the first round. The D-Wave One more or less tied the classical machine, although there were some problem instances where it was doing better. So the following jars of Maple Syrup will soon be shipped to Zürich:
What I was obviously hoping for was a decisively clear performance advantage, but at this point this isn’t the case, unless you compare it to off-the-shelf optimizer software as was done in the test conducted by McGeoch et. al.
This, despite the evidence for quantum entanglement of D-Wave’s machines getting ever more compelling. A paper has just been published in Pysical Review X, that demonstrates eight qubit entanglement. Geordie blogged about it here, and it already generated some great press (h/t Pierre O.), probably the most objective mainstream article on D-Wave I’ve seen yet. It is a welcome change from the drivel the BBC put out on QC in the past.
So will I ever get some Raclette cheese in return for my Maple syrup? The chances for winning the next part of my bet with Matthias hinge on the scaling behavior, as well as on the question if a class of hard problems can be identified where quantum annealing manages to find the ground state significantly faster. For the generic randomly generated problem set, scaling alone does not seem to cut it (although more data will be needed to be sure). So I am counting on D‑Wave’s ingenuity, as well as those bright minds who now get to work hands-on with the machine.
Nevertheless, Matthias is confident he’ll win the bet even at 2000 qubits. He thinks D-Wave will have to improve much more than just the calibration to outperform a single classic CPU. On the other hand, when I had the pleasure of meeting him last year in Waterloo, he readily acknowledged that it was impressive what the company had accomplished so far. After all, this is an architecture that was created within just ten years based on a shoestring budget, compared to the multimbillion dollar, decades mature semiconductor industry.
Unfortunately, when his university, the venerated ETH Zürich (possibly the best engineering school on this planet) came out with this press release, they nevertheless (accidentally?) played into the old canard that D-Wave falsely claimed to have produced a universal quantum computer.
It puts into context the Chinese whisper process as depicted in this cartoon that I put up in an earlier post. Unlike depicted here, where the press gets most of the blame, ever since I started paying attention to university press releases, I am compelled to notice that they are more often than not the true starting point of the distortions.
Henning: Did you multiply all of Troyer’s times by 500? His ‘procedure’ divided all the actual runtimes of his experiments by the number of variables. If you want to compare core to core you need to multiply all his times by N (so 500 for the biggest problems). Try doing this and see who wins.
Hi Geordie, must admit, this is the first time I heard about this fudge factor. Thought that the Troyer et al. methodology was pretty uncontroversial.
Going into this, I was counting on such a decisive advantage that I agreed to Scott as arbiter if there were doubts about how to decide the bet (although Scott is blissfully unaware of this). I.e. this makes pleading my case rather difficult.
At any rate, the way I see it, the McGeoch et. al. paper showed that in terms of practical usage your hardware is already delivering the goods, but when put up against hand-crafted code by top notch numerical programmers, regular machines still hold their own. I am pretty confident that eventually they won’t, and that the difference will be so obvious that there really is no argument to be had.
In the meantime, IMHO any high-tech company that requires HPC numerical simulation is missing out if they aren’t already investing in R&D to explore the new computing paradigm that D-Wave represents.
The business case is hardly affected by this.
Hi Henning: I’m sorry to see you lose your bet!. But, by Geordie’s above calculation, you didn’t!. At any rate, in the last sentence of your third paragraph from the last you say this: “After all, this is an architecture that was created within just ten years based on a shoestring budget, compared to the multimillion dollar, decades mature semiconductor industry.” I would change “compared to the multimillion dollar” to “compared to the multi-billion, or even multi-trillion dollar”, and decades of mature semiconductor industry. Thanks.
Yes, you are absolutely correct, meant to write “multi-billion”. Corrected this now.
that’s ok, I mean maybe they can win the first few rounds based on technicality (random instances?), but D-wave is aiming for a knock-out on the 12th round! Anyways, thanks for the post, love the jugs of maple syrup, hope Troyer enjoys them while they last 😉
Agreed, from my point of view this really changes very little. IMHO an ironclad business case can be made for D-Wave.
Oh, so many billions! Alas, shrouds have no pockets.
It sure adds up
Hi Henning: Here is something novel from D-Wave, collaborating with couple of companies in quantum software development and applications. Thanks.
http://www.marketwatch.com/story/d-wave-systems-building-quantum-application-ecosystem-announces-partnerships-with-dna-seq-alliance-and-1qbit-2014-06-09
Had an opportunity to chat with some of the 1Qbit guys at an informal meeting, they were tremendously committed and energized. Happy to see that this is taken to another level.
Check this out Henning, a very positive coverage of Dwave from Wired magazine:
http://arstechnica.com/science/2014/06/a-quantum-computing-gun-revealed-by-quantum-smoke/
Since the 8-qubit peer review came out, it seems the critics tone are changing, which is kind of weird because this is old news. But is this the beginning of the end of Dwave controversy? Already, Im finding myself no one to argue with in the forums or vehemently explain quantum stuff. I was expecting Scott to come out, guns blazing, dissing the peer review, but nothing all is quiet. It would be interesting to see how much impact a peer review paper can have.