Teacher’s Pet: In Gene Munster’s IQ test, Siri got smarter

From Annual Digital Assistant IQ Test – Siri, Google Assistant, Alexa, Cortana, posted to Loup Ventures subscribers Wednesday.

We asked Siri, Google Assistant, Alexa, and Cortana 800 questions each. This time, Google Assistant was able to answer 86% of them correctly vs. Siri at 79%, Alexa at 61%, and Cortana at 52%…

Over a 15-month period, Google Assistant improved by 11 percentage points, Siri 13 points, and Cortana only 3 points in terms of questions answered correctly. We do not have historical data on the Alexa app. We’re impressed with the speed at which the technology is advancing. Many of the issues we had just last year have been erased by improvements to natural language processing and inter-device connectivity.

iq test Siri

My take: Really? Siri on my iPhone seems dumber to me. Maybe my expectations are too high, but I feel like I’m eating the dog food.

3 Comments

  1. Fred Stein said:
    I’m with you, Philip, Siri is frustrating. That said our expectations of AI are way beyond what is achievable today. As Giannandrea pointed out in the youtube you posted a while back, machine intelligence is narrow. For specific tasks it can surpass humans, but falls apart outside what it has been trained for. For example, in the SF bay area if one says, “mission”, that could be a street or that street’s neighborhood in San Francisco; Or it could be an organizations goals. Humans understand unstated context.

    Siri (and other assistants) need to learn human skills. If you ask me a question, and my answer is off the mark, your response, “No!” tells me you’re annoyed and that I need to get a better understanding of your question. Humans are usually good at asking questions to clarify, rather than Siri’s “I didn’t quite get that”.

    Test scores don’t address our frustration when digital assistants just don’t get it.

    2
    July 25, 2018
  2. Richard Wanderman said:
    I continue to use Siri to great effect in the car, both via CarPlay in my wife’s C-RV and directly on my iPhone in my Tacoma. It’s far from perfect but I can send texts, reply to texts, get stock quotes and quite a bit more without looking at or touching the iPhone.

    CarPlay makes it better because the car’s microphone filters ambient noise and makes for a better sound sample.

    Using Siri and the iPhone through my truck’s Bluetooth connection is a disaster; Bluetooth introduces enough lag so that things get out of synch often.

    I use Siri in the house to set and disable timers for cooking.

    One thing that’s improved my experience with Siri out of the car is to avoid using “hey Siri” and simply holding down the home button to trigger it.

    Using Siri is as much about the user being trained on syntax and cadence as anything else. I guess I’m trainable.

    0
    July 26, 2018

Leave a Reply