Wayne Ma: Why Apple’s prototype headset has 14 cameras

From “Behind the Apple Design Decisions That Bogged Down Its Mixed-Reality Headset” ($) posted Friday by The Information:

Another design decision that has greatly added to the technical challenges for the Apple headset has been the inclusion of its 14 cameras, which allow it to capture everything from images of the outside world to facial expressions and body gestures.

Apple had to build the Bora image signal processor to process the bounty of imagery. But Apple’s engineers have faced technical challenges getting Bora to work with the headset’s main processor, code-named Staten. The back-and-forth communication between the two chips increases latency, which can create nausea for people wearing the headset…

Another technical hurdle for Apple has involved making a key function of the headset—video pass-through, which depends on the cameras—work properly. That function will allow people wearing the headset to see video images of their surroundings on the displays inside the device, a capability intended to reduce the isolation users experience with other VR headsets, as we previously reported.

But because Apple is also putting a display on the outside of the headset—which will show video images of a user’s eyes and expressions to people around them—it couldn’t position the outward-facing cameras roughly where users’ eyes will be.

My take: Wait. Can this be right? Video images of a user’s eyes and expressions displayed on the outside of the headset? I’m already feeling nauseated.

14 Comments

  1. Sounds a costume from a Terry Gilliam movie, ‘The Zero Theorem’ or ‘The Imaginarium of Doctor Parnassus’ maybe. Having a conference room full of trainees looking back at me with digitally-reproduced dilated pupils and mascara eyes while they complete a VR demo? Only Poe could imagine it.

    0
    May 20, 2022
  2. John Blackburn said:
    It depends on the implementation. If done well it would be marvelous—and presumably dubbed “Transparency Mode”, a visual extension of the AirPods feature.

    0
    May 20, 2022
  3. Fred Stein said:
    Speaking of nausea, that’s a potential side effect of VR headsets.

    Maybe that is why the market size is only 11M in 2021, from .5M in 2014. The market doubled in 2021, mainly due to aggressive price cuts by Meta to $299 and $399.

    0
    May 20, 2022
    • Robert Paul Leitao said:
      Fred: Apple must deliver a global-level solution with potential revenue value reaching into the billions of dollars per annum. Whatever Meta might be doing or talking about doing, I don’t think the company is in a position to do anything on Apple’s scale or to the level of sophistication and detail that Apple will achieve. Apple has mastered the ability to provide a solution to a challenge that had yet to be identified until they see Apple’s solution and then it is realized enterprises and consumers needed the solution all along. From what I’ve read, this product will be powered by chips of Apple’s design and operate on an OS created for the product that will work seamlessly in Apple’s eco-system similar to watchOS, iOS, etc.

      3
      May 20, 2022
  4. Greg Lippert said:
    Sounds like its still a tech thats not ready for prime time. I sure Apple will not throw something out there to see if it sticks. They have always been better than that. Hoping that still applies vs grasping for the next hit.

    2
    May 20, 2022
  5. Robert Paul Leitao said:
    In the March quarter Apple’s R&D expenses rose about 21.4% and for the first six months of the fiscal year, R&D expenses rose about 21.8% to $12.693 billion. Obviously, Apple is investing heavily in new product development and whatever the state of prototypes might be today as the company works through concepts, new products are on the way. I suspect development activity is occurring at a fairly brisk pace. Because Apple is a global titan, the addressable market for the product will need to be massive. Along with the product there must be services and applicable solutions. I’m not concerned about what is being described now because features and functionality may continue to morph and there’s no external pressure at this time to deliver a solution to a “problem” that has yet to be readily identified. Of course there will be enterprise-level applications and a consumer market will be created. My point is, there’s much more to product and market development at this time than what is being discussed and disclosed. The engineers and designers are already looking five years out.

    0
    May 20, 2022
  6. Ken Cheng said:
    “Apple is also putting a display on the outside of the headset—which will show video images of a user’s eyes”

    Weird. Some people, ie celebs or people who think they are celebs, go around wearing sunglasses indoors and out, no need for eyes showing on the outside of the lens. There’s that Vogue editor for one.

    0
    May 20, 2022
  7. John Konopka said:
    Huge difference between research prototypes and things they will ship. I’m glad Apple swinging for the fences in the lab. They will learn a lot. I suspect what they ship will be much simpler. Maybe years from now they’ll ship something like this.

    Look up the video “Knowledge Navigator”. (Get the HD version) Apple made it in 1987. It was an iPad with a conversational virtual assistant with an internet connection, before the web came to be. That more than 20 years before the iPad itself was released. The point is that many impossible things are prototyped years before they become real. This also gives lie to the notion tossed about that Steve Jobs just dreamed up the iPhone a year or two before it came out. These sorts of ideas have been around for a long time. Figuring out how to actually make it was genius.

    It’s pretty cool. The “iPad” is very clunky with a huge border, hinge and camera. It is foldable which looks cool. The virtual assistant appears as a visual person which is kind of nice. It has some familiar things like touch screen, no physical keyboard, FaceTime, drag-and-drop, and it has an internet connection. The OS looks somewhat like a traditional Mac OS with a menu bar and desktop icons, including a trash can.

    7
    May 20, 2022
    • Dan Scropos said:
      That video was absolutely incredible. Thanks for sharing.

      0
      May 20, 2022
    • Chris De Armond said:
      John
      Thanks for the video pointer. Very interesting and forward looking.
      Here’s wishing Siri could someday understand like that.

      0
      May 21, 2022
    • David Drinkwater said:
      “Look up the video “Knowledge Navigator”.”

      I agree. This was like Sci-Fi Gold at the time.

      There are other videos I remember, but cannot find, that predicted multi-touch already back in the mid ’90’s. Apple is surfing the future wave.

      As June 30 approaches, which is the end of the academic fiscal (and charitable) year, my broker suggests using AAPL for giving.

      That’s just not going to happen. Especially after the last 6 weeks that we have had.

      0
      May 23, 2022
  8. Alan Trerise said:
    Any new platform will need some critical mass of applications at launch to justify its usefulness. Developers will need to know the specifics of the product to author for it well in advance of it shipping. WWDC should provide a strong signal one way or the other as to how far off the product launch is.

    2
    May 21, 2022
    • Robert Paul Leitao said:
      Alan: Absolutely. I’m not expecting any announcements at this year’s WWDC but I wouldn’t mind being pleasantly surprised. Building a market and inviting content providers and developers into the project takes time and planning. I’d like to know more about the timeline for an anticipated release. I jut think WWDC might be a bit early to debut a product under development and engage developers with likely a new iOS variant. We’ll see. Again, I wouldn’t mind being pleasantly surprised.

      0
      May 21, 2022

Leave a Reply