Apple’s quest to build a mixed reality headset began in 2015 but the project has been plagued by a series of challenges that have occasioned the endless delays and consistently pushed back a lunch date, a detailed report by The Information explains the major bottlenecks the tech giant has faced over the years as it seeks to bring an AR/VR headset into the market.

Apple VR Headset Render
A Previous Apple AR/VR Headset Render

In 2021, three reliable sources, the Bloomberg, The Information, and supply chain analyst Ming-Chi Kuo all released reports to the effect that Apple was planning to launch a premium mixed-reality headset (with AR and VR functions) that features a high-resolution color passthrough.

The Apple AR/VR dream team is christened Technology Development Group (TDG) and is being led by an Apple vice president Mike Rockwell.

The Information report corroborates an earlier Bloomberg report by Mark Gurman which stated that the TDG had initially built a powerful “hub” capable of wirelessly streaming high-fidelity content to a lightweight headset. However, the former Apple Chief Design Officer, Jony Ive, had balked at the idea and insisted on a standalone headset. There was a “standoff” between Ive and Rockwell which lasted months with Apple boss Tim Cook eventually siding with Ive in 2019.

The decision to run with the standalone headset caused “disbelief” among some of the TDG members who had already developed a high-fidelity software capable of running on a powerful external processing unit such as the “hub” and which included photorealistic avatars.

The report goes into detail on the challenges that the team faced as they tried to bring forth an ultra-high-end standalone headset that would pack all the processing onboard without overheating or adversely truncating the device’s battery life. The report says the Apple leadership expects to have an Augmented Reality experience that is “far beyond” what Meta Platforms currently offers and Rockwell is reportedly struggling to deliver the expectations.

A rendering of Apple mixed reality device
A rendering of Apple mixed reality device by The Information

The Apple mixed reality headset features a total of 14 cameras when you factor in those that are used for positional tracking, passthrough, eye tracking, face tracking, body tracking as well as hand tacking and this complexity has caused “headaches” for the company’s hardware and algorithm engineers. To manage this, Apple developed a dedicated image signal processor (ISP) chip christened Bora although Apple’s engineers are struggling to get rid of the latency between this chip and the headset’s main processor, the upcoming M2. To mitigate the latency, Apple has had to add a third chip that acts as a “fast conduit” but the latency issue has still not been fully fixed.

The Apple headset also has a unique feature- a screen on the front that allows other people in the room to see your eyes and upper face. However, the addition of this screen to the design necessitated the passthrough cameras to be positioned in “awkward” places which are far from where the wearer’s eyes should be. The development of the passthrough reprojection algorithms has therefore been a lot more difficult.

Apple mixed reality headset render by Antonio De Rosa based on the drawings by The Information
Apple mixed reality headset render by Antonio De Rosa based on the drawings by The Information

The Information quoted “several people familiar with the headset” referring to Apple’s makeshift solutions as consequences of “overengineering, too-complicated solutions” that often come from “poor planning”.

Jony Ive moved from Apple in 2019 but according to The Information report, he is still working on the project as a consultant and Apple employees need to go to his San Francisco home for his approval on any changes. The report says earlier prototypes of the headset had integrated the battery into the headband but Ive reportedly prefers a design that has a tethered battery that the user wears. The Information says it has no idea what (battery design) approach Apple used in the final design.

The article also says that four people who worked on the Apple headset project had criticized its lack of focus on gaming use-cases. The report says that in internal presentations, the TDG “almost never” talks of games and the team is not building a tracked controller. On the contrary, input methods consist of hand tracking along with a “clothespin-like finger clip” which has previously been revealed in Apple patent filings.

The Information report came out earlier last week and cited five sources saying Apple CEO Tim Cook rarely visits the Apple team that is developing the headset. This is in sharp contrast to Zuckerberg’s almost obsessive commitment to VR and AR projects at Reality Labs. Due to the apparent lack of top-level executive commitment to the project, it has been harder for the TDG to have the required engineering staff and resource allocation compared to what the company is committing to Mac and iPhone.  The team members even had to issue warnings that Apple might permanently lose out on the sector to rivals like Meta and Magic Leap in order to prompt more top-level support for the project.

In a previous report by The Information, the publication had stated that Apple was contemplating a price tag of $3,000 for the headset. However, the report says a top Apple executive Dan Riccio, who oversees the development of the headset, was tasked last year with finding ways to cut the cost of materials so as to make the headset more affordable. Apple’s rivals like Meta Platforms, Sony, HTC, Magic Leap, and Nreal, which have been building and productizing XR hardware over the past decade, are probably better versed with the market demands and pricing expectations for AR/VR hardware. The Apple headset should reportedly compete with Meta’s upcoming Project Cambria which is set to launch later in the year and will be priced “significantly” above $800 according to a company spokesperson.

However, from the reports that have leaked so far, the Apple headset will pack a higher resolution, feature a slimmer design, and run a more powerful processor.

https://virtualrealitytimes.com/wp-content/uploads/2021/12/Apple-XR-Headset-Render-600x375.jpghttps://virtualrealitytimes.com/wp-content/uploads/2021/12/Apple-XR-Headset-Render-150x90.jpgKristin HummelAugmented RealityTechnologyApple’s quest to build a mixed reality headset began in 2015 but the project has been plagued by a series of challenges that have occasioned the endless delays and consistently pushed back a lunch date, a detailed report by The Information explains the major bottlenecks the tech giant has...VR, Oculus Rift, and Metaverse News - Cryptocurrency, Adult, Sex, Porn, XXX