Steam’s VR Hardware Survey is No Longer Reliable
Steam’s regular VR Hardware Survey has had some anomalies lately and as a result, the data is no longer a reliable source of insights into the VR market.
Steam Hardware Survey has been a useful pointer to the market size of the various VR headsets. It has been a particularly important metric given that the leading VR hardware companies such as Meta, Valve, and HTC don’t provide any sales figures and estimates of sales have mostly come from secondary sources.
For years, the Steam Hardware Survey has been the most reliable source of insights into the monthly Steam user base. Steam sends the survey to a random sample of its user base every month. If a user accepts the survey, it will upload their PC specifications and peripherals.
Until March 2020, this survey was based on headsets that were connected via a USB at the time the sampling was taking place. Valve later changed the methodology and began relying on the scanning of users’ SteamVR logs over a month.
This year, the data captured from two months had clearly anomalous figures for ‘Steam Users with VR Headsets’. After Steam introduced the SteamVR log scanning method, this data has stayed constant at 2%. However, in May, this data saw a sudden and unexplained surge to 3.24%. In June, the data reverted to the normal range and in July, it once again surged to 6.67%.
For the month of August, the data once again reverted to the more credible 2.60%. However, it is the headset share data that is anomalous this time.
The survey shows the number of Steam users with a headset as well as the per-headset share. There is also the ‘Other’ section which covers unidentified headsets or those headsets that have less than 0.01% share. The ‘Other’ section usually lies in the range of 1% to 2%. However, in the Steam Hardware Survey for August, this section surged to 13.76%.
The increase apparently consists of the sum of losses from almost all the other headsets that are listed in the survey. For instance, in August, the survey registered an 8% dip in Quest users, which is the most popular headset on the platform.
The survey data shows that Pico Neo 2, Oculus Rift, and the Oculus DK2 headset from 2014 are the only headsets that registered growth in September. This is unlikely the case because these are relatively old headsets and the newer headsets such as the Meta Quest have recorded tremendous growth which leads to the conclusion that there is likely something wrong with the data.
For the past few years, the Steam Hardware Survey has been the most reliable data source on the growth of the headsets and the user base. However, these recent anomalies mean that this data is no longer reliable and insightful. Valve is yet to explain the source of the anomalies.