How does VR software testing work?
Technical director of the company-developer of VR/AR applications “Nomix”
The global demand for high-quality content in VR is growing exponentially. Several factors contribute to this. Firstly, the growth of the market — according to PwC, in 2019 the VR/AR industry amounted to $ 46 billion, and by 2030 it will grow 30 times and reach $1.5 trillion. Secondly, progress in hardware — so, Oculus Quest 2 surpassed the previous model in popularity with customers by 5 times. And thirdly, the interest in the industry is also fueling the attention of IT giants – Apple plans to release its VR device in 2022.
Software testing in the VR industry is difficult to overestimate, not only due to the attention to the market, but also due to the fact that technologies have found their application not only in the usual entertainment sector, but also in industries where the price of bugs in scenarios can be very high — for example, in the military-industrial complex and medicine.
The mechanics of testing VR/AR applications are not fundamentally different from testing the rest of the software. Conditionally, the process can be divided into 2 key parts — performance testing throughout the scenario and functional testing.
At the performance check stage, it is necessary to adjust the frame rate level (frame per second, FPS) throughout the entire application scenario. This indicator is critically important for VR applications, as its stable level provides the same “immersion effect”. In games running on a PC or console, a short-term FPS drawdown may go unnoticed, but in VR it can destroy the sense of presence, as well as cause dizziness.
For VR applications, the gold standard on the Oculus Quest FPS 90 headset. However, all VR content hosting platforms, except Oculus Store, do not require strict compliance with guidelines, and therefore allow content that does not commit to a stable fps or have certain rules for user interaction, only controversial content is moderated, which already falls under the general rules of playgrounds.
During testing, you can check the level manually or using an algorithm.
The task of the algorithm is to check all the most likely (the probability is based on the location of interactive and static objects, possible options for interacting with them) locations that cause fps drawdowns, places and moments of effects or even spawn objects when previously missing geometry appears.
If such an algorithm, once at a certain point or having performed an action in it, receives a long-term decrease in the frame rate, it writes to the position log, a chain of events and objects that led to this and attaches a screenshot of the game camera so that the developer can then easily repeat the required actions.
If we have a very large location, then the algorithm uses a behavioral model in which the algorithm earns a certain number of points for passing the scenario, interacting with objects in the right order, being in the locations of the most likely fps problems. For all abnormal situations and fps drawdowns, the points received are maximum and stimulate the algorithm to repeat actions that lead to application errors.
Functional testing includes the study and verification of the interaction of objects, the patency of the scenario and the movement system, as well as the adjustment of the height of the camera.
This stage can also be performed manually or using an algorithm. It is noteworthy that part of the functional testing of a VR application can be performed on a computer, for example, the logic of passing or searching for visual problems.
Initial testing is performed by the developer himself, without giving new functionality or new entities to other developers until he is convinced of the correctness of behavior in all required situations and interactions. For all these checks, a set of tests is written in the future, allowing, when objects and their behavior change, to make sure in the future that the changes will not affect the already working parts of the application.
The logic of working inside the script after the initial manual check is checked automatically, iteratively running the script over and over again and evaluating, registering all cases when the script cannot be completed in the allotted time or when problematic situations arise.
The height of the camera is another aspect, since different devices can determine the height of a player in different ways — that’s why we test the scenario on different devices for users with different heights, and also test applications from different positions – standing or sitting. Separately, we check all inscriptions and text blocks — the correctness of their location, readability, and the relevance of their appearance.
Testing of a VR application can be enhanced by testing on special focus groups, especially if the application is designed for the mass consumer. This stage allows you to understand how well the idea is generally accepted by the market.
Users are assigned certain tasks that need to be performed inside the application. All actions, reactions and emotions are recorded on audio and video for further analysis for necessary adjustments.
Summing up, at the moment, the main aspects of testing VR software will be monitoring the level of frame frequency and checking the interaction of objects. The quality of the result will depend on the ability to optimally combine the mechanics of manual testing and setting up the necessary automatic algorithms.