ARKit 6. What's new? / Habr

ARKit 6. What's new?

At the recent WWDC 2022, ARKit was updated along with iOS 16 and iPadOS 16. I am glad that Apple continues to invest in the AR field, offering more features and improvements every year. What’s new in ARKit 6? Let’s take a look!

4K video

Despite the fact that the hardware component of the cameras has long allowed you to shoot video in 4K, ARKit video of such high resolution has not been used before.

The image from most of the sensor was used, however, after capture, a process called binning. Its essence is simple: binning it takes a 2×2 pixel area, averages the pixel values and writes them to one pixel, as a result, the image is reduced to 1920×1440 pixels. As a result, each frame of the video consumes less memory and computing power. This allows the device to launch the camera at up to 60 frames per second and frees up resources for rendering.

Now, thanks to the power of the latest hardware stuffing, Apple has included a full-fledged 4K video mode in ARKit. Now the application can take advantage of a higher resolution image by skipping the stage binning and directly addressing it in full 4K resolution. Otherwise, the application will work as before.

The new 4K mode can be enabled by following a few simple steps. Let’s see how it looks in the code.

if let highResFormat = ARWorldTrackingConfiguration.recommendedVideoFormatFor4KResolution {
    
    config.videoFormat = highResFormat
}
            
session.run(config)

ARConfiguration got a new convenient property recommendedVideoFormatFor4KResolution, which returns the 4K video format if this mode is supported on the device. If the device or configuration does not support 4K, this property will return nil. Then you can assign this video format to your configuration and run ARSession with this configuration.

4K video mode is available on iPhone 11 and newer, as well as on any iPad Pro with M1 chip. The resolution is 3840×2160 pixels at 30 frames per second.

When using ARKit, especially in 4K resolution, it is important to follow some guidelines to achieve optimal results:

  • Don’t hold ARFrame too long. This may prevent the system from freeing up memory and will not allow ARKit show new frames, which will be visible by missing frames in rendering. Ultimately, the tracking state ARCamera can go back to limited.
  • Watch out for warnings in the console to make sure you don’t save too many images at a certain point in time.
  • It’s also worth considering whether the 4K video format is really suitable for your application. Such a need may arise, for example, in applications for film production. Working with images and videos with higher resolution requires additional system resources, so for games and other applications that require a high refresh rate, Apple still recommends using Full HD video at 60 frames per second.

Improvements on camera settings

In addition to the new 4K mode, there are some additional improvements that allow us to make better use of the camera’s capabilities.

Capturing highres images in the background

Imagine a situation when you are running an application with ARSession you need to take a photo to capture the scene you see in AR. ARKit 6 has added the ability to request the capture of individual photos on request in the background while the video stream is running continuously. These photos take full advantage of your camera’s sensor. That is, on the iPhone 13, it will be all 12 megapixels of a wide-angle camera. Another use case that would greatly benefit from this API is the creation of 3D models using object capture (Object Capture).

Function Object Capture takes a lot of photos of real objects, such as these sneakers in the illustration, and using algorithms Photogrammetry turns them into realistic 3D models ready for use in augmented reality applications.

Let’s look at how to capture high-resolution photos in code.

if let highResCaptureFormat = ARWorldTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing {
    //установка видео формата, который поддерживает захват изображений
    config.videoFormat = highResCaptureFormat
}

session.run(config)

First, we check the video format that supports hi-res image capture. To do this, use the function recommendedVideoFormatForHighResolutionFrameCapturing. After we make sure that the format is supported, we can install a new video format and run ARSession.

session.captureHighResolutionFrame { (frame, error) in
    if let frame = frame {
        saveHighResImage(frame.capturedImage)
    }
}

After the launch, you can take a picture. ARSession has a new function captureHighResolutionFrame. Calling this function triggers a high-resolution background image capture. Asynchronously in the completion handler we get access to ARFramecontaining the image and all other properties of the frame. In this example, the frame is saved to disk.

HDR

Shooting and experimenting with the camera on the iPhone, we know that when you turn on the HDR mode, some details in the photo are saved much better. Here is an example with clouds:

Let’s see how HDR is enabled for ARSession.

if config.videoFormat.isVideoHDRSupported {
    config.videoHDRAllowed = true
}

session.run(config)

Through the property isVideoHDRSupported we can find out if the video format supports HDR. If HDR is supported, set true for videoHDRAllowed in ARConfiguration and ARSession. It should be understood that the inclusion of HDR will affect performance, so you need to use it only when it is necessary.

Access to AVCaptureDevice

In case you need manual control of settings such as exposure or white balance, ARKit 6 offers a convenient interface for direct access to AVCaptureDevice and changes to any of its settings via the property configurableCaptureDeviceForPrimaryCamera.

if let device = ARWorldTrackingConfiguration.configurableCaptureDeviceForPrimaryCamera {
    do {
        try device.lockForConfiguration()
        //конфигурация настроек AVCaptureDevice
        device.unlockForConfiguration()
    } catch {
        //обработка ошибок
    }
}

EXIF data

ARKit provides access to EXIF data in the application. They are now available in ARFrame.

open class ARFrame: NSObject, NSCopying {
    ...
    @available(iOS16.0, *)
    var exifData: [String: Any] { get }
}

EXIF contains useful information about white balance, exposure, and other settings that can be useful for post-processing.

Improved motion recognition

Motion recognition introduced in ARKit 3 (Motion capture) allows you to track the movements of the human body and use them as input data for augmented reality. With each update, recognition of more body poses and gestures is added. Determining the position of the body and its parts is becoming more accurate, and we can now track movements from a greater distance.

What was added and improved in ARKit 6:

  • Left and right ear position tracking (2D);
  • Improved pose detection (3D). The defined human skeleton has been updated and now has more joints;
  • Significantly lower latency and greater time consistency in general;
  • More stability and accuracy when a body part is closed or a person comes close to the camera.

Improvement in Location Anchors

This functionality was added in ARKit 4, but it has improved significantly in the 6th version. Generally, Location Anchors they are needed to bind AR objects to the coordinates of a certain geolocation in the real world. Thanks to Location Anchors you can, for example, place street names and navigation elements in augmented reality in their exact position.

New cities where they have become available Location Anchors:

  • Vancouver, Toronto, Montreal (Canada)
  • Singapore
  • Tokyo, Fukuoka, Hiroshima, Osaka, Kyoto, Nagoya, Yokohama (Japan)
  • Melbourne, Sydney (Australia)

Later this year will be available:

  • Auckland (New Zealand)
  • Paris (France)
  • Tel Aviv (Israel)

Conclusion

As we can see, Apple did not present a completely new ARKit functionality at WWDC. Apparently, Apple has big plans for 2023 – the presentation of a completely new AR headset. However, we can note that this year we have received significant improvements and extensions to the previously presented functions, such as: support for 4K, HDR, highRes photo capture and access to camera settings. This allows us to make more and more high-quality and sophisticated AR products for professional purposes and entertainment.

Vrtual Reality Company | Outsource IT Services

Go to our cases Get a free quote