Tags: iphone 13, Mobility, Seeing AI
Back in December of 2020, Microsoft’s Seeing AI released an update for the app that included a brand new channel. The new channel, called World, uses an iPhone’s LiDAR sensors to gain a spatial, 3D depiction of the environment in the phone’s camera lens. Using 3D spatial audio, the app notifies the user of the location of different objects within the camera’s field. Although this new channel was released in late 2020, I just recently discovered the new feature.
Unfortunately, the LiDAR sensor is only available on certain models of iPhones running iOS 14. These include iPhone 12, iPhone 12 Pro, iPhone 13 Pro and iPhone 13 Pro Max. When working with a client who had an iPhone 12 a few months ago, I was able to learn about the new channel. While my experience with the World feature is somewhat limited, I feel the technology involved is impressive. The real world application, however, is uncertain.
World channel operates in an automatic fashion. The user navigates to the channel, begins moving the phone around as the app scans the camera field and identifies objects. This works similar to the scene feature, but in real time versus taking a photo.
In order to take full advantage of the 3D sound environment, the app requires compatible headphones (more on that later). Using headphones, the user will hear objects around the room called out with the identifying voice sounding as though it is emanating from its location. For example, if a stove is located to the left of the user, the identifying voice will sound from the left. If the user slowly rotates so the stove is behind them, the voice would sound as though it was coming from behind the user.
An additional feature compiles all objects identified within the camera field into a list in the menu. Once the user has identified an object of interest, the user can navigate to the list of items, select the item of interest and the app will start a navigation task. This process uses the above-described 3D sound and guides the user to the object using beeps.
I try to evaluate and determine the real world uses of every new channel introduced in Seeing AI. Some channels are easy to find a use for like Short Text for checking mail envelopes. Others are less useful such as the Person channel. Facial recognition is impressive, but not useful day to day.
The World channel might not be used frequently by most users, but could come in quite handy in some situations. When in unfamiliar surroundings, the app could scan a room to locate an exit door of a large conference room or front door of a home or business, then navigate to it listening to the 3D beeps.
The World channel is impressive and could potentially be incredibly useful. Unfortunately, the limitations of the channel are certainly a drawback. The restriction to only certain models of iPhones, while not Microsoft or Apple’s fault, limits access for most users. In addition, in order to take full advantage of the 3D sound capabilities, a user must have compatible headphones. A method to test headphone compatibility is located within the Settings menu. I cannot say which headphones are compatible and which are not.
Based on my experience with the World channel, I feel that it is impressive technology with potential real world uses. Due to current technological limitations, it may be a few years before it is accessible to most users.
If you are interested in learning how to use your iPhone with VoiceOver, how to use specific apps for people with vision loss or learn how an iPhone can improve your independence give us a call at 312-236-8569 or send an email to kathy.austin@second-sense.org. We provide both in-person and remote training to adults with vision loss.
Cody is a Certified Vision Rehabilitation Therapist at Second Sense.