The iPhone’s Face ID will soon work with a mask—if you have an Apple Watch
Recognizing you while your face is covered is still pretty tough for a computer.
Photo credit: ROBERT BONET/GETTY IMAGES
APPLE IS FACING our face-masked future. This week, the company started testing some new software for the iPhone that will let device owners unlock the handset while wearing a face covering. There’s a catch, though, one that lines up with Apple’s strategy of locking people in to different Apple products, and it highlights how challenging it can be to develop accurate facial recognition technology: The new face-unlock feature requires an Apple Watch.
The first developer beta of iOS 14.5 includes updates to app tracking controls and Siri alongside the face-mask function. App-makers typically get early access to the newest version of iOS in order to launch or retool their apps well in advance of the formal software release. (Brave souls who don’t mind the risk of potentially bricking their iPhones can also enroll in public beta releases.) The fully baked version of the software is expected to be made available to the general public this spring.
That means by the time most people install the newest version of iOS on their iPhones, we’ll have been wearing masks for a year or more to help prevent the spread of coronavirus. Compared with all the other ways the pandemic has altered our lives, having to use a method other than Apple’s Face ID to unlock your iPhone isn’t a huge inconvenience. Still, it’s frustrating to hold your phone up to your face only to remember Face ID won’t work on account of your mask. The promise of facial recognition technology—which coexists with very valid concerns about its misuse and its error rates for people with darker skin—is that it’s supposed to get smarter and better over time.
With the coming software update, Apple is more or less handing off the authentication load to the Apple Watch. If you’re using a newer iPhone model (one with Face ID) and have installed the iOS 14.5 beta software, and you’re wearing an Apple Watch with watchOS 7.4, raising the locked phone to your face will trigger a bit of communication between the phone and watch. The phone will unlock. The watch will also display a notification that the phone has been unlocked. One iOS developer described it to WIRED as an experience similar to unlocking a Mac with an Apple Watch.
As 9to5 Mac notes, this is the second change Apple has made to its Face ID authentication system to accommodate face masks. Last spring the company released software that made it easier to avoid using Face ID while wearing a mask, by showing the iPhone’s passcode screen after the first time Face ID fails. Still, these updates have their limitations. The new Face ID-with-face-mask feature will work only with phone unlocking. So if you use Face ID for Apple Pay transactions or to log in to third-party apps, you’ll still have to authenticate in some other way.
But the bigger question is why Apple is relying on the Apple Watch to unlock iPhones with face masks instead of releasing software that simply recognizes the uncovered portion of a person’s face. At the time of publication, Apple hadn’t answered WIRED’s queries about this. Experts say there are plenty of ethical and technical considerations when deploying any facial recognition tech, but when it comes to performing facial recognition on faces that are partially covered, it’s particularly challenging.
Anil K. Jain, who researches computer vision, machine learning, and biometrics recognition at Michigan State University, says that despite facial recognition’s progress over the past five to 10 years, it’s still “susceptible to occlusion—that is, what part of the face is not visible. In most cases, the technology assumes the person will remove their glasses and facial coverings, the illumination would be homogenous, and the expression would be neutral, like a passport photo.”
Once those conditions change, the facial recognition tech can become less accurate. Jain says he and his team explored these kinds of facial recognition scenarios several years ago, trying to study just the area around the eye, called the periocular region, rather than scanning the face in its entirety. There was “sufficient discrimination in terms of recognition performance,” Jain says.
Jain also notes that there’s no one standard of masks these days, which could also throw off facial recognition scans, depending on the data sets the technology is trained on. There aren’t just different mask patterns and textures, but different ways of wearing masks and, more recently, some guidance to wear two masks. “There may be portions of two different masks covering larger areas of the face,” he says, and that coverage could vary every time you put on a mask, further challenging the technology.
Shaun Moore, the chief executive and founder of facial recognition company Trueface, says his company has also been testing its tech on both masked and unmasked faces. While he says the industry is getting better overall at recognizing masked faces, and that there’s a lot of “very good data we can extract from around the eyes and the middle of the forehead,” the technology is still imperfect. In a recent test, Trueface’s identification rate for unmasked faces was just over 98 percent. For masked faces, it was 94.6 percent.
Neither Jain nor Moore assumed to know exactly why Apple chose to rely on Apple Watch to unlock the phone rather than refining the Face ID software to recognize both masked and unmasked faces. “I’m surprised they haven’t been able to figure this out, given that they have depth sensors and 3D recognition capabilities on the phone,” says Moore. It’s very possible that the potential degradation in accuracy, as described by both Jain and Moore, was enough for Apple to trust the customer experience would actually be worse if the customer believed Face ID was supposed to work both with and without a mask. This could create even more frustration than just punching in a passcode.
Or … maybe Apple just really wants you to use the Apple Watch. “That means you have to have two devices, right?” Jain said with a laugh. “And they’re not that cheap. That’s not something everyone can afford. So it’s not a solution for everyone.”
