by Jared Shelly
If you’ve tried to unlock your smartphone while wearing a mask, you know the limits of facial recognition.
For the engineers and computer programmers behind the most commonly used facial recognition algorithms, masks are an unforeseen challenge. When COVID-19 made masks mainstream, threat detection technologies were weakened seemingly overnight.
The National Institute of Standards and Technology (NIST) recently examined the top 89 facial recognition algorithms, finding error rates between 5 and 50 percent. The more a mask covers the nose, the lower the algorithm’s accuracy. There were also nuances: round masks cause fewer accuracy problems than wide ones; black masks lead to more errors than blue masks.
Facial recognition algorithms work by analyzing the size of facial features and their distance from each other. When a nose, mouth, and chin are covered by a mask, there are fewer data points to identify a face. In security settings, a “faceprint” can identify people in a known database; perhaps a list of parents approved to visit a school or sex offenders prohibited from school grounds. Taylor Swift’s security team even deployed the technology during her 2018 tour, cross-referencing images of concert-goers with a database of the star’s known stalkers.
The most common error found by NIST researchers came when facial features were blocked, preventing algorithms from making an effective comparison.
“Some algorithms tolerate masks quite well, and some fail quite miserably. It’s a spectrum,” said Patrick Grother, a scientist at NIST and one of the authors of the study. “It sets the development of the industry back three or four years.”
Despite the immense challenge, upcoming threat detection technologies are in the works to detect faces even if they’re partially covered. In the long run, masks could end up making facial recognition stronger, not weaker.
Rank One Computing is developing periocular recognition, a system that analyzes only the eyes and eyebrows to identify a person. Breakthroughs in deep convolutional neural networks allow periocular recognition to identify a person even when they are wearing a mask that covers the nose and mouth. Once the periocular region of the face has been detected, the Rank One system performs tens of millions of mathematical computations to create a vector template that’s compared with others in a database. Then it goes hunting for a match.
“It explicitly excludes any region of the face below the eyes. Thus, masks mostly have no impact on periocular recognition,” said Brendan Klare, co-founder and CEO of Rank One Computing.
Klare said that in controlled settings his company’s periocular recognition achieves nearly 99% accuracy, and in uncontrolled settings, accuracy is approximately 92%.
In the long run, Klare believes that the engineering challenge presented by masks will lead to facial recognition technology that’s stronger and more accurate.
“Anytime edge-cases are exposed (in this case, masks) and researchers and engineers have time to address those cases, the technology strengthens,” he said. “The strong accuracy of our solution was achieved in a short time frame due to the urgent issue of COVID, though it obviously benefited from many years of face recognition research.”
Albert Fox Cahn, the executive director of the Surveillance Technology Oversight Project, agrees. He said the pandemic and masks have forced researchers to continue “a global arms race to refine facial recognition that can work when people wear a mask. This may, in the long run, make it harder for members of the public to avoid facial recognition with face coverings.”
The NIST study focused on algorithms developed before the pandemic — before masks were part of everyday American life. It’s also important to clarify that it added masks digitally to existing images rather than testing the system on real people. NIST plans to analyze algorithms created with masks in mind. The hope is that it will provide more quantifiable data to determine if facial recognition is indeed on track to become more powerful.
In the meantime, Grother — the NIST scientist— said that covered faces should leave the facial recognition industry in “worse shape” than pre-pandemic. Still, he expects developers to push hard to find solutions.
“You can look at the best algorithms now and say they’re probably usable for one-to-one authentication and other developers will get at least there,” he said. “You shouldn’t bet against developers doing smart things.”
Jared Shelly is a freelance writer who writes about business and emerging technology. The opinions and positions expressed in this article do not necessarily reflect the opinions and positions held by Patriot One Technologies and inclusion of persons, companies, or methods herein should not be interpreted as an endorsement.