State Department database crash strands hundreds at the border

Outage caused by corrupted data, which was mirrored to backup—making it useless.


by Sean Gallagher - Jun 23, 2015 11:14am PDT  Ars Technica

Thanks to the failure of a system used to collect and transmit fingerprints and photos, the US State Department has been unable to issue visas to travelers or guest workers for the past two weeks, The New York Times reports. While some of the systems related to visa processing have been restored, biometric information is still not being processed, leaving many travelers from outside the US and hundreds of agricultural "guest workers" stranded.

State Department officials told the Times that the issue was related to a hardware failure. In an e-mail to the paper, Consular Affairs spokesperson Ashley Garrigus said that there had been data corruption caused by a hardware failure, which had been replicated to the biometric database's backup system. “While switching to the backup system," Garrigus wrote, "we discovered that the data was damaged and unusable. We deeply regret the inconvenience to travelers and recognize the hardship to those waiting for visas, and in some cases, their family members or employers in the United States.”

The State Department normally processes about 50,000 visa applications a day, according to astatement on the outage on the agency's website. But that number surges seasonally as employers bring in laborers (mostly from Mexico) to harvest crops. On top of that, the overall number of visas annually has grown. Earlier this year, the State Department put out a "sources sought" call for a new facial recognition service because of the huge growth in visa requests, especially those driven by the H2-A and H2-B visa programs. "At present, the Visa and Passport face galleries contain over 275 million images combined, which are among the largest face recognition datasets in the world and growing annually at an anticipated rate of 23 million images per year," the State Department's procurement officer wrote in the procurement announcement . "Every passport and visa application requires a face recognition search of the legacy and watch-list galleries."

The guest worker visa program itself is a tangled web of systems. First, employers apply through the Department of Labor with candidate workers. Then the US Citizenship and Immigration Service at the Department of Homeland Security screens the individuals petitioning through the employer for visas. USCIS next sends approved petitions to the State Department’s Kentucky Consular Center to be entered into the visa database. And that in turn allows consular and embassy offices outside the US to see that the individuals are authorized to apply for a visa, thus submitting biometric data back to the Kentucky Consular Center's database to check against watch lists and other image and fingerprint data that might catch attempted visa fraud.

Because of the system failure, the State Department has delayed visas for over 1,500 guest agricultural workers in Mexico. "Last week, nearly 1,250 temporary or seasonal workers who had been issued visas in the past were issued new visas in Mexico," a State Department official said in a statement on the outage. "We have issued more than 3,000 visas globally for urgent and humanitarian travel."

According to a Reuters report, the outage is having real economic impact—especially in Washington, where cherry growers have been unable to get workers out to pick before fruit becomes unsellable. The blueberry crop may be affected next, as workers usually stay to harvest them after cherries. Washington Farm Labor Association Director Dan Fazio told Reuters, "Our farmers are all in for the guest worker program, but the government isn’t. We have a lot of cherries that are ruined and it looks like a lot of blueberries are going to be lost.”