Democrats want feds to target the 'black box' of AI bias

Congress is starting to show interest in prying open the "black box" of tech companies' artificial intelligence with oversight that parallels how the federal government checks under car hoods and audits banks.

One proposal introduced Wednesday and co-sponsored by a Democratic presidential candidate, Sen. Cory Booker, would require big companies to test the "algorithmic accountability" of their high-risk AI systems, such as technology that detects faces or makes important decisions based on your most sensitive personal data.

"Computers are increasingly involved in so many of the key decisions Americans make with respect to their daily lives — whether somebody can buy a home, get a job or even go to jail," Sen. Ron Wyden said in an interview with The Associated Press. The Oregon Democrat is co-sponsoring the bill.

"When the companies really go into this, they're going to be looking for bias in their systems," Wyden said. "I think they're going to be finding a lot."

The Democrats' proposal is the first of its kind, and may face an uphill battle in the Republican-led Senate. But it reflects growing — and bipartisan — scrutiny of the largely unregulated data economy — everything from social media feeds, online data brokerages, financial algorithms and self-driving software that are increasingly impacting daily life. A bipartisan Senate bill introduced last month would require companies to notify people before using facial recognition software on them, while also requiring third-party testing to check for bias problems.

Academic studies and real-life examples have unearthed facial recognition systems that misidentify darker-skinned women , computerized lending tools that charge higher interest rates to Latino and black borrowers, and job recruitment tools that favor men in industries where they already dominate.

"There's this myth that algorithms are these neutral, objective things," said Aaron Rieke, managing director at advocacy group Upturn. "Machine learning picks up patterns in society — who does what, who buys what, or who has what job. Those are patterns shaped by issues we've been struggling with for decades."

President Donald Trump's administration is also taking notice and has made the development of "safe and trustworthy" algorithms a major objective of the White House's new AI initiative . But it would do so mostly by strengthening an existing industry-driven process of creating technological standards.

"There's a need for greater transparency and data comparability," and for detecting and reducing bias in these systems, said Commerce Undersecretary Walter Copan, who directs the National Institute of Standards and Technology. "Consumers are essentially flying blind."

Dozens of facial recognition developers, including brand-name companies like Microsoft, last year submitted their proprietary algorithms to Copan's agency so that they could be evaluated and compared against each other. The results showed significant gains in accuracy over previous years.

But Wyden said the voluntary standards are not enough.

"Self-regulation clearly has failed here," he said.

In a bolder move from the Trump administration, the federal Department of Housing and Urban Development has charged Facebook with allowing landlords and real estate brokers to systematically exclude groups such as non-Christians, immigrants and minorities from seeing ads for houses and apartments.

Booker, in a statement about his bill, said that while HUD's Facebook action is an important step, it's necessary to dig deeper to address the "pernicious ways" discrimination operates on tech platforms, sometimes unintentionally.

Booker said biased algorithms are causing the same kind of discriminatory real estate practices that sought to steer his New Jersey parents and other black couples away from certain U.S. neighborhoods in the late 1960s. This time, he said, it's harder to detect and fight.

The bill he and Wyden have introduced would enable the Federal Trade Commission to set and enforce new rules for companies to check for accuracy, bias and potential privacy or security concerns in their automated systems, and correct them if problems are found. It exempts smaller companies that make less than $50 million a year, unless they are data brokers with information on at least 1 million consumers.

New York Democratic Rep. Yvette Clarke, who is introducing a companion bill in the House, said the goal is to fix problems, not just to assess them. She said it makes sense to give the FTC authority to regularly monitor how these systems are performing because it "has the finger on the pulse of what's happening to consumers."

But a think tank backed by major tech companies including Google and Amazon called the bill a misfire, arguing it unfairly singles out algorithms and could discourage the pursuit of AI.

"If a certain decision carries a high risk of harming consumers, it should make no difference whether an algorithm or a person makes that decision," Daniel Castro, vice president of the Information Technology and Innovation Foundation, said in a statement. "To hold algorithms to a higher standard than human decisions implies that automated decisions are inherently less trustworthy or more dangerous than human ones, which is not the case."

Castro also said it unfairly targets big companies and would hurt the product development process if a company needs to conduct a new assessment for every minor software update.