Second wave Algorithmic Accountability: from "What should algorithms do?" to "Should we use an algorithm?"

For ten years, activists and theorists have been developing a critique of "algorithms" (which have undergone numerous renamings over the same time, e.g. "filter bubbles"), with the early critiques focusing on the way that these can misfire with dreadful (or sometimes humorous) consequences, from discrimination in which employment and financial ads get served to the "dark patterns" that "maximized engagement" with services that occupied your attention but didn't bring you pleasure.

Today, a new wave of critiques is emerging, one that doesn't merely ask "What are the problems with how this algorithm does its job?" but also asks, "Should an algorithm do this job?"

The canonical example of this is bias in facial recognition: it's well-understood that facial recognition tools perform worse when asked to identify women and people with darker skin, a circumstance that is very plausibly attributed to the male, white developers of these tools who trained them on people who looked like themselves.

The first-order critique of this is "Garbage In, Garbage Out": the lack of representation and the bias in tech hiring ripples out beyond the workplace and into the products, reproducing discrimination everywhere the products land.

But the second-order critique is more nuanced: "Given that a major application for facial recognition is totalitarian surveillance and control, maybe we should be thinking about limiting facial recognition altogether, rather than ensuring that it is equally good at destroying the lives of women and brown people."

This is a point that was really well articulated by Cindy Cohn at last year's launch party for the EFF/McSweeney's book on privacy. Read the rest