We Built the Cage and Called It Safety

The Flock CEO wants you to know his cameras are accurate. He'd like you to focus on that. He's got numbers, and numbers are reassuring, and when you're selling surveillance infrastructure to police departments across the country, reassurance is the product just as much as the hardware is. What he'd prefer you not focus on is Brandon Upchurch, who is a real person who got mauled by a police dog in Toledo because Flock's camera misread a license plate, and who then lost his job, got evicted, and settled for $35,000, which is the price the system decided his suffering was worth. The CEO has thoughts about database quality. Brandon Upchurch has scars.

The Roseville records are where the sales pitch goes to die. Spreadsheets, emails, internal analyses, the kind of documentation that accumulates when a system is quietly failing and everyone downstream knows it but nobody upstream wants to say so out loud. The cameras can't reliably distinguish a 1 from a 4. License plate frames confuse them. Distance blurs the read. These are not edge cases or exotic failure modes, they're the ordinary conditions of a parking lot, a street corner, a driveway at dusk, and the system chokes on them at a rate that, if a human officer performed this badly, would end careers. A 71% false alert rate is not a rounding error. It's the whole story.

And then the CEO, faced with that number, points at the NCIC database. The database is outdated, he says. The database has bad records. Which is true, actually, NCIC has real problems, and if Flock's cameras were reading plates perfectly and then matching against a flawed database, that would be a coherent criticism worth having. But that's not what's happening. What's happening is the cameras are misreading the plates before they ever touch the database, feeding garbage into a lookup that then returns a warrant hit on a man who has nothing to do with the vehicle in question, and then a dog is biting him in Toledo. Blaming NCIC for that sequence of events is like blaming the recipe because you measured the flour wrong. The information downstream was never the problem.

What's worth sitting with, really sitting with, is how cleanly this whole architecture is designed to absorb its own failures without consequence. Flock sells the system, the department deploys it, the officer acts on the alert, the innocent person gets hurt, the city settles, and Flock's contract renews. Nobody in that chain is structurally incentivized to slow down. The company's interest is in keeping the contracts. The department's interest is in appearing to use modern tools. The city's interest, once the lawsuit lands, is in making it go away quietly enough that the next council meeting isn't awkward. Brandon Upchurch's interest, the one that doesn't have a lobbyist or a sales deck, is in not getting mauled by a dog because a camera couldn't read a plate. That interest lost.

This is the part that should make you angry, not the technology specifically (though the technology is genuinely bad at its stated job), but the fact that we built a pipeline where a private company can feed directly into police use of force with essentially no meaningful checkpoint between a bad camera read and a dog bite. No human looks at the alert and thinks, wait, let me verify this before we escalate. The alert arrives and the response follows, because that's what the system is designed to do, because speed is the selling point, because the whole value proposition is that you don't have to slow down, and slowing down is exactly what would have saved Brandon Upchurch some significant pain and his landlord a lot of paperwork.

We keep doing this. We keep deciding that the machine's output is close enough to trust, and then we build enforcement infrastructure on top of that trust, and then someone gets hurt at the exact point where "close enough" wasn't, and we call it an edge case and move on. It is not an edge case. It is the predictable consequence of treating probabilistic pattern-matching as if it were certain identification, and then attaching consequences to it that a human being has to live with long after the algorithm has already moved on to the next plate. The machine doesn't remember Brandon Upchurch. The machine doesn't know he existed. The machine is already reading the next plate on the next street in the next city where the contract just renewed.

The gardener's job isn't to water whatever grows. It's to know what you're growing and why, and to yank out the thing that's choking everything else before it goes to seed. We planted this. We can still decide what it becomes. But that requires someone with the authority to make that decision to care more about the Brandon Upchurches than the contract renewals, and right now I'm not seeing a lot of evidence that person exists anywhere in this chain. Which means the rest of us have to be loud enough that the silence stops being comfortable.