Judged by a Machine

The last piece was about being steered, a force aimed at what you see and believe, and it still left you plenty of room. This one leaves you less, because here you are not the user of the machine at all. You are its object. A score, a flag, a ranking is generated about you, often without your knowledge, by a system you cannot inspect, using logic no one will explain, with an appeal route that is either missing or for show. This is the most concentrated form of the whole series' question. Not what you can do with a tool, but whether you can talk back to one that has already decided something about you.

When it goes wrong, it goes wrong for everyone at once

Two cases tell you what is really at stake, and neither is a glitch. In the Netherlands, the tax authority used a risk system to hunt childcare benefit fraud, and it treated things like a second nationality or a name that sounded foreign as grounds for suspicion. Around 26,000 families, many of them immigrants, were wrongly branded fraudsters and ordered to repay tens of thousands of euros, often all at once. Lives were destroyed, children were taken into care, and the scandal was severe enough to bring down the national government in 2021. Years on, watchdogs say it still is not properly fixed.

In Australia, a scheme now known as Robodebt used crude automated income averaging to raise debts against roughly four hundred and fifty thousand people, many of whom owed nothing. A royal commission called it crude and cruel, neither fair nor lawful, and connected it to real deaths, and the government has since had to pay back and compensate to the tune of billions. Read those two together and the pattern is clear. These were not computers making rare errors. They were institutions hiding a harsh decision behind a machine and quietly betting that almost no one would have the knowledge or the strength to fight back.

That is the danger unique to automated judgment. A human clerk who is wrong is wrong about one person at a time. A model that is wrong is wrong about everyone it touches, identically and instantly, and the single flawed assumption inside it scales to hundreds of thousands before anyone notices.

It lands hardest on the people with the least room

And it does not land evenly. When journalists got hold of the actual welfare fraud model a Dutch city was using, it scored you as riskier for being a woman, a single parent, or someone not fluent in the language, and the city eventually suspended it. Similar systems have surfaced elsewhere in Europe with the same tilt. In the United States, a fight that has run for years over a tool that scores criminal defendants found that Black defendants who did not go on to reoffend were flagged as high risk at nearly twice the rate of white defendants who did not.

That last case is worth slowing down on, because it is genuinely contested rather than a tidy villain story. The tool's makers could show it was, by another reasonable measure, equally accurate across races. It turns out that when two groups are arrested at different rates to begin with, it is mathematically impossible for a scoring system to be fair in every sense at once. That is not an excuse, it is a deeper problem, it means "is this algorithm biased" often has no clean yes or no, and a company can honestly call its tool fair while it still lands unfairly on you. Fairness is not a setting someone forgot to switch on. It is a genuine dilemma, which is exactly why leaving it to an unaccountable model is the wrong move.

The quiet bet that you won't appeal

The sharpest edge is the appeal, or the lack of one. A class action in the United States alleges that a health insurer used an algorithm to deny elderly patients' rehabilitation care, and that when denials were challenged they were overturned about nine times in ten, yet only a tiny fraction of patients, on the order of one in five hundred, ever appealed. Sit with that shape. If almost every appeal wins and almost no one appeals, the system is not really predicting anything. It is running on the certainty that you will accept the verdict. The insurer says the tool only guides its staff, and these figures are still allegations being fought in court, but the structure is the thing to see, because it repeats everywhere. Gig workers are "deactivated," fired by software, with no human to ask why. The whole model is built on your silence.

Both sides, honestly

Now the other half, because the fix is not as simple as "ban the machines." The human systems these replaced were also bad, and often biased in their own quiet ways. A caseworker with thirty thousand files applies tired, uneven, sometimes prejudiced judgment, and a loan officer's hunches had documented racial patterns long before any algorithm existed. Automated scoring can be more consistent, can be checked in principle, can be faster and can widen access, an instant decision beats waiting three weeks for a lot of people. The problem is not that institutions use models to make decisions at scale. They have to, and they always did. The problem is that they do it without recourse. And even attempts made in good faith to build a fair one have failed, which tells you this is hard, not that it is hopeless.

What stays yours

So your agency here is smaller than the tech optimists admit and larger than the doom story allows. You usually cannot opt out of being scored, that ship is not in your harbour. But you can know it is happening, you can demand the explanation the law increasingly says you are owed, and you can appeal, because these systems are quietly built on the bet that you will not.

And the law is, slowly, starting to take your side. Europe's top court has ruled that an automated credit score is itself a decision you have rights over, its data rules give you a right not to be subject to purely automated decisions that seriously affect you, and its new AI law puts hiring, credit, benefits and justice tools in its strictest category, with human oversight required. The catch, and it is a real one, is that these protections are largely European. In much of the world, including the United States, there is no federal equivalent, and the recourse is thinner. So the honest instruction is this. Learn to notice when a machine, not a person, just decided something about your money, your job, your health or your freedom, and treat that as the trigger to ask how, and to push back, with your eyes open, because the whole apparatus assumes you never will.

That was a decision aimed squarely at you, one person at a time. The last of these outward pieces pulls all the way back, to the costs that are not aimed at anyone in particular, the ones the whole world pays whether or not it ever asked to.