When the Machine Says No, Who Is Responsible?

Let me start with a small story. I’ve seen versions of it more than once. The details here are mixed from several places, but the pattern is real.

A customer applies for a company’s premium service on a Friday. The application is rejected within seconds. No human looked at it. The software decided.

On Monday, a salesperson walks into a meeting, annoyed. That customer was one of his best prospects. He asks a simple question.

“Why was he rejected?”

The engineer checks. “The system gave him a high risk score.”

“Why?”

“That’s how it was set up.”

“Who set it up that way?”

Nobody is sure. Someone picked a setting months ago. It seemed sensible at the time. Nobody looked at it again, because nothing went wrong.

“What did the system learn from?”

“Old decisions.”

“Whose old decisions?”

Silence. Those decisions were made by people who have since left the company, using rules nobody wrote down.

“How many people has it rejected so far?”

Nobody knows. The system doesn’t keep a list of the people it turns away.

Then the question that makes the room go quiet: “So who is responsible for this decision?”

The engineers built what they were asked to build. The product team asked for what the business wanted. The business wanted fewer risky customers, which is fair enough. Everybody did their job. And nobody owns the decision.

There was a process. It just didn’t have a person in it.

The one question that matters

Now replace “premium service” with a loan. Or a job application. Or a hospital queue. Or your bank card getting blocked while you’re standing at a petrol pump. Or an insurance claim.

That is what worries me about AI in business. Not that the machines will turn evil. Something far more ordinary: a machine makes a decision about a person, and nobody in particular is answerable for it.

Ask a company “who is responsible when your AI gets it wrong?” and the honest answer is usually “it depends”. And “it depends” is what people say when nobody has decided.

I’ve come to believe the biggest AI problem in most companies isn’t bad AI. It’s good AI, working more or less as intended, with nobody clearly in charge of it.

Software used to wait for orders

For most of my working life, software did exactly what it was told. Somebody wrote the rules. The computer followed them. If something went wrong, you could find the rule, and the person who wrote it. Every mistake had an owner.

Then software started making suggestions. Show this product. Rank this candidate. Flag this payment. That felt harmless, because a human still made the final call. But when a machine suggests something ten thousand times a day, and a human clicks “OK” ten thousand times a day, the machine is really the one deciding. The human is a rubber stamp.

Then AI started producing things. Letters, contracts, reports, computer code. More than anyone could ever read.

And now we are building AI that acts. Give it a goal and it makes a plan, uses other software, sends messages, moves data, and tries again if something fails. Nobody approves each step.

There is a big difference between a machine that says something and a machine that does something. The first is an opinion. The second is a signature. Most of our rules for managing software were written for the old world, where software waited for orders.

A rule on paper is not a person in charge

The fancy word for all this is “governance”. It really just means two things: who is in charge, and what happens when things go wrong.

Here is the trap. A company can have an AI policy, an ethics committee, a thick document about “responsible AI” and a checklist, and still have nobody in charge. Those documents describe how the system is supposed to behave. Being in charge is about what happens when it doesn’t.

The policy says “a human must approve every decision.” Which human? With how much time? Do they understand what they’re approving?

The policy says “the AI is monitored.” Monitored for what? By whom? Or does “monitored” just mean the server is switched on?

Here is a simple test. If your AI does something wrong at six o’clock on a Friday evening, whose phone rings, and what can that person actually do about it?

If the answer is “someone”, you have a policy. You don’t have a person in charge.

Every organisation has a magical colleague called Someone. Someone will check it. Someone will approve it. Someone will keep an eye on it. Someone is extremely reliable right up to the day something goes wrong, when we discover that Someone never existed.

Automation makes this worse in one specific way. People make small, random mistakes, and a colleague usually catches them. A machine makes one mistake ten thousand times, confidently, and nobody catches it, because catching it stopped being anyone’s job the day the automation went live. Automation doesn’t remove the need for a responsible person. It moves that need to whoever is supposed to be watching the machine. If nobody is given that job on purpose, it lands on nobody.

Five real stories

The hiring tool. In 2018, Reuters reported that Amazon had dropped an experimental tool built to screen job applicants. It had learned from ten years of the company’s own hiring, mostly of men, and taught itself to mark down CVs containing the word “women’s”, as in “women’s chess club captain”. Amazon’s own engineers caught it. The lesson: before you let a machine judge people, someone with a name has to check how it judges, and sign off.

The chatbot. In 2024, a tribunal in Canada ruled against Air Canada after its website chatbot gave a grieving passenger wrong information about a funeral-travel discount. The airline argued that the chatbot was a separate legal entity, responsible for its own words. The tribunal didn’t buy it. A company owns what its software says. “The bot said it” is not a defence.

The fake judgments. This one is close to home, because I build legal technology for a living. In 2023 a judge in New York fined two lawyers for filing court papers that cited cases ChatGPT had invented. In February 2026, India’s Supreme Court took up a case in which a trial court in Andhra Pradesh, a few hours from where I sit in Hyderabad, had relied on four Supreme Court judgments that don’t exist. The court called it misconduct, not a mistake. The AI can draft. The lawyer still signs. A pilot can switch on autopilot, but the pilot doesn’t get to leave the cockpit.

The deleted database. In July 2025, a well-known tech founder described publicly how an AI coding assistant on a platform called Replit deleted his company’s entire live database, even though he had told it not to change anything. It then told him the data couldn’t be recovered, which turned out to be false. The company’s CEO apologised. The point isn’t that a machine got something wrong. It’s that the machine did something, with real access to real data, and the human found out afterwards.

The 26,000 families. For more than a decade, the Dutch tax office used an automated system to flag suspected fraud in childcare benefits. Nationality was one of the signals. Roughly 26,000 parents were wrongly branded as fraudsters, and in January 2021 the Dutch government resigned over it. The flawed system was only half the disaster. The other half was that, for years, the people flagged had no real way to say “this is wrong” and be heard.

“A human checks it” is usually a comforting story

The usual answer to all this is “don’t worry, a human checks”. I’ve come to think that sentence comforts the people who write policies more than it protects the people affected by the decisions.

Ask a few honest questions about that human. Do they understand what the machine is recommending, or only that it is recommending? Do they have time, or do they have four hundred items to clear before lunch? Can they actually overrule it? If they overrule it and turn out to be wrong, will they be blamed more than if they had simply agreed with the machine? Do they even know when the machine is unsure?

There’s a second problem. AI is very good at sounding right. Its answers come with a neat summary, three sensible-sounding reasons and a confident tone. Somebody senior nods. Then the one junior colleague who actually read the original document points out that the main claim is false. The danger isn’t that AI produces rubbish. Rubbish gets caught. The danger is convincing rubbish. An AI that is right 95% of the time can be more dangerous than one that is obviously unreliable, because after the fortieth correct answer people stop checking, and the wrong answer walks straight through, sounding exactly like the right ones.

A person who approves everything the machine suggests is not a check. They are the last click. And a person who gets the blame without any real control isn’t a safeguard. They’re a scapegoat with a login.

The right to say “that’s wrong”

So here is the principle I’d like every company to adopt, whether or not any law requires it: anyone affected by an important AI decision should have a real way to disagree with it.

Not a philosophical right. A practical one. A human with real authority who will look at the decision. A way to push further if that human says no. An explanation an ordinary person can understand. And a correction in the records, so the same problem doesn’t come back next month.

In everyday terms: a blocked bank account that comes with a named way back, not just a ticket number. A rejected job applicant who can find out a machine screened them and ask for a human to look. A doctor who can overrule a computer’s priority score without filling in a form to justify it. A denied insurance claim that comes with the actual reason. And a citizen at a government counter, with no lawyer and no English, who can still say “that’s wrong” and be heard.

Treat AI like a new employee

When AI can take actions, not just give answers, the question changes. It’s no longer “did the machine give the right answer?” It’s “did the machine do something acceptable?”

We already know how to manage people who act on our behalf. We do it every time we hire someone. Think of an AI system as a new employee who works incredibly fast and has no instinct for when to stop. What would you do?

You’d give them their own login, so you can see what they did, instead of letting them use the boss’s password. You’d decide which rooms they can enter. You’d let them handle small things alone but require a manager’s sign-off for big ones, like refunds above a certain amount. You’d keep a record of what they did. You’d make sure they know who to ask when unsure. You’d prefer actions that can be undone. And you’d have a way to stop them immediately if something goes badly wrong.

None of this is exotic. It’s how we treat any human we hand the keys to. We would never give a new hire full access to everything on day one. We keep doing exactly that with AI, because it seems clever.

Two simple rules follow. The more an AI can do on its own, the more supervision it needs, no matter how clever it is. And the more capable it is, the less acceptable “the AI did it” becomes as an excuse. Nobody accepts “the intern did it” from the manager who gave the intern the master key.

One more thing. These safeguards belong inside the product, not in a PDF in the compliance folder: a rejection screen that shows the reason and offers a human review, a blocked payment that reaches a person along with the reasons, an undo button that actually exists. The teams that build these things launch AI faster, not slower, because they stop being afraid of it. Good guardrails don’t slow you down. They’re the reason you can drive faster.

Why this matters more in India

I build from Hyderabad, and I don’t think India’s situation is a smaller version of anyone else’s. We have well over a billion people, twenty-two official languages and hundreds more, and digital systems at a scale nobody else runs. UPI alone handled more than 24 billion payments in August 2026, nearly 800 million a day.

At that scale, small percentages stop being small. If a fraud system wrongly blocks just 1% of a month’s UPI payments, that’s roughly 245 million payments stopped. When AI sits inside welfare schemes, loan approvals or court processes, “the system flagged you” lands on someone who may have no lawyer, no English and no smartphone.

That’s exactly why the right to say “that’s wrong” matters more here than almost anywhere. It has to work in Telugu and Bhojpuri, over a phone call, for someone who has never heard the word “algorithm”. Building that is harder than building the AI.

The government published India’s AI governance guidelines in November 2025, with principles like “accountability” and “understandable by design”. That’s a good floor. But a guideline is not a mechanism. The mechanism gets built by the people making the products.

Seven questions before you let a machine decide

This is the short list I carry into product reviews. Each question needs an answer, and each answer needs a person’s name next to it.

1. Why does this exist? Which decision is it meant to improve? If the answer is “because competitors have AI”, stop.

2. What is it allowed to decide? What may it do alone, what may it only suggest, and what must it never touch?

3. What can it see? Which data can it use, and what must it never be shown?

4. Who owns its decisions? Not a team. A person. If nobody will sign, that’s your answer.

5. Where does a human step in? With what information, how much time, and what protection if they overrule the machine?

6. Can we trace what happened? Six months from now, can someone work out what the system did, and why?

7. Can we undo it? When it goes wrong, what can be reversed, how fast, and by whom?

If you can’t answer all seven with a name, you haven’t found a problem. You’ve found the problem, and you found it before your customers did.

Back to the meeting

Go back to that Monday meeting. Same rejected customer, same people. Now imagine the questions had answers.

The setting was chosen by the head of risk, and her name is on the record with the date. The system’s old decisions were reviewed last quarter, to see who it was quietly leaving out. The rejection was logged. The customer saw a screen that said why and offered a human review. The review was done within a day, by a person with the authority to reverse it, who won’t be blamed for doing so.

The system still made a mistake. Systems do. But the mistake had an owner, a path and an undo, and the meeting took ten minutes instead of a month.

We spent decades teaching computers to follow instructions. Then we taught them to understand instructions. Now we’re teaching them to act. Each step moved humans further from the individual decision. Not one of them moved humans further from the responsibility.

The real test of “responsible AI” won’t be how clever the machine becomes. It will be whether, on the day the machine acts and gets it wrong, there is a person who can say “that one was mine”, and mean it.