How Much Authority Should You Give AI in Your Software?

Ask your team what the AI in your product is allowed to do, and you will usually get a list of features. It drafts support replies and flags accounts that look likely to cancel. 

Ask who decided it could send those replies without a person reading them first, and you will often find that it happened gradually, one reasonable shortcut at a time.

The better approach is to give each AI feature only the authority its job requires, decided on purpose, with a named person behind the result.

‍

Should AI be allowed to do everything it is capable of?

AI should not be allowed to do everything it is capable of, because the same capability can protect a business or harm one depending on who is using it and why.

Anthropic applies that thinking to its own models. Its Cyber Verification Program gives vetted security organizations access to Claude capabilities that stay restricted for everyone else, because the work that helps a defender understand an attack could also help someone carry one out. On July 9, 2026 Mitiga announced it had joined, describing the program as a vetting process for "legitimate dual-use defensive work."

Every customer gets the same model, and Anthropic decided on purpose who could use the restricted parts of it and under which controls. Businesses rarely make the equivalent decision about the AI inside their own software, even though their customers are the ones who carry the risk when it goes wrong.

‍

How does AI end up with more authority than anyone intended?

AI usually gains authority through small operational shortcuts, each of which makes sense on the day someone takes it.

Your support team starts using AI to draft replies, and a person reads each one before it goes out. The drafts are good and the queue gets long, so someone suggests sending the simple ones automatically. Then the definition of simple widens, because every reply that goes out untouched is time saved. A few months later the AI is answering customers on your company's behalf, and there was never a meeting where anyone decided it should.

The risk in any of this depends far more on what the AI is allowed to touch than on which model you use. An AI that reads your support history can give you a wrong answer. An AI that sends email can give your customer one.

In April 2025 the AI agent Cursor used as the first filter for support email, signing its replies "Sam," told users their subscription allowed only one active session. That policy did not exist. Some users canceled before co-founder Michael Truell posted "We have no such policy," and the company began labeling AI-written support replies. The bot was doing the job it had been given, and that job included speaking for the company to paying customers with no person checking what it said.

‍

What are the levels of AI autonomy in business software?

AI autonomy can be described in five levels, and naming the level for each feature is what keeps its authority from drifting upward without anyone deciding. Here is the same support feature at each one:

1. Observe. The AI reads a week of tickets and summarizes what customers keep asking about.
2. Recommend. The AI suggests which ticket to answer first and which help article fits it.
3. Prepare. The AI drafts the reply, and a person reads it and presses send.
4. Act within limits. The AI sends replies on its own for a defined set of questions, like password resets and shipping status, and routes everything else to a person.
5. Operate independently. The AI answers any customer question, issues credits, and updates the account across several systems with little human involvement.

The support team's drift was a jump from level three to level five that no one wrote down. Level four is where a lot of teams would have been comfortable, and it only exists if someone defines the limits. Each level higher also needs better logging and a faster way to undo a mistake, and those belong in place before the feature moves up, because teams rarely go back and add them afterward.

‍

When should AI need human approval before it acts?

AI should need human approval wherever a mistake would be costly or hard to undo, and that approval only protects you if the person reviewing has the time and context to say no.

A manager working through a long queue of AI-drafted refunds with one click each is signing off on the AI's judgment without applying their own. The business pays for a human in the loop and gets none of the protection, and the approval log makes it look covered.

Approval works best when each action gets a checkpoint that matches what a mistake would cost:

• Formatting a report can run on its own, because a bad format costs someone a minute.
• Issuing a refund can run automatically below a set amount and wait for a manager above it, so the manager's attention goes to the refunds that matter.
• Changing a customer's contract should wait for a person every time, with the full account history in front of them.

‍

Who is responsible when an AI action goes wrong?

A named person should own every AI action the way they would own a process their team runs, including the decision about whether the AI keeps that permission after a mistake.

Once systems are connected, one AI action can reach several departments. Say an AI marks an account as churned because usage dropped for a month. Billing stops invoicing it, marketing adds it to a win-back campaign, and next month's report counts it as lost revenue. The customer was on vacation. Every downstream system did exactly what it was told, and each team assumed the account flag was someone else's to check.

Before a feature like that goes live, your team should be able to point to the person who notices the mistake and the person who decides whether the AI should still be allowed to set that flag.

‍

How should AI permissions be built into software?

AI permissions should be designed the same way user roles are, before the feature is built, with the AI given only the access its current level requires.

We treat an AI feature like another user of the system. It gets a defined role, a written list of what it can read and change, and a decision about whether it acts with the permissions of the person using it or with its own. Its level on the ladder goes into the spec alongside everything else, so moving it up becomes a change someone has to approve, the same as any other change to what the software does. Everything it does is logged, and anything it changes can be reversed. We covered the agent side of this in an earlier article.

For Witherspoon Rose Culture we delivered an AI integration for reporting alongside Quickrose, the operations platform we built to run their wholesale rose business. Reporting fits naturally on the lower levels of the ladder, because the AI's job is helping people see the business clearly, and the decisions about what to do next stay with the people running it.

We believe that business is built on transparency and trust, and that good software is built the same way, which for AI means anyone on your team should be able to see what it did and why.

‍

What should you check before giving AI access to a business system?

Before AI goes into a workflow, your team should be able to answer eight questions in writing. If an answer is missing, the feature is not ready.

1. Level. Which level of autonomy is this feature starting at, and who has to approve moving it up?
2. Access. What information does it actually need to do the job?
3. Permissions. Can it only read, or can it also change, delete, or trigger other systems?
4. Boundaries. What should it never be allowed to do?
5. Approval. Which actions wait for a person, and does that person have what they need to say no?
6. Identity. Does it act with the user's permissions or its own?
7. Logging and recovery. Can you see exactly what it did, and can you undo it?
8. Ownership. Who is responsible when it gets something wrong?

‍

How much autonomy should AI have in your business?

AI should get a level of autonomy that matches what its mistakes would cost, and that level should rise only after the feature has proven itself in use.

The strongest AI features we see belong to teams that decided exactly what the AI could touch, and moved it up a level only after it had earned that trust.

Pick one AI feature in your product this week and write down which level it sits on today. Then ask when it got there, and who decided.