As AI becomes more capable, the future of work will not be shaped only by which tasks AI can perform. It will also depend on which human capabilities we consciously preserve.
If human agency is not protected by design, repeated reliance on AI decision-support may gradually weaken people’s ability to question, interpret and independently judge important situations. The real danger appears when AI fails: the human may still be responsible for the decision, but may no longer have the judgement capability needed to recognise the failure and intervene effectively.
In today’s world, artificial intelligence is becoming increasingly capable of making decisions for us. It can recommend which course a student should take, shortlist candidates for a job, assess credit risk, identify patients who may need medical attention, evaluate documents and increasingly act with limited human intervention.
Much of the current discussion asks whether such systems are accurate, safe, transparent or unbiased. But there is another question that may become even more important: How should AI systems be designed so that, while becoming more capable, they do not gradually make human beings less capable of deciding?
This question is addressed very directly in a recent paper by Stefan Buijsman, Sarah Carter and Juan-Pablo Bermúdez, titled “Autonomy by Design: Preserving Human Autonomy in AI Decision-Support.” The paper argues that AI systems should not merely improve decision performance. They should be designed in ways that preserve human autonomy within the specific domain in which AI is being used. The authors focus particularly on maintaining human competence and ensuring that people continue to act according to values that are genuinely their own.
This article is primarily inspired by that paper. I am using its arguments to explore a related question that I have been working on: When should AI decide, and when should final decision authority remain with humans?
It is now common to say that important AI systems should keep a human “in the loop.” But this can be misleading. A person may technically remain responsible for a decision while having very little real control over it.
Suppose an AI system recommends that a student should not receive a scholarship. A faculty member is allowed to reject the recommendation. Formally, the human is still in control.
But what if she does not understand why the recommendation was generated? What if she has no clear reason to doubt it? What if she gradually becomes accustomed to approving AI recommendations? Or what if the institution itself creates pressure to follow the system?
In that situation, human oversight exists on paper, but meaningful human agency may already be weakening. My own work treats human agency as the practical ability to understand a situation, exercise judgement, question an AI output, intervene when necessary and remain responsible for the outcome.
Before building an AI application, organisations often begin with the technology. What can the model automate? What can the agent do? How much efficiency can we gain?
Perhaps the sequence should be reversed. We should first ask what kind of decision is involved and how much authority AI should receive in that decision. There is a large difference between AI recommending an elective course and AI rejecting a student’s admission. Similarly, AI suggesting a medical test is different from AI deciding whether treatment should be denied. The technology may be similar. The consequences are not.
In my own framework, I have been looking at four broad questions: How serious are the consequences of the decision? How much authority is being transferred to AI — information, recommendation, decision or execution? Can the human still understand, question, challenge and override the AI? And who remains accountable, particularly when a person is adversely affected?
The central idea is simple. Greater AI autonomy may be acceptable where decisions are routine, low-risk and easily reversible. Stronger human control is required where consequences are serious, difficult to reverse or highly dependent on judgement.
But deciding the boundary is only the first step. Once we decide what AI is allowed to do, we must also design the human–AI interaction properly. This is where Autonomy by Design becomes particularly useful.
One of the most important arguments in Autonomy by Design concerns what the authors describe as the lack of reliable failure signals. Most conventional technologies give us some indication when something is going wrong. A machine makes an unusual noise. A dashboard displays a warning. A medical instrument may show that the input quality is poor. AI is different. A wrong answer can look remarkably similar to a correct answer. The user may therefore not know when to become cautious.
The paper argues that users need signals that give them a reason to question an AI output. For example, the system may indicate that a case is very different from the data on which it was trained, or that important contextual information may be missing. Such signals help the user move from automatic acceptance to more careful judgement.
Good AI design should therefore help us know not only what the system recommends, but also when we should stop trusting it automatically and start thinking more carefully ourselves.
Another important argument in Autonomy by Design concerns the division of work between humans and AI. A common arrangement is that AI performs most of the task while the human remains responsible for checking the result.
This sounds efficient. But it creates a strange role for the human: watch the machine continuously and intervene only when it makes a rare mistake. Over time, attention declines. Skills may weaken. And the human may become less capable of detecting exactly the kinds of errors that she is expected to identify. Buijsman and colleagues therefore suggest that, where possible, humans and AI should perform complementary tasks rather than identical tasks. The human should continue doing work that requires contextual understanding, judgement and interpretation, while AI contributes where its capabilities are strongest.
Consider higher education. Instead of asking AI to evaluate a struggling student and asking a faculty member merely to approve the recommendation, AI might analyse academic patterns, identify unusual changes and organise relevant information. The faculty member can then understand the student’s circumstances, discuss alternatives and make the consequential judgement.
Perhaps the most important long-term issue is deskilling. Suppose AI helps me perform a task better today, but after several years I can no longer perform that task without AI. Has AI really augmented my capability? Or has it created dependence?
Autonomy by Design argues that repeated AI assistance can weaken existing skills and may also prevent new skills from developing. The authors therefore suggest deliberate training arrangements where people periodically perform important tasks independently of AI.
This does not mean that every human skill must be preserved forever. Few of us worry that calculators have reduced our ability to perform long calculations mentally. But some capabilities remain essential. A physician must retain enough diagnostic understanding to recognise when an AI recommendation does not make sense. A teacher must retain enough academic judgement to question an automated evaluation. A researcher using AI must still understand enough about research to judge whether an AI-generated interpretation is meaningful.
So every serious AI implementation should ask: which human capabilities must remain strong even after AI becomes widely used? The system should then be designed to protect them.
Perhaps we need to change the way we think about responsible AI adoption. The question is not only whether an AI system is responsible. We also need to ask how much decision authority we should give to AI in a particular situation. We must also decide how the human–AI system should be designed so that people can still understand, question, learn, decide and take responsibility. The goal is not to keep humans involved in every decision. It is also not to automate everything AI can do. The goal is simple: use AI where it strengthens human capability, but set clear boundaries where increasing AI authority starts to weaken meaningful human agency
Buijsman, S., Carter, S. E., & Bermúdez, J.-P. (2025). Autonomy by design: Preserving human autonomy in AI decision-support. Philosophy & Technology, 38, Article 97.
European Parliament & Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).
Prunkl, C. (2024). Human autonomy at risk? An analysis of the challenges from AI. Minds and Machines, 34, Article 26.
