AI Between Autonomy and Oversight: Why Humans Must Remain in Control

Artificial intelligence is no longer an experiment, but an integral part of economic processes. It decides on loan approvals, sorts job applications, assesses security risks, and writes code. The crucial question is therefore no longer whether, but how AI can be integrated safely and responsibly.

However, with increasing performance come new vulnerabilities. Generative models not only deliver impressive texts or images, but also tools for fraud, manipulation, and disinformation. The more autonomously systems operate, the more important human control becomes.

When reality becomes negotiable

With generative AI, the lines between perception and manipulation begin to blur. Deepfake videos, synthetic voices, and deceptively real chat dialogues make a person’s identity in the digital space increasingly uncertain.

Attackers exploit this development in a targeted manner: fake video conferences in which supposed executives give approvals, or AI-supported phishing campaigns that imitate the language and style of real colleagues.

In one documented case, such a deepfake call led to a transfer of USD 10 million – triggered by a virtual but believably animated copy of a CFO. Such incidents mark a turning point: authenticity is no longer a technical property, but a question of governance.

The blind spot of automation

AI has long been considered a key factor in the security industry.

Systems detect anomalies, assess risks, and respond in seconds. But machine precision is no substitute for judgment. Models can form erroneous correlations, respond to manipulated inputs, or simply misinterpret what they see. A purely automated defense therefore carries a structural risk: it responds to patterns, not motives. When context is missing, efficiency becomes a danger.

Human in the Loop – control as a system principle

The solution lies not in more AI, but in smarter integration. The concept of Human in the Loop describes an architecture in which humans remain a conscious part of the decision-making and control loop. In practice, this means that analysts review the results of models before automated measures are triggered. Analysts feed this feedback into the systems to correct misclassifications and adaptively improve models. Security-related decisions – such as incident response or financial approvals – remain the responsibility of humans. This creates a balance between the speed of machine processes and the plausibility of human judgment.

AI as a defender

The duality of AI is obvious: the same technologies that create deepfakes can also expose them. Machine learning models help to detect synthetic content, uncover unusual communication patterns, and stop attack chains early on. In modern security operation centers, AI-supported systems filter billions of log entries, prioritize threats, and automate responses. But the decisive strength only emerges when they’re combined: AI detects – humans evaluate.

Governance is a prerequisite

The upcoming EU AI Act shifts responsibility to companies.

It demands transparency, traceability, and human oversight – precisely the principles that are necessary in the security context anyway.

Governance is therefore not a regulatory addition, but an operational necessity. Clear responsibilities, documented decision-making processes, and defined control points prevent automation from becoming a black box.

AI systems must remain verifiable, correctable, and explainable – not only technically, but also organizationally.

Ethical responsibility for data and decisions

Artificial intelligence is changing company security architectures – and with them, the understanding of responsibility. Automation creates efficiency, but it must never be an end in itself. Trust arises where machines calculate and humans decide. The human-in-the-loop approach is not a nostalgic brake, but the necessary counterpoint to a technology that lies ever more realistically.

In addition to technical security and governance, the ethical dimension of AI is also central. The quality and integrity of the data used to train systems determine how fair and trustworthy their decisions are. Principles such as non-harm, fairness, data protection, and accountability must therefore be consistently applied. The German UNESCO Commission warns that AI can reinforce existing stereotypes – for example, regarding gender, origin, or age – if training data is unbalanced. This is why diversity-conscious development teams, transparent data processes, and independent controls are needed. Only in this way can it remain clear how a system arrives at its results. Human autonomy, privacy, and a focus on societal welfare must remain the guiding principles of all AI development – so that progress does not come at the expense of dignity.

This article is based on the white paper “Artificial Intelligence in IT Security – Opportunities, Risks, and Protective Measures” by the Security Competence Group at eco – Association of the Internet Industry, which will be published at the end of 2025 here.

Olaf Pursche is an independent IT security consultant, trusted advisor, journalist and keynote speaker. Since 2025, he has headed the Security Competence Group of the eco Association, prior to which he advised eco on IoT security. For 15 years, he was responsible for IT security for all European editions of COMPUTER BILD and wrote for BILD, Welt, iX, kes and IT-Sicherheit, PC Professionell and other magazines and newspapers. He was a member of advisory boards at the BSI, the GDV and other expert committees. He was also responsible for communications, press relations and marketing for 10 years as CCO of the AV-TEST Institute and as Head of Marketing & Communications at the SITS Group.