Learning to Trust Anew in the Age of AI Phishing

Deepfaked calls, personalized scam messages, hyper-realistic phishing: the tools of social engineering are changing fast, making it harder than ever to catch them. Typos or strange grammar are no useful giveaways anymore. In a world where fraud is harder to detect than ever, I wanted to see just how easily that trust can be exploited, so I ran a small experiment on myself.

A few weeks ago, I asked an AI chatbot to scrape whatever it could find about me online and turn it into a phishing message. What came back was a LinkedIn message: a researcher at a well-known institute, assembling a review panel for a study on AI-powered spear-phishing, convinced that my background would be a great fit – could he send over the details? Nothing about it seems like a mass phishing attempt. It's tailored precisely to what would make me, its target, curious and flattered: a personalized scam, ready in a few clicks.

So this isn’t really about inventing new tricks. Cybercriminals have always relied on trust, urgency, and emotion. What's changing is the speed, scale, and precision with which those old tricks can now be deployed. And, even more worrying: the very signals we are used to rely on, a familiar voice, a recognizable face, a professional-sounding message, can now be faked well enough to fool even the most vigilant of us.

From bottleneck to open tap

For years, a social engineering attack sophisticated enough to fool a careful person required a small team: someone to write convincing messages, someone to script the interaction, someone to run the infrastructure. That need for coordination was itself a kind of natural defense by capping how many people a single operation could realistically target.

AI removes that bottleneck. In one documented case, a phishing scenario begins not with a link but with a phone call: an email claims the recipient's Google account has been blocked and gives a number to call for support. Whoever calls this number is connected to an AI voice agent, responding in real time, that walks them through a fake “recovery” process. This ends in a request for the six-digit code, ultimately handing attackers the account. Behind the scenes, all of this can be monitored by just one human operator, attacking targets at scale. What once needed a team is now controlled with a single dashboard.

Aimed, not blasted

The same technology that scales an attack can also tailor it. A LinkedIn profile, a conference appearance, a job title, or a recent publication is often enough for an AI system to draft a message that feels personally relevant, which is exactly what happened in my own experiment above. For a researcher, that might be a panel invitation. For a sales leader, a message referring to a recently attended event. For someone in accounting, a request impersonating a known supplier. The goal is always the same: a hook personal enough to seem legitimate, making the recipient act instead of looking closely.

What still counts as proof

Professional language, correct spelling, a recognizable voice, a familiar face, a polished-looking interface: these used to be reasonably reliable proxies for “this is legitimate.” None of them are anymore, because all of them can now be produced convincingly by AI. That doesn't mean employees should trust no one; an organization that operates on default suspicion doesn't function. It means they need to ask a different question.

Instead of “does this look professional,” the more useful question is: which signal am I actually trusting right now, and is that signal reliable in this context? A voice on the phone is a signal. A face in a video call is a signal. Professional language in an email is a signal. And all three can now be faked. An established process, by contrast, can be a genuinely strong signal, because it doesn't depend on anyone's subjective judgment. If payments require a defined approval path, a request that bypasses it gets rejected, even if it appears to come from the CEO.

The two examples below show why surface signals fail in different ways and why process is the thing that holds up in both cases.

When the face on the call isn't real

Deepfakes, i.e. AI-generated voices, images, and video convincing enough to imitate a real person, have earned their attention. But in practice, they are rarely the whole attack; they're usually one layer in a longer chain.

A documented case: victims receive a calendar invite for a video call and join what looks like an already ongoing executive meeting. No one in the meeting can hear the victim, seemingly an audio issue. Someone in the meeting sends a chat message suggesting a quick software update to fix it. That "update" is actually an infostealer. The deepfaked faces make the scene credible, but the real compromise happens because the victim is nudged into acting under mild pressure, in a moment that feels routine rather than risky.

The practical implication is that “spot the deepfake” isn't a durable defense. Unnatural blinking or garbled audio may be indicators today and invisible next year, because the technology keeps improving faster than any list of visual giveaways. What holds up is asking process questions instead: is this really the platform we normally use for this kind of call? Would this person actually ask me to install something this way? Does anything here skip a step our process requires?

The tool you already trust

The last example is a different kind of risk, because no one is impersonating a person here: the AI-generated content itself is the trap. Someone searches for a simple task, say how to free up disk space on a Mac. One of the top results looks like a shared ChatGPT conversation: clean formatting, a friendly tone, a short numbered list ending in “copy this command into your terminal.” Running it doesn't clean anything though, it installs malware.

This works because people have started to associate the chatbot format itself with trustworthiness, because it is concise, well-structured, and sounds reasonable. But that format is now something attackers can produce as easily as anyone else. Two simple habits can help not falling for the scam. First, as an immediate rule: never run a command, script, or download without understanding exactly what it does. The broader advice is that AI-generated content, including content that looks like it came from a chatbot, deserves the same scrutiny as an email or a link, not more trust because of how tidy it looks.

What this means in practice

None of this calls for panic, but it does call for an update to how organizations prepare people. Three shifts matter most.

First, training has to move at the speed of the threat. A library of last year's phishing examples won't prepare anyone for an AI voice agent or a deepfaked meeting invite. Learning needs to adapt dynamically to modern threat patterns rather than relying on outdated annual cycles.

Second, the training itself should teach context over surface features. The useful habit isn't spotting a typo; it's pausing to ask whether a request fits the channel, the process, and the moment it arrived in.

Third, that habit only works if people have somewhere to take it. Employees need a clear, low-friction way to report suspicious activity and explicit permission to verify requests. Treat human resilience as a core part of security maturity rather than a standalone exercise, building instincts directly through realistic simulations.

Learning trust anew

AI didn't invent social engineering, it just made the old version faster, larger, and considerably harder to spot. The signals people have relied on for years, a familiar voice, a recognizable face, a polished message, are no longer proof of anything on their own.

The task in front of organizations isn't to eliminate trust; it's to make it more deliberate. That means knowing which signals can be faked, which processes actually protect you, and where a request can still be verified. The organizations that adapt fastest won't be the ones that teach people to spot mistakes. It will be the ones that equip people as active defenders to pause, verify, and follow a clear process, even when the attack looks familiar, professional, and entirely convincing.

Dr. Gundula Zerbes is an Adversarial AI Researcher at SoSafe, where she studies how attackers use generative AI to scale and sharpen social engineering. With a doctorate in cognitive neuroscience, she approaches phishing, vishing and deepfake fraud as problems of human decision-making as much as of technology, analysing the psychological mechanisms that make manipulation work. She translates her findings into practical defences against emerging AI-enabled threats.