Table of contents
ARTIFICIAL INTELLIGENCE & SOCIETY
PART 2
The series so far
Part 1 asked what happens when an AI agent and a compromised identity network keep acting as you after you lose control of the channels. Account recovery is not identity restoration. Institutions often trust the active session more than the human on the phone. [10]
One thing up front. The topic of AI invites panic, and I want to stay out of it. I use AI heavily, most days, for real work. I am not writing this to argue that AI is bad or that we should stop. We cannot put the genie back in the bottle. Progress does not pause because verification got harder. AI is a tool. Preparation is the point. If you understand which trust assumptions are weakening, you can decide what to build before the weakness is common.
Convincing faces already exist. That part is true enough to matter.
The real subject is trust architecture.
A lot of identity verification was built on a quiet assumption: an attacker might steal a password or a phone, but they could not convincingly reproduce several human identity signals at the same time. Face. Voice. Government ID. Live presence. Supporting channels that look like they belong to the same person.
AI changes that assumption. When synthetic signals meet a compromised identity network, weak proofs can reinforce each other until a system treats the wrong person as more credible than the real one.
This part of Artificial Intelligence & Society stays on that problem: what happens to verification when looking like you stops being enough.
Direct Answer
Yes, deepfakes can fool common identity verification steps when an attacker synthesizes a matching face, voice, or ID image and presents them together. Remote proofing trusted that several human signals at once were hard to fake. AI weakens that assumption. Strong verification now needs injection detection, device-bound keys, and human review for high-stakes checks.
A room that trusted the faces in it
In early 2024, an employee at the engineering firm Arup joined a video call. The chief financial officer was on it. So were other familiar colleagues. Over the course of the meeting, the employee approved a series of transfers worth about twenty-five million dollars. [2]
Everyone on that call except the employee was fake. The faces and voices were synthetic. Arup later confirmed that fake voices and images were used in the fraud. [1][2]
Hold the money aside for a moment and look at what the situation actually trusted. It trusted that live faces and familiar voices, moving and responding in real time, meant the real people were present. That is the same assumption a lot of identity verification rests on. Not because verification teams are naive, but because, until recently, faking several people live was genuinely hard.
Carry that assumption into remote identity proofing. This next step is a near-term projection, not a report. The Arup case was executive fraud on a conference call, not a defeat of a named identity-proofing service. I am not claiming any specific verification vendor has been beaten end to end. The trust assumption is still the same: a face that matches an ID photo, a short video selfie, a spoken phrase, a blink or a head turn on cue.
A verification system does not see a person. It sees a package of signals and decides whether the package looks complete. If live presence can be synthesized well enough to move millions across a boardroom, the practical question for identity proofing is simple: what happens when a similar package is presented to the check that decides you are you?
How remote identity verification proves you are you
Remote identity proofing usually combines a government ID, a face match, a liveness check, supporting account signals, and a human fallback. Strip the marketing language and most high-assurance remote flows still trust that short stack.
A government ID image. Front of a license or passport. Machine vision reads the fields. Rules check whether the document looks like a known template.
A face that matches the ID photo. A selfie or short video compared to the portrait on the document.
Some form of liveness. Proof that the face is not a printed photo or a frozen frame. Sometimes a head turn. Sometimes a spoken phrase. Sometimes a passive model watching for spoof artifacts.
Supporting channels. Phone ownership. Email ownership. Device reputation. Address or Social Security number checks. Fraud and watchlist screens.
A human path when automation fails. A video call with a trained agent. An in-person option for the highest stakes.
You can see the pattern in public materials. A widely used commercial verifier describes it plainly in its help documentation: upload a government ID, take a video selfie so the face can be matched to the ID, enter personal information, and escalate to a short video call with a human when self-service fails. [3] Login.gov, the federal sign-in service, reached a comparable bar a different way. It uses a photo ID plus a live selfie matched against it. It was certified to Identity Assurance Level 2 (IAL2), NIST’s label for a strong remote proofing process, and it offers in-person proofing at the Post Office as an alternative path. [6][7] Those public descriptions are useful models of what “prove you are you” often means in practice.
NIST describes the same stack in standards language. Its digital identity guidelines use those Identity Assurance Levels to separate weaker remote checks from stronger ones, up to attended, on-site proofing with biometric collection. [4][5] Revision 4, finalized in 2025, wrote injection attacks and forged media, including deepfakes, into the remote-proofing standard. The gap is no longer only a vendor marketing problem. [4]
The old trust story was simple. If someone has your password, make them show a face. If someone has a stolen ID image, make them match a live selfie. If someone has a photo of you, make them move. Stack the signals. Raise the cost.
That only works while the simultaneous-signal assumption holds.
How deepfakes attack identity verification, and what is still hard
Better looking fakes were never the real issue. What changed is what a matching face, voice, or ID photo actually proves.
A convincing ID image is no longer scarce. In a federal case, the operator of the site OnlyFake pleaded guilty after prosecutors said it sold more than ten thousand digital fake identification documents, marketed to help people get past identity-document upload checks. [8] A photo of a document is now a cheap, mass-produced input.
A face match no longer means a live human is present. Synthetic faces and real-time face reenactment attack the meaning of the match itself. The face can look right without the person being there. Arup is the documented version of that at the high end. [1][2]
Liveness does not always mean the camera saw a real face. Presentation attack detection (PAD) is the standardized defense for spoofs shown to a capture device: photos, masks, or replays held in front of a real camera. [9] Injection is a different problem. It feeds synthetic or recorded video into the software path as if it came from the camera. PAD tests what appears before a camera. Injection detection asks whether the camera supplied the media at all. A system can be strong against presentation attacks and still be weak against injection. NIST naming injection and forged media in Revision 4 put that gap on the record. [4]
Voice from short public audio is now cheap to clone. Commercial tools can synthesize a recognizable voice from a short sample of someone speaking, the kind of audio available from any podcast, video, or recorded meeting. Exact sample length varies by vendor. The direction is what matters.
Still, several things remain genuinely hard for a synthetic attacker.
Digital IDs that carry a seal from the agency that issued them, like a mobile driver’s license a DMV can digitally sign, are hard to forge in a way that changes the whole game. I will come back to those. A security key or a passkey tied to your phone still proves you control a specific device, which a cloned face does not. Rules that require a real camera, and that look for fake video fed into the system without ever hitting a lens, now written into NIST guidance, raise the bar for careful verifiers. [4] A trained person reviewing the check still catches things automation misses.
One limit on the alarm: I have not found a clean public measurement of how often production liveness and injection defenses actually fail in the wild, as opposed to in a lab or a vendor demo. Lab and research results show the attacks work. They do not prove every deployed system is already broken. Treat lab success as a warning about design, not as proof of collapse.
Put those pieces together and the change is architectural. Verification still asks for signals. The attacker does not need one stolen secret. They need a coherent package the system was built to trust.
When weak signals vouch for each other
Synthetic media by itself is a forgery problem. Synthetic media plus a compromised identity network is a trust-architecture problem. What follows is my near-term projection built on documented pieces. It is not a case file.
If an attacker only has a fake face, a good system can still demand a phone you control, an email you control, a device history the attacker does not have, or a human review that notices inconsistency.
But if the attacker already controls recovery channels, trusted devices, or account history from a Part 1-style compromise, those extra checks stop being independent. [10] They become confirmation of the same compromised graph.
That is the reinforcement loop. Face matches document. Document matches breached personal data. Phone matches the hijacked number. Email matches the hijacked inbox. Device looks familiar. Account history looks continuous. Each weak signal makes the next one look stronger. The real person calling support sounds like the outlier.
NIST treats injected, forged video paired with a stolen or fabricated identity document as a threat to remote proofing. [4] What I am adding is the network layer around it. When the signals that are supposed to cross-check each other are all inside a compromised network, cross-checking stops being a safeguard and starts being an echo.
Part 1 still holds: account recovery is not identity restoration. [10] Part 2 adds the next sentence. If verification trusts appearance packages that can be assembled without the person, restoration gets harder, not easier.
Could this really happen, and how soon
Separate the tiers. Do not blur them.
Documented now. Digital fake-ID markets exist and have drawn federal prosecution. [8] Deepfake voice and video have already been used in high-value fraud. [1][2] NIST revised its digital identity guidance to address forged media and injection. [4]
Commercially available. Voice cloning and face-swap tools are sold and marketed. Quality varies. Access does not require a research lab.
Demonstrated in controlled settings. Security research and vendor testing keep showing presentation, replay, and injection attacks against biometric capture. That is a design warning, not proof of production collapse. The wild-failure measurement is still missing, as I said above.
Possible when several conditions line up. The full Part 2 failure mode needs more than one trick: a synthetic face or document, enough personal data, control of supporting channels, and timing against a specific verification flow. Nobody does that on a whim. The architecture fails when those conditions meet.
Near-term projection. As generation quality rises and identity networks stay connected, relying on appearance as the last line of defense gets weaker. Not overnight. Not evenly. Directionally.
Push back is fair. A conference-call fraud is not the same as defeating a certified identity-proofing service at scale. Standards, mobile credentials, and phishing-resistant sign-in are already closing gaps. That objection is correct as far as it goes. The narrower point still stands: the assumption that live human signals are scarce is eroding, and a lot of architecture is still built on it, including systems certified at high Identity Assurance Levels. Certification raises the bar. It does not freeze the trust assumptions underneath.
Verification is not dead. Systems that treat looking like you as decisive proof are betting on an assumption AI is actively weakening.
Why a better face matcher will not save it
If the response is only “buy a better face matcher,” you are still inside the old architecture.
A better matcher makes the synthetic face compete harder with the real one. It does not restore the assumption that the face belongs to a person who also controls an uncompromised life. Injection stays unsolved, because the forged video never has to pass in front of a real camera. [4] A fake document that matches breached fields stays unsolved. A trusted phone that already belongs to the attacker stays unsolved.
Appearance is one signal. Proof of a person runs through issuers, devices, institutions, and time, not through a prettier match score. Tightening one biometric threshold, when the whole package can be assembled without the person, is an arms race inside a broken frame.
Safeguards worth building now
Preparation beats panic here. Verification is not suddenly impossible. Several controls still provide real assurance when they are designed and described honestly. I will mark what already exists against what is still mostly proposed. Blurring those two is its own kind of dishonesty.
Layered checks still raise cost (deployed). Document plus selfie plus phone plus fraud signals is harder to beat than any one of them alone. Harder is not solved. It is still better than a single selfie.
Human escalation still matters (deployed, uneven). A trained reviewer on a video call, or an in-person proofing session, can catch inconsistencies automation misses. NIST’s highest-assurance processes still pull toward attended proofing for a reason. [5]
Presentation attack detection still blocks some spoofs (deployed, scoped). Photos and masks shown to a camera are a real threat that PAD addresses. [9] It is not the same problem as injection. Vendors should say which one they test for. Buyers should ask.
Hardware-backed, phishing-resistant authentication still proves possession (deployed, uneven). A security key or device-bound passkey is not a face. It answers a different question: does this authenticator belong to this account? Federal cybersecurity guidance has pushed this direction for years. [15] That proves control of a credential. It does not prove presence by appearance.
Issuer-signed mobile credentials change the document problem (standard exists, rollout uneven). A mobile driver’s license (mDL) built to the ISO/IEC 18013-5 standard is not a picture of a plastic card. It is data signed by an issuing authority, so a verifier can check origin and integrity rather than trusting that an uploaded image looks right. [13] The infrastructure to distribute and trust those issuer signatures across jurisdictions is being built now, not finished. [14] This is the most promising direction, and it is not yet everywhere. Where it exists, prefer it over a photo of plastic.
What still works shares a pattern. It trusts something a synthetic person cannot casually recreate from public video: the issuer’s signature, a key on a real device, a person who can still say no, or a rule that refuses to treat one coherent-looking package as final.
What the law and institutions should require
Law will lag. It usually does. Existing identity-document and wire-fraud statutes already reach many of the outcomes when someone uses a synthetic identity to steal. [16] Regulators have started naming the harm more directly. The Federal Trade Commission has moved to extend impersonation protections to cover AI-enabled impersonation of individuals, and it has warned that biometric technologies, including deepfakes built from biometric data, create risks it intends to scrutinize. [11][12] Synthetic impersonation is a consumer-protection problem now, not a novelty.
The hard questions are operational, not rhetorical. Who has to restore the person of record when the winning package was synthetic, and how fast? What must a verifier disclose about testing for injection as well as presentation attacks? What logging, preservation, and appeal rights exist when appearance itself is contested? Most of that is still proposed rather than settled. Part 1 made the same argument about duties after credible compromise notice. [10] Part 2 adds the verification layer: logs, appeal rights, and preserved disputed sessions matter most when the fake package looks complete.
What should replace looking like you
Go back to the meeting, and to every remote check built on the same assumption. One package of signals looked complete. Whether that package belonged to a person was a separate question the system was not really asking.
If looking like you is no longer enough, the replacement is not one gadget. It is a shift in what counts as decisive proof.
Prefer issuer-backed credentials over document images where they exist. Verify signatures and check status instead of trusting that a photo of plastic is an issued credential. [13] Prefer phishing-resistant, hardware-backed authentication for account control. Keep “can operate this account” separate from “uploaded a matching selfie.” [15] Treat injection and forged media as first-class threats in remote proofing, the way NIST now does, and expect verifiers to say what they actually test for. [4] Keep a human path for identity-critical failures. Someone has to reopen the case when the real person is locked out by a coherent fake. Bind high-risk changes, new payees, recovery rewrites, credential rebinding, to confirmation through a channel the contested session does not already control.
None of that requires believing AI is evil. The genie is out. We do not get to rewind capability. We get to decide whether proof keeps up. These checks were built when faking a face, a voice, an ID, and live presence all at once was rare. That is getting less rare every year.
Looking like you is no longer enough as the last line of defense. What replaces it has to be harder to recreate from public media and a compromised inbox: the issuer’s signature, a key you hold, a second channel the contested session does not already control, and a person who can still say the coherent package is wrong.
The systems that survive will not be the ones with the prettiest face matcher. They will be the ones that stop confusing a matching appearance with a person.
Frequently Asked Questions
Can deepfakes defeat remote identity verification and selfie checks?
Sometimes, under conditions. Deepfake video and voice have already been used in a documented twenty-five million dollar fraud on a live call. [1][2] Digital fake-ID markets have drawn federal prosecution. [8] Researchers keep demonstrating presentation and injection attacks in controlled settings. What has not been shown publicly is a certified identity-proofing service beaten end to end at scale. Lab success warns about design. It does not prove every deployed check is already broken. [4]
What is layered identity verification?
Layered identity verification stacks more than one check: a government ID, a face match, a liveness check, supporting account signals, and a human fallback when the remote stack fails. [4] One strong layer does not save a weak stack. Document plus selfie plus phone plus fraud signals is harder to beat than any single selfie. Harder is not solved. Biometrics and liveness can still be attacked through presentation or injection, which is why NIST’s digital identity guidelines treat both as threats remote proofing has to address. [4][9]
What is the difference between presentation attack detection (PAD) and injection attacks?
Presentation attack detection tests what appears in front of a real camera: printed photos, masks, replayed video held up to the lens. [9] Injection skips the camera. It feeds synthetic or recorded video into the software path as if the camera had captured it. A system can be strong against presentation attacks and still weak against injection. That is why NIST’s 2025 digital identity guidelines name both as threats remote proofing has to address. [4]
What is a mobile driver’s license (mDL), and why does the cryptography matter?
An mDL built to the ISO/IEC 18013-5 standard is not a photo of a plastic card. It is identity data signed by the issuing authority, so a verifier can check where the data came from and whether it was altered. [13] That moves trust from appearance to issuer signature. The infrastructure to distribute and trust those signatures across jurisdictions is still being built. [14]
What still works when looking like you is not enough?
Controls a synthetic attacker cannot casually recreate from public media: issuer-signed credentials instead of document images [13]; hardware-backed, phishing-resistant authentication that proves possession of a key rather than a matching face [15]; attended human proofing for the highest stakes [5]; and verifiers that test for injection as well as presentation attacks, and say which one they test for. [4]
How is this different from Part 1 of the series?
Part 1 examined what happens after account takeover: recovery flows, hijacked channels, and institutions trusting the active session over the person on the phone. [10] Part 2 moves one layer down, to verification itself. If the package that proves you are you can be assembled without you, restoration gets harder, because the check meant to rescue the real person is the same check the fake package was built to pass.
References
[1] The Guardian. “UK engineering firm Arup falls victim to £20m deepfake scam” (May 17, 2024). https://www.theguardian.com/technology/article/2024/may/17/uk-engineering-arup-deepfake-scam-hong-kong-ai-video
[2] Financial Times. “Arup lost $25mn in Hong Kong deepfake video conference scam.” https://www.ft.com/content/b977e8d4-664c-4ae4-8a8e-eb93bdf785ea
[3] ID.me Help Center. “Verify your identity with ID.me Self-Service” (public flow description; example only). Retrieved July 16, 2026. https://help.id.me/hc/en-us/articles/9198013708439-Verify-your-identity-with-ID-me-Self-Service
[4] NIST. Digital Identity Guidelines, SP 800-63-4 (final, 2025), including injection and forged-media controls. https://pages.nist.gov/800-63-4/ · https://csrc.nist.gov/pubs/sp/800/63/4/final
[5] NIST. Digital Identity Guidelines: Identity Proofing and Enrollment, SP 800-63A-4. https://pages.nist.gov/800-63-4/sp800-63a.html
[6] U.S. General Services Administration. “GSA’s Login.gov announces certification of IAL2” (Oct 9, 2024). https://www.gsa.gov/about-gsa/newsroom/news-releases/gsas-logingov-announces-certification-of-ial2-10092024
[7] Login.gov. “Our services” (identity verification: photo ID plus selfie; in-person option). https://www.login.gov/partners/our-services/
[8] U.S. Attorney’s Office, Southern District of New York. “Creator Of ‘OnlyFake’ Charged And Pleads Guilty To Selling More Than 10,000 Digital Fake Identification Documents.” https://www.justice.gov/usao-sdny/pr/creator-onlyfake-charged-and-pleads-guilty-selling-more-10000-digital-fake
[9] ISO/IEC 30107-3:2023. Information technology — Biometric presentation attack detection — Part 3: Testing and reporting (scope: attacks at the capture device during presentation). https://www.iso.org/standard/79520.html
[10] Andrew Drasen. “The AI Identity Problem: Can AI Agents Hijack Your Digital Identity?” A Vision of Hope Insights (Part 1). /insights/artificial-intelligence-society/the-ai-identity-problem
[11] Federal Trade Commission. “FTC Proposes New Protections to Combat AI Impersonation of Individuals” (Feb 15, 2024). https://www.ftc.gov/news-events/news/press-releases/2024/02/ftc-proposes-new-protections-combat-ai-impersonation-individuals
[12] Federal Trade Commission. Policy Statement on Biometric Information and Section 5 of the FTC Act. https://www.ftc.gov/system/files/ftc_gov/pdf/p225402biometricpolicystatement.pdf
[13] ISO/IEC 18013-5:2021. Personal identification — ISO-compliant driving licence — Part 5: Mobile driving licence (mDL) application. https://www.iso.org/standard/69084.html
[14] American Association of Motor Vehicle Administrators. “Mobile Driver’s License (mDL) Digital Trust Service.” https://aamva.org/identity/mobile-driver-license-digital-trust-service
[15] Cybersecurity and Infrastructure Security Agency. “Implementing Phishing-Resistant MFA” (fact sheet). https://www.cisa.gov/sites/default/files/2023-01/fact-sheet-implementing-phishing-resistant-mfa-508c.pdf
[16] 18 U.S.C. §§ 1028, 1343 (identity-document fraud; wire fraud). https://www.law.cornell.edu/uscode/text/18/1028