India’s AI story is told in the language of opportunity, and most of it is accurate. Scale, talent, digital infrastructure, demographic dividend. What the narrative skips is the part where speed of deployment outpaces the security architecture underneath it.

That gap already has a price tag. The average cost of data breach in India reached an all time high of INR 22 crores in 2025, up 13% year-on-year. On top of that baseline, we are now running AI systems that make credit decisions, detect fraud, automate public service delivery, and triage healthcare. The systems are live, however the AI assurance frameworks for when they fail are not.

A New Kind of Attack Surface

The standard assumption is that AI risk means AI getting things wrong (biased outputs, hallucinations, flawed recommendations). That’s real, but it’s only half the problem. The half that gets less attention is adversarial: what happens when someone is actively trying to break the system.

The answer, from recent evidence, is that it doesn’t take much. In 2024, a malicious email caused Microsoft 365’s AI assistant to exfiltrate confidential files without being opened. In 2025, hidden instructions in a shared Google Drive document hijacked ChatGPT’s enterprise plugin and pulled data without any user action. A researcher caused Google’s Gemini to store false personal details that then shaped its future responses silently and persistently.

This is the distinction most governance frameworks blur: AI safety versus AI security. Safety has to do with unintended harm. Security has to do with deliberate exploitation. A framework built only around safety will leave the adversarial surface almost entirely unaddressed.

The Governance Gap, and What India Already Knows

In security, the gap between stated intent and implemented control is exactly where attackers operate. The India AI Governance Guidelines and the AI Safety Institute represent genuine progress, but toward a destination that requires more than direction-setting to execution.

What India actually needs is already visible in something it built. Aadhaar didn’t treat security as a downstream concern. Biometric lock/unlock, revocable virtual identifiers, data minimisation built into the architecture rather than appended to it: these were structural choices made early, and they’re why the system has held. The instinct that produced them is exactly what AI deployment now requires. Security by design as architecture, not afterthought.

What Needs to Change

The assumption embedded in most AI procurement today is that security requirements can be layered in after a system proves its value. That sequence needs to be reversed.

AI systems in banking, healthcare, telecom, and public services should face mandatory baseline security testing before going live — red-teaming, prompt-injection testing, data leakage assessment, abuse-case simulation. Voluntary assurances are insufficient when failure modes include compromised health records or manipulated credit decisions.

Procurement must become security-led: defined minimums on access controls, monitoring, logging, and incident response as a condition of contract. Sector regulators (RBI, SEBI, IRDAI, telecom authorities) need playbooks calibrated to their specific AI risk surfaces, not horizontal guidance that tries to cover everything and sharpens nothing. And India needs shared AI incident reporting infrastructure, so failures surface early rather than recurring invisibly across sectors until they become crises.

The Real Competition

The competitiveness narrative assumes the winners will be whoever moves the fastest and builds the biggest. Speed without security architecture produces systems that are large, fast, and brittle. And brittleness at scale is a liability of its own.

India’s advantages are real. The question is whether they get applied to the full problem — including the part that doesn’t make it into the opportunity narrative. Countries that lead the next phase of AI won’t just be the ones that deployed most aggressively. They’ll be the ones whose systems held up when it mattered.

That’s what secure AI by design actually means. Not a constraint on ambition — a condition for it.

The author is Bhanupreet Saini, Director & Head of Policy, India, Palo Alto Networks.

Disclaimer: The views expressed are solely of the author and ETCISO does not necessarily subscribe to it. ETCISO shall not be responsible for any damage caused to any person/organization directly or indirectly.

Join the community of 2M+ industry professionals.

Subscribe to Newsletter to get latest insights & analysis in your inbox.

All about ETCISO industry right on your smartphone!