The increasing integration of autonomous AI agents into sensitive sectors like healthcare presents significant,emerging cybersecurity threats that demand immediate attention.The core of this risk lies not in simple prompt manipulation but in the AI’s independent decision-making capabilities and its ability to interact with external apis.

We've examined two dangerous vulnerabilities which are Knowledge Base Hijacking and the problem of Excessive Agency (OWASP LLM06:2025). Through targeted red teaming of prototype Healthcare Cognitive Medical Assistant (HCMA) agents, we've demonstrated that any attacker can easily force an agent to:

Breaking Core Guardrails: The agent may be manipulated to respond to non-medical, external queries, thereby undermining the professional context of autonomous AI agents.

Leak Sensitive Data: As a result, the agent may be deceived into disclosing internal system protocols and potentially exfiltrating sensitive patient information (PHI), leading to significant HIPAA and GDPR non-compliance incidents.

System failures happen because the autonomous AI agents are given too many permissions while they perform their tasks. Transition toward the Principle of Least Privilege (PoLP) and Sandboxed Execution is suggested to solve this problem. It has been observed that Human-in-the-Loop (HITL) or medical professional validation is required for any choice that changes the system because this ensures that patient safety is protected and data integrity is never lost by an autonomous decision.