AI & AgenticAI Security

An answer to a stupid question needs to be stupid as well.

 

Absolute security: now no one will be able to break the door of my house…

Kindergarten Habits

When IT starts creating something new to it, people demonstrate a kindergarten syndrome, which I can articulate as “I am the first on the planet who figured it out”. Another is a boom syndrome: “Everything around me should incorporate or be viewed from what I do.”

This article is about security means for AI (generative AI) and Agents utilising this AI.

A quite popular opinion is that previously existing security threats, concerns and means are not enough for an AI.  Really? Let’s set the case and see what holds the truth when one ‘graduates’ from a kindergarten. Let’s start with threats. The major security (?) threats unique to AI are:

The last childish ‘belief’ is that AgenticAI is fundamentally different from AI. In reality, an AgenticAI is a class of artificial intelligence, i.e., an AI “designed to act autonomously, sense their environment, make decisions, and take actions without constant human oversight. However, their autonomy and potential to operate with minimal human intervention raise unique challenges in terms of regulation, particularly under the EU AI Act”. So, everything that relates from the AI security perspective applies to the Agentic AI security, which contains even more aspects.

Notably, “What sets agentic AI apart from monolithic (‘stand-alone”) LLMs is their ability to interact with external systems. … Agentic AI can retrieve real-time information, access databases, interact with other software tools, and take initiative based on such context. This dynamic interaction allows agentic AI to adapt to changing circumstances and perform tasks that go beyond the limitations of traditional LLMs“.

Security versus Inertia of Thinking

Since IT looks at everything through a data lens, we try to assess which threats to AI are security-related, and that represents just a data-centric way of thinking.

Deliberate attacks on AI

As we know, AI refers to an artificial intelligence system that, analysing human-created content – words, phrases, images – can uncover non-obvious interdependencies between content parts and create new content based on patterns learnt from data it is trained on using machine learning algorithms (MLA). Training data is usually acquired by the AI creator and used privately. The AI models are computational compositions of interdependent content elements that are recomposed in different orders in line with the learnt patterns. Therefore, the AI outcome can sound like a human-created one.

In this case, what does an “attack on AI” mean?

If it is an attack on the training data, it is not an AI problem – this data must be secured by known means in the creator company. If this is an attack on the model, this means an assault on the MLA. Again, it is not an AI but the company security’s problem.   The only attack on AI that I can recognise can be conducted by either creator’s stakeholders or AI developers. That is, a deliberate attack on AI is exclusively the creator’s internal business; AI users have no concerns about them because the AI creator is fully legally accountable for the AI-based products.

Jailbroken LLMs or prompt injection

LLMs are linguistic AI models. They are trained the same way as non-linguistic ones. ”Jailbreaking a model using prompt injection means manipulating the input to an AI model to bypass its built-in safeguards and instructions, causing it to generate responses it was not intended to provide. It’s like tricking the AI into ignoring its programming and delivering unwanted or malicious outputs.”

In AI, jailbreaking means modifying the firmware of electronic devices. This excellent explanation points to a serious “problem”:

The UN’s ethical frameworks for AI assume that the AI user has some kind of obligation to the AI creator called “ethical use”. It is a nonsense in technology – a user of the product may do anything with the product; otherwise, the user is in “jail”. The fundamental technology rule is that a good product prevents potential distractions caused by one user from impacting other users. Advanced technologies enabling multiple users working together must be resilient, i.e., an instance of the AI engine may not be shared concurrently. This principle is known from the early days of technology and then was applied to client-server models and, finally, to distributed computing. This means that if an AI can be modified using prompts, it is a bad AI because. Period. It is the fault of the AI creator.

The AI user has all rights for applying any prompts, i.e., the right to use an official contracted interface to the device. Only lazy or illiterate developers can call this “prompt injection”. Such an injection is a standard IT product test for product quality, which should be applied before the product release. It is another creator’s problem that automated testing frequently omits these “rainy day” tests.

If “manipulating the input to an AI model can bypass its built-in safeguards and instructions, causing AI to generate responses it was not intended to provide”, here are two possible conclusions:

a) This AI product must be called off the market and repaired.

b) The AI as a technology is inconsistent and cannot reliably serve the users.

Recent research tends to favour option b) and to disqualify “prompt injection” as an AI threat.

Automated malware creation and data mining

Malware attack prevention is a legitimate security task, regardless of its creation being automated or not. Since the prevention means are reactive, automated creation causes results in the latency of reaction.   If AI is used for creation automation, this can result in more sophisticated and widespread cyber-attacks.

Unregulated Corporate AI Use

Regulated or unregulated use of AI in the company, especially in the AI-creating company, is an internal problem of the company that has accountability to the AI users.

After so many years of research, development and usage of AI, some people still believe that regulating AI creation can help the AI users. If regulation is not for the user benefits, then why is it executed at all? A shared opinion is that cybersecurity can be compromised by “impersonating trusted platforms, biased resume screening in employment decisions, and harmful errors in healthcare applications”. Which one of these compromise examples can be avoided if AI were regulated in the company? Can any regulation prevent submitting and screening biased resumes or prevent healthcare applications from errors? I think here is an almost obvious speculation about trusted platforms. Who defined a trust for any platform other than the platform author or owner? What proves, except suggestions and recommendations, that a platform is trusted? A conclusion from my personal research states that AI used in the socio-cultural, political and mass-media areas are trustless by design. I think that those are unrelated examples, at least.

Wikipedia explains that “the basic approach to [AI] regulation focuses on the risks and biases of machine-learning algorithms at the level of the input data, algorithm testing, and decision model.  It also focuses on the explainability of the outputs. There have been both hard law and soft law proposals to regulate AI.” Well, each algorithm has associated risks; if a regulation states that the risk should be eliminated, it will not strip the algorithm of its natural risk. If the regulation requires mitigating the algorithm risk, this is usually done by other software means that possess their own risks. What “biases machine-learning algorithms”are no one knows, as well as what the bias is because it is an exclusively subjective matter. All “the input data, algorithm testing, and decision model” ought to be regulated because they form their creation process, i.e., it is a routine that should go without saying.

Any self-respecting designer or developer will not start working on AI without incorporating regulation of the AI creation process. However, “a recent survey by Compliance Week revealed that nearly 70 per cent of organisations use AI but do not have adequate AI governance. This is shocking. But the most alarming part is that these organisations do not perceive that lack of governance as a high risk. … In effect, these organisations are adopting a high-risk position without the safeguards to manage those risks and fail to grasp the current and potential threats posed by unregulated AI.” What is going around will come around.

Poisoned data

Poisoned data is an omni-excuse for the bad or even wrong work in an AI creation. It is the creator company that obtains training data. So, it is the creator company that is responsible for this data. This is not a security but a data quality control problem.

If the company manipulates training data to gain a certain outcome or synthesis data, the result might be received, but the model would become unrealistic and immediately fail in the real-world environment. Such a result is highly probable for socio-cultural and mass-media spheres. In so-called “natural science domains” where the data can be simulated based on exact scientific knowledge, synthetic data are acceptable. However, the data may be poisoned in this domain only inside the creator company, deliberately. This is the only security use case related to data protection control.

In contrast, the most popular complaints about poisoned data come from the use of linguistic models (LLM) in the social-cultural areas. The LLM-based AIs are trained on the people’s words and language patterns, i.e., the models reflect what people say. If one does not like what the people say, do not train your AI on this data. It is the problem of personal acceptance because bias and fairness in the AI outcome exist in your opinion, and another user may and likely would not take this outcome as you do. In other words, the data in the public and private sources in the socio-cultural and mass-media may be or are poisoned, but the creators do not know what and how it had been poisoned. This means that cleansing LLMs to protect from bias is a useless and hopeless exercise from the consumer viewpoint.  The extremely rich variety of social groups, people, cultures, customs and habits leads to a high probability of mismatch between individual take and the ethics of the AI creator that tries to insert it in the AI model.

Inconsistent corporate policies for unsanctioned AI models 

A problem of inconsistent governance over the AI product creation is well known, and we have addressed it earlier. However, it has one additional aspect I’d like to outline. An obsession with minimising time-to-market and poorly controlled deployment automation (the inheritance from agile, scrum development of the past) produced mistaken over-reliance on the developer teams that deploy unsanctioned or authorised AI models and AI products, often referred to as “shadow AI”. These releases may , and usually do, include unmitigated security risks, user privacy violations and breaches of compliance.

Hand-made dependency

The same weakened control over the development teams has raised a problem of reckless dependency inserted in the products. This problem is known from the time of shared services and APIs and not smoothly transitioned into AI. Why? – Because development teams have no official business responsibilities but are allowed to act on behalf of the company’s business.

Precisely, developers freely include references or links to the external services/components or AIs in their products without requesting business and security permissions for this. Yes, these requests usually take some time, causing the delays to the release of the developed stuff and “compromising” the development performance. In essence, external links constitute external business dependencies that, if done without approval, are unmitigated. That is, they can cause serious financial and reputational troubles for the company.

I can agree that this is a partial security problem, though it roots in the thoughtless governance.

Unclear accountability

In my opinion, it is about the business operational model rather than about security. According to all AI frameworks I am aware of at the moment, the creator or provider organisation is accountable to the AI users for the AI outcome and behaviour. This is irrelevant to the types of usage and data environment. If, for example, an AI is created for use in the construction work domain but applied in biology, the documentation and prompt analyser should prevent such application by denying the service.

An accountability problem arises inside the creator’s company between the designers, stakeholders/assessors and developers. However, this is an organisation, not a security problem. If this problem is not resolved or has a weak resolution, the security issues may occur as a result, but they will be only derivatives.

Counterfeit AI development platforms & bogus publicly available AI services

This is the real threat based on the recklessness, naivety and primitivism of many AI developers, which I have to admit with regrets. Related security means should work like a gateway to the company’s IT, across and above all IT management, because it constitutes an existential risk for the company. “The fact that we can do something does not mean that we should“[by quizlet.com].

This security problem has roots shared with the “hand-made dependency”, with the only difference being that counterfeit AI platforms and bogus AI services have been created on purpose in order to screw the insensible AI enthusiasts, the majority of whom are young people.

Unfortunately, some real AIs, especially from big tech vendors, have very similar targets – to hook “useful idiots” [by V.I. Lenin] in IT and, when the time comes, start dictating the company’s (hooked via its IT) business and political behaviour. This is known as vendor-locking, which companies should avoid by all means, but their IT frequently do not care about this (”time-to-market is above all”).

General amnesia – ignorance of the existing experience

While this is not AI specific but rather a common sin of every new generation (though previous generations were ‘accompanied’ with strict punishment for even accidental mistakes), in combination with weak governance and respect for unaccountable development (aka neo-liberal norm), this becomes a quickly growing problem for companies and for the AI realm, particularly.

It is a misconception that security for AI and AgenticAI is something new and may be properly constructed from a greenfield. Because of human impacts and influence on life decision-making, the security requirements for AI are much harder than IT knew before, and retribution for mistakes will be stronger. So, if the new development wants to be faster, it should be aware of the security experience that existed before AI became that popular. Otherwise, all time and money won on “bicycle discovery” will be wasted on the problem resolutions, compensations and redevelopment.

More security for AgenticAI

The hype of AgenticAI has eclipsed security in the same way as the old hype of microservices eclipsed its security. Many development teams still have not grasped those security requirements and survived only on the mercy of lazy attackers. However, the nature of AI makes old customs not only absolute but dangerous now.

If you can recall SOA, it said that each service is an independent application that can have its own creators, owners and providers. In other words, two or more SOA services working together on solving a shared task may encounter several different owners with different security mechanisms and providers. The AgenticAI are of the exact same nature from the security and ownership viewpoints. Moreover, a need or relatively complex process of AI training for AgenticAI makes multi-ownership combination even more likely. 

The security consequences of this are crucial. When an AgenticAI Alpha decides it needs to interact with another AgenticAI Beta, the following security questions should be answered and addressed in implementation:

• Should an owner or provider of Beta know up-front or at runtime that Alpha wants to interact with Beta?
• Should an owner or provider of Alpha know up-front or at runtime that Alpha might want to interact with Beta?
• May Alpha initiate interaction with Beta?
• Should Beta be aware that Alpha might interact with it?
• When and how does Beta permit any interactions from other AgenticAIs, including Alpha?
• How is a permission for Alpha to interact with Beta actually authorised?
• How would Alpha and Beta recognise each other for the interaction?
• Who and how should control that the Alpha and Beta risks are compatible, and what should be done if they are not?
• How would Alpha know at runtime that it should contact Beta rather than Gamma AgenticAI for a particular Alpha’s task?

All these and similar questions relate to two major security domains: entity identity (in the absence of a human end-user) and entity entitlement (and related access control). All that I read and heard about AgenticAI was about how to enable interaction, i.e., about the capability of interoperability between AgenticAIs and possibly related applications. As in usual IT practice, security is considered an additional concern (and undesirable trouble).

Nonetheless, the AgenticAI concept has at least one, but not the last, more problem to consider. While AgenticAI can and are assumed to work autonomously, without human oversight, some compositions of AgenticAIs will eventually result in interaction with human users in socio-cultural, political and mass-media spheres. This can open a ‘can of worms’, if not a Pandora’s box, related to the potential kaleidoscope of risks of different AgenticAIs in the compositions. The level of concerns in these spheres is so high that a possibility of regulated restrictions is under revision. That is, certain AhenticAIs or their compositions may be prohibited from the aforementioned humanitarian areas. 

In the end, I have to remind about an old open question regarding end-user identity propagation through the chain or graph of applications, services and AgenticAI invocations. As in Microservice compositions, propagation of the initiator’s identity requires strong justification because 1) the further this identity is propagated , the more risks of interception it acquires (security risks); and 2) it is almost obvious that the initiator has no clue what applications, services, APIs and agentic AI may be involved in the task solutions as well as what data sources might be needed to engage, i.e., not even a trace of an initiator’s entitlement to those resources exists.

Summary

I have collected twelve issues claimed as AI security problems. The provided analysis demonstrates that only 30% of them relate to AI security. Others are faults of AI development to different degrees. Also, only one case is AI-specific, while others are known before and simply inherited by AI due to the extremely “short memory” of the AI creators.

The category of AgenticAI has inherited the security concerns and needs as were understood and defined for SOA services, then for Microservices (though not fully implemented), then for APIs. There is nothing new except that compositions of AgenticAI quite likely will have more multi-ownership and multi-provider constructs than we’ve seen before.