The State of Attacks on GenAI: Industry-First Analysis of Real-World Interactions

I am excited to share a big milestone for Pillar Security and the AI industry at large. Today, we release the "State of Attacks on GenAI" report—the industry's first comprehensive analysis based on real-world data from over 2,000 LLM applications. This groundbreaking research provides unparalleled insights into the evolving landscape of AI security threats, offering a data-driven perspective that moves beyond theoretical risks and hypothetical scenarios.

‍

The rapid development and deployment of new AI models and applications—sometimes within mere months—have propelled us into an era of unprecedented technological advancement. However, this swift progression brings with it significant challenges. Security and AI leaders are now grappling with the task of balancing these substantial gains against potential setbacks, particularly security vulnerabilities.

‍

Although numerous theoretical studies, surveys, and potential scenarios exist, there has been limited analysis of real-world attacks and risks. For the first time, our report bridges this gap by providing insights grounded in tangible data interactions from production AI-powered applications over the past few months.

‍

Key Findings and Insights from Our Research

Our extensive analysis has uncovered several critical findings and insights that I believe would be essential for anyone involved in AI development, deployment, or security. These insights highlight the multifaceted vulnerabilities inherent in LLMs:

‍

1. High Success Rate of Data Theft

• 90% of successful attacks resulted in the leakage of sensitive data. Attackers are primarily motivated to access and steal proprietary business data, user inputs, and Personally Identifiable Information (PII).
• 20% of jailbreak attack attempts successfully bypassed GenAI application guardrails. This high success rate indicates that existing security measures are often insufficient against sophisticated attack techniques.

2. Swift Attack Execution

• Adversaries require only an average of 42 seconds to complete an attack. This highlights the incredible speed at which vulnerabilities can be exploited, leaving minimal time for detection and response. Attacks are happening at machine speed.
• Minimal Interaction Required: Attackers need just five interactions on average with GenAI apps to achieve a successful attack. This efficiency underscores the sophistication of modern attack strategies.

3. Top Adversary Goals and Motivations‍

• Attackers are primarily motivated to access and steal proprietary business data, user inputs, and Personally Identifiable Information (PII), and generate malicious content such as disinformation, hate speech, phishing messages, or malicious code.

4. Vulnerabilities at Every Interaction Point

• Attacks can exploit vulnerabilities at every stage of interaction with LLMs, including inputs, instructions, tool outputs, and model outputs.
• This underscores the importance of implementing comprehensive security measures throughout the entire interaction pipeline, not just at isolated points.

5. Top Predominant Jailbreak Techniques Observed

• Ignore Previous Instructions: Attackers direct AI systems to disregard their initial programming, potentially causing the AI to generate harmful content, violate ethical guidelines, and inflict reputational damage.
• Strong Arm Attack: Persistent and forceful requests pressure AI into compliance, which can lead to the AI revealing sensitive information or performing unauthorized actions, resulting in data breaches or system compromise.
• Base64 Encoding: Malicious prompts are encoded to evade security filters, allowing attackers to bypass safeguards and potentially execute malicious code or extract protected data.

6. Consequences of Successful Jailbreaks

• Due to the ease of launching attacks, malicious actors persistently attempt to jailbreak GenAI applications—often dozens of times.
• Attackers sometimes use specialized tools to generate large volumes of attack variations, making robust defenses imperative.
• Successfully jailbroken applications can be abused for the attackers' purposes, including generating disinformation, hate speech, phishing messages, or automating sophisticated attacks.

7. Disparity Between Open-Source and Commercial Models

• There is a significant gap in resilience to attacks between open-source and commercial LLMs.
• Open-source models, while fostering collaboration and innovation, inadvertently lower the barrier for malicious exploitation due to their transparency.
• Commercial models tend to be more secure, benefiting from substantial resources dedicated to security, highlighting the importance of considering security features and support when deploying LLMs in critical applications.

‍

2025 Outlook 

By examining the adversary mindset alongside the evolving technology landscape, this report provides valuable insight to help readers better understand the future: 

• Shift from Chatbots to Autonomous Agents: The next wave of AI adoption will move beyond simple conversational interfaces to autonomous agents capable of complex tasks and decision-making, offering sophisticated applications but also expanding attack surfaces for malicious actors due to increased capabilities and system access.
• Proactive Security Measures and "Secure by Design" Approach: Implementing tailored red-teaming and resilience exercises specific to AI applications is essential amid rising threats. Early adoption of a "secure by design" approach in Generative AI development enhances security and reduces future costs by addressing vulnerabilities proactively.
• Need for Dynamic, Context-Aware Security Measures: Organizations must move beyond static security controls to implement dynamic, context-aware security measures that evolve with AI systems. These measures should be model-agnostic, align with governance frameworks, and anticipate and respond to emerging threats in real time.
• Proliferation of Small Models and Local Deployment: Smaller, more efficient AI models are rapidly emerging, with advances in model distillation and hardware making local deployment on personal devices feasible. This democratizes AI access but also expands the attack surface, introducing new security challenges that organizations must address.

‍

Our Commitment to AI Security

Pillar Security was founded by a team of seasoned security leaders approaching AI Security with a practitioner's mindset and frontline experience. Our focus with this report is to deliver genuine insights that truly matter to organizations grappling with AI security.

We recognize that as AI technologies continue to evolve, so too do the threats posed by malicious actors. Pillar is dedicated to addressing these challenges, enabling swift and secure AI adoption through our comprehensive approach. Our unified AI security layer provides continuous red teaming, runtime protection, adaptive guardrails, and granular governance controls, empowering organizations to confidently harness AI's potential across their entire infrastructure.

‍

Moving Forward Together

The release of the "State of Attacks on GenAI" report is not just a milestone for Pillar Security but a call to action for the entire industry. We must collectively acknowledge the risks and work towards implementing robust, adaptive security measures that protect our organizations and users.

We invite you to read the full report to gain a deeper understanding of these critical issues. By sharing our findings and insights, we hope to empower you to make informed decisions about AI security within your organizations.

‍

You can download the full report here: https://www.pillar.security/resources/the-state-of-attacks-on-genai 

‍