AI Risks and Attacks: Escalating from Misuse to Abuse
Cybercriminals and AI: Reality vs. Hype
"AI will not replace humans in the near future. However, individuals who are proficient in using AI will outpace those who are not," states a Chief Security Strategist. "Attackers are also utilizing AI to enhance their capabilities."
Nonetheless, much of the discourse surrounding AI's involvement in cybercrime tends to be exaggerated. News articles often sensationalize AI threats, employing dramatic terms such as "Chaos-GPT" and "Black Hat AI Tools," and even suggesting they aim to threaten humanity. Such articles predominantly instill fear rather than accurately depict significant risks. For example, discussions in underground forums revealed that several so-called "AI cyber tools" were essentially just rebranded versions of basic public large language models (LLMs) lacking advanced functions. In fact, many attackers have labeled these products as scams.
How Hackers Are Utilizing AI in Cyber Attacks
In practice, cybercriminals are still learning how to effectively employ AI. They encounter the same challenges and limitations as legitimate users, such as hallucinations and restricted capabilities. Experts predict it will be several years before these criminals can utilize generative AI effectively for hacking purposes. Currently, generative AI tools are mainly used for straightforward tasks like crafting phishing emails and generating code snippets for integration into attacks. Additionally, there have been observations of attackers feeding compromised code to AI systems for analysis, attempting to "normalize" such code as benign.
Using AI to Misuse AI: The Introduction of GPTs
Customizable versions of well-known AI chat systems, introduced recently, enable users to input specific commands, link external APIs, and incorporate tailored knowledge sources. This functionality allows for the creation of specialized applications, including tech support bots, educational tools, and more. Furthermore, developers are offered opportunities to monetize these custom models through a dedicated marketplace.
Potential for Misuse
These custom AI models raise a series of security concerns. A significant risk is the potential exposure of sensitive guidelines, proprietary information, or API keys integrated within the custom AI. Malicious entities may exploit AI, particularly through prompt engineering, to replicate a custom GPT and exploit its monetization features.
Attackers can manipulate prompts to extract knowledge sources and instructions. Simple requests might include asking the custom AI to enumerate all uploaded files and commands or seek debugging information. More sophisticated requests might involve asking the AI to compress a PDF and generate a downloadable link or to present its functionalities in a structured table.
"Even the safeguards implemented by developers can be circumvented, allowing for the extraction of all information," remarks a Threat Intelligence Researcher.
These risks can be mitigated by:
Not uploading sensitive information
Implementing instruction-based safeguards, although these might not be foolproof. "It's essential to consider all potential scenarios an attacker could exploit," the researcher notes.
Utilizing available protective measures from the AI provider.
AI Attacks and Associated Risks
There are numerous frameworks available to assist organizations engaged in AI software development:
NIST Artificial Intelligence Risk Management Framework
Google’s Secure AI Framework
OWASP Top 10 for LLMs
OWASP Top 10 for LLM Applications
The newly launched MITRE ATLAS
LLM Attack Surface
Six critical components of large language models (LLMs) are potential targets for attackers:
Prompt - Techniques like prompt injections manipulate the AI's outputs.
Response - The misuse or leakage of sensitive information in AI-generated outputs.
Model - Theft, poisoning, or manipulation of the AI model.
Training Data - Inserting harmful data to skew the AI's behavior.
Infrastructure - Attacks on the servers and services that power the AI.
Users - Misleading or exploiting individuals or systems dependent on AI outputs.
Actual Examples of LLM Exploitation
To illustrate, here are some instances of LLM manipulations that could be weaponized:
Customer Service Prompt Injection - In one case, a car dealership's AI chatbot was manipulated by a researcher who prompted it to agree to all customer statements, leading to a fraudulent purchase at an excessively low price, thereby exposing a significant vulnerability.
Hallucinations Resulting in Legal Issues - Air Canada faced legal scrutiny when its AI chatbot provided erroneous information about refund policies. A customer relied on this information for a claim, resulting in the airline's liability.
Leak of Proprietary Data - Employees at a major company inadvertently disclosed sensitive information while using an AI system to analyze code, highlighting the dangers of submitting confidential data to third-party AI platforms.
AI and Deepfake Technology in Fraud - Cybercriminals also exploit AI for purposes beyond text generation. A bank in Hong Kong suffered a $25 million loss when attackers employed live deepfake technology in a video call, impersonating bank officials and tricking the victim into transferring funds to a fraudulent account.
Conclusion: The Role of AI in Cyber Crime
AI serves as a powerful tool for both defenders and attackers. As cybercriminals continue to experiment with AI, it is crucial to understand their mindset, the tactics they employ, and the options available to them. This knowledge can better equip organizations to defend their AI systems against potential misuse and exploitation.
Reach out for a free consultation on how to protect your environment
Related Blogs
