The Dark Side of Generative AI: Part 2 — Mitigating the Security and Privacy Risks

To secure your project, you need a thorough assessment and mitigation of every possible risk. Here is how to approach this mission.

Note: This is part 2 of the 3-part series that will explore the pitfalls that developers of the Generative AI-based application need to know and try to mitigate.


Have you ever wondered how risk assessment for generative AI applications differs from the risk assessment performed for other applications? Generative AI applications have unique attributes related to their architecture, usage, and user interaction that pose risks that are not necessarily present in other applications. It is essential to recognize these applications’ specific risks and take steps to mitigate them in order to ensure their security.

There is much room for creativity with the ingredients the chef can use to make a meal. However, developers must address every potential risk when making applications safe and secure. Attackers only need to find one vulnerability to exploit the application, whereas defenders must ensure they do not create a single mistake. The same principle applies when assessing the risks generative AI application, or any other application, is exposed to.

In this article we will provide a guide for identifying and mitigating the key dangers, along with a few interesting examples of what can and has gone wrong in the past. We will also give examples of the tools that can be used to mitigate the risks.

Main pitfalls in each risk category

The previous blog series installment introduced five risk categories: Security, Privacy, Misuse, Regulation, and Service Quality. This installment will delve into each category’s dangers and potential downfalls.


Security risk is one of the toughest challenges that application developers face. Ignoring these risks can lead to severe consequences, such as the loss or theft of sensitive information, financial losses, reputational damage, and legal implications. Therefore, it is crucial to prioritize security measures and ensure correct implementation to avoid any negative impact on the application or its users.

Security risk refers to the potential exploitability of a system. Developers must prioritize implementing robust security measures such as encryption, access controls, and regular vulnerability assessments to mitigate these risks and protect against cyber threats.

Here are a few examples of security pitfalls that demonstrate some of the security risks:

Recent research shows interesting examples of potential attacks on the Open AI GPT-4 agent (full paper) using its three public APIs.

This research provides an example of the security risks for a generative AI application like OpenAI ChatGPT, but most risks are relevant to a broader set of applications.

Many more attacks are already known, like the exfiltration of sensitive files and the insertion of a backdoor into Open AI Code Interpreter. Attacks can be used in different modalities, like in the case of Open GPT-4 Vision Prompt Injection.

This document does not claim to provide a complete list of potential threats but instead emphasizes the need for thorough risk assessment and threat modeling.


Keeping data under tight control, protecting it from unauthorized access, auditing its usage, etc., is a challenging task the industry has dealt with for a long time. Failing to do so leads to many risks, including privacy violations. The task becomes even more complex with the sophistication and volume of the data typically being managed for training and operating generative AI applications.

The following research demonstrates an attack carried out by a white-hat security researcher on the OpenAI Chat GPT application: ChatGPT Vulnerability Allows Training Data to be Accessed by Telling the Chatbot to Endlessly Repeat a Word.


Many generative AI applications’ capabilities can be used for illegal or unesthetic activities, and application vendors need to do their best to prevent that to reduce the risk of being liable for assisting bad actors in their tasks.

An example of the above risks is WormGPT - a new AI tool that allows cybercriminals to launch sophisticated phishing attacks by launching sophisticated phishing and business email compromise campaigns. It’s speculated that it uses the open-source GPT-J language model developed by EleutherAI. Another example is FraudGPT, a new AI tool tailored for sophisticated attacks like crafting spear phishing emails, creating cracking tools, carding, etc.


Regulatory and legal risks can pose a significant threat to businesses, potentially resulting in major losses, bankruptcy, or even personal legal liability for company executives, as demonstrated by the following examples.

Service Quality

A generative AI application’s quality of service depends on various aspects, such as efficient system maintenance and prompt resolution of any technical issues that may arise. The user experience, functionality, and overall performance of the application are also crucial factors that can impact service quality.


Application developers should not feel helpless when faced with various pitfalls that can pose a great risk to the success of their applications. Instead, they should systematically apply mitigation measures based on a thorough risk assessment of their application before releasing it. Here are some of the main mitigations applications developers should consider.

Risk Assessment

Risk Assessment and Risk Management are among the first steps needed as the assessment will help prioritize and uncover the following mitigations required. There are several industry frameworks that can help with this step, some general and some more specific to generative AI applications:


Governance programs help organizations ensure that they are operating in a transparent, ethical, and compliant manner. These programs establish policies and procedures that guide decision-making, risk management, and overall operations. By implementing effective governance programs, organizations can minimize the risk of legal or reputational harm while also improving efficiency and accountability.


Usage policies for generative AI applications are essential to ensure responsible and ethical use of this technology. Such policies should provide a legal framework that outlines the limitations of use, potential risks, and liability for any harm caused by the AI-generated content. By establishing clear guidelines, we can prevent the misuse of generative AI and promote its safe and beneficial use for all.


Most generative AI applications are interested in saving user data for future model training. However, it’s essential to allow users to opt out of their data being saved and used to improve the model to prevent leakage of sensitive users’ data. This will also help to show transparency and contribute to user trust in the application.

ML Operation

Proper implementation of MLOPS (Machine Learning Operations) industry best practices is essential to deal with data drift, continuous monitoring, retraining, and tuning of the model to ensure that it continues to perform accurately over time. There are many mature products that exist on the market that can help build state-of-the-art MLOPS infrastructure, so most organizations do not need to develop it in-house.


There are several types of guardrail mechanisms that can be implemented to mitigate different kinds of risks.

Resource Utilization

Generative AI applications require a lot of computation power, typically in the form of GPUs, to train models or perform inference at scale. It is essential to have proper control and thresholds on the consumed compute power to ensure that one application function does not starve other functions of resources. Additionally, consumption should not exceed the limit the company can afford.

User Education

As generative AI applications have some unique characteristics, like a level of confidence in the accuracy of generated content or data privacy, it is essential to educate end users (enterprise employees or consumers) about the risks associated with generative AI platforms and provide guidelines for the appropriate use of these tools.


Like any other application, a generative AI-based application can be vulnerable to various security attacks. To mitigate this risk, essential security disciplines like identity and access management, data protection, privacy and compliance, application security, and threat modeling must be implemented during both the development and operation phases. As the industry has been tackling this issue for many years, there are numerous mature frameworks and tools available to help. Recently, some of these frameworks have even released versions specifically designed for generative AI applications and their potential threats.


Making sure compliance with regulations in the relevant location is critical. As of today, the regulation might be quite vague in some areas, but we definitely see major progress in regulation clarity across many countries. For cases where regulation is not clear enough at the moment, it’s essential to take a proactive approach and engage in conversation with policymakers and industry experts to understand where it is going and try to minimize the risk of non-compliance in the future.

Call to Action

In conclusion, as the adoption of generative AI continues to grow, it is crucial to address the security concerns associated with this technology. From data loss and unauthorized access to the creation of deepfakes and fake news, the risks are real. However, by implementing essential security measures, assessing risks, and implementing effective governance strategies, organizations can mitigate these threats.

At Atchai, we specialize in generative AI security solutions that combat unique AI system threats. If you’re embarking on a new generative AI project and want to make sure you develop it responsibly and protect your users’ data, we will be happy to discuss how we can help.

Let’s build together products that are both capable and conscientious.