Cloud and Threat Report:
AI Apps in the Enterprise

The starting point for genAI app risk are the applications themselves. Practically, application controls are also the starting point for controlling that risk, typically implemented as allow or block lists within an inline SWG or proxy.

Most organizations have restricted the use of genAI apps to protect their data, with 77% of organizations blocking at least one genAI app as of June 2024, which is a 45% increase from 53% of organizations in June 2023.

This trend indicates good maturity and adoption in basic controls around genAI application usage. Controlling which apps are used in an organization is a required starting point for reducing risk. However more granular controls will be required to be effective. The specific use of an application often determines whether the activity should be allowed or not. For example, a general search in ChatGPT should be allowed, while the submission of source code should not.

Looking in more detail at blocked applications, the median number of applications blocked for all users in an organization has also been increasing, from 0.6 apps in June 2023 to over 2.6 apps in June 2024. Having so few organization-wide bans on genAI apps compared to the hundreds of genAI apps on the market points toward the popularity of other more nuanced controls, which we will discuss in more detail later in this report.

The most blocked genAI applications do track somewhat to popularity, but a fair number of less popular apps are the most blocked. Of those organizations blocking genAI apps, 28% block Beautifal.ai (making it the most commonly blocked genAI app), and 19% block Perplexity AI, which is the 10th most commonly blocked app. Blocks can often be a temporary measure as new apps are evaluated to determine if they serve any legitimate business purposes and are safe for certain use cases.

The specific applications that are blocked will vary by organizational policy, but when percentages are high enough, as in the above top 10 list, it’s worthwhile for all organizations to review if the specific applications are used within their own environment, as well as whether to adjust controls around the category of applications. The following table shows that the most blocked apps span a variety of different use cases in the genAI space.

Most Blocked GenAI App Categories

As organizations mature beyond allowed applications lists, they tend to begin enacting more fine-grained controls around the usage of the allowed applications. Unsurprisingly, DLP controls are growing in popularity as a genAI data risk control. Data loss prevention has increased in popularity from 24% in June 2023 to over 42% of organizations using DLP to control the types of data sent to genAI apps in June 2024, more than 75% year-over-year growth.

The increase in DLP controls reflects an understanding across organizations about how to effectively mitigate data risk amidst the larger, broad trend of increasing genAI app use. Within DLP policies, organizations are looking to control specific data flows, especially to block genAI prompts containing sensitive or confidential information. In organizations with data protection policies, source code accounts for nearly half (46%) of all DLP violations, with regulated data driven by industry regulations or compliance requirements at 35%, and intellectual property at 15%. Regulated data has been a top violation area pre-genAI and reinforces the challenges with manual training of users in the improper sharing of regulated data.

Posting sensitive data to genAI apps not only reflects the current priorities among organizations, but also shows the utility of various genAI apps. For example, GitHub Copilot’s popularity, coupled with Microsoft’s market share as a software development platform, may drive higher use of genAI for code generation in the future.

While obviously malicious applications or activities are served well by controls that block and alert, data risk is often more of a gray area. Coaching controls can be a highly effective technique to deal with gray areas, especially in early fast-moving technology cycles like genAI. Furthermore, organizations can use coaching to inform and refine security controls without blocking productivity with false positives and slow approval processes.

Coaching controls provide a warning dialog to the user while interacting with genAI apps, allowing them to cancel or proceed with their actions. It functions like safe browsing features built into browsers. The advantages are it provides a friendlier experience for the user, not blocking their work, but informing and enabling users to improve their behavior.

For organizations with policies to control genAI application usage, 31% of them used coaching dialogs as of June 2024 compared to 20% of organizations in June 2023, an over 50% increase in adoption.

This tracks with growing sophistication of organizations in applying similar coaching controls from other security domains to newer ones such as genAI app risk. While the growth has flattened, coaching controls are relatively new as compared to outright blocks and alerts based on DLP or application. We expect the adoption to continue growing as more organizations understand how to use coaching to manage the grayer risks associated with data.

When we break down the actual user response to the coaching dialog alerts that have occurred, we see a measure of efficacy. For all users that received coaching dialog alerts for genAI apps, in 57% of those cases, users chose to stop the action they were performing, which reduced risk by users avoiding sending sensitive data in genAI app prompts or by avoiding the use of unknown or new genAI apps. 57% is high enough to bolster the case that coaching can be an effective control to complement explicit application blocks and DLP policies. Furthermore, coaching also enables feedback. When deciding to proceed, most organizations put policies in place that require users to justify their actions during the coaching interaction.

This is not to suggest that user decision-making be the basis for security policy. Rather, it indicates that for organizations that utilize coaching, approximately half of the genAI app usage that is not being outright blocked may be further reduced by user decisions. Just as important, user decisions in response to coaching dialogs can and should inform security policy review and adjustment. While user decisions can be faulty and risky, the two categorizations form the basis for further review. Applications that a user has decided not to use based on a coaching dialog should be analyzed for an outright block list. Applications that a user decided to use should be reviewed to be put on an allowed corporate standards list or perhaps blocked if there’s a better or more acceptable app. User responses to coaching prompts could also be used to refine more nuanced policies, like DLP policies, to be more targeted in their application.

We are seeing early signs of advanced detections of suspicious user behavior with regards to data movement that were detected by behavioral detection engines. Suspicious data movement often comprises multiple indicators of suspicious or unexpected behavior by a user with respect to the user’s or organization’s normal activity baseline. The indicators might involve anomalous download or upload activity, new or suspicious sources or targets of data such as new genAI apps, as well as other suspicious behavior such as use of an unexpected or obfuscated IP address or user agent.

One example of a detected, advanced behavioral alert for suspicious data movement involves the movement of data from an organization-managed application and uploaded to an unapproved, personal genAI app.

Similar to coaching, the list of apps appearing in these alerts can also be used to prioritize control efforts with genAI application usage. For example, two commonly used apps may overlap in functionality, and that alone might prioritize efforts to reduce the number of apps.

The use of behavioral alerts show the early awareness and adoption of organizations to find harder to detect threats, which has traditionally included compromised credentials, insider threats, lateral movement, and data exfiltration activities by malicious actors.

When we look closer at the sensitive data movements, we find that the top applications where the sensitive data originated reflect popularity of corporate cloud apps with OneDrive (34%) and Google Drive (29%) at the top, followed by SharePoint (21%) and Outlook (8%) and Gmail (6%).

The top 3 apps are all cloud storage and collaboration apps and were the source of the sensitive data 84% of the time, while the major email cloud apps were the source 14% of the time. By knowing which specific applications are involved more frequently in sensitive data movements, security teams can then adjust controls appropriately, for example, placing additional DLP controls around file-sharing applications. Organizations should prioritize security risk assessment and control implementation around data movement between apps, especially moving from managed apps to unmanaged genAI apps, as it is both becoming more common and potentially having large data loss impact.