Clio: A Real-World AI Usage Insight System for Privacy Protection
What do people use AI models for? Despite the rapidly growing popularity of big language models, until now we have lacked insight into exactly how they are used.
It's not just a matter of curiosity or even sociological research. Understanding how people actually use language models is critical to security: service providers conduct extensive testing before deployment and use trust and security systems to prevent abuse. However, the variety and scale of functions that language models can perform makes understanding their use - let alone comprehensive security monitoring - difficult.
There's another key factor that gets in the way of a clear understanding of how AI models are being used: privacy. At Anthropic, our Claude Model DefaultsWill not use user dialog data for trainingand we take the protection of user data very seriously. So how do we study and observe the use of our systems while strictly maintaining user privacy?
Claude insights and observations, or "Clio" for short, is our attempt to answer this question. clio is an automated analytics tool that analyzes real-world usage of language models in a privacy-preserving way. It's similar to Google Trends in that it gives us insights into how claude.ai is used on a day-to-day basis, and also helps us improve our security measures. In this post (withFull Research Paper), we will describe Clio and some of its preliminary results.
How Clio Works: A Large-Scale Privacy-Preserving Analysis
Traditional top-down approaches to security (e.g., evaluation and red-team testing) rely on knowing what to look for ahead of time.Clio takes a different approach, enabling bottom-up pattern discovery by distilling conversations into abstract, easy-to-understand clusters of topics. At the same time, it protects user privacy: data is automatically anonymized and aggregated, and only higher-level cluster information is visible to human analysts.
Summary example of the Clio analysis step, illustrated using a fictional dialog example.
Below is a brief overview of the Clio multi-stage process:
- Extracting Properties: For each conversation, Clio extracts multiple "attributes" - specific properties or metadata, such as the topic of the conversation, the number of round trips in the conversation, or the language used.
- semantic clustering: Automatically group similar conversations together by theme or general topic.
- Cluster Description: Each cluster is given a descriptive title and summary that distills common themes in the raw data while excluding private information.
- Building a hierarchical structure: Clusters are organized into a multi-level structure that facilitates exploration. They can then be presented in an interactive interface that Anthropic's analysts can use to explore patterns in different dimensions (themes, languages, etc.).
These four steps are performed entirely by Claude, not by human analysts. This is part of Clio's privacy-first design, with multiple levels of "defense-in-depth". For example, Claude is instructed to exclude private details when extracting relevant information from conversations. We also have minimum thresholds for the number of unique users or conversations to ensure that low-frequency topics, which may be specific to a particular person, are not accidentally exposed. Finally, Claude verifies that cluster summaries do not contain overly specific or recognizable information before displaying them to human users.
All of our privacy protections have been extensively tested and are detailed atResearch papersThe
How People Use Claude: Insights from Clio
With Clio, we were able to gain high-level insights into how people actually use claude.ai. While the likes of WildChat cap (a poem) LMSYS-Chat-1M Public datasets like this provide useful information about people's use of language models, but they only capture specific contexts and use cases. clio gives us an idea of the full range of scenarios in which claude.ai is used in the real world (which may be different from the use of other AI systems because of differences in user groups and model types).
Key use cases for Claude.ai
We used Clio to analyze 1 million conversations on claude.ai (both free and pro versions) to identify the main tasks that users use Claude for. This shows that users are particularly focused on programming-related tasks: the "Web and mobile app development" category accounts for more than 10% of all conversations. Software developers use Claude for tasks ranging from debugging code to explaining Git operations and concepts.
The most common types of conversations users have with Claude, across all languages. The area of the circle corresponds to the percentage of conversations; the headings are summaries generated by Clio after analyzing 1 million randomly selected conversations.
Educational use is another important category, accounting for more than 7% of the total conversations, with content primarily related to teaching and learning. Business strategy and operations (including tasks such as writing professional communications and analyzing business data) accounted for close to 6% of conversations.
Clio also identifies thousands of smaller clusters of conversations, demonstrating the diverse uses of Claude. Some of these uses may be surprising, including:
- Interpreting Dreams;
- Analyzing soccer games;
- Disaster Response Preparedness;
- Provides crossword puzzle hints;
- Dungeons & Dragons Games;
- Count the letter "r" in the word "strawberry".
The use of Claude varies from language to language
The use of Claude varies significantly across languages, reflecting different cultural contexts and needs. We calculated the basic frequency of occurrence of each language in the overall conversation, and from this we identified certain themes that appeared significantly more frequently in specific languages. Some examples for Spanish, Chinese and Japanese are shown below.
Conversation topics that appear more frequently in the three selected languages (relative to the underlying frequency of that language), insights provided by Clio.
How we used Clio to improve our security system
In addition to training our language models to reject harmful requests, we use specialized trust and security enforcement systems to detect, block, and respond to requests that may violate our Usage Policy Clio added to this work by helping us understand where we can improve and enhance these systems.
We have implemented strict privacy access controls in our use of Clio to further strengthen our policy, as this may require a review of individual accounts. Our Trust & Safety team can identify areas that may indicate a violation of our usage policies through topic cluster reviews. For example, a cluster titled "Generate Misleading Fundraising Email Content" or "Incite Hateful Behavior" describes activities that we prohibit. Our Trust and Safety team can use this bottom-up review approach to identify individual accounts that require further review and, where appropriate, action in accordance with our Terms and Policies. We strictly limit such reviews to those with legitimate Trust & Safety needs. Our Research papers Contains more information about these processes.
We are still in the process of rolling out Clio to all of our fulfillment systems, but so far it has proven to be a useful part of our security toolkit, helping us to identify areas where we need to strengthen our protection measures.
Identifying and stopping coordinated abusive behavior
Clio is highly effective at identifying coordinated, complex patterns of abuse that may be undetectable from individual conversations alone and may evade simpler detection methods. For example, in late September, we discovered a set of automated accounts that used a similar prompt structure to generate spammy content for search engine optimization. While no individual conversations violated our Usage PolicyWe removed this group of accounts, but the pattern of behavior between the accounts revealed a type of coordinated platform abuse that our policy explicitly prohibits. We also used Clio to identify other accounts that we were Usage Policy Prohibited activities, such as attempting to resell unauthorized access to Claude.
Enhanced monitoring for high-risk events
Clio also helps us monitor new usage patterns and potential risks during periods of uncertainty or high-risk events. For example, during the launch of our new Computer use Prior to the feature, we conducted extensive security testing while using Clio to screen for emergent capabilities and hazards that may have been missed.Clio provided additional security here and insights that will help us to continually improve security measures as the feature is rolled out and in future versions of the system.
Clio also helps us monitor for unknown risks on the eve of major public events, such as elections or major international events. In the months leading up to the 2024 U.S. elections, we'reUsing Clio Identifying clusters of activity related to political, voting and related issues in the U.S. and preventing any potential risk or abuse, Clio's ability to detect the "unknown unknowns" complements our proactive security measures and helps us respond quickly to new challenges.
Reduction of false negatives and false positives
Typically, there is agreement between Clio and our existing trust and security classifiers on the risk determination of session clusters. However, there are disagreements for certain clusters. One improvement opportunity is to reduce false negatives (i.e., the system does not flag sessions that may actually be harmful as potentially harmful content). For example, when users asked Claude to translate from one language to another, our system sometimes failed to flag the offending content, and Clio recognized those sessions.
The scatterplot shows session clusters (one dot represents one cluster) and the extent to which they are labeled as risky by the Trust & Safety classifier (x-axis) and Clio (y-axis). In the upper left corner are clusters that may have been underestimated as risky by the Trust & Safety classifier: false negatives that contain worrisome content but are not labeled as such. In the lower right corner are clusters that may be over-labeled: false positives that may not contain worrisome content. The overall correlation between the Trust & Safety classifier and the Clio classification is r = 0.71, indicating high overall agreement between the two.
We also use Clio to investigate false positives, another common challenge when developing trust and safety classifiers, where the classifier incorrectly labels harmless content as harmful. For example, sessions in which job seekers request resume advice are sometimes mislabeled as containing personal information. Programming issues related to security, networking, or web crawling are sometimes mislabeled as potential hacking. Even the above Dungeons & Dragons In-game sessions about combat statistics may also trigger our hazard detection system. We use Clio to highlight these false positives and help our security system trigger only on content that does violate policy, while keeping intervention in the user's legitimate use to a minimum.
Ethical Considerations and Mitigation Measures
Clio provides valuable insights into improving the security of deployed large language models. However, it also raised some important ethical considerations during its development, which we evaluated and acted upon accordingly:
- False Positive: In the context of trust and security, we have implemented key safeguards against potential false positives. For example, we currently do not use Clio's output for automated execution of measures and have extensively validated its performance under different data distributions, including the multilingual tests detailed in our paper.
- Clio's risk of abuse: Systems like Clio may be used inappropriately for surveillance behavior. In addition to strict access control and privacy techniques, we mitigate this risk by implementing a strict data minimization and retention policy: we collect and retain only the minimum amount of data necessary for Clio.
- User Privacy: While Clio performs well on our privacy assessments, as with any real-world privacy system, certain types of private information may go unrecognized. To mitigate this potential risk, we regularly audit Clio's privacy protections and assessments to ensure that its protections meet expectations. Over time, we also plan to use the latest Claude model in Clio to continually improve the performance of these protections.
- User Trust: Despite the extensive privacy protections we provide, some users may find systems like Clio intrusive or interfere with their use of Claude. We choose to be transparent about Clio's uses, capabilities, limitations, and the insights we gain from it. As mentioned earlier, Clio recognizes false positives (i.e., activities that appear to violate the usage policy but actually do not) in our standard security classifier, which may allow us to intervene less when interfering with legitimate uses of the model.
reach a verdict
Clio is an important step toward empirically supported AI security and governance. By supporting privacy-preserving analysis of real-world AI usage, we can better understand how these systems are actually used. Ultimately, we can use Clio to make AI systems more secure.
AI providers have a dual responsibility: to maintain the security of their systems and to protect user privacy, and Clio demonstrates that these two goals are not mutually exclusive - with careful design and implementation, we can achieve both at the same time. By publicly discussing Clio, we aim to set a positive norm around the responsible development and use of such tools.
We are continuing to develop and improve Clio and want others to expand upon it. To learn more about the technical details of Clio, including our privacy verification and assessment methods, see Full Research PaperThe
We are currently recruiting for the Social Impact team. If you are interested in Clio or related research questions, we look forward to your application. For more information about the position, please visitthis linkThe
footnotes
^1 ^ During security surveys, we also run Clio on select first-party API traffic, with results limited to authorized employees. Certain accounts are excluded from the analysis, including trusted organizations with whom we have zero retention agreements. For more information on our policies, see Appendix F of the research paper.
