Tencent Research Institute Large Model Research Team
Currently, the identification ofAI-generatedThe technical means for content are not yet mature. How to strike a reasonable balance between potential risks, governance costs, and target effectiveness is critical. It is recommended that small steps be taken to explore scientific risk management options.
An extension of content governance logic:
Human original or AI generated?
For the proportion of AI-generated content in the total information content in the future, the caliber of prediction varies among different organizations, ranging from 20%-90% [2]. But it cannot be denied:With the popularization of generative AI technology applications, the proportion of AI-generated content is gradually climbing.One study shows that the number of web pages containing AI-generated content surged by 28,48% in just one year from 2023 to 2024 [3]. At the same time, changes in content production models are driving a quiet change in the logic of content governance, expanding from the past focus on the nature of the content-whether it is illegal and harmful-to the source of the content-whether it is AI-generated.
In the early stages of AI-generated content, large model vendors have attempted to work on labeling from the goals of improving model transparency and supporting rights protection. Especially in terms of copyright, although there is still a big controversy over the copyrightability of AI-generated content, clarifying the source of the nature of its content still helps possible right holders to claim their rights and interests, and incentivizes the public to utilize the new tool for content creation; at the same time, it is also conducive to clarifying the demarcation of rights and responsibilities between the model vendors and the use of the subject, with the former having more control capabilities in the generation stage and the latter needing to take more responsibility for the subsequent At the same time, it is also conducive to clarifying the division of rights and responsibilities between model manufacturers and users.
The push for logos by governments and the public stems largely from the potential risk of AI-generated content to the order of information dissemination.Cases of AI-generated falsified false information occur from time to time. Such as synthesizing false disaster and news to disturb public order [4], using AI to change faces to synthesize false pornographic pictures and videos to violate personal reputation, etc. [5]. Harmful information, whether human-generated or AI-generated, can be covered by traditional content governance, and the same measures such as deletion and blocking can be taken to maximize the elimination of its impact. HoweverThe more important considerations driving the expansion of content governance are: Generative AI dramatically improves the efficiency of content production, with richer multimodal content and more realistic interactions, and if once applied on a large scale to the creation of false content such as rumors, it will potentially cause confusion among the public about the real information, triggering a general distrust of the public towards the media [6].Despite the fact that, to this day, AI-generated technologies have not been used to the extent previously expected in the news media and other content industries, nor have we seen an actual impact of AI-generated content on the media communication order.But as AI continues to advance, this concern remains real.
How do you differentiate AI-generated content?
Distinguishing AI-generated content is first and foremost a technical issue[7]. Currently, the technical path of identification mainly includes two directions: generated content detection and source data tracking. The former mainly determines whether the content is generated or tampered by AI by looking for the generative features contained in the digital content. The latter indirectly reflects the nature of digital content by independently recording the relevant information (whether it is generated or modified by AI) throughout the life cycle of digital content.However, for AI content recognition, there is a lack of mature and reliable technical solutions.
1. Content detection path
Content detection is the most intuitive solution. While AI-generated content is now comparable to human-created content at the level of human senses, at the level of detail there are still features that can be perceived by machines or technologists.
In image content, when it comes to the processing of image edges, textures and other details, AI-generated content will show pixel-level inconsistencies; when it comes to realistic physical features such as scale and symmetry, illumination and shadows, AI-generated content will show minor errors; in video content, when it comes to the trajectory of object movement, changes in illumination and shadows, AI-generated content will show slight unnaturalness, lack of coherence or abnormal physical patterns; similarly, in audio, text and other AI-generated information, there are similar minor differences. regularity anomalies; similarly, similar minor differences exist in audio, text, and other AI-generated information.
However, even though AI-generated content differs from human content in numerous ways, mature, efficient and reliable synthetic content detection techniques are not yet available.Evaluating techniques for AI-generated content detection mainly requires consideration of the following elements: generalizability, interpretability, efficiency, robustness, and computational cost. In the image domain, techniques have emerged that utilize a variety of models such as deep learning models, machine learning models, and statistical models for detection, but generalizability and robustness generally perform poorly. The accuracy obtained using different methods on different training and testing subsets is reported to be only from 61% to 70%.When synthetic images are post-processed (e.g., compressed and resized), the detection accuracy will be further reduced, making it difficult to operate reliably in practice.
2. Data tracking path
Source data tracking is an indirect solution to "content characterization".Source data tracking does not rely on the content itself, but rather reflects the authenticity and integrity of the content by recording changes (generation, modification, etc.) to the content. Current source data tracking methods mainly include explicit and implicit identification.
(1) Explicit marking
The most important feature of explicit marking is that it can be directly perceived by people, and the effect of prompting and informing is obvious, but its practical effect is still to be evaluated. Explicit logos include content labels and visible watermarks. Content labels exist separately from digital content (e.g., peripheral hints in specific scenes), and cannot play a distinguishing effect in the whole life cycle of generating synthetic content; while visible watermarks are limited to a part of the content, and are easy to be cut or removed; when visible watermarks are applied to a large part of the whole content, it will reduce the quality of digital content.
(2) Implicit marking
Implicit marking refers to the marking added in the generation of synthetic content or data that cannot be directly perceived by the user but can be processed by technical means. At present, there are two main technical paths: digital watermarking and metadata recording.
Digital watermarks are machine-readable watermarks that can embed additional source information by perturbing the content in a way that is not visible to the naked eye. Based on the different ways of being perturbed, it can be categorized into LSB-based watermarking, Discrete Cosine Transform (DCT) watermarking, LLM watermarking, and so on.But the effectiveness of digital watermarking is equally questionable.Watermarks generated by complex algorithms require a large amount of computational resources to read and are cost efficient; watermarks generated by simple algorithms are easily removed and tampered with and are not sufficiently secure.
Metadata logging is another approach, where metadata generated by content changes are stored independently in the same file as the digital content to provide information about its content attributes, source, etc. Based on the characteristics of independent storage, this method is more efficient but has obvious drawbacks.first, metadata needs to be stored for long periods of time, and resources need to be invested in management and query optimization, increasing costs;Secondly, metadata can in principle be arbitrarily added, modified, or erased, making it difficult to ensure integrity and authenticity. Although security can be enhanced by digital fingerprinting or signature technology, this also brings additional costs;one more time, the threshold for circumventing metadata logging is low. Users can bypass metadata logging through non-download methods such as taking screenshots or shots from external devices.
Whether it is digital watermarking or metadata, the biggest challenge facing privacy markers is that the realization of their technological goals places high demands on the governance ecosystem.Considering the complexity of the network communication link, writing implicit identifiers, reading and verifying them, and ultimately prompting users, completing this closed loop requires a high degree of collaboration among ecological subjects, taking into account the confidentiality of the algorithm and cross-platform interoperability for identification. If there is a lack of mature technology and governance norms, not only can the purpose of source identification not be realized, but also may exacerbate the risk of deception or confusion.
Industry for AI-generated content
Spontaneous Exploration of Logos
Globally, AI companies and large online platforms have spontaneously started exploring around the labeling of AI-generated content based on principles such as transparency and trustworthiness.For large models, ChatGPT-generated image content is labeled using metadata records [8]; images created or edited by Meta AI contain visible watermarks [9]. AI systems developed by domestic enterprises such as Yuanbao, Doubao, and Wen Xiaoyan have all added explicit logos to the images they generate.For Internet platformsMeta requires users to mark content shared that contains realistic video or simulated audio generated or modified by digital means (e.g., using AI) [10], and Meta is also experimenting with adding markers to images detected as having been generated by the platform's AI. [11] X will add explicit markers to media content that utilizes AI to fictionalize or simulate real people, or media content that is altered by AI to distort its meaning to provide additionalBackground information or just delete the content[12]. Domestic platforms such as Xiaohongshu and Weibo have also gone online with the user self-declaration function [13].
The exploration practices at home and abroad reflect the following common points: first, the first choice to try in the image, video and other areas that are most likely to produce confusion and misrecognition, and the big model enterprises explicitly identify the content in the generation stage; second, the dissemination platform prompts the user, actively declares when the user shares realistic content generated using AI, and at the same time, explores the path of identification based on metadata and other technologies. In addition, foreign countries are more embodied in the spontaneous formation of industrial alliances by enterprises to promote the formation of open technical standards [14].
AI-generated content logos:
Exploring dynamic risk-based governance
AI-generated content poses a completely different set of risks than in the past, driving attempts to clarify the boundaries between AI-generated and human-created. However, no mature technical solution has yet been developed for labeling AI-generated content. In general, due to the risk prevention idea of "prevention is better than cure", the labeling work is in a state of spontaneous exploration. Correspondingly, at the level of global governance rules, there are mostly some general requirements in principle, and there are no detailed provisions on the realization of marking, thus leaving more room for practical exploration.
1. Explore reasonable risk management programs in the process of continuous trial and error verification
It is suggested that an open recommendation approach be adopted to encourage relevant subjects to actively explore a variety of technical approaches, including content detection, digital watermarking and so on. For the cross-subject reading and verification of metadata, it should be continuously improved through AB experiments, and then gradually expanded after obtaining a technical framework recognized by general practice. For the anti-deletion and anti-tampering attack and defense of the logo, it is a process of "the devil is one foot high and the road is ten feet high", which requires the industry to make concerted efforts to deal with it. In addition, the public's understanding of the logo and the application of pain points, but also decided to mark the work will be in the dynamic to seek the best practices.
2. Differentiate the governance roles of different subjects based on scenarios
For AI-generated content, providers and deployers of AI-generated technology have clear subject role differences and need to adapt different rules. For example, the European Union's Artificial Intelligence Act establishes different marking norms based on the different subject roles. Article 50(2) stipulates that providers of AI systems should realize that their output content can be marked in a machine-readable format; and paragraph 4 stipulates that the deployer of an AI system that generates or manipulates image, audio or video content that constitutes a deep forgery should disclose that the content is generated or manipulated by human beings. As can be seen, the former emphasizes that the "developer" of the technology focuses on providing a "machine-readable" technical solution, while the "deployer" focuses on "disclosing" the content of the deepfake. "Disclosure" of its nature.
3. Avoiding the big picture and focusing governance resources on "real risk areas"
(a) Comprehensive marking sounds "big and beautiful", but in essence it may hinder the realization of the purpose of marking by integrating truly risky content into a sea of universally marked information, which may easily overload the public with information and greatly reduce the effectiveness of risk control.In this context, consideration could be given to limiting the scope of the marking.
First, it is limited in the field.Given the generality of AI generation technologies, there are a large number of generation applications outside the information dissemination domain (e.g., data synthesis to satisfy model training, AI generation to serve processing touch-up purposes such as maps, game rendering; AI office scenarios on the B-side, etc.), and in these domains, where the risk of content dissemination is low, the marking work is not of priority urgency or can be explored in a less burdensome way;
The second is the limitation on the content of the logo.Concentrate limited resources on areas of higher risk. As China's "Regulations on the Administration of Algorithmic Recommendation of Internet Information Services" takes the idea - "For those that lead to confusion or misidentification by the public.The information content generated or edited should be marked prominently at a reasonable location or area to alert the public to the depth of the synthesis. This also reflects the current consensus of domestic and international labeling practices. As Meta's content policy states, "When we are confident that some AI content will not violate our policies, we remove it from our review queue. This allows our review resources to focus more on content that may violate our rules."
Negative externalities of excessive labeling have already surfaced. It has been reported that minor operations such as removing dust and blemishes from a photo by the AI function of image editing software can result in the photo being labeled as "AI-generated" when it is uploaded to social media platforms [15]. Such labeling may lead to "reverse confusion", whereby the public may mistake human-created content for "generated synthetic content", which may adversely affect intellectual property rights, the protection of personal rights and interests, and even public trust. In order to avoid excessive labeling, it is also necessary to focus on exploring the "exception rules for labeling".
4. Cultivating public "information literacy" in the AI era
How perfect identification rules, and ultimately can only play the role of auxiliary judgment, can not replace the public on the content of the information for the final judgment of the authenticity of the individual will always be their own "choose to believe that the content of the" final gatekeeper. In the era of information explosion, the more we need to enhance the sensitivity to the authenticity of information. Just as Internet natives are more cautious in scrutinizing online information than their predecessors, in the age of AI, it is even more important for people to say goodbye to the judgmental rule of "no picture, no truth". Taking the labeling rules as an opportunity to cultivate the public's rational judgment in the face of online content, and to form personal "information literacy" in the age of AI, is a more critical part of content governance.
Annotated Source:
[1] The articles in this issue were mainly completed by: Rong Wang, Ze Li, Qiang Wang and Yuan Yuan.
[2] Gartner: "Predicts 2021"; Forbes: Is AI quietly killing itself - and the Internet? https://www.forbes.com.au/news/innovation/is- ai-quietly-killing-itself-and-the-internet/
[3] Copyleaks: Copyleaks Analysis Reveals Explosive Growth of AI Content Across the Web https://copyleaks.com/about-us/media/copyleaks-analysis -reveals-explosive-growth-of-ai-content-across-the-web
[4] Fujian Daily: Zhejiang Shaoxing police successfully detected "the use of generative artificial intelligence rumor" "AI face" and other cases https://china.huanqiu.com/article/ 4ESTJ7yZmlX
[5] 21st Century Business Herald South Korea's 'AI face-swap' pornography criminalized, production, preservation, distribution of all aspects of the criminalization https://m.21jingji.com/article/20241008/herald/ 1bc218eb1719fb416dcd688b8c032b57.html
[6] AI. Gore: the assault on reason 2017 updated edn
[7] Unless otherwise noted, portions of this section draw primarily on the National Institute of Standards and Technology report: Overview of Technical Approaches to Transparency in Digital Content (NIST AI 100-4), and the European Commission report: Transparency in Generative Artificial Intelligence: Identifying Machine-Generated Content. https://publications.jrc.ec.europa. eu/repository/handle/jrc137136
[8] OpenAI to add tags to AI-generated images: https://siliconangle.com/2024/02/07/openai-will-now-add-labels-ai-generated-images-following-meta/
[9] Meta: Responsibly Building Generative AI Features: https://about.fb.com/news/2023/09/building-generative-ai-features-responsibly/
[10] ins: tagging AI content on instagram: https://help.instagram.com/761121959519495
[11] Meta: Tagging AI-generated images on Facebook, Instagram, and Threads https://about.fb.com/news/2024/02/labeling-ai-generated-images-on-facebook-instagram -and-threads
[12] X: Synthesized and manipulated media policies https://help.x.com/en/rules-and-policies/manipulated-media
[13] Nandu Big Data Institute: AI-generated synthetic content may have to be tagged! Reporters test social platform recognition https://m.mp.oeeee.com/a/BAAFRD0000202409141000167.html
[14] For example, The Coalition for Content Provenance and Authenticity (C2PA), https://c2pa.org/about/.
[15] turns out instagram may label your photos as 'made with AI' even when they're not https://www.designboom.com/technology/instagram-label- photos-made-with-ai-when-theyre-not-06-07-2024/