Cross-border research collaboration reveals gaps in how Brazilian news websites shield content from AI crawlers

São Paulo, 12 March, 2026: The International Fund, together with the Journalism Relay Project and Momentum - Journalism and Tech Task Force, have published a new report—The Protocol Gap: Brazil—that begins to uncover critical gaps in how news sites are managing AI access to their content.

Press Releases

No items found.

As of November 2025, only 7.2% of Brazilian news sites block at least one “AI crawler” through their robots.txt files, even though 75% have the file installed. AI crawlers are automated bots that systematically browse and collect data from websites to train large language models and power AI assistants and search engines. The findings show that the small number of outlets restricting access focus mainly on well-known companies such as OpenAI, Google, Common Crawl, ByteDance, Amazon, Apple, Meta, and Huawei.

Amid surging demand for data to train large language models (LLMs), develop AI assistants, and power AI-driven search engines, the findings suggest that many news websites are not yet using robots.txt as a strategic tool to signal their preferences regarding AI companies scraping their content.

Without mechanisms to control AI crawlers, media organizations are unable to effectively protect or monetize content reused by AI systems—further straining outlets already facing declining traffic and visibility from digital platforms.

Robots.txt is a simple text file placed on a website’s root domain that provides instructions to web crawlers about which pages or sections they are allowed to access.  While robots.txt compliance is not legally binding and has its limitations, it is one of the few free and widely recognized tools available for publishers to signal their preferences regarding AI scraping.

‍

The impact of AI crawlers on media business models was one of the key issues highlighted during the 2025 CTRL+J conference series, co-hosted and sponsored by the International Fund with regional partners. Participants emphasized the need for stronger technical defense mechanisms that allow publishers to control AI crawler access and potentially monetize it.

Produced by the Journalism Relay Project, Momentum - Journalism and Tech Task Force, and the International Fund for Public Interest Media, this first report is part of a broader research collaboration focused on Brazil, Indonesia, and South Africa. Through technical and qualitative research, it aims to increase awareness among media organizations on how AI systems access their content and help to inform strategies and policies to manage AI crawlers in ways that best serve newsroom interests. The report includes a detailed methodology for all those who may want to replicate this research on robots.txt in other countries and regions.

Read the full report in English, Portuguese or Spanish.

Also see

Latest news
International Fund to Support Ten Independent Media Outlets in the Philippines
Paris, 13 February, 2026: The International Fund has made new grants to local or regional independent news outlets in the Philippines and will fund ten media organizations as of 2025. The grants, ranging from US$10’000 to US$220’000 per media outlet will support a diverse group of independent newsrooms including digital start-ups, TV stations, newspapers and a variety of other media outlets.
Read more
International Fund Announces New Call for Grant Proposals from Independent Media in Armenia
Paris, 12 January: The International Fund has launched an open call for proposals to support independent media in Armenia.
Read more
Open Call for Funding Announced: The International Fund Invites Grant Proposals by Exiled Media from Belarus and Azerbaijan
Friday, 4 July, Paris: The International Fund for Public Interest Media cordially invites independent media organizations from Belarus and Azerbaijan operating from exile to participate in a call for proposals for grant support. The Open Call for Belarusian exiled media is coordinated jointly with the JX Fund.
Read more
Thirty-four Governments Join Forces in Defense of Access to Free and Reliable Information Worldwide
Historic Paris Declaration on Multilateral Action Lays Out Commitments by Leading Democracies
Read more
From Challenges to Innovative Solutions: Bringing IFPIM Latin American Grantees Together for Collective Learning
In the first edition of a gathering of all IFPIM Latin American grantees, 25 media organizations and systemic change initiatives from Argentina, Bolivia, Brazil, Colombia, Costa Rica, Ecuador and Paraguay came together on the 24th of July in Bogotá, Colombia to showcase their impact, connect with each other and share collective learnings.
Read more
Sweden Awards Grant of SEK120 million (€10.5 million) to Support Independent Public Interest Journalism
Paris, 6 December: Sweden’s International Development Cooperation Agency (Sida) has this week confirmed grant support for the International Fund for Public Interest Media (IFPIM), providing SEK120 million in new funding for independent journalism.
Read more
Why it Matters
The Threats