Newsful-Bot
Newsful-Bot is the crawler behind Newsful, a news reader that shows each person only the stories they can act on. If you run a news site and found us in your logs, this page is for you.
What it fetches
It reads the RSS feeds of the outlets we cover, about once an hour.
When a feed item carries only a short description, it may also fetch that one article page to read the full story. Once it has read an article, it does not fetch it again, and it never follows links elsewhere on your site.
What we do with your article
We read it to decide who it matters to: where it applies, what a reader could do about it, and by when. Readers see a short summary we write and a link that takes them straight to your page. We never republish your article text, and we never show it to readers.
Making that decision involves a language model. That is why we treat opt-outs aimed at AI crawlers as covering us too.
How it behaves
Every request identifies itself as Newsful-Bot and links to this page.
Before fetching an article page it reads your robots.txt and obeys rules for Newsful-Bot and for all crawlers. If your robots.txt disallows the common AI crawlers — GPTBot, ClaudeBot, Google-Extended and similar — from that page, we do not fetch it, unless you have told us otherwise.
It fetches one page at a time from your site, at least two seconds apart, and honors Crawl-delay.
How to opt out
To stop article-page fetches, add a group for Newsful-Bot to your robots.txt: “User-agent: Newsful-Bot” followed by “Disallow: /”.
To stop us reading your feed as well, or to arrange anything different, email us and we will act on it within a week.
Contact
privacy@newsful.org reaches the people who build Newsful.