Our crawler, TruthSignal

What it is

TruthSignal is the crawler of Truth Signal, which checks whether a business online is real. When someone asks about a business, or its report is due a refresh, it reads that business’s own public pages: the home page and the pages about contact, the company, returns, delivery, cancellation, terms and privacy. From them we read the company name and number, the address, phone and email, and the policies, and the report cites each page.

The name it sends

Every visit names us, from our page reader and from the Chrome browser we use for pages that only appear once their scripts run:

robots.txt

Before it reads a page it checks your robots.txt and follows the rules under User-agent: TruthSignal, or under User-agent: * when none name it. A page your rules close is not read. To keep it out of your whole site, add:

To let it in while your file keeps other crawlers out, add:

How often it visits

One check reads a handful of pages, one after another, and only when someone asks about the business or its report is due a refresh. It reads only the pages named above, never your whole site.

Where it comes from

Our browser visits from one fixed address, also listed as plain text at /bot/ips.txt:

Our page reader visits from our hosting provider’s shared addresses, which are not ours alone and can change. When a site does not answer our own reader, we may ask Firecrawl, a page-reading service, to read the page; that visit comes from Firecrawl. We also test a wider read, of a site’s home page and the pages its menu links to, with copies of the same browser, name and robots.txt rules, from short-lived servers whose addresses are not on the list.

Contact

For a question about TruthSignal, or a visit you want us to look at, write to hello@truthsignal.com.