Grant Discovery Bot

A public-data crawler in service of nonprofits.

What this crawler does

Grant Discovery Bot collects publicly available information about grant programs, foundations, and federal awards from sources such as the IRS, Grants.gov, SAM.gov, USASpending, and ProPublica’s Nonprofit Explorer. Its purpose is to help nonprofit organizations identify funding opportunities they would otherwise need expensive commercial databases to find.

It is operated by TheAlmoner LLC, an independent project dedicated to making philanthropic information accessible to the organizations that serve the poor and afflicted.

How to identify this crawler

Every request the bot makes carries two unambiguous identifiers:

User-Agent header: Mozilla/5.0 (compatible; GrantDiscoveryBot/1.0; +https://bot.thealmoner.com)
From header: bot@thealmoner.com

The token GrantDiscoveryBot inside the User-Agent string is the technical identifier — this is what appears in your access logs and what robots.txt rules should match. “Grant Discovery Bot” is the human-readable name for the same crawler; they refer to one and the same program.

Both values are stable and will not churn between versions. The crawler runs from a single fixed egress IP, advertised in machine-readable form at /ips.json. Each IP reverse-resolves to a hostname of the form crawl-N.thealmoner.com, so a webmaster auditing access logs can confirm the IP belongs to this operator.

Sub-identities

The same operator runs a lightweight URL-reachability probe under a distinct sub-identity, following the Googlebot / Googlebot-Image naming convention. It performs only HEAD or short-GET requests to check whether source URLs are still live; it never fetches body content for storage.

Reachability probe User-Agent: Mozilla/5.0 (compatible; GrantDiscoveryBot-Reachability/1.0; +https://bot.thealmoner.com)

This sub-identity shares everything else with the main bot — same operator, same From address, same egress IPs, same opt-out path. A robots.txt rule targeting GrantDiscoveryBot also covers it through token-prefix match.

Rendering policy

The main bot uses plain HTTPS clients only — no JavaScript execution, no fingerprint spoofing, no headless browsers. This matches the convention followed by Googlebot, Bingbot, ClaudeBot, GPTBot, and other reputable crawlers.

For the small set of portals that cannot be reached by plain HTTPS because the underlying data is rendered exclusively client-side and no public API is available, the operator reserves the right to deploy a clearly-identified rendering sub-identity:

Rendering sub-identity (reserved; not currently in use): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 GrantDiscoveryBot-Render/1.0 (+https://bot.thealmoner.com)

If and when this sub-identity is deployed, the affected portals will be enumerated here — following the convention of Google’s Web Rendering Service, which is publicly disclosed at Google’s crawler documentation. As of today, the list is empty and no headless browser runs in production.

How this crawler behaves

robots.txt is honored. Disallow rules, the GrantDiscoveryBot user-agent token, and Crawl-delay directives all apply.
Rate is conservative. No more than one request to a given host every 5–10 seconds; exponential backoff on 429 and 503 responses.
No headless browsers by default. Plain HTTPS clients; no JavaScript execution, no fingerprint spoofing. Any portal that genuinely cannot be reached without rendering will be served by the clearly-identified GrantDiscoveryBot-Render/1.0 sub-identity (see Rendering policy above) and listed there by name.
Opt-out is immediate. A request to the contact address below removes a host from the crawl queue within one business day.

How to block or limit this crawler

To stop the crawler from visiting your site, add the following to /robots.txt. The user-agent token GrantDiscoveryBot is what the bot literally sends, so rules must match that token:

User-agent: GrantDiscoveryBot
Disallow: /

To allow access only to specific paths, use standard Disallow rules:

User-agent: GrantDiscoveryBot
Disallow: /private/
Disallow: /api/
Allow: /

To request a longer minimum delay between requests:

User-agent: GrantDiscoveryBot
Crawl-delay: 30

Contact

For any concern — opt-out requests, audit-log inquiries, questions about the data we collect — write to bot@thealmoner.com. Mail is read by the operator; a human replies.