Bot and Crawling Policy

Undergrove · UndergroveBot/1.0 · Last updated: 12 August 2026 Contact: undergrove@undergrovescholar.com

If you run a website and you've seen our crawler, this page explains what it does. If you'd like us to stop, email us and we will.

What we do

We read scholarship providers' own web pages to find out what an award requires, when it closes and how much it's worth, so we can tell students accurately.

How we behave

  • We identify ourselves. Every request carries UndergroveBot/1.0 (+https://undergrovescholar.com/bot).
  • We respect robots.txt. If you disallow us, we stop.
  • We're slow on purpose. No more than one request per second to any single domain. Many scholarship providers are small organisations on shared hosting.
  • We don't create accounts and we don't log in. We only read pages available to anyone without credentials.
  • We don't bypass anything. No CAPTCHA solving, no rate-limit evasion, no rotating IPs to avoid blocks. If you block us, we're blocked.
  • We fetch a page, not a site. We follow a link to one scholarship page; we don't spider.
  • We back off. A 403 or 429 means stop, not retry harder.

What we store

Facts: name, deadline, amount, eligibility requirements, application URL. Facts about scholarships aren't ours and aren't anyone's.

Our own words: every description a student reads is written by us from those facts. We don't copy your text.

What we don't do

  • We don't crawl scholarship aggregator or listing sites. We read the scholarship provider's own page.
  • We don't republish your content.
  • We don't use your pages to train AI models.
  • We don't collect personal information about anyone from your site.

Ask us to stop

Add us to robots.txt:

User-agent: UndergroveBot
Disallow: /

Or email undergrove@undergrovescholar.com and we'll remove you and stop fetching. No argument, no form.

If we got something wrong

If we've listed your scholarship inaccurately — wrong deadline, wrong amount, listed as closed when it isn't — please tell us. Accuracy is the whole product, and a correction from the provider is the best source there is. We fix these quickly.