YepBot

Version

1.0

Bot type

Good (identifies itself, has an official moniker)

Category

Search engine crawler

Obeys robots.txt

Yes

Obeys crawl delay

Yes

User-agent string

Mozilla/5.0 (compatible; YepBot/1.0; +http://yep.com/yepbot/)

Reverse DNS suffix

ahrefs.com

YepBot requests come from the Ahrefs crawler IP ranges: see the current list or fetch it from the API.

What is YepBot

YepBot is a web crawler that, along with AhrefsBot, builds the index of web pages behind Yep, a web search engine, and Ahrefs, an online data toolset.

Our bots collect fresh information about websites, their content, and how they link to each other. That data is what makes it possible to find content on Yep and Ahrefs.

YepBot obeys robots.txt

YepBot strictly follows robots.txt, for both disallow and allow rules.

Customizing YepBot access

Because YepBot follows the robots.txt file on your website, you control what it does.

To change how often YepBot visits your site, set the minimum delay between two consecutive requests in your robots.txt file:

User-agent: YepBot
Crawl-Delay: [value]

The Crawl-Delay value is a time in seconds.

To stop YepBot from visiting your site, add these two lines to your robots.txt file:

User-agent: YepBot
Disallow: /

YepBot may need some time to pick up changes to your robots.txt file.

If your robots.txt contains errors, YepBot may parse it partially and follow only the rules it was able to recognize.

Read more about robots.txt at www.robotstxt.org.

Contact

If you think YepBot is misbehaving on your website, or if you have any questions, contact [email protected] and we will do our best to help.