Question 1

What is a robots.txt file?

Accepted Answer

A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot access. It's placed in the root directory of your website.

Question 2

Does robots.txt block pages from being indexed?

Accepted Answer

No, robots.txt only prevents crawling. To prevent indexing, use a noindex meta tag instead.

Robots.txt Generator

Related Tools

Robots.txt Checker

XML Sitemap Generator

Noindex Tag Test

Frequently Asked Questions