learning$web

robots.txt

PrevNext
Term side of the Card
Topic: Search Engine Optimization
Unit title: robots.txt
Categories: File
Explanation side of the Card

Explanation: The text file used by websites to communicate with web crawlers. It specifies which pages or sections of the site should or should not be crawled or indexed by search engines.

Edit on GitHubContributions in our content repository are 🤗 welcome from everyone.Link leads to the Contributions guidelines on GitHubLink leads to the content repository on GitHub